In 2016, Dreamix marks the 6th year of our successful partnership with one of our major clients – a leading international full-service provider of payment solutions and mobile value added services. I`m happy to announce that we`re still widely involved in the development of new analytics and reporting features, deploying system releases as well as supporting and customizing the existing functionality. The solution processes data from over 10 app store markets (including Google, Facebook, Microsoft, Sony) from over 20 end client telecoms around the world, in more than 15 countries.
Background
Dreamix has been involved in since the initial business analysis activities in 2011. We designed the solution and deployed the first release of the payment analytics and reporting tools in production early in 2012, processing data from a single telecom. Since then, the system has gone through several major releases incorporating new extensions and large modifications and customizations. But the road to success is not straight. The team faced many challenges and here I offer you a chance to become part of our journey to creating reliable BI solution for payment data and mobile value added services.
The top 5 Challenges
- Large amount of data
No longer after the client enabled their payment services for several telecoms, a decision was taken that payment data must be collected and analysed. Initial attempts made internally, showed that the current databases, configured for operational purposes (OLTP), were incapable of serving analytic queries involving large amounts of data. Basic database queries took hours to execute while others couldn’t execute within days leaving database servers partially or completely unresponsive. An advanced analytics solution was necessary, capable of storing and analysing all history information. - RealÂtime SLA Monitoring
Apart from collecting and analyzing large amounts of current and history payment messages, the solution should monitor the operational systems and provide an updated information on their performance every few minutes and preserve it for the last several months, tracking internal and external response goals and service Âlevel agreements (SLA). This operational information, along with the allÂtime payment analytical data, must be incorporated into a single web-based portal where different users will have access to both types of information simultaneously. - ETL execution time
Business data from payment messages must be updated once a day. Processing the transactions of 10 App Stores for more than 20 telecoms worldwide requires a huge amount of preÂprocessing. It is rather difficult to keep these batch processing cycles under 24 hours each so newly processed data is available for the users on the next day. - Data access
The analytics and reporting system should be capable of SSO, using LDAP login. Multiple roles must be supported with different levels of access to data and functionality. Apart from our direct client internal users, who may have access to most of the data, users from different telecoms must also be granted access. Those company external users must though be restricted to data specific to their own telecom. - Technology limitations
In addition to the company application standards that had to be followed strictly, the client had several technological limitations. The BI environment that is to be built must work under Red Hat Linux servers. Database servers must use the same database vendor PostgreSQL and version. All tools used must be nonÂcommercial open source libraries and frameworks.
Currently, after 4.5 years in production, the system has stored and processed over one billion transactions, loading and processing gigabytes of data from over 20 telecoms around the world each day.
Next week I will discuss our approach to coping with the above-mentioned problems, so stay tuned.
