Background
Dreamix has been involved in since the initial business analysis activities in 2011. We designed the solution and deployed the first release of the payment analytics and reporting tools in production early in 2012, processing data from a single telecom. Since then, the system has gone through several major releases incorporating new extensions and large modifications and customizations. But the road to success is not straight. The team faced many challenges and here I offer you a chance to become part of our journey to creating reliable BI solution for payment data and mobile value added services.The top 5 Challenges
- Large amount of data No longer after the client enabled their payment services for several telecoms, a decision was taken that payment data must be collected and analysed. Initial attempts made internally, showed that the current databases, configured for operational purposes (OLTP), were incapable of serving analytic queries involving large amounts of data. Basic database queries took hours to execute while others couldn’t execute within days leaving database servers partially or completely unresponsive. An advanced analytics solution was necessary, capable of storing and analysing all history information.
- Realtime SLA Monitoring Apart from collecting and analyzing large amounts of current and history payment messages, the solution should monitor the operational systems and provide an updated information on their performance every few minutes and preserve it for the last several months, tracking internal and external response goals and service level agreements (SLA). This operational information, along with the alltime payment analytical data, must be incorporated into a single web-based portal where different users will have access to both types of information simultaneously.
- ETL execution time Business data from payment messages must be updated once a day. Processing the transactions of 10 App Stores for more than 20 telecoms worldwide requires a huge amount of preprocessing. It is rather difficult to keep these batch processing cycles under 24 hours each so newly processed data is available for the users on the next day.
- Data access The analytics and reporting system should be capable of SSO, using LDAP login. Multiple roles must be supported with different levels of access to data and functionality. Apart from our direct client internal users, who may have access to most of the data, users from different telecoms must also be granted access. Those company external users must though be restricted to data specific to their own telecom.
- Technology limitations In addition to the company application standards that had to be followed strictly, the client had several technological limitations. The BI environment that is to be built must work under Red Hat Linux servers. Database servers must use the same database vendor PostgreSQL and version. All tools used must be noncommercial open source libraries and frameworks.
