Modern Big Data Analytics platform for AdTech StartUpBig Data
The client is a young AdTech Start-up that helps over 300 Consumer brands gain deeper insights about its target audience, by incentivizing consumer attention. The Start-Up rewards millennials with free mobile airtime for completing AdTasks – Consumer brand related survey/task/trial etc. With 8+ MN app downloads and 3+ MN monthly user engagement, the client targeted to hit 30 MN monthly user engagement in 3 years. To that end, the client aimed to move towards a Big Data Analytics platform that leverages flexibility of open-source systems, while taking advantage of the cost & scalability benefits of AWS.
Lambda architecture was devised to support batch and stream processing of user events such as app install, wallet credit etc.
- Streaming data events from mobile apps (JSON) were retrieved using Apache Kafka queue producer
- Events from queue were processed at real-time for various topologies using Apache Storm, and were stored in Cassandra NoSQL DB. Daily batch analysis was done using Amazon EMR
- HA best practices for Storm, Kafka & Cassandra implemented
- Storm Web UI used to monitor & configure Storm topologies
- Scalability & performance bottlenecks identified & resolved
- 10x performance uptick
- Pay as you go model
- Advanced Analytics ready flexible Big Data Analytics platform