Problem Statement
The client’s goal was to improve operational efficiency, provide real-time updates to customers, and enhance data security.
Challenges
- High-Volume Data Handling: Manage 4-5 lakh KYC and Rupee Seed data records per day.
- Event Processing: Handle 40k-50k triggered events daily through concurrent threads.
- Notifications Service: Notify clients about events (e.g., MTF, Futures, Options) through SMS, WhatsApp, and app notifications.
- Search Functionality: Enable efficient querying of stock data (~20k records) for buy/sell decisions.
Languages and Technology Stack
- Programming Languages: Java, JavaScript (ReactJS)
- Message Broker: Apache Kafka
- Database and Caching: Redis, Elasticsearch
- Notification: Twilio (SMS), Firebase
Solution Design
The solution was designed as a scalable, event-driven platform leveraging the following technologies:
Backend Architecture:
- Built using Java and Spring Boot to handle business logic and microservices.
- Kafka integrated to process high-volume KYC and event data.
- REDIS is used for caching stock-related data for low-latency retrieval.
Data Processing:
- Kafka’s producer-consumer model enabled ingestion of 4-5 lakh daily records with topic partitioning for scalability.
- Java’s ExecutorService was used for efficient thread management, processing 40k-50k events per day.
Notifications Service:
- Notification types included SMS, WhatsApp, and app notifications for 25-30 event types.
- APIs from Twilio (SMS), WhatsApp Business, and Firebase (app notifications) were integrated
Search Engine:
- Migrated stock data (~20k records) from Excel to Elasticsearch for faster querying and scalability.
| Challenges | Solutions |
| Scaling to handle large data volumes | Kafka with topic partitioning for horizontal scalability |
| Reliable multi-channel notifications | Integrated APIs for SMS, WhatsApp, and app notifications with schedulers |
| Transitioning from Excel for search | Migrated to Elasticsearch with optimized indexing |
Results
- Increased Efficiency:
- The platform processed over 4.5 lakh records and 50k events daily without downtime.
- Improved Customer Engagement:
- Notifications delivered seamlessly across multiple channels improved client satisfaction. Delivered 25-30 event notifications on time with 95% delivery accuracy.
- Enhanced Security:
- Encrypted data storage and secure authentication protocols ensured compliance with industry standards.
- Faster Search and Visualization:
- Transitioning to Elasticsearch reduced query times significantly, while Redis caching enabled real-time data rendering. Search query time was reduced by 90%, from ~10 seconds to under 1 second.
Conclusion
The new platform enabled the client to excel in delivering real-time, data-driven financial services. By adopting a scalable event-driven architecture, the firm successfully addressed its operational requirements while positioning itself to adapt seamlessly to future growth and evolving market demands
