The customer is a leading housing finance company, one of the most diversified NBFCs in the country with 19 product lines across consumer, commercial and SME finance, which makes it one of the most profitable companies in the category. The main objective of the company is to provide long-term finance to individuals as well as corporate entities for purchase, construction, and renovation of homes, plots or commercial spaces. Mainly, it provides finance to builders and developers engaged in the construction of homes. The customer wanted to build a serverless data warehouse on the AWS cloud.
The Need for A Scalable, Cost-Effective Data Warehouse System
The customer needed to build their separate Data Warehouse and Analytical system from scratch. Their analytical department had the challenge of building such a system that will meet the existing analytical reporting requirements as well as to have a flexible and scalable system for future requirements. They needed the system to be cost-effective, scalable, fail-over capable and co-operative with a sudden spike in workload. The project expected to go live within 7 to 8 months, hence, they did not want to spend time and resources for managing on-premise hardware infrastructure.
Blazeclan’s Proposed a AWS Cloud-based Solution
Keeping the key objectives like cost-effectiveness, scalability and fastest delivery in mind, the Blazeclan team proposed AWS cloud-based solution for the requirement. Blazeclan’s solution architect team designed the below phases/components of the project.
-
Source Data Capture: Being a financial company, the customer had a variety of pennant-based data sources like LMS (Loan management system), LOS (Loan Origination system), VAS (Value Added Services), and Campaigning Data and CIBIL Data. The team created an Oracle-based staging database (AWS RDS) that would have only incremental data extraction from above OLTP sources. Later, this incremental data is migrated to AWS S3 using the AWS DMS (Data migration system). Another source for data was API and CSV which were processed via AWS lambda and API and later put into S3.
-
Data Storage and Security: Data captured via DMS, Lambda and API were later put on S3 file storage. For the customer, being a financial organization, the top priority was the security of the data. The Blazeclan team leveraged IAM policies and KMS encryption to meet security expectations.
-
Data processing: As soon as files are put on S3, wrapper Lambda’s for each module (LMS, LOS, VAS, and campaign, etc) get triggers that invoke respective glue jobs. These glue jobs perform configurable data validations processing, like referential integrity checks, NULL checks, and Data type checks and later, classify them in fact and dimension tables.
-
Data Warehousing/Publishing: These glue jobs process the data and populate them in respective fact and dimension tables in a data mart, created in the AURORA Postgres database. The monitoring, logging, and troubleshooting are performed in AWS Cloudwatch.
-
Visualization/Notification: Visualization of reports and business insights were designed in tableau installed on a reserved EC2 instance. The data mart was used as a source for these visualizations. Most of the scheduled reports were automated using AWS Cloudwatch daily and delivered to the management via mail. Some part of the data was also published using APIs. The team effectively used Lambda functions for generating these reports, as per the requirement of the business.
Benefits to the Customer
-
Data Archival: After processing of data residing at S3 storage, it was automated to be transferred to AWS Glacier for cost optimization.
-
Fastest Delivery: Due to the cloud-based solution with serverless and managed services, the customer got the solution delivered within 7 months.
-
Cost Reduction: Serverless and managed services eliminated the extra cost to be spent on on-premise hardware infrastructure and its maintenance.
-
Automation: Variety of the reports to be sent to the customer’s management team was automated using CloudWatch and Lambda’s time and event-based invocation reduced a lot of manual work.
-
Enriched Visualizations: With Tableau, the customer got interactive and enriched visualizations.
