Digital Channel Submission Rate

Migrated legacy Hive workflows to an automated PySpark and Airflow pipeline powering Tableau dashboards on digital customer activity.

Problem

Reporting on digital channel metrics such as payments and account openings depended on legacy Hive workflows that took around 12 hours to run and needed manual intervention.

Approach

Migrated the legacy Hive workflows to an automated PySpark pipeline on Hadoop.
Introduced Apache Airflow for scheduling and orchestration, making the ETL fully automated and reliable.
Optimised the output datasets and integrated them into Snowflake for fast downstream querying.
Delivered curated data to Tableau dashboards used by stakeholders across the business.

Results

Processing time dropped from 12 hours to under 2 hours.
Near real-time visibility into digital metrics like payments and account openings.
Faster stakeholder decision-making, with the dashboards contributing to a 30% improvement.