Job Description
- Lead the Azure Data Engineering team and provide day-to-day technical direction, delivery oversight, and issue resolution.
- Design, develop, orchestrate, and maintain scalable ADF pipelines for batch ingestion, data integration, transformation, and downstream processing.
- Develop complex and reusable data-transformation frameworks using PySpark and Azure Databricks notebooks.
- Design metadata-driven ingestion, transformation, validation, reconciliation, logging, and restart/recovery components.
- Build and maintain solutions across ADLS Gen2, Azure Databricks, SQL databases, storage accounts, Azure Key Vault, and upstream/downstream enterprise systems.
- Migrate RDBMS and traditional ETL workloads to modern Data Lake, ELT, and cloud-native processing patterns.
- Optimize PySpark workloads for partitioning, caching, shuffle reduction, file sizing, scalability, and efficient processing of large datasets.
- Implement data-quality rules, control totals, exception handling, auditability, monitoring, alerting, and operational support mechanisms.
- Support data lake and SQL Server control/governance frameworks, including standards, templates, naming conventions, and directory structures.
- Analyze defects, incidents, change requests, and enhancement needs; perform root-cause analysis and implement sustainable fixes.
- Convert functional and business requirements into technical designs, estimates, implementation plans, and deployment activities.
- Perform code reviews and enforce engineering standards, security practices, source-control discipline, CI/CD, and documentation quality.
- Coordinate with onsite architects, business analysts, QA teams, application teams, production support teams, and customer stakeholders.
- Own offshore deliverables, dependencies, risks, status reporting, release readiness, and post-production stabilization.
- Mentor team members, build technical capability, and ensure high-quality, timely completion of deliverables.
Insurance / Reinsurance Domain Expectations: