• Design and implement enterprise-scale data pipelines using Databricks on AWS, leveraging both cluster-based and serverless compute paradigms • Architect and maintain medallion architecture (Bronze/Silver/Gold) data lakes and lakehouses • Develop and optimize Delta Lake tables for ACID transactions and efficient data management • Build and maintain real-time and batch data processing workflows • Create reusable, modular data transformation logic using DBT to ensure data quality and consistency across the organization • Develop complex Python applications for data ingestion, transformation, and orchestration • Write optimized SQL queries and implement performance tuning strategies for large-scale datasets • Implement comprehensive data quality checks, testing frameworks, and monitoring solutions • Design and implement CI/CD pipelines for automated testing, deployment, and rollback of data artifacts • Configure and optimize Databricks clusters, job scheduling, and workspace management • Implement version control best practices using Git and collaborative development workflows • Partner with data analysts, data scientists, and business stakeholders to understand requirements and deliver solutions • Mentor junior engineers and promote best practices in data engineering • Document technical designs, data lineage, and operational procedures • Participate in code reviews and contribute to team knowledge sharing