Lead design and build of scalable AWS data lake/lakehouse solutions using S3 and Iceberg. Develop and optimize PySpark ETL/ELT pipelines, orchestrate with Airflow, integrate Snowflake, ensure data quality/governance, lead technical design, and mentor a team of data engineers.
Key Responsibilities:
- Design and implement scalable data lake/lakehouse architectures on AWS using S3 and Iceberg.
- Develop and optimize ETL/ELT pipelines using PySpark and orchestrate workflows with Apache Airflow.
- Integrate and manage Snowflake as the enterprise data warehouse and business layer for analytics.
- Ensure data quality, governance, security, and performance across the data platform.
- Lead technical design discussions, define best practices, and drive data engineering standards.
- Mentor and guide a team of data engineers, fostering a collaborative and high-performance culture.
- Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders.
Key Responsibilities:
- Design and implement scalable data lake/lakehouse architectures on AWS using S3 and Iceberg.
- Develop and optimize ETL/ELT pipelines using PySpark and orchestrate workflows with Apache Airflow.
- Integrate and manage Snowflake as the enterprise data warehouse and business layer for analytics.
- Ensure data quality, governance, security, and performance across the data platform.
- Lead technical design discussions, define best practices, and drive data engineering standards.
- Mentor and guide a team of data engineers, fostering a collaborative and high-performance culture.
- Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders.
Required Skills & Qualifications:
- 9+ years of experience in data engineering or related roles.
- Proven expertise in AWS services, especially S3-based data architectures.
- Strong hands-on experience with Apache Iceberg, PySpark, and Apache Airflow.
- Solid experience implementing Snowflake as a data warehouse/business layer.
- Strong programming skills in Python and distributed data processing frameworks.
- Experience leading and mentoring engineering teams.
- Excellent problem-solving, communication, and collaboration skills.
EXL Chennai, Tamil Nadu, IND Office
Chennai, India
Similar Jobs
Information Technology • Database • Consulting
Lead data engineering projects using AWS and Snowflake. Design solutions, guide development and testing, ensure production deployments, perform performance tuning, manage stakeholders, and mentor team members to deliver iterative, high-quality Big Data solutions.
Top Skills:
Aws (Architecture)Aws AuroraAws Cloud MicroservicesAws EmrAws GlueAws RdsAws S3PysparkPythonSnowflakeSQL
Healthtech • Information Technology • Software • Telehealth
Maintain and audit a comprehensive provider database: extract and analyze data with SQL, merge sources to build accurate provider profiles, update large datasets in Excel/Sheets, verify provider information via outreach, and ensure data privacy and compliance.
Top Skills:
Ai-Powered ToolsGoogle SheetsExcelSQL
Healthtech • Information Technology • Software • Telehealth
Zocdoc seeks to connect with candidates interested in improving healthcare. This general interest role emphasizes collaboration and innovation.
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
