Top Data Engineer Jobs in Chennai

Operate and improve production data infrastructure across large-scale distributed cloud systems. Monitor performance, manage alerts, respond to incidents, perform root cause analysis, automate repetitive tasks, support CI/CD pipelines, document runbooks, and assist with capacity planning. The role requires participation in an on-call rotation and collaboration with Engineering to maintain reliability, efficiency, and uptime.

Build and enhance enterprise data pipelines into Databricks Unity Catalog and Delta Lake. Improve remediation and entity-matching layers (including D&B DataBlocks), integrate APIs, and support LLM-assisted remediation and CMG redesign projects.

Design and implement distributed data systems for Apache Hudi and Onehouse’s managed data lakehouse. Responsibilities include developing concurrency controls, transactional capabilities, indexing schemes, metadata and data-access systems, and optimization algorithms for Kubernetes-based distributed processing. The role involves analyzing complex data, prototyping performance improvements, deploying solutions safely to production, and serving as a subject matter expert in databases, query engines, storage, and stream processing.

Build and productionize distributed data infrastructure for Onehouse’s managed lakehouse platform. Responsibilities include developing Apache Hudi transactional capabilities, concurrency control, indexing, metadata and data-access systems, and optimization algorithms for distributed processing on Kubernetes. The role involves analyzing complex data, prototyping performance improvements, collaborating with engineers, and safely deploying solutions. Candidates should bring strong object-oriented programming, distributed systems, algorithm, and relational database expertise.

Design, deploy, and maintain AWS and Databricks data infrastructure, data lakes, and scalable ETL pipelines. Build custom ingestion services and connectors using Airbyte, Go, and Python; optimize performance, reliability, monitoring, and data quality. Implement cloud data governance, security, access controls, encryption, and compliance practices while collaborating with platform engineers, analysts, and stakeholders.

Designs and operates end-to-end data products on Databricks, including ingestion, transformation, streaming pipelines, APIs, applications, and curated datasets. Partners with stakeholders to clarify needs, prototype solutions, and deliver secure, scalable production systems. Responsibilities include data governance, testing, CI/CD, monitoring, optimization, troubleshooting, technical design, code reviews, and mentoring. The role requires strong SQL, Python, PySpark, cloud, event-streaming, backend, and data-modeling expertise.

The Data Engineer role involves designing and maintaining ETL pipelines, optimizing data storage solutions, collaborating with teams, and ensuring data accuracy and integrity.

Designs, builds, and maintains ETL/ELT pipelines feeding Microsoft Fabric Lakehouse platforms. Responsibilities include data ingestion, bronze/silver/gold layer development, CDC, incremental loading, data quality, metadata governance, performance tuning, monitoring, automation, and low-level design documentation. The role collaborates with architects, data modelers, and business analysts to deliver reliable, reusable datasets and promote self-service analytics while applying security, resilience, compliance, and cost-optimization standards.

Design, build, and maintain scalable Microsoft Fabric and Azure-based data platforms, ETL/ELT pipelines, lakehouse and data warehouse architectures. Implement orchestration, monitoring, data quality, CI/CD, and support analytics, AI, and production operations while following Specification-Driven Development practices.

Lead and contribute to design, architecture, and hands-on development of scalable data platforms and ETL pipelines on cloud (GCP). Drive best practices, CI/CD, integrations, API specs, code reviews, and collaborate with product and stakeholders to deliver high-availability, big-data solutions and BI reporting.

Build and maintain scalable batch and real-time data pipelines using Kafka, Maxwell, Spark and dbt; design medallion-based data models; implement cloud-native data platforms (S3, Trino, BigQuery); optimize performance, ensure data quality, observability, and SLA adherence; enable analytics through trusted datasets and Metabase dashboards.

Lead test engineering teams to design and execute system integration and automated tests across UI, services, and APIs. Develop and maintain reusable automation frameworks, test plans, and procedures; partner with product, development, and DevOps; track execution, risks, and environment/data dependencies; ensure process compliance and mentor junior engineers.

Lead Azure Data Engineer to design, build, and operate petabyte-scale data systems on Azure. Lead a data engineering team to develop ETL pipelines (ADF, Synapse, Databricks), perform data modelling, migrations, monitoring, security, and collaborate with BI/ML/Product teams while mentoring junior engineers and improving analytics and dashboarding.

Design, implement, and optimize Apache DolphinScheduler workflows and ETL pipelines. Integrate with Hadoop, Spark, Flink, Hive, and Kafka. Develop custom plugins, troubleshoot performance and scalability issues, monitor job execution and resource allocation, and maintain CI/CD, version control, and infrastructure automation while collaborating with data engineers and DevOps teams.

Design, build, and optimize ELT data pipelines using Python, Airflow DAGs, and Spark in cloud environments. Convert and debug Microsoft BI ETLs into scalable big-data workflows, tune Spark resources, collaborate with business partners, and participate in Agile ceremonies with Pacific-time overlap.

Design and implement data quality processes, profile and cleanse data, deploy cloud-based data quality solutions (Azure/AWS/GCP), document issues and remediation, report metrics, and collaborate with data teams and stakeholders.

Design, develop, test, and deploy high-performance Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and Cloudera platform performance, integrate HDFS/Hive/Impala/HBase/Kafka, ensure data integrity/security, and implement version control and CI/CD for production pipelines.

Design, build, and optimize large-scale ETL pipelines and distributed data applications (batch and streaming) using Spark and Scala. Ensure data quality and performance, deploy and manage workloads in cloud/container environments, troubleshoot production issues, and collaborate with product, engineering, and analytics teams to deliver scalable AdTech data solutions.

Design, build, and operate scalable batch and real-time data pipelines and analytics platforms. Partner with enterprise customers to define requirements, own technical delivery from discovery to adoption, and present architectures to stakeholders. Drive data governance, observability, CI/CD, and platform optimization while mentoring junior engineers and collaborating with Product, Engineering, and Customer Success teams to enable analytics and AI/ML applications.

Senior technical leader responsible for designing, building, and scaling cloud-native data platforms and pipelines. Partners with stakeholders, mentors engineers, enforces engineering best practices, ensures data modeling, governance, and quality, debugs production issues, and contributes to architecture, design, and Agile delivery to support business outcomes.

Design, build, and maintain scalable ETL/ELT data pipelines and data warehouse solutions using Snowflake. Optimize data models and query performance, ensure data quality, integrate multiple sources, implement security and governance, and collaborate with BI and product teams to deliver reliable datasets.

Build and maintain scalable ETL/ELT pipelines and data platforms, optimize queries and databases, design analytics-ready data models, ensure data quality and governance, collaborate with stakeholders, and support data products and automation for business intelligence and analytics.

Design, build, and support ETL workflows to migrate and transform legacy EHR data into athenahealth systems. Develop Python utilities, validate and reconcile data, support HL7/FHIR interoperability and EHI export tasks, troubleshoot issues, maintain documentation, and collaborate across product, sales, and operations to improve migration processes and automation.

Triage and resolve Master Data Management (Party, Risk, Reference) issues in Stibo STEP, run targeted data-cleansing sprints, monitor data quality dashboards, support releases/UAT, and maintain documentation and playbooks to prevent recurring data issues.

Lead a two-month project to assess the current data environment and design, build, and optimize a scalable cloud data warehouse. Deliver automated ETL/ELT pipelines, dimensional models, analytics-ready datasets, documentation, and handover while partnering with analysts and stakeholders for BI and reporting.