Design, build, and optimize enterprise-scale data pipelines and modern data platforms. Develop ETL/ELT workflows with Python, PySpark, Scala, SQL, Informatica, Databricks, and cloud services. Manage databases, orchestration, streaming, containers, CI/CD, governance, and multi-cloud environments. Collaborate with BI teams, troubleshoot performance issues, and maintain technical documentation.
We are looking for a highly skilled Lead Data Engineer / Architect to design, build, and optimize scalable data pipelines and modern data platforms. This role is critical in transforming raw data into meaningful insights and enabling enterprise-level analytics.
Key Responsibilities
- Design, develop, and optimize ETL/ELT pipelines using Python, PySpark, Scala, and SQL
- Build and manage data workflows using Informatica, Databricks, AWS Glue, Snowflake
- Work across multi-cloud platforms (AWS, Azure, GCP, Microsoft Fabric)
- Manage and optimize databases such as Redshift, SQL Server, AWS RDS
- Orchestrate workflows using Apache Airflow
- Implement real-time streaming solutions using Apache Kafka
- Deploy and manage containerized applications using Kubernetes
- Automate processes using Unix shell scripting
- Implement CI/CD pipelines and ensure data governance
- Collaborate with BI teams for Tableau, Looker, Power BI
- Troubleshoot and optimize performance of pipelines and queries
- Maintain technical documentation
Requirements
- Bachelor’s/Master’s degree in Computer Science or related field
- 7+ years of experience in Data Engineering
- Strong skills in Python, PySpark, Scala, SQL
- Experience with Informatica, Databricks, AWS Glue
- Expertise in Snowflake & Redshift
- Experience in AWS / Azure / GCP
- Experience with Apache Airflow
- Knowledge of Apache Kafka & Kubernetes
- Experience with CI/CD pipelines
- Exposure to BI tools (Tableau, Looker, Power BI)
Similar Jobs
Fitness • Healthtech • Payments • Software
Lead data engineering design and delivery for CRM data platforms supporting member aggregation, segmentation, audiences, campaigns, reporting, and Connect use cases. Build and optimize SQL workflows across MySQL and Redshift, operate Iceberg-based lakehouse models, own pipeline and infrastructure reliability, and mentor engineers. Partner with Product and engineering teams using Agile, Lean, DevOps, CI/CD, and modern data quality practices.
Top Skills:
AgileAmazon RedshiftApache IcebergAWSCi/CdDbtDevOpsEltETLEvent-Driven ReplicationGoJavaKafkaLeanMySQLSQLSqsTerraformTrino
Information Technology • Software • Analytics
Lead the architecture and development of a Risk and Entity Intelligence Platform. Build scalable data pipelines, ETL and ingestion systems, entity resolution capabilities, graph database solutions, and risk intelligence products. Apply compliance, KYC/AML, sanctions, and multilingual data expertise while making technical decisions and mentoring engineers. The role also involves working with APIs, web scraping, cloud infrastructure, workflow orchestration, and large-scale data acquisition.
Top Skills:
AirflowAPIsAWSDagsterDedupeElasticsearchFollowthemoneyGCPMemgraphMlMoody'S GridNeo4JNlpOpencorporatesOpensanctionsOpensearchPrefectPythonSayariSenzingSplinkSQLTigergraphWeb ScrapingYenteZingg
Fintech
Leads the architecture, design, and delivery of enterprise data products and domain data assets. Establishes engineering standards, frameworks, governance, reliability, observability, and security practices. Guides cloud and data platform decisions, translates business priorities into technical roadmaps, mentors engineers, supports hiring, and drives modernization, cost optimization, and technical debt reduction. Builds trusted data products for AI, machine learning, and advanced analytics.
Top Skills:
AgileAnalytical DatasetsCi/CdCloud Data PlatformsData ModelingData WarehousingDatabricksDataopsDevOpsDistributed Processing FrameworksFeature StoresPythonScalaSemantic ModelsSnowflakeSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
