Lead design, build, and optimize scalable cloud data platforms and ETL/ELT pipelines on GCP. Deliver batch and real-time processing with Apache Beam/Dataflow, Java backends, BigQuery warehousing, and Kafka streaming. Ensure data quality, performance optimization, monitoring, CI/CD deployments, and collaborate with architects, DevOps, analysts, and stakeholders to support enterprise analytics and reporting.
This role is for one of the Weekday's clients
Salary range: Rs 1500000 - Rs 2500000 (ie INR 15-25 LPA)
Experience: 7+ yrs
Location: Chennai, Coimbatore, Bangalore, Pune
Jobtype: full-time
We are looking for a highly skilled Lead Data Engineer to design, build, and optimize scalable cloud-based data platforms and processing pipelines for enterprise-grade applications. This role is ideal for someone who enjoys solving large-scale data engineering challenges and has deep expertise in Google Cloud Platform (GCP), Apache Beam/Dataflow, Java, and BigQuery.
As a Lead Data Engineer, you will play a critical role in building robust batch and real-time data processing systems that enable analytics, reporting, and data-driven decision-making across the organization. You will work closely with architects, analysts, DevOps teams, and business stakeholders to develop reliable, scalable, and high-performance data solutions.
The ideal candidate is passionate about modern cloud data architectures, distributed systems, and building efficient ETL/ELT pipelines capable of handling large-scale enterprise workloads. This role requires strong technical expertise, problem-solving ability, and the capability to drive engineering best practices across data platforms.
RequirementsKey Responsibilities
- Design and develop scalable ETL/ELT pipelines using Google Cloud Platform services
- Build and maintain real-time and batch data processing pipelines using Apache Beam and Google Dataflow
- Develop backend processing components and data transformation services using Java
- Work extensively with BigQuery for data warehousing, analytics, querying, and performance optimization
- Integrate data from multiple sources including APIs, relational databases, streaming systems, and cloud platforms
- Build reliable and scalable streaming data pipelines using technologies such as Kafka and cloud-native services
- Optimize pipeline performance, scalability, reliability, and cloud infrastructure costs
- Ensure high standards of data quality, governance, monitoring, security, and operational excellence
- Collaborate with cross-functional teams including Architects, Analysts, DevOps, and business stakeholders to deliver data-driven solutions
- Troubleshoot production issues, perform root-cause analysis, and implement long-term scalable fixes
- Implement CI/CD practices, version control workflows, and automated deployment pipelines for data engineering solutions
- Design and maintain scalable data models, schemas, and warehouse structures
- Participate in architecture discussions and contribute to improving overall cloud data platform capabilities
- Drive best practices in distributed data processing, pipeline optimization, and cloud-native engineering
- Strong hands-on experience with Google Cloud Platform (GCP) services
- Deep expertise in Apache Beam and Google Dataflow for large-scale data processing
- Strong programming skills in Java with experience building backend processing systems
- Hands-on experience with BigQuery and cloud-based data warehousing solutions
- Experience building and maintaining batch and real-time streaming data pipelines
- Strong understanding of SQL, data modeling, and distributed data processing concepts
- Experience working with Kafka or similar streaming technologies
- Familiarity with CI/CD pipelines, Git workflows, and Agile development methodologies
- Strong analytical, troubleshooting, and debugging capabilities
- Understanding of scalable cloud architecture, performance optimization, and reliability engineering
- Ability to work collaboratively with cross-functional engineering and business teams
- Experience handling enterprise-scale datasets and complex integration requirements
- Strong communication skills with the ability to translate technical solutions into business impact
- Passion for building modern cloud-native data platforms and solving large-scale engineering challenges
Similar Jobs
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Design, build, and scale cloud-native data and BI solutions on Databricks and Snowflake. Develop high-performance data pipelines, enable multi-cloud interoperability (AWS/Azure), implement governance and secure access patterns, maintain CI/CD and infrastructure-as-code for platform provisioning, and lead technical design while mentoring engineers.
Top Skills:
Adls Gen2AksApache IcebergAWSAzureAzure DatabricksCi/CdDatabricksDelta LakeEmrEnterprise Identity IntegrationGlueIamInfrastructure As CodeRbacS3Service PrincipalsSnowflake
Artificial Intelligence • HR Tech • Professional Services • Software
Design, build, and optimize scalable AWS-based data platforms and ETL/ELT pipelines using Python/PySpark and SQL. Implement data warehouses, data lakes/lakehouse architectures, ensure data quality/governance, optimize performance, enable analytics/ML use cases, monitor reliability, and mentor junior engineers while supporting deployments and platform improvements.
Top Skills:
Amazon AuroraAmazon OpensearchAmazon RedshiftAmazon S3Aws GlueAws LambdaAws SnsAws Step FunctionsCi/CdData LakeData WarehouseDevOpsEltETLEvent-Driven ArchitectureInfrastructure As CodeLakehousePl/SqlPysparkPythonSQLWorkflow Orchestration
Artificial Intelligence • HR Tech • Professional Services • Software
Lead design, build, and optimize scalable AWS-based data platforms and ETL/ELT pipelines. Develop data warehouses, dimensional models, and large-scale PySpark/Python processing. Ensure data quality, governance, security, performance tuning, monitoring, and platform reliability. Collaborate with stakeholders and support AI/ML use cases while mentoring engineers and driving best practices.
Top Skills:
Amazon AuroraAmazon OpensearchAmazon RedshiftAmazon S3Aws GlueAws LambdaAws SnsAws Step FunctionsCi/CdData LakeData WarehouseDevOpsEltETLInfrastructure As CodeLakehousePl/SqlPysparkPythonSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
