Design, build, and maintain scalable batch and streaming data pipelines across cloud and distributed environments. Integrate multiple data sources, optimize ETL/ELT workflows, ensure data quality and governance, and collaborate with product and analytics teams to deliver production-grade data solutions using Snowflake, Kafka, Spark, and cloud data services.
This role is for one of the Weekday's clients
Salary range: Rs 1500000 - Rs 3000000 (ie INR 15-30 LPA)
Experience: 6+ yrs
Location: Chennai
Job type: full-time
We are looking for a highly experienced Data Engineer to design and build scalable, high-performance data systems that power enterprise-grade analytics and real-time data-driven applications. The role involves working on complex data architectures, integrating multiple data sources, and building reliable pipelines that support both batch and streaming use cases.
You will be responsible for developing robust data engineering solutions across cloud and distributed environments, ensuring seamless data flow, high performance, and strong data quality standards. The role requires strong hands-on expertise in Python, Snowflake, and modern data engineering ecosystems.
This position is ideal for someone who enjoys solving large-scale data challenges, working with modern data warehouses, and building systems that support real-time decision-making across the organization.
RequirementsKey Responsibilities
- Design, develop, and maintain scalable data pipelines for ingestion, transformation, and integration across multiple data sources
- Build and optimize ETL/ELT workflows to support batch and real-time data processing requirements
- Work with relational databases such as MySQL and PostgreSQL as well as NoSQL systems like MongoDB and Cassandra
- Develop and manage data pipelines using modern data warehouse platforms such as Snowflake and Redshift
- Process large-scale structured and unstructured data using AWS services such as S3, Athena, and Glue
- Implement real-time streaming pipelines using Kafka for data ingestion and event-driven architecture
- Utilize Apache Spark for distributed data processing and large-scale analytics (batch and streaming)
- Collaborate with cross-functional teams including product, analytics, and engineering to translate business requirements into technical data solutions
- Ensure data reliability, accuracy, consistency, and governance across all systems
- Monitor, troubleshoot, and optimize pipeline performance for scalability and cost efficiency
- Build reusable data engineering components and improve existing data infrastructure
- Implement best practices for data modeling, storage optimization, and pipeline architecture
- Work closely with cloud and DevOps teams to manage data infrastructure on AWS, GCP, or Azure environments
- Continuously evaluate and adopt new technologies and frameworks in the data engineering ecosystem
- 6–9 years of hands-on experience in data engineering and large-scale data pipeline development
- Strong proficiency in Python for data processing and backend engineering tasks
- Solid experience working with Snowflake and modern data warehouse platforms
- Strong understanding of SQL and relational databases such as MySQL and PostgreSQL
- Experience working with NoSQL databases like MongoDB or Cassandra
- Hands-on experience with Kafka for real-time streaming and event-driven data systems
- Strong understanding of ETL/ELT processes, data modeling, and pipeline optimization
- Experience with big data frameworks such as Apache Spark is highly preferred
- Familiarity with cloud platforms such as AWS, GCP, or Azure is an added advantage
- Experience working with AWS services like S3, Athena, and Glue for data processing
- Strong analytical and problem-solving skills with attention to detail
- Ability to collaborate effectively with cross-functional teams in fast-paced environments
- Strong communication skills with the ability to explain technical concepts clearly
- A mindset focused on building scalable, reliable, and production-grade data systems
Similar Jobs
Information Technology • Marketing Tech • Analytics
Lead design and delivery of scalable data platforms and production-grade pipelines using Databricks and cloud providers. Build batch and real-time ETL, implement data models for analytics, ensure data quality and governance, integrate AI/GenAI capabilities, mentor junior engineers, and drive client-facing technical discussions and architecture reviews.
Top Skills:
Ai/MlSparkAws Ec2Aws GlueAws IamAws LambdaAws S3AzureCi/CdData Pipeline OrchestrationDatabricksDbrxDelta LakeDevOpsGCPGenaiLakehouseLlmsMedallion ArchitectureMosaic AiPower BIPysparkPythonRagSQLTableau
Fintech • Financial Services
The Senior Data Engineer will architect and implement cloud data platforms, leading a team to deliver high-performance solutions and collaborate with product teams on data transformation initiatives.
Top Skills:
APIsBig DataConfluenceGCPGitJavaJenkinsJIRAKafkaNoSQLPythonRdbmsScalaSparkSplunkXlr
Cloud • Information Technology • Security • Software • Cybersecurity
The Sr. Staff Integration Engineer will onboard cybersecurity data sources, normalize security data, troubleshoot pipeline health, and communicate with stakeholders to enhance product integration and performance.
Top Skills:
APIsPythonSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
