Design, build, optimize, and maintain scalable ETL/ELT pipelines and complex SQL transformations. Translate business needs into reusable data solutions, automate workflows with orchestration tools, and ensure reliability through testing, validation, monitoring, and documentation. Collaborate with analysts, data scientists, and stakeholders to deliver trusted datasets. Troubleshoot production issues, improve performance and cost efficiency, and follow coding, version control, and peer review practices.
Duties & Responsibilities
Design, develop, and maintain scalable ETL/ELT data pipelines to support business and analytics needs
Write, tune, and optimize complex SQL queries for data transformation, aggregation, and analysis
Translate business requirements into well-designed, documented, and reusable data solutions
Partner with analysts, data scientists, and stakeholders to deliver accurate, timely, and trusted datasets
Automate data workflows using orchestration/scheduling tools (Airflow, ADF, Luigi, etc.)
Develop unit tests, integration tests, and validation checks to ensure data accuracy and pipeline reliability
Document pipelines, workflows, and design decisions for knowledge sharing and operational continuity
Apply coding standards, version control practices, and peer code reviews to maintain high-quality deliverables
Proactively troubleshoot, optimize, and monitor pipelines for performance, scalability, and cost efficiency
Support function rollouts, including being available for post-production monitoring and issue resolution
Requirements
Basic Qualifications
Bachelor’s degree in computer science, Information Systems, Engineering, or a related field
2–5 years of hands-on experience in data engineering and building data pipelines
At least 3 years of experience in writing complex SQL queries in a cloud data warehouse/ data lake environment.
Solid hands-on experience with data warehousing concepts and implementations
At least 1 year of experience with Snowflake or another modern cloud data warehouse
At least 1 year of hands-on Python development.
Familiarity on Data modeling and Data warehousing concepts
Experience with orchestration tools (e.g., Airflow, ADF, Luigi)
Familiarity with at least one cloud platform (AWS, Azure, or GCP)
Strong analytical, problem-solving, and communication skills
Ability to work both independently and as part of a collaborative team
Preferred Qualifications
Experience with DBT (Data Build Tool) for data transformations
Exposure to real-time/streaming platforms (Kafka, Spark Streaming, Flink)
Familiarity with CI/CD and version control (Git) in data engineering projects
Exposure to the e-commerce or customer data domain
Understands the technology landscape, up to date on current technology trends and new technology, brings new ideas to the team
Similar Jobs
Digital Media • Information Technology • News + Entertainment
Designs, develops, and manages advanced data architectures, ingestion frameworks, and pipelines. Ensures data quality, lineage, integrity, accessibility, privacy, and regulatory compliance across AWS, Databricks, Kubernetes, and Teradata environments. Builds APIs, database views, and data extracts; selects appropriate storage platforms; applies transformation rules; monitors data quality; and collaborates with cross-functional teams to optimize data processing and resolve issues. Requires independent judgment and availability for variable schedules, including nights and weekends.
Top Skills:
Apache AirflowAPIsAWSDatabricksKubernetesPythonTeradata
Digital Media • Information Technology • News + Entertainment
Design, implement, and administer SAP Analytics Cloud (SAC) planning and reporting solutions. Lead SAC modelling, story creation, SAC admin, and migrations from BPC to SAC. Collaborate with data, product, and functional teams to define data acquisitions, virtual models in HANA, pipelines, and security. Provide support, training, and stabilization efforts while managing multiple projects and mentoring team members.
Top Skills:
AbapAWSAzureEpmGCPSap Analytics Cloud (Sac)Sap AoeSap BpcSap BtpSap BwSap DatasphereSap HanaSap S/4 Hana
Hardware • Software
Designs, develops, and tests scalable data management solutions, including acquisition, integration, and transformation pipelines. Builds data structures and infrastructure across cloud and edge environments using Azure, Databricks, and SAP Data Services. Manages DevOps pipelines, continuous deployment, and containerization. Collaborates with technical leads, analysts, architects, product owners, and business stakeholders to define requirements, deliver high-quality code, and enhance applications.
Top Skills:
AgileAzure Data Factory (Adf)Azure Data Lake Storage (Adls)Ci/CdContainerizationDatabricksDevOpsAzureSap Data ServicesSQLTabular Models
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
