Leads the design, development, optimization, and support of ETL/ELT pipelines, data warehouses, data models, and cloud-based data solutions. Writes advanced SQL, implements testing and CI/CD practices, monitors production systems, resolves incidents, and improves performance and cost efficiency. Partners with analysts, data scientists, and stakeholders to deliver reliable datasets while documenting solutions and mentoring associate engineers.
Duties & Responsibilities
Lead the design, development, and optimization of ETL/ELT pipelines and workflows
Write, tune, and optimize advanced SQL queries for large-scale data processing
Build and maintain data warehouses and data models (relational, dimensional, star schema)
Translate business requirements into robust, well-architected, and reusable data solutions
Partner with analysts, data scientists, and stakeholders to deliver trusted, actionable datasets
Implement and maintain unit tests, integration tests, and validation frameworks to ensure pipeline reliability
Document workflows, and design decisions to support knowledge sharing and operational continuity
Apply coding standards, CI/CD practices, version control, and peer code reviews to ensure high-quality deliverables
Proactively monitor, optimize, and troubleshoot pipelines for performance, scalability, and cost efficiency
Support deployments and handle post-production monitoring and incident resolution
Mentor associate engineers, providing technical guidance and feedback
Requirements
Basic Qualifications
Bachelor’s degree in computer science, Information Systems, Engineering, or a related field
6-10 years of hands-on experience in data engineering or related fields
Expertise in writing complex SQL queries, optimizing, using advanced functionals and turning the performance on cloud data warehouse environment.
Strong experience with data warehousing concepts, design, and implementation
Minimum 6 years of hands-on experience with Snowflake or modern cloud data warehouse
Minimum 3 years of hands-on experience in data modeling
Minimum 2 year of hands-on Python development (ETL/ELT scripting, OOP, automation)
Strong knowledge of at least one major cloud platform (AWS, Azure, or GCP)
Experience with orchestration tools (e.g., Airflow, ADF, Luigi) for workflow management
Demonstrated ability to lead and mentor teams while managing multiple priorities
Strong communication and stakeholder management skills
Strong hands-on experience with data warehousing concepts, design, and implementation
Preferred Qualifications
Hands-on experience with DBT (Data Build Tool) for data transformations
Familiarity with DevOps and CI/CD best practices in data engineering
Exposure with real-time/streaming platforms (Kafka, Spark Streaming, Flink)
Exposure to the e-commerce domain or large-scale B2B/B2C environments
Has the ability to break down problems and estimate time for development tasks
Experience mentoring and guiding junior engineers
Understands the technology landscape, up to date on current technology trends and new technology, brings new ideas to the team
Learns organization vision statement and decision-making framework. Able to understand how team and personal goals/objectives contribute to the organization vision
Similar Jobs
eCommerce • Retail • Software
Lead the design, development, and scaling of a digital experience analytics platform. Build data foundations and AI-driven decision systems using Azure, Databricks, dbt, Python, SQL, and cloud data technologies. Guide junior engineers, collaborate with product and analytics teams, support data architecture, modeling, governance, quality, and operationalization, and coordinate cloud deployments and technical delivery across onsite and offshore teams.
Top Skills:
Adobe AnalyticsAgentic AiAzureAzure Data EngineeringAzure Data FactoryAzure Data LakeCi/CdDatabricksDbtDevOpsGCPInfrastructure As CodeLinuxPower BIPythonSnowflakeSQL
Blockchain • Database • Analytics
Lead the design, development, and optimization of scalable Databricks ETL/ELT pipelines. Integrate data from databases, S3, files, and REST APIs; implement Delta Lake medallion architecture, Unity Catalog, dimensional models, data quality controls, and Spark performance tuning. Schedule and monitor Databricks workflows, collaborate cross-functionally, and guide engineers while driving technical decisions.
Top Skills:
AirflowAmazon S3SparkAuto LoaderAWSCi/CdDatabricksDatabricks Lakehouse PlatformDbtDelta LakeGitKafkaPysparkPythonRest ApisSQLUnity Catalog
HR Tech • Professional Services
Designs, builds, and optimizes scalable batch and real-time data pipelines on GCP using Dataflow, Apache Beam, Java, and BigQuery. Integrates diverse data sources, improves pipeline performance and reliability, supports data governance and security, troubleshoots production issues, and collaborates with architects, analysts, DevOps, and business stakeholders.
Top Skills:
Apache BeamSparkBigQueryCi/CdCloud ComposerCloud StorageDataprocGitGoogle Cloud Platform (Gcp)Google DataflowJavaKafkaPub/SubPythonSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
