Designs and maintains scalable Azure-based ETL/ELT pipelines using Data Factory, Databricks, Python, SQL, PySpark, and Spark. Integrates diverse data sources, develops data models and warehouse structures, implements CDC and batch processing, and optimizes queries and distributed workloads. Manages orchestration, monitoring, error handling, CI/CD, documentation, and production support using Airflow, Git, GitHub, and Azure DevOps. Collaborates with cross-functional teams to troubleshoot issues and deliver reliable data solutions.
- Design,
develop, and maintain scalable ETL/ELT data pipelines using Azure Data
Factory and Azure Databricks.
- Develop
data ingestion and transformation workflows using Python, SQL, and
PySpark.
- Work
with Apache Spark and PySpark for large-scale distributed data processing.
- Implement
batch processing, incremental loading, full loading, and Change Data
Capture (CDC) mechanisms.
- Integrate
data from multiple sources, including relational databases, APIs, and
cloud storage.
- Work
with Azure Data Lake Storage Gen2 and Azure Blob Storage.
- Develop
and optimize data models, fact tables, dimension tables, and data
warehouse structures.
- Perform
data cleansing, validation, deduplication, and aggregation.
- Implement
and manage workflows using Azure Data Factory and Apache Airflow.
- Optimize
SQL queries, Spark jobs, and data pipelines for better performance.
- Work
with Azure Synapse Analytics and relational databases.
- Implement
error handling, logging, monitoring, and alerting for production
pipelines.
- Manage
version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.
- Troubleshoot
production issues, perform root cause analysis, and provide production
support.
- Maintain
technical documentation and collaborate with cross-functional teams.
Requirements
Programming Languages
- Python,
SQL, PySpark.
- Pandas,
NumPy, OOP concepts, exception handling, and API integration.
Big Data Technologies
- Apache
Spark and PySpark.
- Spark
DataFrames, Spark SQL, transformations, actions, joins, aggregations, and
window functions.
- Partitioning,
caching, and performance optimization.
Azure Cloud Technologies
- Azure
Data Factory (ADF).
- Azure
Databricks.
- Azure
Data Lake Storage Gen2 (ADLS Gen2).
- Azure
Blob Storage.
- Azure
Synapse Analytics.
- Azure
Key Vault and Azure Monitor.
ETL & Data Engineering
- ETL/ELT
pipeline development.
- Data
ingestion, transformation, and migration.
- Change
Data Capture (CDC).
- Incremental
loading and full loading.
- Batch
and distributed processing.
Database Technologies
- Microsoft
SQL Server, PostgreSQL, MySQL, and Oracle.
- Complex
SQL queries, indexing, stored procedures, and query optimization.
Data Modeling & Warehousing
- Dimensional
and relational data modeling.
- Fact
and dimension tables.
- Data
normalization and denormalization.
- Schema
design and data warehouse development.
Workflow Orchestration & DevOps
- Azure
Data Factory and Apache Airflow.
- Git,
GitHub, Azure DevOps, and CI/CD pipelines.
File Formats
- CSV,
JSON, XML, Parquet, and Delta Lake.
- 4–5
years of relevant experience in Data Engineering.
- Strong
hands-on experience with Azure Data Factory, Azure Databricks, PySpark,
Python, and SQL.
- Experience
developing and optimizing scalable data pipelines.
- Good
understanding of cloud-based data engineering and data warehousing
concepts.
- Strong
analytical, debugging, and problem-solving skills.
- Good
communication and collaboration skills.
Similar Jobs
Information Technology • Consulting
Designs and maintains scalable Azure Databricks data pipelines using Python, Scala, PySpark, Delta Lake, and SQL. Develops ETL/ELT workflows, optimizes processing performance, ensures data quality and governance, manages Azure Data Lake and Unity Catalog integrations, and supports cloud architecture, migration, modernization, troubleshooting, documentation, and cross-functional delivery.
Top Skills:
Apache AirflowAzure Data FactoryAzure Data LakeAzure DatabricksCi/CdDelta LakeEltETLInformaticaOraclePl/SqlPysparkPythonRdbmsScalaSQLUnity Catalog
Blockchain • Database • Analytics
Designs and maintains scalable Databricks ETL/ELT pipelines using Python, PySpark, SQL, Spark, and Delta Lake. Manages Databricks infrastructure, Jobs, Workflows, clusters, access controls, and performance optimization. Implements data governance through Unity Catalog, data quality checks, lineage, metadata management, cataloging, and governed access. Integrates diverse data sources, troubleshoots pipeline issues, supports data security and compliance, and collaborates with teams on enterprise data management practices.
Top Skills:
Amazon S3SparkCi/CdDatabricksDelta LakeGitPysparkPythonRest ApisSQLUnity Catalog
Artificial Intelligence • Cloud • Information Technology • Machine Learning • Natural Language Processing • Consulting
Build, optimize, and scale end-to-end data pipelines using Databricks, SQL, Python, and Spark. Improve ETL/ELT performance, reliability, and cost efficiency; implement data quality, governance, and security practices; troubleshoot distributed data workflows; collaborate with engineers, analysts, and stakeholders; and support cloud data platform migration and modernization initiatives.
Top Skills:
SparkAWSAzureDatabricksDatabricks WorkflowsDelta LakeGitJfrog ArtifactoryPythonShell ScriptingSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.


