InFynd Logo

InFynd

Senior Data Engineer

Posted 9 Days Ago
Be an Early Applicant
In-Office
Coimbatore South, Coimbatore, Tamil Nadu
Senior level
In-Office
Coimbatore South, Coimbatore, Tamil Nadu
Senior level
Designs and maintains scalable Azure-based ETL/ELT pipelines using Data Factory, Databricks, Python, SQL, PySpark, and Spark. Integrates diverse data sources, develops data models and warehouse structures, implements CDC and batch processing, and optimizes queries and distributed workloads. Manages orchestration, monitoring, error handling, CI/CD, documentation, and production support using Airflow, Git, GitHub, and Azure DevOps. Collaborates with cross-functional teams to troubleshoot issues and deliver reliable data solutions.
The summary above was generated by AI

  • Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
  • Develop data ingestion and transformation workflows using Python, SQL, and PySpark.
  • Work with Apache Spark and PySpark for large-scale distributed data processing.
  • Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.
  • Integrate data from multiple sources, including relational databases, APIs, and cloud storage.
  • Work with Azure Data Lake Storage Gen2 and Azure Blob Storage.
  • Develop and optimize data models, fact tables, dimension tables, and data warehouse structures.
  • Perform data cleansing, validation, deduplication, and aggregation.
  • Implement and manage workflows using Azure Data Factory and Apache Airflow.
  • Optimize SQL queries, Spark jobs, and data pipelines for better performance.
  • Work with Azure Synapse Analytics and relational databases.
  • Implement error handling, logging, monitoring, and alerting for production pipelines.
  • Manage version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.
  • Troubleshoot production issues, perform root cause analysis, and provide production support.
  • Maintain technical documentation and collaborate with cross-functional teams.


Requirements

Programming Languages

  • Python, SQL, PySpark.
  • Pandas, NumPy, OOP concepts, exception handling, and API integration.

Big Data Technologies

  • Apache Spark and PySpark.
  • Spark DataFrames, Spark SQL, transformations, actions, joins, aggregations, and window functions.
  • Partitioning, caching, and performance optimization.

Azure Cloud Technologies

  • Azure Data Factory (ADF).
  • Azure Databricks.
  • Azure Data Lake Storage Gen2 (ADLS Gen2).
  • Azure Blob Storage.
  • Azure Synapse Analytics.
  • Azure Key Vault and Azure Monitor.

ETL & Data Engineering

  • ETL/ELT pipeline development.
  • Data ingestion, transformation, and migration.
  • Change Data Capture (CDC).
  • Incremental loading and full loading.
  • Batch and distributed processing.

Database Technologies

  • Microsoft SQL Server, PostgreSQL, MySQL, and Oracle.
  • Complex SQL queries, indexing, stored procedures, and query optimization.

Data Modeling & Warehousing

  • Dimensional and relational data modeling.
  • Fact and dimension tables.
  • Data normalization and denormalization.
  • Schema design and data warehouse development.

Workflow Orchestration & DevOps

  • Azure Data Factory and Apache Airflow.
  • Git, GitHub, Azure DevOps, and CI/CD pipelines.

File Formats

  • CSV, JSON, XML, Parquet, and Delta Lake.
  • 4–5 years of relevant experience in Data Engineering.
  • Strong hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.
  • Experience developing and optimizing scalable data pipelines.
  • Good understanding of cloud-based data engineering and data warehousing concepts.
  • Strong analytical, debugging, and problem-solving skills.
  • Good communication and collaboration skills.


Similar Jobs

5 Days Ago
In-Office
Chennai, Tamil Nadu, IND
Senior level
Senior level
Information Technology • Consulting
Designs and maintains scalable Azure Databricks data pipelines using Python, Scala, PySpark, Delta Lake, and SQL. Develops ETL/ELT workflows, optimizes processing performance, ensures data quality and governance, manages Azure Data Lake and Unity Catalog integrations, and supports cloud architecture, migration, modernization, troubleshooting, documentation, and cross-functional delivery.
Top Skills: Apache AirflowAzure Data FactoryAzure Data LakeAzure DatabricksCi/CdDelta LakeEltETLInformaticaOraclePl/SqlPysparkPythonRdbmsScalaSQLUnity Catalog
12 Days Ago
Hybrid
Senior level
Senior level
Blockchain • Database • Analytics
Designs and maintains scalable Databricks ETL/ELT pipelines using Python, PySpark, SQL, Spark, and Delta Lake. Manages Databricks infrastructure, Jobs, Workflows, clusters, access controls, and performance optimization. Implements data governance through Unity Catalog, data quality checks, lineage, metadata management, cataloging, and governed access. Integrates diverse data sources, troubleshoots pipeline issues, supports data security and compliance, and collaborates with teams on enterprise data management practices.
Top Skills: Amazon S3SparkCi/CdDatabricksDelta LakeGitPysparkPythonRest ApisSQLUnity Catalog
17 Days Ago
In-Office
Senior level
Senior level
Artificial Intelligence • Cloud • Information Technology • Machine Learning • Natural Language Processing • Consulting
Build, optimize, and scale end-to-end data pipelines using Databricks, SQL, Python, and Spark. Improve ETL/ELT performance, reliability, and cost efficiency; implement data quality, governance, and security practices; troubleshoot distributed data workflows; collaborate with engineers, analysts, and stakeholders; and support cloud data platform migration and modernization initiatives.
Top Skills: SparkAWSAzureDatabricksDatabricks WorkflowsDelta LakeGitJfrog ArtifactoryPythonShell ScriptingSQL

What you need to know about the Chennai Tech Scene

To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account