Kumaran Systems Logo

Kumaran Systems

Senior Data engineer

Posted 5 Days Ago
Be an Early Applicant
In-Office
Chennai, Tamil Nadu, IND
Senior level
In-Office
Chennai, Tamil Nadu, IND
Senior level
Designs and maintains scalable Azure Databricks data pipelines using Python, Scala, PySpark, Delta Lake, and SQL. Develops ETL/ELT workflows, optimizes processing performance, ensures data quality and governance, manages Azure Data Lake and Unity Catalog integrations, and supports cloud architecture, migration, modernization, troubleshooting, documentation, and cross-functional delivery.
The summary above was generated by AI


Key Responsibilities

 

Data Pipeline Development


• Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.

• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.

• Develop and maintain data processing workflows using Delta Lake architecture.

• Ensure efficient data ingestion, transformation, and loading processes.


Data Management & Optimisation

• Optimise performance of data pipelines and processing workloads.

• Ensure data quality, reliability, and consistency across data platforms.

• Write and optimise complex SQL queries for data processing and validation.

• Implement data governance and access controls.


Cloud & Data Platform Integration

• Work with Azure Data Lake for storing and managing large datasets.

• Manage data cataloging and governance using Unity Catalog.

• Support scalable and secure cloud-based data architectures.


Collaboration & Migration Projects

• Work closely with cross-functional teams to deliver data-driven solutions.

• Participate in data migration and modernisation initiatives.

• Support troubleshooting and resolution of data pipeline issues.

• Document data architecture, pipelines, and processes.


 Required Skills & Experience 

·        Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills.

·        Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.

·        Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL).

·        Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.

·        Optimise performance of data processing and ensure data quality standards.

·        Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.

·        Experience with orchestration tools like Airflow or Azure Data Factory is a plus.

·        Have working experience on a migration project. 

·        Preferable to have experience in Informatica (although not mandatory)

  

Relevant Experience Preferred 

• Hands-on experience developing data pipelines using Azure Databricks and PySpark.

• Experience implementing ETL/ELT solutions for enterprise data platforms.

• Experience optimizing data processing performance and data workflows.

• Experience working with Azure Data Lake and cloud-based data architectures.

• Experience working on data migration or modernization projects.



Kumaran Systems Tamil Nādu, IND Office

SIPCOT IT Park, Plot No.28/A-41, III Main Road, 6th Cross, , , Tamilnadu , India, 603 103,

Kumaran Systems Chennai, Tamil Nadu, IND Office

III Main Road, 6th Cross,SIPCOT IT Park, Siruseri, ,, Chennai, India, 603103

Kumaran Systems Chennai, Tamil Nadu, IND Office

8th Floor (Level 9), “ALTIUS C” Block, Olympia Technology park, SIDCO industrial Estate, Guindy, – , Chennai, India, 600032

Similar Jobs

9 Days Ago
In-Office
Senior level
Senior level
Information Technology • Software • Analytics
Designs and maintains scalable Azure-based ETL/ELT pipelines using Data Factory, Databricks, Python, SQL, PySpark, and Spark. Integrates diverse data sources, develops data models and warehouse structures, implements CDC and batch processing, and optimizes queries and distributed workloads. Manages orchestration, monitoring, error handling, CI/CD, documentation, and production support using Airflow, Git, GitHub, and Azure DevOps. Collaborates with cross-functional teams to troubleshoot issues and deliver reliable data solutions.
Top Skills: Apache AirflowSparkAzure Blob StorageAzure Data FactoryAzure Data Lake Storage Gen2Azure DatabricksAzure DevopsAzure Key VaultAzure MonitorAzure Synapse AnalyticsCi/CdCsvDelta LakeGitGitJSONMicrosoft Sql ServerMySQLNumpyOraclePandasParquetPostgresPysparkPythonSQLXML
12 Days Ago
Hybrid
Senior level
Senior level
Blockchain • Database • Analytics
Designs and maintains scalable Databricks ETL/ELT pipelines using Python, PySpark, SQL, Spark, and Delta Lake. Manages Databricks infrastructure, Jobs, Workflows, clusters, access controls, and performance optimization. Implements data governance through Unity Catalog, data quality checks, lineage, metadata management, cataloging, and governed access. Integrates diverse data sources, troubleshoots pipeline issues, supports data security and compliance, and collaborates with teams on enterprise data management practices.
Top Skills: Amazon S3SparkCi/CdDatabricksDelta LakeGitPysparkPythonRest ApisSQLUnity Catalog
17 Days Ago
In-Office
Senior level
Senior level
Artificial Intelligence • Cloud • Information Technology • Machine Learning • Natural Language Processing • Consulting
Build, optimize, and scale end-to-end data pipelines using Databricks, SQL, Python, and Spark. Improve ETL/ELT performance, reliability, and cost efficiency; implement data quality, governance, and security practices; troubleshoot distributed data workflows; collaborate with engineers, analysts, and stakeholders; and support cloud data platform migration and modernization initiatives.
Top Skills: SparkAWSAzureDatabricksDatabricks WorkflowsDelta LakeGitJfrog ArtifactoryPythonShell ScriptingSQL

What you need to know about the Chennai Tech Scene

To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account