Key Responsibilities
Data Pipeline Development
• Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.
• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop and maintain data processing workflows using Delta Lake architecture.
• Ensure efficient data ingestion, transformation, and loading processes.
Data Management & Optimisation
• Optimise performance of data pipelines and processing workloads.
• Ensure data quality, reliability, and consistency across data platforms.
• Write and optimise complex SQL queries for data processing and validation.
• Implement data governance and access controls.
Cloud & Data Platform Integration
• Work with Azure Data Lake for storing and managing large datasets.
• Manage data cataloging and governance using Unity Catalog.
• Support scalable and secure cloud-based data architectures.
Collaboration & Migration Projects
• Work closely with cross-functional teams to deliver data-driven solutions.
• Participate in data migration and modernisation initiatives.
• Support troubleshooting and resolution of data pipeline issues.
• Document data architecture, pipelines, and processes.
Required Skills & Experience
· Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills.
· Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.
· Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL).
· Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
· Optimise performance of data processing and ensure data quality standards.
· Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.
· Experience with orchestration tools like Airflow or Azure Data Factory is a plus.
· Have working experience on a migration project.
· Preferable to have experience in Informatica (although not mandatory)
Relevant Experience Preferred
• Hands-on experience developing data pipelines using Azure Databricks and PySpark.
• Experience implementing ETL/ELT solutions for enterprise data platforms.
• Experience optimizing data processing performance and data workflows.
• Experience working with Azure Data Lake and cloud-based data architectures.
• Experience working on data migration or modernization projects.
Kumaran Systems Tamil Nādu, IND Office
SIPCOT IT Park, Plot No.28/A-41, III Main Road, 6th Cross, , , Tamilnadu , India, 603 103,
Kumaran Systems Chennai, Tamil Nadu, IND Office
III Main Road, 6th Cross,SIPCOT IT Park, Siruseri, ,, Chennai, India, 603103
Kumaran Systems Chennai, Tamil Nadu, IND Office
8th Floor (Level 9), “ALTIUS C” Block, Olympia Technology park, SIDCO industrial Estate, Guindy, – , Chennai, India, 600032


