Design and optimize data pipelines using Azure Databricks and Spark, ensuring data reliability and quality through ETL/ELT workflows and CI/CD automation.
Key Responsibilities Design, develop, and optimize data pipelines using Azure Databricks. Implement ETL/ELT workflows leveraging Spark (PySpark/Scala). Integrate and manage data from multiple sources such as Azure Data Lake, SQL DB, Synapse, and external APIs. Design and enforce data quality rules, validation checks, and monitoring frameworks. Work with data quality tools to profile, cleanse, and validate data. Ensure data pipelines follow best practices for performance, scalability, and cost optimization. Collaborate with data engineers, analysts, and business stakeholders to ensure data reliability. Implement CI/CD pipelines and automation for Databricks deployments. Maintain documentation for data processes, pipelines, and quality rules.
Hexaware Technologies Chennai, Tamil Nadu, IND Office
MCN Nagar Ext, MCN Nagar Extension, Thoraipakkam, Chennai, Tamil Nadu, India, 600097
Similar Jobs
Information Technology • Consulting
Design and optimize scalable data pipelines and analytics platforms using Snowflake, Airflow, and SQL. Ensure data quality and drive performance optimization.
Top Skills:
AirflowAWSAzureDbtDebeziumDockerFivetranGCPGitKafkaKinesisKubernetesPub/SubPythonSnowflakeSQLTerraform
Information Technology • Consulting
Design, build, and optimize scalable data pipelines and analytics solutions on Snowflake. Collaborate with teams for data products and enforce security and governance.
Top Skills:
SnowflakeSQL
Information Technology • Consulting
The role requires leading big data projects with hands-on experience in Azure Databricks, Pyspark coding, advanced SQL, and data warehouse management.
Top Skills:
Azure DatabricksPysparkPythonSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
