The Lead Azure Data Engineer will design and optimize data pipelines in Databricks, develop ETL workflows, implement Delta Lake, and collaborate with architects for scalable data solutions. Key responsibilities include managing data quality and leveraging machine learning tools as needed.
Role : Azure Data Engineer - Lead
Exp: 8+ yrs
Loc: Bangalore / Noida (Hybrid)
Job Description :
- Data Engineering with Databricks:
- Design, implement, and optimize data pipelines in Databricks to process large-scale structured and unstructured data.
- Develop ETL workflows for data ingestion, transformation, and integration using Databricks and other cloud-based tools.
- Implement Delta Lake for efficient data management and real-time data processing.
- Collaborate with data architects and stakeholders to build scalable and reusable data solutions.
- Leverage Databricks' MLflow for managing and deploying machine learning workflows, if required.
- Ensure data quality, integrity, and compliance with organizational standards and governance policies.
Required Skills & Qualifications:
- Education: Bachelor’s or master’s degree in computer science, Data Analytics, Information Technology, or a related field.
- Technical Proficiency:
- Hands-on experience with Databricks: Spark, Pyspark, Scala, or SQL
- Strong understanding of database concepts and working experience with SQL/NoSQL databases.
- Analytical Skills: Strong ability to analyse, interpret, and visualize large datasets.
- Problem Solving: Demonstrated ability to troubleshoot complex data workflows and reporting challenges.
- Soft Skills: Strong written and verbal communication, collaboration, and stakeholder management.
Top Skills
Pyspark
Scala
Spark
SQL
Similar Jobs
Be an Early Applicant
As an Azure Data Engineer at Haleon, you will participate in fostering an inclusive work environment, helping to harness diverse perspectives to enhance data management and analytics processes. This role emphasizes flexibility and collaboration within a diverse workforce, ensuring the development of equitable opportunities.
Be an Early Applicant
As a Data Engineer - Azure, you will innovate and maintain technology services for large-scale data engineering projects, focusing on Azure components while managing teams and problem-solving to deliver complex client solutions effectively.
Be an Early Applicant
As an Azure Data Engineer, design and optimize data pipelines using Databricks for large-scale data processes. Develop ETL workflows, implement Delta Lake for real-time data management, and collaborate with stakeholders to build scalable data solutions. Ensuring data integrity and leveraging MLflow for machine learning deployment are also key responsibilities.
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
