Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.
We are looking for a highly experienced Azure Databricks (ADB) Lead with 10+ years of experience in data engineering, cloud platforms, and big data processing. The ideal candidate will have expertise in Azure Data Services, Databricks, Spark, and data pipelines to design, develop, and optimize large-scale data solutions.
Experience- 10+ years
Location- Kochi/Chennai/Coimbatore
Key Responsibilities:
• Lead the design, architecture, and implementation of Azure Databricks solutions for data processing and analytics.
• Develop and optimize ETL/ELT pipelines using Apache Spark, Python (PySpark), Scala, and SQL.
• Work closely with data engineers, data scientists, and business stakeholders to ensure efficient data workflows.
• Implement data governance, security, and compliance best practices in Databricks and Azure Data Lake.
• Optimize performance tuning, cost management, and scalability of Databricks workloads.
• Automate CI/CD pipelines for Databricks notebooks using DevOps tools like Azure DevOps, Terraform, or GitHub Actions.
• Troubleshoot and resolve issues related to big data processing, Spark jobs, and cloud storage.
• Stay updated with Azure and Databricks best practices and mentor junior team members.
• Required Skills & Experience:
10+ years of experience in data engineering, big data, or cloud-based data solutions.
• Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.
• Expertise in Python (PySpark), Scala, SQL, and data transformation techniques.
• Deep understanding of Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory, etc.).
• Experience with data modeling, ETL/ELT design, and distributed data processing.
• Proficiency in DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and version control (Git).
•Familiarity with MLflow, Unity Catalog, and Databricks SQL is a plus. • Strong analytical, problem-solving, and communication skills.
Preferred Qualifications:
• Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
• Experience with containerization (Docker, Kubernetes) and orchestration tools.
• Knowledge of real-time data streaming using Kafka, Event Hubs, or similar technologies.
• Exposure to Machine Learning (ML) workflows in Databricks is a plus.
What we provide:
• Competitive Salary
• Performance based Variable Pay
• Health, Vision, Dental, and Life Insurance
• Comprehensive Employee Wellness Plan
• Retirement plan
• Paid Holidays and Vacation Time Accrual
• Encouraging and highly energetic environment
• Personal and Professional Growth Opportunities
Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.
Candidate Privacy Policy
Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:
- What information we collect during our application and recruitment process and why we collect it;
- How we handle that information; and
- How to access and update that information.
Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.
Orion Innovation Chennai, Tamil Nadu, IND Office
Ambit IT Park, Ambit Park Road Ambattur Industrial Estate, Chennai, India, 600 058
