Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.
Job overview:
We are seeking a Senior Data Engineer with strong expertise in building and optimizing large-scale data platforms to support advanced analytics, AI, and Generative AI (GenAI) use cases. The ideal candidate will have hands-on experience with Snowflake and Databricks, combined with the ability to design scalable data pipelines and enable AI/ML integration. This role requires strong data engineering fundamentals, cloud expertise, and collaboration with data scientists, AI architects, and business stakeholders.
Key Responsibilities:
- Design, build, and maintain large-scale data pipelines for batch and real-time processing.
- Develop and optimize data models, ETL/ELT processes, and data warehouses/lakes on Snowflake and Databricks.
- Ensure enterprise data quality, lineage, governance, and security.
- Apply best practices for cost efficiency, scalability, and observability in cloud-native platforms.
- Collaborate with AI/ML teams to prepare datasets for training, fine-tuning, and evaluation.
- Integrate pipelines with AI/GenAI workflows including LLMs, embeddings, vector databases, and RAG.
- Enable feature engineering and feature store integration for ML applications.
- Support responsible AI by ensuring datasets are accurate, diverse, and compliant.
- Work with architects, data scientists, and product owners on end-to-end solutions.
- Mentor junior engineers, enforce coding standards, and create reusable frameworks.
- Contribute to design sessions, architecture reviews, and sprint planning.
Key Skills:
- 7+ years of experience in data engineering with large-scale, enterprise-grade systems.
- Strong expertise with Snowflake (data modeling, query optimization, performance tuning).
- Hands-on experience with Databricks (PySpark, Delta Lake, Unity Catalog).
- Proficiency in SQL, Python, and Spark for data processing and transformation.
- Experience with cloud platforms (AWS, Azure, GCP) and associated data services.
- Knowledge of data governance, lineage, metadata management, and security best practices.
- Familiarity with CI/CD pipelines, orchestration tools (Airflow, Dagster, Prefect), and DevOps practices.
Preferred Qualifications
- Exposure to AI/GenAI use cases, including RAG pipelines, embeddings, and model-ready data pipelines.
- Experience with vector databases (Pinecone, FAISS, Weaviate, Milvus) and feature stores.
- Familiarity with data quality frameworks (Great Expectations, dbt tests, Monte Carlo).
- Prior experience supporting real-time streaming data (Kafka, Kinesis, EventHub).
- Certifications in Snowflake, Databricks, or cloud platforms (AWS, Azure, GCP).
Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.
Candidate Privacy Policy
Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:
- What information we collect during our application and recruitment process and why we collect it;
- How we handle that information; and
- How to access and update that information.
Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.
Top Skills
Orion Innovation Chennai, Tamil Nadu, IND Office
Ambit IT Park, Ambit Park Road Ambattur Industrial Estate, Chennai, India, 600 058
