Design and optimize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Build agentic workflows for data engineering automation, implement data quality and monitoring controls, optimize workloads, and support testing, deployment, troubleshooting, and production operations. Collaborate with business, data engineering, and platform teams to identify automation opportunities and develop reusable frameworks.
We are looking for a Data Engineer with strong Databricks and GCP experience to develop scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
The mandatory requirements are 5+ years of experience with Databricks and PySpark, advanced SQL, hands-on GCP experience, and experience with Delta Lake and modern lakehouse architectures.
RECRUITMENT PROCESS
1. Application — Share a few details about your experience and background.
2. Coding Challenge — If applicable, complete it in the coding language you are most comfortable with.
3. Video Interview — Record a short video introduction in English.
4. Technical Interview or Hiring Manager Interview — Discuss your experience and fit for the role.
5. Offer — If it’s a match, you’ll receive an offer.
WHAT YOU’LL GAIN
• Remote work — Work from where you feel most productive.
• Local presence in India — Work in a structured and compliant environment aligned with Indian regulations.
• Competitive compensation in INR — Receive compensation in INR, plus support for learning, education, and wellness.
• Exciting projects — Work with modern technologies for global clients and fast-growing companies.
MUST HAVES
• 5+ years of strong hands-on experience with Databricks and PySpark.
• Advanced SQL and data-processing skills.
• Hands-on experience with GCP, particularly BigQuery.
• Experience with Delta Lake and modern data lake/lakehouse architectures.
• Strong understanding of ETL/ELT, data pipeline design, performance optimization, and data quality.
• Experience building reliable, scalable, production-grade data solutions.
• Strong analytical and troubleshooting skills.
• Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
• Upper-intermediate English level.
NICE TO HAVES
• Experience developing or integrating AI/agentic workflows, AI agents, or workflow automation solutions.
• Experience applying AI to automate data engineering, validation, troubleshooting, or operational processes.
• Familiarity with orchestration and automation frameworks.
• Experience developing reusable data engineering frameworks and platform components.
• Exposure to productionizing AI-enabled solutions with appropriate validation, monitoring, and human oversight.
WHAT YOU WILL DO
• Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
• Build and operationalize agentic workflows to automate data engineering and operational processes such as data validation, issue identification, troubleshooting, and workflow execution.
• Integrate agentic capabilities with existing Databricks, GCP, BigQuery, and Delta Lake environments.
• Develop data pipelines and reusable frameworks and components to support new business requirements and datasets across multiple use cases.
• Implement data quality checks, monitoring, validation, exception handling, and production controls.
• Optimize PySpark and SQL workloads for performance, reliability, and scalability.
• Support testing, deployment, productionization, troubleshooting, and ongoing enhancement of data and agentic solutions.
• Collaborate with business, data engineering, and platform teams to identify further automation opportunities.
Similar Jobs
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Provides technical leadership for aerospace navigation database production and customer support. Engages internal and external stakeholders to resolve technical issues, delivers tailored solutions, and supports production lifecycle activities. Develops quality assurance processes aligned with DO-200 standards, conducts reviews and audits, coordinates with source suppliers, improves workflows, and mentors team members. Requires extensive aerospace and flight operations knowledge, including FMS, ARINC 424, navigation procedures, flight planning, and aeronautical charts.
Top Skills:
Aerospace Navigation DatabasesArinc 424Do-200Flight Management Systems (Fms)Honeywell Navigation Database Formats
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Designs, develops, integrates, and validates aerospace engine, power, avionics, and actuation control systems. Responsibilities include requirements engineering, system architecture, control-system analysis, MATLAB/Simulink modeling, embedded C development, sensor and actuator integration, SIL/HIL testing, verification, validation, and aerospace certification compliance. The role supports model-based development and real-time embedded applications using aerospace standards and communication protocols.
Top Skills:
AfdxArinc 429Arm CortexArp4754AArp4761C++CanDo-160Do-178CDo-254DoorsDoors NgEmbedded CHilMatlabMbsePowerpcRs232Rs422/485RtosSilSimulinkStateflowSysmlTi C2000
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Manages export compliance operations, including transaction screening, audits, risk assessments, export licenses, metrics, regulatory procedures, training, and internal controls. Ensures global trade activities comply with restricted-party, embargo, EAR, ITAR, OFAC, BIS, SCOMET, and Military End User requirements. Develops policies, coordinates license renewals, supports compliance systems access, and communicates procedural updates across the organization.
Top Skills:
BisEarItarMilitary End User RegulationsOcr EaseOfacSap GtsScomet
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
