Designs and maintains scalable Databricks and Spark data pipelines, including ETL/ELT workflows, Delta Lake transformations, data quality checks, and performance optimization. Works with cloud platforms, data warehousing, modeling, CI/CD, monitoring, and production support. Troubleshoots large-scale data processing issues and collaborates with engineers, data scientists, analysts, and business teams. Preferred experience includes Unity Catalog, orchestration tools, streaming technologies, infrastructure automation, and Databricks certifications.
We are looking for a Senior Databricks Engineer with 7+ years of experience in data engineering and strong hands-on expertise in Databricks, Apache Spark, and Claude Agent. The ideal candidate will design and build scalable data pipelines, optimize large-scale data processing, work with cloud-based data platforms, and leverage Claude Agent for AI-powered data engineering and workflow automation.
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Develop and optimize PySpark/Spark jobs for large-scale data processing.
- Build reliable ETL/ELT pipelines using Databricks workflows, Delta Lake, and SQL.
- Implement data transformations, data quality checks, and performance optimization.
- Work with cloud data platforms such as AWS, Azure, or GCP.
- Optimize Spark jobs, cluster configurations, partitioning, joins, and data processing performance.
- Design, develop, and integrate Claude Agent-based solutions for AI-powered data engineering, workflow automation, and intelligent data processing.
- Collaborate with data engineers, data scientists, analysts, and business teams to deliver data solutions.
- Implement best practices for CI/CD, version control, monitoring, and production support.
- Troubleshoot data pipeline and performance issues in production environments.
Required Skills- 7+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks.
- Strong expertise in Apache Spark / PySpark.
- Experience with Delta Lake, Databricks Workflows, and Spark SQL.
- Strong proficiency in Python and SQL.
- Experience building and optimizing large-scale ETL/ELT pipelines.
- Experience with at least one cloud platform: AWS, Azure, or GCP.
- Good understanding of data warehousing, data modeling, and distributed data processing.
- Experience with Git and CI/CD practices.
- Experience with Unity Catalog and Databricks governance.
- Experience with Azure Data Factory, AWS Glue, or similar orchestration tools.
- Experience with Claude Agent / Claude-based AI agents for data engineering or workflow automation.
- Knowledge of streaming technologies such as Kafka or Spark Structured Streaming.
- Experience with Infrastructure as Code or cloud automation.
- Databricks certifications are a plus.
Similar Jobs
Information Technology • Database • Consulting
Design, build, and optimize Databricks-based lakehouse solutions for financial crime use cases. Manage Databricks workspaces, clusters, Unity Catalog, and jobs. Develop scalable batch and streaming PySpark/Spark SQL pipelines (Kafka/Event Hubs/Kinesis) using Delta Lake, DLT, and Medallion architecture. Ensure data governance, performance tuning, error handling, and integration with cloud storage (ADLS/S3/GCS) and secret management.
Top Skills:
Amazon S3Aws Secrets ManagerAzure Data Lake Storage (Adls)Azure Event HubsAzure Key VaultChange Data Feed (Cdf)DatabricksDatabricks JobsDatabricks SecretsDatabricks Unity CatalogDelta LakeDelta Live Tables (Dlt)Google Cloud Storage (Gcs)KafkaKinesisPysparkSpark SqlStructured Streaming
Digital Media • Information Technology • News + Entertainment
Designs and customizes software applications, manages releases, gathers requirements, and improves functionality and user experience. Collaborates with stakeholders, QA, and cross-functional teams to deliver reliable solutions. Tracks performance metrics, maintains technical documentation, researches industry trends, and leads development, prototyping, and design sessions. Mentors junior engineers and exercises independent judgment while supporting variable work schedules.
Digital Media • Information Technology • News + Entertainment
Develops and optimizes data structures, ingestion frameworks, ETL pipelines, warehouses, and data lakes. Ensures data quality, governance, lineage, and compliance while supporting analytics and machine learning initiatives. Designs storage solutions across on-premises and cloud platforms, manages migrations, develops APIs, monitors data flows, collaborates with technology and analytics teams, and participates in code reviews and model deployments.
Top Skills:
Amazon RedshiftAPIsAws S3DatabricksETLKubernetesMinioTeradata
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
