Designs, builds, and optimizes scalable batch and real-time data pipelines on GCP using Dataflow, Apache Beam, Java, and BigQuery. Integrates diverse data sources, improves pipeline performance and reliability, supports data governance and security, troubleshoots production issues, and collaborates with architects, analysts, DevOps, and business stakeholders.
We are looking for a highly skilled Senior Data Engineer with strong expertise in Google Cloud Platform (GCP), Apache Beam/Dataflow, Java, and BigQuery. The ideal candidate will be responsible for designing, building, and optimizing scalable data pipelines and cloud-based data processing solutions for enterprise applications.
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines using GCP services.
- Build and maintain real-time and batch data processing pipelines using Dataflow and Apache Beam.
- Develop robust backend processing solutions using Java.
- Work extensively with BigQuery for data warehousing, analytics, and performance optimization.
- Integrate data from multiple sources including APIs, databases, and streaming platforms.
- Optimize pipeline performance, cost, scalability, and reliability on GCP.
- Collaborate with cross-functional teams including Architects, Analysts, DevOps, and Business stakeholders.
- Ensure data quality, governance, monitoring, and security best practices.
- Troubleshoot production issues and provide long-term scalable solutions.
Requirements
- Strong hands-on experience with GCP services.
- Expertise in Google Dataflow / Apache Beam.
- Strong programming skills in Java.
- Hands-on experience with BigQuery.
- Experience in building batch and streaming data pipelines.
- Good knowledge of SQL and data modeling concepts.
- Experience with CI/CD, Git, and Agile methodologies.
- Strong problem-solving and debugging skills.
Good to Have
- Experience with Pub/Sub, Cloud Composer, Cloud Storage, Dataproc, or Kafka.
- Exposure to Python or Spark.
- Experience in handling large-scale enterprise data platforms.
Preferred Qualifications
- Bachelor’s/Master’s degree in Computer Science, Engineering, or related field.
- GCP certifications are an added advantage.
Similar Jobs
eCommerce • Retail • Software
Leads the design, development, optimization, and support of ETL/ELT pipelines, data warehouses, data models, and cloud-based data solutions. Writes advanced SQL, implements testing and CI/CD practices, monitors production systems, resolves incidents, and improves performance and cost efficiency. Partners with analysts, data scientists, and stakeholders to deliver reliable datasets while documenting solutions and mentoring associate engineers.
Top Skills:
AirflowApache FlinkAWSAzureAzure Data FactoryCi/CdDbtGCPGitKafkaLuigiPythonSnowflakeSpark StreamingSQL
eCommerce • Retail • Software
Lead the design, development, and scaling of a digital experience analytics platform. Build data foundations and AI-driven decision systems using Azure, Databricks, dbt, Python, SQL, and cloud data technologies. Guide junior engineers, collaborate with product and analytics teams, support data architecture, modeling, governance, quality, and operationalization, and coordinate cloud deployments and technical delivery across onsite and offshore teams.
Top Skills:
Adobe AnalyticsAgentic AiAzureAzure Data EngineeringAzure Data FactoryAzure Data LakeCi/CdDatabricksDbtDevOpsGCPInfrastructure As CodeLinuxPower BIPythonSnowflakeSQL
Blockchain • Database • Analytics
Lead the design, development, and optimization of scalable Databricks ETL/ELT pipelines. Integrate data from databases, S3, files, and REST APIs; implement Delta Lake medallion architecture, Unity Catalog, dimensional models, data quality controls, and Spark performance tuning. Schedule and monitor Databricks workflows, collaborate cross-functionally, and guide engineers while driving technical decisions.
Top Skills:
AirflowAmazon S3SparkAuto LoaderAWSCi/CdDatabricksDatabricks Lakehouse PlatformDbtDelta LakeGitKafkaPysparkPythonRest ApisSQLUnity Catalog
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
