The Senior Data Engineer will design and implement scalable data systems, focusing on AWS-based architectures, data pipelines, and performance tuning, while mentoring other engineers.
Description
We are building a greenfield analytics platform supporting both batch and real-time data processing. We are looking for a Senior Data Engineer who can design, implement, and evolve scalable data systems in AWS.
This role combines hands-on development, architectural decision-making, and platform ownership.
Core Responsibilities:
- Design and implement batch and streaming data pipelines using Apache Spark.
- Build and evolve a scalable AWS-based data lake architecture.
- Develop and maintain real-time data processing systems (event-driven pipelines).
- Own performance tuning and cost optimization of Spark workloads.
- Define best practices for data modeling, partitioning, and schema evolution.
- Implement monitoring, observability, and data quality controls.
- Contribute to infrastructure automation and CI/CD for data workflows.
- Participate in architectural decisions and mentor other engineers.
Required Qualifications:
- 5+ years of experience in Data Engineering.
- Strong hands-on experience with Apache Spark (including Structured Streaming).
- Experience building both batch and streaming pipelines in production environments.
- Proven experience designing AWS-based data lake architectures: S3, EMR, Glue, Athena.
- Experience with event streaming platforms such as Apache Kafka or Amazon Kinesis.
- Experience implementing lakehouse formats such as Delta Lake.
- Strong understanding of partitioning strategies and schema evolution.
- Experience using SparkUI and AWS CloudWatch for profiling and optimization.
- Strong understanding of Spark performance tuning (shuffle, skew, memory, partitioning).
- Proven track record of cost optimization in AWS environments.
- Experience with Docker and CI/CD pipelines.
- Experience with Infrastructure as Code: Terraform, AWS CDK.
- Familiarity with monitoring and observability practices.
- Experience in the Financial domain.
- Experience running Spark workloads on Kubernetes.
- Experience implementing data quality frameworks or metadata/lineage systems.
- English - B2, Ukrainian- Native MUST
Top Skills
Amazon Kinesis
Apache Kafka
Spark
Athena
AWS
Aws Cdk
Aws Cloudwatch
Ci/Cd
Delta Lake
Docker
Emr
Glue
S3
Sparkui
Terraform
Similar Jobs
Information Technology • Internet of Things
The Senior Data Engineer designs, builds, and maintains data infrastructure for large data sets, focusing on ETL processes and collaboration with stakeholders on data projects.
Top Skills:
AirflowAthenaAWSEc2EmrFirehoseKinesisLambdaPythonS3ScalaSpark
Information Technology • Consulting
As a Senior Data Analyst, you'll extract insights from data, advise stakeholders, conduct analyses, and manage tracking and A/B tests to enhance the user experience for a car market platform.
Top Skills:
A/B TestingAWSAws Quick SightBig DataDbtDockerGa4GtmPythonQlikSalesforceSnowflakeSQL
Software
As a Senior Data Engineer, you will design and implement data ingestion pipelines, manage data orchestration, and support machine learning workflows, contributing to an integrated platform for the public sector.
Top Skills:
AirflowApache AtlasApache RangerSparkCloudera Data PlatformCloudera Machine LearningHdfsHiveIcebergImpalaKafkaPythonS3ScalaSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
