Yadimen Consulting Limited Logo

Yadimen Consulting Limited

Data Engineer — PySpark + AWS Glue

Posted Yesterday
Be an Early Applicant
In-Office
Chennai, Tamil Nadu, IND
Senior level
In-Office
Chennai, Tamil Nadu, IND
Senior level
Design, develop, optimize, and maintain scalable ETL pipelines using PySpark and AWS Glue. Build data ingestion and transformation workflows, orchestrate processes with AWS Step Functions, and develop Python-based AWS Lambda functions. Support AWS data lake architectures, analytical use cases, monitoring, reliability, and performance optimization. The role also involves SQL and may include Java microservices, REST APIs, and backend integrations.
The summary above was generated by AI

Data Engineer — PySpark + AWS Glue

ETL & Data Pipelines | 2 Openings

 

 

Location: Remote (UK)

Employment Type: Contract / Permanent

Experience Level: 4–8 Years

Openings: 2

 

About the Role

We are looking for a skilled Data Engineer with solid hands-on experience in PySpark, AWS Glue, and ETL development to build and maintain scalable, production-grade data pipelines on AWS. You will be part of a delivery-focused team working on complex data engineering challenges, contributing to data lake architecture and end-to-end pipeline development.



Requirements

Key Responsibilities

       Design, develop, and maintain scalable ETL pipelines using PySpark and AWS Glue

       Build and optimise data ingestion, transformation, and loading workflows

       Orchestrate data workflows using AWS Step Functions

       Develop serverless functions using AWS Lambda (Python)

       Work with data lake architectures on AWS to support analytical use cases

       Ensure pipeline reliability, monitoring, and performance optimisation

 

Required Skills & Experience

       Strong hands-on experience with PySpark and AWS Glue

       Proven track record in ETL pipeline development and optimisation

       Experience orchestrating workflows with AWS Step Functions

       Proficiency in serverless development using AWS Lambda (Python)

       Good SQL skills and a solid understanding of data processing principles

 

Nice to Have

       Exposure to Java-based microservices

       Understanding of REST APIs and backend service integrations

       Experience working with AWS-based data lakes



Similar Jobs

21 Hours Ago
Remote or Hybrid
India
Expert/Leader
Expert/Leader
Cloud • Information Technology • Security • Software • Cybersecurity
The Country Director will lead Cloudflare's operations in SAARC, driving revenue growth, customer service, and team effectiveness while collaborating across functions in the region.
21 Hours Ago
Remote or Hybrid
India
Senior level
Senior level
Cloud • Information Technology • Security • Software • Cybersecurity
Drive partner-focused technical lifecycle for Cloudflare platform adoption (CF1 and AI). Lead pre-sales validation, build technical champions, architect solutions, accelerate partner services delivery, and automate using Cloudflare AI tools to expand partner-influenced revenue.
Top Skills: Agent EchoAgent PatAi GatewayApplication SecurityCloudflare One (Cf1)Cloudflare OsDnsHybrid CloudLlm Orchestration FrameworksMulti-CloudSaseSseVector DatabasesWorkers AiZero Trust
21 Hours Ago
Remote or Hybrid
India
Senior level
Senior level
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Work as a Business Analyst on banking digital transformation programs: elicit and document requirements as user stories/epics/features in Jira, collaborate in Agile teams, engage stakeholders across markets, challenge product owners, and apply domain expertise in Cards, Payments, Accounts, or Security/Authentication.
Top Skills: AgileJIRA

What you need to know about the Chennai Tech Scene

To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account