Design, build, and support production-grade AWS data pipelines that transform operational data into secure, high-quality, AI-ready datasets. Responsibilities include distributed data processing, Parquet curation, privacy-preserving transformations, orchestration, data quality monitoring, schema management, CI/CD, infrastructure as code, metadata and lineage management, troubleshooting, and reliable backfills.
This is a remote position.
Role OverviewWe are looking for Data Engineers at Senior and Mid-Level to join our team in building a privacy-preserving data platform where data engineering meets production-grade software engineering.
You will work on designing, developing, and maintaining reliable data pipelines that transform operational data into high-quality, secure, and AI-ready datasets.
- Build and maintain production-grade data pipelines on AWS.
- Extract, transform, validate, and curate large-scale Parquet datasets.
- Implement data de-identification, masking, and privacy-preserving transformations.
- Design and maintain data pipeline orchestration, scheduling, retries, and backfill mechanisms.
- Implement comprehensive data quality checks, monitoring, and alerting.
- Work with workflow orchestration tools such as Airflow, Dagster, or AWS Step Functions.
- Contribute to CI/CD pipelines and Infrastructure as Code (IaC) practices.
- Manage schema evolution and schema drift across data sources and pipelines.
- Provide production support, troubleshooting, and root cause analysis for data pipeline issues.
- Maintain data catalogs, metadata, and data lineage.
- Follow software engineering best practices including Git, code reviews, automated testing, and maintainable code.
- Build reliable and idempotent data pipelines capable of handling retries and large-scale backfills.
- Strong proficiency in Python and SQL.
- Hands-on experience with AWS data services and production data pipelines.
- Experience with Apache Spark or equivalent distributed data processing technologies.
- Practical experience with Airflow, Dagster, AWS Step Functions, or similar orchestration tools.
- Strong understanding of data pipeline architecture, ETL/ELT, and data transformation.
- Experience working with Parquet and large-scale datasets.
- Understanding of data quality, schema management, monitoring, and alerting.
- Strong software engineering practices including:
- Git and version control
- Code reviews
- Automated testing
- Idempotency
- Error handling
- Retries and backfills
- Git and version control
- Experience supporting and troubleshooting production data pipelines.
- Ability to work effectively with cross-functional engineering and data teams.
Requirements Required Skills
Python | SQL | AWS | Data Engineering | Data Pipelines | ETL/ELT | Apache Spark | Parquet | Airflow | Dagster | AWS Step Functions | Data Orchestration | Data Quality | Schema Management | Data Transformation | Production Support | Git | CI/CD | Automated Testing | Data De-identification | Data Lineage | Data Catalog
Debezium | AWS DMS | Apache Iceberg | Delta Lake | Apache Hudi | Data Masking | Data Tokenization | Terraform | CloudFormation | ML/AI Training Data | Privacy-Preserving Data
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Build and scale machine learning systems for search, including ranking models, feature engineering, model deployment, monitoring, experimentation, and online inference. Design fault-tolerant distributed infrastructure, optimize search latency and throughput, and develop reliable ML services from proof of concept through production. Collaborate across engineering and product teams, communicate technical decisions, review code, document systems, and mentor junior engineers.
Top Skills:
AWSAzureCi/CdDistributed SystemsGCPGoJavaKotlinMachine LearningMlopsPythonTypescript
Digital Media • Information Technology • News + Entertainment
Designs and customizes software applications, manages releases, gathers requirements, and improves functionality and user experience. Collaborates with stakeholders, QA, and cross-functional teams to deliver reliable solutions. Tracks performance metrics, maintains technical documentation, researches industry trends, and leads development, prototyping, and design sessions. Mentors junior engineers and exercises independent judgment while supporting variable work schedules.
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Manage SSD quality and reliability during high-volume production, including reliability evaluations, deviation management, containment, failure analysis, data analysis, and outgoing quality monitoring. Develop evaluation plans, assess material dispositions, analyze MES data, report reliability metrics, and collaborate with engineering, manufacturing, quality, and customer teams globally. Provide technical consultation, training, audit and customer support, while using generative and agentic AI tools to improve engineering workflows.
Top Skills:
Agentic AiChatgptClaudeDevice PhysicsGenerative AiMamMicron Manufacturing Execution System (Mes)Microsoft CopilotNandNvmePcieSataSemiconductor Reliability StatisticsSolid State Drives (Ssd)Wafer Fabrication
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
