Design and maintain scalable pipelines for visual and voice data, including ingestion, processing, validation, testing, observability, and automation. Build cloud, on-premise, and containerized data infrastructure; optimize storage, retrieval, and caching for large media assets; debug complex data issues; and collaborate with data scientists, operators, and product teams to support machine learning workflows.
As a Data Engineer, you’ll architect and maintain the pipelines that power our products and services. You’ll work at the intersection of ML, media processing, and infrastructure; owning the data tooling and automation layer that enables scalable, high-quality training and inference. If you’re a developer who loves solving tough problems and building efficient systems, we want you on our team.
Key Responsibilities
- Design and maintain scalable pipelines for ingesting, processing, and validating datasets with main focus visual and voice data.
- Work with other teams to identify workflow optimisation potential, design and develop automation tools, using AI-driven tools and custom model integrations and scripts.
- Write and maintain tests for pipeline reliability.
- Build and maintain observability tooling in collaboration with other engineers to track data pipeline health and system performance.
- Collaborate with data scientists, operators, and product teams to deliver data solutions.
- Debug and resolve complex data issues to ensure system performance.
- Optimise storage, retrieval, and caching strategies for large media assets across environments.
- Deploy scalable data infrastructure using cloud platforms as well as on-premise and containerization.
- Deepen your knowledge of machine learning workflows to support AI projects.
- Stay current with industry trends and integrate modern tools into our stack.
Must Haves
- 3+ years in data engineering or related backend/infrastructure role.
- Strong programming skills in Python or similar languages.
- Experience with software development lifecycle (SDLC) and CI/CD pipelines.
- Proven experience building and testing data pipelines in production.
- Proficiency in Linux.
- Solid SQL knowledge.
- Experience with Docker or other containerisation technologies.
- Proactive approach to solving complex technical challenges.
- Passion for system optimisation and continuous learning.
- Ability to adapt solutions for multimedia data workflows.
Nice to Have
- Experience with Kubernetes (k8s).
- Knowledge of machine learning or AI concepts.
- Familiarity with ETL tools or big data frameworks.
- Familiarity with cloud platforms (e.g., AWS, GCP, Azure).
About You
- Innovative
- Like challenges
- Adaptable
- Calm under pressure
- Strong communication abilities
Similar Jobs
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Develop and maintain Azure Databricks data pipelines using PySpark, Spark SQL, SQL, and Delta Lake. Support Informatica-to-Databricks modernization and HR data integrations, build source-to-target mappings and dimensional models, perform profiling, testing, reconciliation, and defect resolution, and create technical documentation. Apply data governance and access controls through Unity Catalog, participate in agile ceremonies and code reviews, and escalate risks while working under senior direction.
Top Skills:
SparkAzure Data FactoryAzure Data Lake StorageAzure DatabricksAzure DevopsAzure SqlAzure SynapseCi/CdDelta LakeGitGithub CopilotInformaticaMicrosoft PurviewOracle Hcm CloudPysparkPythonSpark SqlSQLUnity Catalog
Fintech • Information Technology • Financial Services
Build, deploy, and support production-grade data pipelines and platforms for structured and semi-structured datasets. Develop Snowflake data models and advanced SQL ELT transformations, implement workflow orchestration and data-quality controls, and improve reliability, observability, lineage, and performance. Partner with product teams and data specialists, troubleshoot production issues, modernize legacy pipelines, and deliver maintainable solutions using Python and cloud technologies.
Top Skills:
Amazon Web Services (Aws)APIsAzureContinuous Integration And Continuous Delivery (Ci/Cd)Dag Workflow OrchestrationData ModelingData WarehousingDistributed Data Analytics EnginesEltEvent Streaming And MessagingGitPythonSnowflakeSQL
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Design, develop, optimize, and support scalable batch and real-time data pipelines using Kafka, Spark, Scala/PySpark, SQL, Python, and orchestration tools. Build reliable ETL/ELT workflows, optimize distributed processing, manage cloud data lake storage, troubleshoot production incidents, perform root cause analysis, conduct code reviews, and mentor junior engineers. Independently deliver complex data engineering solutions with emphasis on scalability, performance, reliability, and maintainability.
Top Skills:
Amazon S3Apache AirflowApache KafkaApache NifiSparkAzure Data FactoryAzure Data Lake StorageData LakesPysparkPythonScalaSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
