Design, implement, and optimize Apache DolphinScheduler workflows and ETL pipelines. Integrate with Hadoop, Spark, Flink, Hive, and Kafka. Develop custom plugins, troubleshoot performance and scalability issues, monitor job execution and resource allocation, and maintain CI/CD, version control, and infrastructure automation while collaborating with data engineers and DevOps teams.
Key Responsibilities:
- Develop, configure, and optimize Apache DolphinScheduler workflows for data processing and automation.
- Design and implement ETL pipelines, job scheduling, and workflow orchestration solutions.
- Troubleshoot and resolve performance, scalability, and reliability issues in DolphinScheduler.
- Integrate DolphinScheduler with big data tools such as Hadoop, Spark, Flink, Hive, and Kafka.
- Work closely with data engineers, DevOps, and software teams to enhance workflow automation.
- Develop custom plugins and extensions for DolphinScheduler as needed.
- Monitor and optimize job execution, resource allocation, and workflow efficiency.
- Maintain best practices for CI/CD, version control, and infrastructure automation.
Required Skills & Qualifications:
- Strong experience with Apache DolphinScheduler in a production environment.
- Proficiency in Java, Python, or Scala for workflow scripting and automation.
- Experience with big data technologies like Hadoop, Spark, Hive, Flink, and Kafka.
- Understanding of workflow orchestration, DAG execution, and scheduling strategies.
- Strong problem-solving skills and ability to work in a fast-paced environment.
Photon Chennai, Tamil Nadu, IND Office
DLF IT Park 1/124 Mount Poonamallee Road Sivaji Gardens Manapakkam , Chennai, India, 600089
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Build and scale machine learning systems for search, including ranking models, feature engineering, model deployment, monitoring, experimentation, and online inference. Design fault-tolerant distributed infrastructure, optimize search latency and throughput, and develop reliable ML services from proof of concept through production. Collaborate across engineering and product teams, communicate technical decisions, review code, document systems, and mentor junior engineers.
Top Skills:
AWSAzureCi/CdDistributed SystemsGCPGoJavaKotlinMachine LearningMlopsPythonTypescript
Digital Media • Information Technology • News + Entertainment
Designs and customizes software applications, manages releases, gathers requirements, and improves functionality and user experience. Collaborates with stakeholders, QA, and cross-functional teams to deliver reliable solutions. Tracks performance metrics, maintains technical documentation, researches industry trends, and leads development, prototyping, and design sessions. Mentors junior engineers and exercises independent judgment while supporting variable work schedules.
.jpeg)
Artificial Intelligence • Hardware • Information Technology • Machine Learning
Manage SSD quality and reliability during high-volume production, including reliability evaluations, deviation management, containment, failure analysis, data analysis, and outgoing quality monitoring. Develop evaluation plans, assess material dispositions, analyze MES data, report reliability metrics, and collaborate with engineering, manufacturing, quality, and customer teams globally. Provide technical consultation, training, audit and customer support, while using generative and agentic AI tools to improve engineering workflows.
Top Skills:
Agentic AiChatgptClaudeDevice PhysicsGenerative AiMamMicron Manufacturing Execution System (Mes)Microsoft CopilotNandNvmePcieSataSemiconductor Reliability StatisticsSolid State Drives (Ssd)Wafer Fabrication
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
