3Pillar Global
Lead Data Engineer (Azure PaaS & Product Engineering) (10+ Years of Exp)
Be an Early Applicant
Lead the design and implementation of scalable data engineering solutions on Azure, optimizing performance and mentoring teams. Ensure data quality and security while delivering analytics-ready datasets through collaboration.
We are building a customer-facing data platform from the ground up. The platform requires real-time data ingestion, high availability, and integration with Generative AI.
You will lead architecture decisions on Azure, sitting between Backend Engineering and Data Engineering. You will own a code-first approach: Python-based pipelines and APIs, not drag-and-drop ETL tools.
Responsibilities
- Design event-driven ingestion architectures. Select appropriate Azure compute (Functions, Durable Functions, containers) based on workload characteristics.
- Design OLTP schemas for sub-second application retrieval, not just offline reporting.
- Architect data flows for RAG workflows. Ensure unstructured text is processed and indexed for vector search.
- Build reusable Python frameworks. Enforce unit testing, CI/CD, and code reviews within the data team.
- Own synchronization between operational databases (SQL/NoSQL) and analytical workloads. Design for consistency, replay capability, and clear separation between serving and analytical layers.
Required Skills
- 10+ years in IT, with 5+ years as a Lead or Architect.
- Software engineering background (APIs, microservices) before moving into data. You understand that cloud data engineering is software development.
- Expert Python skills for backend processing, API integration, and automation. Clean, testable, production-grade code.
- Hands-on Azure experience: Functions (Consumption/Flex), Event Grid, Event Hubs. You understand tradeoffs between serverless, container-based, and dedicated compute.
- Experience building pipelines that feed LLMs or vector databases.
- Lakehouse architecture: medallion patterns, Delta Lake semantics, separation of storage and compute. Experience with Microsoft Fabric or Azure Databricks.
- Data modeling for both transactional systems and analytics. You know the difference between designing for an app vs. a data lake.
Good to Have
- DP-203, DP-600, or Databricks Data Engineer Associate certification.
- Azure AI Search experience.
- Commercial SaaS product background.
Tech Stack
- Python, SQL, PySpark, Spark, Azure Data Lake (ADLS Gen2), Azure Functions, Event Grid, Event Hubs, Delta Lake, Microsoft Fabric, Azure Databricks, REST APIs, Kafka, Airflow, Docker, Kubernetes, Git, Terraform, Data Pipelines, Streaming, Big Data, Data Governance, Data Quality
Regards
Rajan Paul(Linkedin Profile)
Talent Acquisition
3Pillar Global
Top Skills
Azure Data Factory
Azure Data Lake
Microsoft Fabric
Power BI
Python
SQL
Synapse
Similar Jobs
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Drive enterprise-wide automation initiatives by designing and implementing solutions using N8N and Tray.ai. Collaborate with teams to automate processes, integrate systems, and ensure high availability of workflows while mentoring others and reporting on impact.
Top Skills:
Gemini EnterpriseJSONMcpN8NPythonRest ApisSQLTray.AiXML
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
The role involves designing and implementing automation solutions using N8N and Tray.ai, developing AI-powered workflows, and integrating various applications while collaborating with cross-functional teams.
Top Skills:
Ci/CdCoupaDevOpsGemini EnterpriseJSONMcpN8NNetSuitePythonRest ApisSalesforceSAPSnowflakeSQLTray.IoWorkdayXML
AdTech • Artificial Intelligence • Big Data • Machine Learning • Marketing Tech • Mobile • Software
As a Product Analyst at Liftoff, you will analyze data to guide product decisions, design A/B tests, and support ML teams with performance metrics.
Top Skills:
JupyterLookerNumpyPandasPythonSQLTableau
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
