The Middle Data Engineer will design and maintain data pipelines, manage cloud-based solutions, and resolve data issues while collaborating with teams.
N-iX is looking a proactive Middle/Senior Big Data Engineer to join our vibrant team.
As a Big Data Engineer, you will play a critical role in designing, developing, and maintaining sophisticated data pipelines, Ontology Objects, and Foundry Functions within Palantir Foundry. The ideal candidate will possess a robust background in cloud technologies, data architecture, and a passion for solving complex data challenges.
Key Responsibilities:
- Collaborate with cross-functional teams to understand data requirements, and design, implement and maintain scalable data pipelines in Palantir Foundry, ensuring end-to-end data integrity and optimizing workflows.
- Gather and translate data requirements into robust and efficient solutions, leveraging your expertise in cloud-based data engineering. Create data models, schemas, and flow diagrams to guide development.
- Develop, implement, optimize and maintain efficient and reliable data pipelines and ETL/ELT processes to collect, process, and integrate data to ensure timely and accurate data delivery to various business applications, while implementing data governance and security best practices to safeguard sensitive information.
- Monitor data pipeline performance, identify bottlenecks, and implement improvements to optimize data processing speed and reduce latency.
- Troubleshoot and resolve issues related to data pipelines, ensuring continuous data availability and reliability to support data-driven decision-making processes.
- Stay current with emerging technologies and industry trends, incorporating innovative solutions into data engineering practices, and effectively document and communicate technical solutions and processes.
Tools and skills you will use in this role:
- Palantir Foundry
- Python
- PySpark
- SQL
- TypeScript
Requirements:
- 3-4+ years of experience in data engineering, preferably within the pharmaceutical or life sciences industry;
- Strong proficiency in Python and PySpark;
- Proficiency with big data technologies (e.g., Apache Hadoop, Spark, Kafka, BigQuery, etc.);
- Hands-on experience with cloud services (e.g., AWS Glue, Azure Data Factory, Google Cloud Dataflow);
- Expertise in data modeling, data warehousing, and ETL/ELT concepts;
- Hands-on experience with database systems (e.g., PostgreSQL, MySQL, NoSQL, etc.);
- Proficiency in containerization technologies (e.g., Docker, Kubernetes);
- Effective problem-solving and analytical skills, coupled with excellent communication and collaboration abilities;
- Strong communication and teamwork abilities;
- Understanding of data security and privacy best practices;
- Strong mathematical, statistical, and algorithmic skills.
Nice to have:
- Certification in Cloud platforms, or related areas;
- Experience with search engine Apache Lucene, Webservice Rest API;
- Familiarity with Veeva CRM, Reltio, SAP, and/or Palantir Foundry;
- Knowledge of pharmaceutical industry regulations, such as data privacy laws, is advantageous;
- Previous experience working with JavaScript and TypeScript.
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Top Skills
Apache Hadoop
Aws Glue
Azure Data Factory
BigQuery
Docker
Google Cloud Dataflow
Kafka
Kubernetes
MySQL
NoSQL
Palantir Foundry
Postgres
Pyspark
Python
Spark
SQL
Typescript
Similar Jobs
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
Lead change management for S4/o9 transformation in MEU, driving stakeholder engagement and developing change plans to improve business processes.
Top Skills:
Change Management MethodologySap S/4 HanaTraining Programs
Digital Media • Gaming • Information Technology • Software • Sports • Esports • Big Data Analytics
Design and build AI-powered tools to enhance engineering workflows, mentor engineers, and integrate intelligent agents to boost productivity.
Top Skills:
C#GoJavaScriptPythonTypescript
Artificial Intelligence • Information Technology • Machine Learning • Natural Language Processing • Productivity • Software • Generative AI
As a Compliance Program Manager, you will oversee audit execution, control testing, and vendor risk management while unifying compliance strategies. You will manage audit cycles and coordinate with cross-functional teams to ensure compliance across multiple frameworks, impacting organizational compliance posture significantly.
Top Skills:
AWSCi/CdGitIso 27001Iso 27017Iso 27018Iso 27701Iso 42001Pci DssSoc 2
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
