Looking for an offshore Lead Databricks/PySpark Developer who is willing to learn new technologies if needed and able to work with team. This position is long term and will likely be renewed annually.
ESSENTIAL DUTIES AND RESPONSIBILITIES
- Design and development of data ingestion pipelines (Databricks background preferred).
- Performance tune and optimize the databricks jobs
- Evaluated new features and refractors existing code
- Mentor junior developers and makes sure all patterns are documented
- Perform data migration and conversion activities.
- Develop and integrate software applications using suitable development methodologies and standards, applying standard architectural patterns, taking into account critical performance characteristics and security measures.
- Collaborate with Business Analysts, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments).
- Perform end to end automation of ETL process for various datasets that are being ingested into the big data platform.
- Maintain and support the application.
- Must be willing to flex work hours accordingly to support application launches and manage production outages if necessary
- Ensures to understand the requirements thoroughly and in detail and identify gaps in requirements
- Ensures that detailed unit testing is done, handles negative scenarios and document the same
- Work with QA and automation team.
- Works on best practices and documenting the process
- code merges and releases (Bitbucket)
- Works with architect and manager on designs and best practices
- Good data analysis skills
EDUCATION AND/OR EXPERIENCE
- Must be a team player.
- Must have following
SCALA
SQL
Spark/Spark Streaming
Big Data Tool Set
Linux
Python/PySpark
Kafka
- Experience collaborating with dev team, project managers, and engineers.
- Excellent communication and teamwork skills
About us
Whether it’s awareness, advocacy, engagement, or efficacy, we move brands forward with work that connects with audiences and delivers results. Through meaningful analytics, engaging communications and innovative technology solutions, we help clients tackle their most ambitious projects and overcome their biggest challenges.
We are an equal opportunity employer
We believe that Innovative ideas and solutions start with unique perspectives. That’s why we’re committed to providing every employee a workplace that’s free of discrimination and intolerance. We’re proud to be an equal opportunity employer and actively search for like-minded people to join our team.
We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform job functions, and to receive benefits and privileges of employment. Please contact us to request accommodation.
- Safeguard the company’s assets.
- Adhere to the company’s compliance program.
- Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
- Maintain a focus on customer-service, efficiency, quality, and growth.
- Collaborating with additional team members
- Other duties as assigned.
