Design, develop, and maintain scalable data pipelines using AWS and other technologies, ensuring high performance, reliability, and efficiency for data operations.
About PayPay India
PayPay, a fintech company providing a service enjoyed by over 66 million users (as of Oct 2024) merely 7 years since its launch in 2018 in Japan. The company is now home to a very diverse team of members from more than 50 countries. We grew to a team of several thousand employees in Japan but are far from over. We are still in the Day 1. Every day, new members join us from all over the world to create new value and deliver it to society.
Why India ?
To build our Payment services, we got technical cooperation from Paytm (A large payment service company in India). And based on their customer-first technologies , we created and expanded the smartphone payment service in Japan. Therefore, we have decided to establish a development base in India, because it is a major IT country with many talented engineers, as evidenced by the fact that cutting-edge mobile payments can continue to be generated.
OUR VISION IS UNLIMITEDWe dare to believe that we do not need a clear vision to create a future beyond our imagination. PayPay will always stay true to our roots and realise a vision (future) that no one else can imagine by constantly taking risks and challenging ourselves. With this mindset, you will be presented with new and exciting opportunities on a daily basis and have the opportunity to grow and reach new dimensions that you could never have imagined.
PayPay's rapid growth necessitates the expansion of its product teams and underscores the critical need for a resilient Data Engineering Platform. This platform is vital to support our increasing business demands. The Data Pipeline team is tasked with creating, deploying, and managing this platform, utilizing leading technologies like Databricks, Delta Lake, Spark, PySpark, Scala, and the AWS suite.
We are actively seeking skilled Data Engineers to join our team and contribute to scaling our platform across the organization.
Main Responsibilities
- Create and manage robust data ingestion pipelines leveraging Databricks, Airflow, Kafka, and Terraform.
- Ensure high performance, reliability, and efficiency by optimizing large-scale data pipelines.
- Develop data processing workflows using Databricks, Delta Lake, and Spark technologies.
- Maintain and improve the Data Lakehouse, utilizing Unity Catalog for efficient data management and discovery.
- Construct automation, frameworks, and enhanced tools to streamline data engineering workflows.
- Collaborate across teams to facilitate smooth data flow and integration.
- Enforce best practices in observability, data governance, security, and regulatory compliance
Qualifications
- Minimum 5 years as a Data Engineer or similar role.
- Hands-on experience with Databricks, Delta Lake, Spark, and Scala.
- Proven ability to design, build, and operate Data Lakes or Data Warehouses.
- Proficiency with Data Orchestration tools (Airflow, Dagster, Prefect).
- Familiarity with Change Data Capture tools (Canal, Debezium, Maxwell).
- Strong command of at least one primary language (Scala, Python, etc.) and SQL.
- Experience with data catalog and metadata management (Unity Catalog, Lakeformation).
- Experience in Infrastructure as Code (IaC) using Terraform.
- Excellent problem-solving and debugging abilities for complex data challenges.
- Strong communication and collaboration skills.
- Capability to make informed decisions, learn quickly, and consider complex technical contexts.
Top Skills
Airflow
Apache Hudi
Aws Athena
Aws Glue
BigQuery
Dagster
Databricks
Delta Lake
Java
Lambda
Prefect
Python
Scala
Spark
SQL
Step Functions
Terraform
Similar Jobs
Big Data • Cloud • Fintech • Financial Services • Conversational AI
You will develop high-performance software for BlackRock's Aladdin investment system, ensure software resilience, and lead project teams using agile methodologies.
Top Skills:
AngularApache CassandraAWSAzureAzure DevopsGCPGrpcHibernateIgniteJavaJavaScriptJunitKafkaMavenMicroservicesMongodbReactRedisSpringTypescript
Enterprise Web • Fintech • Financial Services
As an Onboarding and Implementation Consultant, you will manage client relationships, facilitate onboarding, and ensure successful adoption of Morningstar technology.
Top Skills:
CRMMS OfficeExcelSaaS
Big Data • Cloud • Software • Database
As a Staff Product Manager, you'll lead enterprise management capabilities for MongoDB, ensuring customer success with self-hosting solutions and collaborating across teams.
Top Skills:
Advanced Management SystemsAPIsDatabasesSoftware Products
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
