AWS Developer (Python/PySpark)
Job Overview
We are seeking an experienced AWS Developer proficient in Python and PySpark to design, develop, and maintain scalable, serverless data processing and workflow automation solutions on AWS. The ideal candidate will build Lambda functions, Step Functions, Glue ETL jobs, and integrate various AWS services to support complex data pipelines and backend logic.
Key Responsibilities
- Develop, deploy, and maintain AWS Lambda functions using Python to implement backend logic, file transformations, API integrations, and event-driven triggers from S3, EventBridge, DynamoDB, and SNS.
- Implement robust error handling, retry mechanisms, timeout management, and custom logging within Lambda functions to ensure resilience and observability.
- Design and manage AWS Step Functions to orchestrate complex workflows that involve multiple Lambda functions, AWS Glue jobs, and external API calls.
- Configure parallel executions, error states, and conditional branches within Step Functions to optimize workflow reliability and efficiency.
- Integrate Step Functions with external systems and AWS services such as SNS and EventBridge using native service integrations.
- Develop and maintain AWS Glue jobs and PySpark scripts for batch ETL processes, data transformations, and format conversions (e.g., JSON to CSV).
- Utilize Glue Crawlers to automate metadata cataloging and manage datasets within the AWS Glue Data Catalog.
- Perform data processing workflows moving data from raw to enriched states using AWS Glue, S3, RDS, DynamoDB, Athena, and other services as required.
- Collaborate with cross-functional teams to implement scalable data solutions and troubleshoot production issues.
AWS Services Experience
- AWS Glue (jobs, crawlers, Data Catalog)
- AWS Lambda
- AWS Step Functions
- Amazon S3
- Amazon EventBridge
- Amazon SNS & SQS
- Amazon DynamoDB
- Amazon Athena
- Amazon RDS (SQL Server)
- Amazon EC2, EKS, Network Load Balancer (NLB)
Required Skills & Qualifications
- Strong proficiency in Python and PySpark programming for data processing and ETL development.
- Solid understanding of core programming concepts, including error handling, retry logic, and asynchronous workflows.
- Hands-on experience developing serverless applications using AWS Lambda and related services.
- Experience building and orchestrating complex workflows using AWS Step Functions.
- Expertise in creating and managing AWS Glue ETL pipelines, including writing Glue scripts in PySpark and Python shell jobs.
- Familiarity with AWS data services like S3, DynamoDB, Athena, and RDS for storage and querying.
- Ability to design scalable, fault-tolerant, and maintainable cloud-native data solutions.
- Experience working in Agile environments and collaborating with technical and business stakeholders
Matrix Global Services is a leading multinational corporation that provides innovative and comprehensive solutions in technology, consulting, and outsourcing. With a history of over 30 years, Matrix has established itself as a trusted partner for businesses across various industries, consistently delivering exceptional results.
We're a network of firms in 10+ countries with over 13,000 people. At Matrix, we pride ourselves on our commitment to excellence and our ability to adapt to the ever-changing needs of our clients. Our team of highly skilled professionals is adept at understanding complex business challenges and tailoring solutions that drive sustainable growth and profitability.
Our wide range of services includes cutting-edge technology solution services, strategic consulting, digital transformation, cloud computing, cybersecurity, and managed services. Whether it's developing customized software applications, streamlining business processes, implementing robust IT infrastructure, or managing complex projects, our expertise and industry knowledge enable us to deliver value-added solutions that meet each client's unique requirements.
Come and join a winning team! You'll be challenged, have fun, and be a part of a highly respected organization! Matrix offers a competitive base salary and a full benefit package. Benefits include medical, dental, 401K, STD, HSA, PTO, and more.
EQUAL OPPORTUNITY EMPLOYER: Matrix is an Equal Opportunity Employer and Prohibits Discrimination and Harassment of Any Kind. Matrix is committed to the principle of equal employment opportunity for all employees and to providing employees with a work environment free of discrimination and harassment. All employment decisions at Matrix are based on business needs, job requirements, and individual qualifications, without regard to race, color, religion or belief, family or parental status, or any other status protected by the laws or regulations in our locations. Matrix will not tolerate discrimination or harassment based on any of these characteristics. Matrix encourages applicants of all ages.
Requirements
None
