Lead design and implementation of scalable big data architecture and pipelines using Spark, Hadoop, Kafka, and related technologies. Mentor engineers, optimize distributed processing, enforce data governance and security, benchmark performance, and translate business needs into technical solutions.
The Senior Big Data engineering lead will play a pivotal role in designing, implementing, and optimizing large-scale data processing and analytics solutions. This role requires a visionary leader who can drive innovation, define architecture strategy, and ensure the scalability and efficiency of our big data infrastructure.
- Key Responsibilities
- Lead the design and development of a robust and scalable big data architecture handling exponential data growth while maintaining high availability and resilience.
- Design complex data transformation processes using Spark and other big data technologies using Scala, Pyspark
- Design and implement data pipelines that ensure data quality, integrity, and availability.
- Collaborate with cross-functional teams to understand business needs and translate them into technical requirements.
- Evaluate and select technologies that improve data efficiency, scalability, and performance.
- Oversee the deployment and management of big data tools and frameworks such as Hadoop, Spark, Kafka, and others.
- Provide technical guidance and mentorship to the development team and junior architects.
- Continuously assess and integrate emerging technologies and methodologies to enhance data processing capabilities.
- Optimize big data frameworks, such as Hadoop, Spark, for performance improvements and reduced processing time across distributed systems.
- Innovate in designing, developing, and refining data pipeline architectures to enhance data flow and ensure data processing capabilities.
- Implement data governance frameworks to ensure data accuracy, consistency, and privacy across the organization, leveraging metadata management and data lineage tracking.
- Conduct benchmarking and stress testing of big data solutions to validate performance standards and operational capacity.
- Ensure compliance with data security best practices and regulations.
Qualifications:
- Bachelor's or Master’s degree in Computer Science, Information Technology, or related field.
- Atleast 8-10 years overall software development experience on majorly working with handling application with large scale data volumes from ingestion, persistence and retrieval.
- Deep understanding of big data technologies, including Hadoop, Spark, Kafka, Flink, NoSQL databases, etc.
- Experience with Bigdata technologies Developer Hadoop , Apache Spark ,Python, PySpark
- Strong programming skills in languages such as Java, Scala, or Python.
- Excellent problem-solving skills with a knack for innovative solutions.
- Strong communication and leadership abilities.
- Proven ability to manage multiple projects simultaneously and deliver results.
Preferred Skills:
- Experience with data modeling and ETL/ELT processes.
- Experience in moving ETL frameworks from proprietary ETL technologies like Abinitio to Apache Spark
- Familiarity with machine learning and data analytics tools.
- Knowledge of core banking/financial services systems
------------------------------------------------------
Job Family Group: Technology------------------------------------------------------
Job Family:Applications Development------------------------------------------------------
Time Type:Full time------------------------------------------------------
Most Relevant Skills Please see the requirements listed above.------------------------------------------------------
Other Relevant Skills For complementary skills, please see above and/or contact the recruiter.------------------------------------------------------
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.
View Citi’s EEO Policy Statement and the Know Your Rights poster.
Citi Chennai, Tamil Nadu, IND Office
C P Ramaswamy Road, Chennai, Tamil Nadu, India, 600018
Similar Jobs
Digital Media • Information Technology • News + Entertainment
Design, develop, and execute ETL test strategies and automated validation for data ingestion, transformation, and loading. Validate data quality, perform reconciliation and lineage checks, develop SQL/Python validation scripts, monitor ETL workflows, collaborate with engineering and product teams, participate in code reviews and defect triage, and support data migrations and platform upgrades.
Top Skills:
AirflowAPIsAutomation Test FrameworkCi/CdData LakesData WarehousesDatabricksDbtETLJavaPythonSnowflakeSQL
Digital Media • Information Technology • News + Entertainment
Design, develop, test and integrate software and web applications. Produce technical documentation, train operations staff, mentor junior engineers, collaborate with QA, present architectural choices, and support software releases while applying project management and independent judgment.
Digital Media • Information Technology • News + Entertainment
Design, build, and optimize scalable data pipelines and platforms (Snowflake, Databricks, Spark). Ensure data quality, lineage, and observability. Mentor junior engineers, lead cross-team initiatives, implement CI/CD, and improve internal data engineering tools and architectures.
Top Skills:
AirflowSparkAWSAws S3AzureCi/CdDatabricksDbtGCPIcebergPythonSnowflakeSQL
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
