As a Senior Data Engineer, you will architect security data ecosystems by designing data lakehouse architectures, implementing real-time streaming pipelines, and enabling AI/ML features. You will manage data ingestion patterns and ensure system integrity through automation and observability.
As a Senior Data Engineer, you will be the architect of our security data ecosystem. Your primary mission is to design and build high-performance data lake architectures and real-time streaming pipelines that serve as the foundation for COGNNA's Agentic AI initiatives. You will ensure that our AI models have access to fresh, high-quality security telemetry through sophisticated ingestion patterns.
Key Responsibilities1. Data Lake & Storage Architecture
- Architectural Design: Design and implement multi-tier Data Lakehouse architectures to support both structured security logs and unstructured AI training data.
- Storage Optimization: Define lifecycle management, partitioning, and clustering strategies to ensure high-performance querying while optimizing for cloud storage costs.
- Schema Evolution: Manage complex schema evolution for security telemetry, ensuring compatibility with downstream AI/ML feature engineering.
2. Real-Time & Streaming Processing
- Streaming Ingestion: Build and manage low-latency, high-throughput ingestion pipelines capable of processing millions of security events per second in real-time.
- Unified Processing: Design unified batch and stream processing architectures to ensure consistency across historical analysis and real-time threat detection.
- Event-Driven Workflows: Implement event-driven patterns to trigger AI agent reasoning based on incoming live data streams.
3. AI/ML Enablement & Feature Engineering
- Vector Data Foundations: Architect the data infrastructure required to support semantic search applications and variants of RAG architectures for our generative AI models.
- Feature Management: Design and maintain a centralized repository for ML features, ensuring consistent data is used for both model training and real-time inference.
- AI Pipeline Orchestration: Build automated workflows to handle data preparation, model evaluation, and deployment within our cloud AI ecosystem.
4. DataOps & Systems Design
- Infrastructure as Code: Utilize declarative tools (e.g., Terraform) to manage the entire lifecycle of our cloud data resources and AI endpoints.
- Quality & Observability: Implement automated data quality frameworks and real-time monitoring to detect "data drift" or pipeline failures before they impact AI model performance.
Requirements
- Experience & Education: 5+ years in Data Engineering or Backend Engineering, focused on large-scale distributed systems. B.S. or M.S. in Computer Science or a related technical field.
- Cloud Architecture: Deep architectural mastery of the Google Cloud Platform ecosystem, specifically regarding managed analytical warehouses, serverless compute, and identity/access management. Proven track record of deploying enterprise-scale Data Lakehouses from scratch.
- Real-Time Mastery: Expertise in building production-grade distributed messaging and stream processing engines (e.g., managed Apache Beam/Flink environments) capable of handling high-velocity telemetry.
- AI Enablement: Strong understanding of how data architecture impacts AI performance. Experience building embedding pipelines, feature stores, and automated workflows for model training and evaluation.
- Software Fundamentals: Expert-level Python and advanced SQL. Proficiency in high-performance languages like Go or Scala is highly desirable.
- Operational Excellence: Advanced knowledge of CI/CD, containerization on Kubernetes, and managing cloud infrastructure through code to ensure reproducible environments.
- Experience with dbt for modern analytics engineering.
- Understanding of cybersecurity data standards (OCSF/ECS).
- Previous experience in an AI-first startup or a high-growth security tech company.
Benefits
💰 Competitive Package – Salary + equity options + performance incentives
🧘 Flexible & Remote – Work from anywhere with an outcomes-first culture
🤝 Team of Experts – Work with designers, engineers, and security pros solving real-world problems
🚀 Growth-Focused – Your ideas ship, your voice counts, your growth matters
🌍 Global Impact – Build products that protect critical systems and data
Top Skills
Apache Beam
Apache Flink
Dbt
Go
Google Cloud Platform
Kubernetes
Python
Scala
SQL
Terraform
Similar Jobs
Information Technology
As a Data Quality Analyst, you will analyze data sources, maintain datasets for AI training, and prepare quality reports. You will collaborate with engineers on dataset creation and maintain evaluation datasets with domain experts.
Top Skills:
AWSPandasPolarsPythonSQL
AdTech • Digital Media
As a Junior Project Manager, you'll manage small to mid-level projects, coordinating timelines, resources, and documentation with a focus on communication and budget management.
Top Skills:
Google SuiteNotionProductiveSlack
Sales • Consulting
The role involves creating compelling ad strategies and creatives for paid campaigns on Meta, TikTok, and Google Search aimed at clients in UAE and Saudi Arabia, with a focus on performance metrics and client collaboration.
Top Skills:
Google SearchMetaTiktok
What you need to know about the Chennai Tech Scene
To locals, it's no secret that South India is leading the charge in big data infrastructure. While the environmental impact of data centers has long been a concern, emerging hubs like Chennai are favored by companies seeking ready access to renewable energy resources, which provide more sustainable and cost-effective solutions. As a result, Chennai, along with neighboring Bengaluru and Hyderabad, is poised for significant growth, with a projected 65 percent increase in data center capacity over the next decade.
