This role is for one of Weekday’s clients
Salary range: Rs 4500000 - Rs 6500000 (ie INR 45 - 65 LPA)
Min Experience: 7+ years
Location: Bengaluru
JobType: full-time
We are looking for an experienced and technically proficient AI Data Engineer to design, develop, and manage scalable data infrastructure that powers advanced Artificial Intelligence (AI), Machine Learning (ML), and Generative AI applications. The ideal candidate will have strong expertise in data engineering, distributed computing, cloud platforms, and modern data architectures, along with an understanding of AI/ML workflows and data-intensive systems.
You will be responsible for building robust data pipelines, developing scalable data platforms, and ensuring the availability, quality, and reliability of structured and unstructured data for AI-driven applications. This role requires someone who can work across engineering, data science, and AI teams to translate complex business requirements into efficient data solutions.
Requirements
Key Responsibilities
1. Data Engineering & Pipeline Development
•Design, develop, and maintain scalable ETL/ELT pipelines for processing large volumes of structured and unstructured data.
•Build reliable batch and real-time data processing systems to support AI/ML applications.
•Develop data ingestion, transformation, integration, and orchestration frameworks.
•Optimize data workflows for performance, scalability, and cost efficiency.
2. AI & Machine Learning Data Infrastructure
•Build and maintain data infrastructure supporting machine learning, deep learning, and Generative AI applications.
•Develop data pipelines for model training, validation, testing, and deployment.
•Work with vector databases, embeddings, and retrieval-augmented generation (RAG) architectures.
•Prepare, clean, transform, and validate datasets for AI model development.
•Support feature engineering, feature stores, and model data management.
3. Data Architecture & Cloud Infrastructure
•Design scalable data architectures using modern data platforms and distributed computing frameworks.
•Work with cloud environments such as AWS, Microsoft Azure, or Google Cloud Platform.
•Implement data lakes, data warehouses, and lakehouse architectures.
•Leverage technologies such as Apache Spark, Kafka, Airflow, Databricks, Snowflake, and Delta Lake.
4. Data Quality, Governance & Security
•Establish data quality checks, monitoring, validation, and observability frameworks.
•Ensure data consistency, integrity, security, and compliance.
•Implement access controls, metadata management, and data governance practices.
•Identify and resolve data pipeline bottlenecks and infrastructure issues.
5. Collaboration & Technical Leadership
•Collaborate with AI engineers, ML engineers, data scientists, and product teams.
•Lead technical discussions, architecture decisions, and engineering best practices.
•Mentor junior engineers and drive continuous improvements in data engineering processes.
•Evaluate emerging AI and data technologies to improve platform capabilities.
Required Skills & Qualifications
•7–18 years of experience in Data Engineering, AI Data Engineering, or related engineering roles.
•Strong proficiency in Python, SQL, and data processing frameworks.
•Hands-on experience with Apache Spark, Kafka, Airflow, and distributed data systems.
•Experience building large-scale ETL/ELT pipelines and modern data architectures.
•Strong understanding of AI/ML data workflows and Generative AI infrastructure.
•Experience with cloud platforms (AWS, Azure, or GCP).
•Knowledge of vector databases, embeddings, RAG pipelines, and unstructured data processing.
•Experience with data warehouses, data lakes, and lakehouse platforms.
•Strong understanding of database technologies, data modelling, and optimization techniques.
•Familiarity with Docker, Kubernetes, CI/CD, and infrastructure automation.
Good-to-Have Skills
•Experience working with LLMs and AI-powered applications.
•Exposure to LangChain, LlamaIndex, and model-serving infrastructure.
•Knowledge of MLOps, feature stores, and model lifecycle management.
•Experience with Databricks, Snowflake, BigQuery, or Redshift.
•Familiarity with data observability and governance tools.