lvl100Sign in

Weekday AI

Lead Data Engineer

  • Delhi, India
  • Full-time
  • Posted

Skills

  • AWS
  • Apache Spark
  • CI/CD
  • Communication
  • Compliance
  • Docker
  • Java
  • Kubernetes
  • Leadership
  • Microsoft Azure
  • Python
  • Root Cause Analysis
  • SQL
  • Snowflake

Job description

๐—ง๐—ต๐—ถ๐˜€ ๐—ฟ๐—ผ๐—น๐—ฒ ๐—ถ๐˜€ ๐—ณ๐—ผ๐—ฟ ๐—ผ๐—ป๐—ฒ ๐—ผ๐—ณ ๐˜๐—ต๐—ฒ ๐—ช๐—ฒ๐—ฒ๐—ธ๐—ฑ๐—ฎ๐˜†'๐˜€ ๐—ฐ๐—น๐—ถ๐—ฒ๐—ป๐˜๐˜€

๐—ฆ๐—ฎ๐—น๐—ฎ๐—ฟ๐˜† ๐—ฟ๐—ฎ๐—ป๐—ด๐—ฒ: ๐—ฅ๐˜€ ๐Ÿฐ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ - ๐—ฅ๐˜€ ๐Ÿฒ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ๐Ÿฌ (๐—ถ๐—ฒ ๐—œ๐—ก๐—ฅ ๐Ÿฐ๐Ÿฌ-๐Ÿฒ๐Ÿฌ ๐—Ÿ๐—ฃ๐—”)

Experience: 6+ yrs

Location: NCR, Delhi, India

As a Lead Data Engineer, you will spearhead the design, development, and optimization of large-scale data pipelines that support both batch and realโ€‘time data processing initiatives across the organization. You will collaborate closely with data scientists, product managers, and infrastructure teams to ensure data integrity, scalability, and high performance at every stage of the pipeline, while driving continuous improvement through automation, monitoring, and best practices. Your leadership will shape the data architecture, protect data quality, and enable timely, actionable insights that inform business decisions. In a fastโ€‘moving environment, you will balance technical excellence with strategic vision, fostering a culture of collaboration and innovation within the data engineering team.

Requirements

๐—ž๐—ฒ๐˜† ๐—ฅ๐—ฒ๐˜€๐—ฝ๐—ผ๐—ป๐˜€๐—ถ๐—ฏ๐—ถ๐—น๐—ถ๐˜๐—ถ๐—ฒ๐˜€

โ€ขDesign and implement robust, endโ€‘toโ€‘end batch data pipelines using tools such as Apache Spark, Hadoop, and Airflow.
โ€ขDevelop realโ€‘time streaming pipelines with technologies like Apache Kafka, Flink, or Spark Structured Streaming to deliver lowโ€‘latency analytics.
โ€ขOptimize pipeline performance through partitioning, indexing, and efficient data format selection (e.g., Parquet, ORC).
โ€ขEstablish data governance, quality checks, and automated monitoring to ensure data reliability and compliance.
โ€ขLead capacity planning and scaling strategies for data infrastructure, including cluster management and cost optimization.
โ€ขMentor and coach junior data engineers, fostering a collaborative and growthโ€‘oriented team culture.
โ€ขCollaborate with Data Scientists to translate analytical requirements into scalable pipeline solutions.
โ€ขDrive integration of data pipelines with downstream applications, dashboards, and reporting tools.
โ€ขStay current with emerging data engineering technologies and evaluate their applicability to business needs.
โ€ขParticipate in incident response and root cause analysis to quickly resolve data pipeline failures.

๐—ช๐—ต๐—ฎ๐˜ ๐— ๐—ฎ๐—ธ๐—ฒ๐˜€ ๐—ฌ๐—ผ๐˜‚ ๐—ฎ ๐—š๐—ฟ๐—ฒ๐—ฎ๐˜ ๐—™๐—ถ๐˜

โ€ข6+ years of experience in data engineering with a proven track record in both batch and realโ€‘time processing.
โ€ขDeep knowledge of distributed computing frameworks such as Hadoop, Spark, and Flink.
โ€ขProficiency in message queuing and stream processing systems like Kafka, Pulsar, or RabbitMQ.
โ€ขStrong scripting skills in Python, Scala, or Java, and experience with SQL on large datasets.
โ€ขHandsโ€‘on experience with cloud data services (AWS Glue, Redshift, BigQuery, Snowflake, or Azure Data Factory).
โ€ขFamiliarity with containerization (Docker, Kubernetes) and CI/CD pipelines for data workloads.
โ€ขDemonstrated ability to architect and scale data pipelines to handle petabyteโ€‘scale volumes.
โ€ขExcellent communication skills, capable of translating technical concepts for nonโ€‘technical stakeholders.
โ€ขA proactive problemโ€‘solver with a passion for data quality, governance, and automation.
โ€ขLeadership aptitude, with experience building, mentoring, and growing data engineering teams.

Browse all jobs by role, city or company