Position Summary
We are seeking a highly experienced, hands-on Data Engineering Lead to design, develop, and own end-to-end enterprise data solutions supporting business applications, analytics, reporting, and AI initiatives.
The ideal candidate brings strong technical expertise in modern data engineering, particularly Snowflake, dbt, Python, SQL, and orchestration frameworks, along with the ability to independently architect and deliver scalable production solutions.
This is a hands-on technical leadership role requiring active involvement in solution design, coding, troubleshooting, engineering standards, and delivery. The individual will work closely with business, analytics, AI, and technology teams to build reliable, reusable, and future-ready data products.
A technically strong, delivery-oriented engineering lead who can take a complex business problem, independently design the end-to-end data solution, develop critical components, guide implementation, and own the solution through production.
The candidate should be equally comfortable writing advanced SQL and Python, designing enterprise data models, troubleshooting production pipelines, collaborating with analytics teams, and enabling emerging AI use cases.
This is not a purely managerial or architecture-only role. Continued hands-on engineering involvement is essential.
In-Office Requirement: 4 days a week
Primary Responsibilities
•End-to-End Solution Ownership: Lead the architecture, design, development, deployment, and ongoing support of enterprise data engineering solutions, from source-system integration through data consumption.
•Data Ingestion & Integration: Design and implement scalable ingestion frameworks supporting databases, enterprise applications, APIs, SaaS platforms, files, and third-party data providers.
•Modern Data Pipelines: Build and optimize batch and incremental ETL/ELT pipelines, including change data capture (CDC), with the ability to evaluate and implement near-real-time and event-driven ingestion patterns as business needs evolve.
•Data Transformation & Modeling: Develop reusable dbt transformations, dimensional models, fact and dimension tables, bridge tables, and curated data marts supporting enterprise reporting and analytics.
•Snowflake Engineering: Design and optimize Snowflake data solutions, including complex SQL transformations, query tuning, workload performance, data sharing, and cost optimization.
•Workflow Orchestration: Develop, manage, and troubleshoot data workflows using managed Apache Airflow or equivalent orchestration frameworks, including dependencies, scheduling, retries, failure recovery, and operational monitoring.
•Data Quality & Reliability: Establish automated data validation, reconciliation, freshness monitoring, alerting, observability, lineage, and production support standards.
•Analytics Enablement: Partner with BI and analytics teams to deliver performant, governed datasets and reusable business metrics for dashboards, reporting, and self-service analytics.
•AI Enablement: Collaborate with AI engineering teams to build trusted, secure, AI-ready data products, including structured and unstructured data integration, reusable data-access interfaces, and data services for AI applications.
•Engineering Excellence: Implement Git-based development, CI/CD, automated testing, code reviews, documentation, and consistent engineering standards across data pipelines and models.
•Technical Leadership: Lead technical design discussions, mentor engineers, resolve complex engineering challenges, and take direct accountability for successful production delivery while remaining actively involved in coding and implementation.
Requirements
Education & Certificates
•Bachelor’s Degree, required
•Degree in Computer Science, Data Science, Finance, Business, or a related field, preferred
Professional Experience
•Minimum 8 years of professional experience in data engineering, data warehousing, or related software engineering roles, with significant hands-on technical leadership experience.
•Proven experience independently designing, building, deploying, and owning end-to-end data engineering solutions.
•Hands-on experience developing and maintaining ETL/ELT pipelines, including batch ingestion, incremental processing, source-to-target mapping, and data reconciliation.
•Experience with Apache Airflow, including managed Airflow environments, or equivalent production orchestration frameworks.
•Strong experience integrating multiple data sources through databases, REST APIs, enterprise applications, and file-based ingestion.
•Experience designing solutions for large datasets, complex transformations, schema changes, historical tracking, and efficient data processing.
•Demonstrated ability to lead engineers technically while remaining hands-on with development, code reviews, debugging, and problem resolution.
•Experience with Fivetran, SnapLogic, or similar enterprise data integration platforms.
•Experience designing near-real-time or streaming ingestion solutions, including CDC, event-driven architectures, Kafka, Snowpipe Streaming, or equivalent technologies.
•Experience with Power BI, including semantic models, DAX, Import versus DirectQuery, refresh strategies, and performance optimization.
•Experience using AI-assisted development tools, such as Cursor, GitHub Copilot, or equivalent tools, to improve engineering productivity.
•Experience with Snowflake Cortex, Snowpark, Dynamic Tables, Streams, Tasks, or Openflow.
•Experience with data observability, cataloging, lineage, and governance platforms such as Atlan, Monte Carlo, or equivalent solutions.
•Experience integrating Salesforce, CRM systems, financial applications, or other enterprise SaaS platforms.
•Financial services or investment management industry experience.
Competencies & Attributes
•Strong hands-on expertise in Snowflake, advanced SQL, Python, and dbt, including production development, performance optimization, and troubleshooting.
•Strong understanding of enterprise data architecture, dimensional modeling, data warehousing, data marts, and reusable data product design.
•Strong understanding of production-grade engineering practices, including Git, CI/CD, automated testing, data quality, monitoring, alerting, and incident resolution.
•Ability to translate business requirements into technical architecture, data models, implementation plans, and production deliverables.
•Strong communication, problem-solving, and cross-functional collaboration skills.
•Familiarity with AI/LLM applications, including RAG, vector search, Model Context Protocol (MCP), AI agents, and governed data access for AI solutions.
•Familiarity with unstructured data ingestion, document processing, and data preparation for enterprise AI applications.
•Experience with modern data architectures, including medallion architecture, data contracts, and reusable data products.
Benefits/Compensation
The compensation range for this role is specific to New York and takes into account a wide range of factors including but not limited to the skill sets required/preferred; prior experience and training; licenses and/or certifications.
The anticipated base salary range for this role is $180,000 to $200,000.
In addition to the base salary, the hired professional will enjoy a comprehensive benefits package spanning retirement benefits, health insurance, life insurance and disability, paid time off, paid holidays, family planning benefits and various wellness programs. Additionally, the hired professional may also be eligible to participate in an annual discretionary incentive program, the award of which will be dependent on various factors, including, without limitation, individual and organizational performance.
Due to the high volume of candidates, please be advised that only candidates selected to interview will be contacted by Carlyle.
Who We Are
When people, ideas, and capital come together, opportunity expands across private markets. At Carlyle, this belief has shaped how we invest for decades, fueling growth for companies and delivering performance for investors.
Visit our website to learn more about our firm and the ways our platform is shaping private markets.