Data Engineer – Python, PySpark, SQL & ETL
- Bengaluru - Bellandur (GTP)
- Full-time
- Posted
Skills
Job description
Job Summary
Synechron is seeking a Data Engineer with 5 years of experience in building data marts, ETL pipelines, and large-scale data-processing solutions.
The role will focus on designing, developing, optimizing, and supporting scalable data pipelines using Python, PySpark, Spark, SQL, and data-warehousing technologies. The position will cover the end-to-end software development lifecycle, including development, testing, UAT support, production deployment, issue resolution, and post-production support.
The role will contribute to business objectives by delivering reliable and efficient data solutions, improving data quality and accessibility, supporting informed decision-making, and enabling consistent data operations across enterprise environments.
Software Requirements
Required Software Skills
•Python: Strong hands-on experience developing data-processing applications, ETL workflows, automation scripts, and production services.
•PySpark and Spark: Strong experience building distributed data-processing and ETL solutions.
•Pandas: Experience with data preparation, analysis, transformation, validation, and profiling.
•SQL: Strong experience with data extraction, transformation, validation, analysis, and query optimization.
•Oracle SQL: Strong knowledge of Oracle SQL, complex queries, joins, aggregations, subqueries, stored procedures, and performance optimization.
•ETL development: Experience designing, developing, testing, deploying, and supporting ETL and data-pipeline solutions.
•Data warehousing: Strong understanding of data-warehouse concepts, dimensional modelling, data marts, transformations, fact tables, dimensions, and data quality.
•Hadoop, Hive, and MapReduce: Experience working with distributed data-processing and big-data technologies.
•SQL and NoSQL databases: Experience working with structured, semi-structured, and unstructured data.
•Jupyter Notebook: Experience using notebooks for data analysis, development, validation, troubleshooting, and technical exploration.
•CI/CD pipelines: Familiarity with automated build, testing, deployment, and release processes.
•Testing and data validation: Experience validating data transformations, pipeline results, data quality, and application behavior.
•Software engineering practices: Understanding of coding standards, version control, documentation, code reviews, testing, and maintainable development practices.
Preferred Software Skills
•Experience with banking or financial-services data and reporting requirements.
•Exposure to cloud-based data platforms and data-processing services.
•Familiarity with workflow scheduling, orchestration, monitoring, and alerting tools.
•Experience with automated data-quality checks, reconciliation, lineage, and metadata management.
•Exposure to infrastructure automation, containerization, and modern DevOps practices.
•Familiarity with tools supporting data pipeline observability and performance analysis.
Version and Experience Guidance
•Specific software versions are not mandated in the requirements.
•Candidates should have experience with current, production-supported versions of Python, PySpark, Spark, Pandas, Oracle SQL, Hadoop, Hive, MapReduce, and related tools.
•Candidates should be able to assess compatibility, manage dependencies, support upgrades, and follow version-control practices.
•Experience should include production data engineering, pipeline support, and end-to-end SDLC participation rather than only academic or prototype development.
Overall Responsibilities
•Design, develop, maintain, and optimize scalable ETL and data-pipeline solutions using Python and PySpark.
•Build and enhance data marts and data-transformation processes.
•Develop distributed data-processing solutions using Spark, Hadoop, Hive, and MapReduce.
•Work with structured, semi-structured, and unstructured data from multiple sources.
•Develop and optimize Oracle SQL queries for data extraction, transformation, analysis, and validation.
•Analyze and debug data issues using Oracle SQL, PySpark, Pandas, and structured data-analysis techniques.
•Perform end-to-end SDLC activities, including development, testing, UAT support, bug fixing, production deployment, and post-production support.
•Collaborate with cross-functional teams to define data requirements, validate solutions, resolve defects, and improve data quality.
•Implement data validation, reconciliation, completeness, accuracy, and consistency checks.
•Support CI/CD implementation for data pipelines, applications, and deployment workflows.
•Monitor production environments and troubleshoot pipeline failures, data-quality issues, performance problems, and processing delays.
•Create and maintain technical documentation, data mappings, transformation logic, support procedures, and operational runbooks.
•Apply software engineering best practices, coding standards, testing practices, and maintainable design principles.
•Contribute to performance improvements through query optimization, efficient transformations, appropriate partitioning, resource management, and pipeline tuning.
•Identify opportunities to improve automation, reliability, data quality, maintainability, and sustainable use of processing resources.
•Contribute to measurable outcomes, including reliable data delivery, reduced processing time, improved data accuracy, fewer production incidents, and successful release execution.
Technical Skills (By Category)
Programming Languages
Essential
•Strong Python development experience.
•Strong PySpark and Spark programming experience.
•Experience using Pandas for data analysis, transformation, profiling, and validation.
•Ability to write modular, maintainable, testable, and production-ready code.
•Ability to develop reusable data-processing components, ETL logic, validation utilities, and automation scripts.
Preferred
•Experience with additional programming or scripting languages used in data engineering and distributed processing.
•Familiarity with asynchronous or event-driven data-processing patterns.
•Experience developing reusable libraries and shared data-engineering components.
Databases/Data Management
Essential
•Strong Oracle SQL experience.
•Experience with SQL and NoSQL databases.
•Understanding of data-warehouse concepts and data-mart design.
•Experience working with structured, semi-structured, and unstructured data.
•Understanding of data modelling, dimensional modelling, schemas, fact tables, dimensions, keys, joins, aggregations, and transformations.
•Experience with data quality, validation, reconciliation, completeness, accuracy, and consistency checks.
•Ability to optimize queries and investigate data-related application or pipeline issues.
•Understanding of data lineage, source-to-target mapping, metadata, data retention, and controlled data access.
Preferred
•Experience with banking or financial-services data structures and reporting requirements.
•Familiarity with data lakes, analytical data stores, and distributed data architectures.
•Experience with partitioning, indexing, compression, schema evolution, and high-volume data processing.
•Exposure to automated data-quality frameworks, data reconciliation, and metadata management.
Cloud Technologies
Essential
•Understanding of scalable data-processing and data-pipeline deployment environments.
•Ability to support development, testing, UAT, staging, and production data environments.
•Understanding of resource management, performance, availability, monitoring, and operational support for data platforms.
Preferred
•Experience with cloud-based data platforms and distributed processing services.
•Familiarity with cloud storage, managed data-processing services, data warehouses, and orchestration tools.
•Exposure to cloud-native CI/CD, infrastructure automation, and containerized data workloads.
•Understanding of processing optimization, resource utilization, and sustainable data-engineering practices.
Frameworks and Libraries
Essential
•Python.
•PySpark and Spark.
•Pandas.
•Oracle SQL.
•Hadoop.
•Hive.
•MapReduce.
•SQL and NoSQL database technologies.
•Jupyter Notebook.
•ETL and data-pipeline development frameworks.
•CI/CD tools and automated deployment workflows.
•Data validation, testing, monitoring, and troubleshooting utilities.
Preferred
•Workflow orchestration and scheduling frameworks.
•Data-quality, reconciliation, lineage, and metadata-management tools.
•Pipeline observability, logging, metrics, and alerting frameworks.
•Libraries supporting performance profiling, distributed processing, and automated testing.
•Tools supporting cloud-based data integration and analytics.
Development Tools and Methodologies
Essential
•Experience across the data-engineering lifecycle, including requirements analysis, design, development, testing, UAT support, deployment, monitoring, troubleshooting, and production support.
•Experience with ETL development and data-mart implementation.
•Familiarity with version control, code reviews, coding standards, technical documentation, and collaborative development practices.
•Experience implementing data validation, unit testing, integration testing, regression testing, and pipeline testing.
•Experience supporting CI/CD implementation for data applications and deployment workflows.
•Ability to create data mappings, transformation specifications, test plans, support procedures, and operational runbooks.
•Familiarity with Agile or iterative delivery practices.
•Experience performing root-cause analysis and resolving production data-pipeline issues.
Preferred
•Experience with automated data-pipeline deployment and release management.
•Familiarity with infrastructure as code, containers, and DevOps practices.
•Experience with pipeline monitoring, operational dashboards, alerting, and service-level reporting.
•Exposure to automated rollback, recovery, retry, and failure-handling patterns.
•Experience supporting continuous improvement and process standardization.
Security Protocols
Essential
•Understanding of secure data access, authentication, authorization, and least-privilege principles.
•Ability to protect sensitive business, customer, financial, and operational data.
•Understanding of secure database connections, credential management, data masking, encryption, and controlled data handling.
•Ability to follow security, privacy, compliance, change-management, and access-control requirements.
•Awareness of risks related to unauthorized access, data leakage, insecure scripts, exposed credentials, and uncontrolled data movement.
•Ability to maintain auditability and traceability across data transformations, deployments, and production changes.
Preferred
•Experience supporting data-security reviews, audit activities, access reviews, and compliance controls.
•Familiarity with secrets management, encryption in transit and at rest, and security monitoring.
•Experience integrating security and quality checks into CI/CD pipelines.
•Knowledge of security requirements for banking and financial-services data.
Experience Requirements
•Overall experience: Minimum 5 years of professional experience in data engineering, ETL development, data warehousing, big-data engineering, or a related technical role.
•Data-engineering experience: Strong hands-on experience building data marts, ETL pipelines, data transformations, and large-scale data-processing solutions.
•Technical experience: Strong experience with Python, PySpark, Spark, Pandas, SQL, Oracle SQL, Hadoop, Hive, MapReduce, and SQL and NoSQL databases.
•Data-warehouse experience: Experience with data-warehouse concepts, data modelling, data marts, source-to-target mapping, data transformations, and data-quality controls.
•SDLC experience: Experience across development, UAT support, bug