Data Engineer

Design, develop, and support batch and incremental data ingestion pipelines into the data lake.

Develop and optimize complex SQL queries, data transformations, and ETL workflows.

Build, maintain, and monitor Airflow DAGs for orchestration and workflow automation.

Analyze and troubleshoot data quality, data reconciliation, and pipeline execution issues.

Perform end-to-end data lineage analysis and trace data from source systems to downstream consumers.

Debug data discrepancies by investigating source systems, transformation logic, and target datasets.

Support data migration and modernization initiatives from legacy platforms to cloud-based data platforms.

Work closely with business stakeholders, architects, and upstream/downstream teams to understand data requirements.

Validate data completeness, accuracy, and consistency across multiple systems.

Participate in production support activities and resolve data-related incidents within SLA timelines.

Develop reusable data engineering components and recommend process improvements.

Required Skills

Strong Expertise

SQL / PL/SQL Complex joins, CTEs, analytical functions

Query optimization and performance tuning

Data modeling and database concepts

BigQuery Query development and optimization

Partitioning and clustering concepts

Data loading and transformation techniques

ETL/Data Engineering Data ingestion patterns (Batch and Incremental)

Data transformations and validations

Data quality checks and reconciliation

Good to Have

Apache Airflow Developing and troubleshooting DAGs

Scheduling, monitoring, and dependency management

Python Basic to intermediate scripting

Data processing and automation

Cloud Platforms (Preferably GCP) Cloud Storage

BigQuery

Composer/Airflow

Preferred Candidate Profile

Strong analytical and problem-solving skills.

Ability to independently investigate and resolve data issues.

Good communication and stakeholder management skills.

Experience in cloud data lake or warehouse environments.

Exposure to customer, transactional, or enterprise-scale datasets is preferred.

Screening Focus Areas

Advanced SQL/PLSQL

ETL & Data Pipeline Design

BigQuery Concepts

Data Debugging & Root Cause Analysis

Airflow DAG Development

Python Basics

Data Migration & Data Validation

Source-to-Target Traceability and Data Lineage

Must Have: SQL/PLSQL, BigQuery/Cloud Data Warehouse, ETL, Data Analysis & Debugging

Good to Have: Airflow DAG Development, Python, GCP Services

Kafka, Pub/Sub optional