Application Developer- Airflow / Astronomer Migration

Job Description Application Developer- Airflow / Astronomer Migration Job Location: Pune Location Flexibility: Primary Location Only Req Id: 10313 Posting Start Date: 7/20/26 Role- Application developer Job Description - Data Engineer Consultant - Airflow / Astronomer Migration Job Title: Data Engineer Consultant - Airflow / Astronomer Migration Experience: 3 to 5 years Location / Shift: Location: Pune, India / Remote / Hybrid as per project need Working Mode: Full-time assignment Client working hours may apply Role Summary: You will work as a Data Engineer Consultant . You will support the migration of orchestration workflows from Azure Data Factory (ADF) to Astronomer / Apache Airflow . You will design, build, test, and document data workflows. You will work closely with the Principal Data Engineer and the Data Engineering team. You will also suggest improvements for performance, cost, reliability, and maintainability. Primary Skills: Must Have Apache Airflow / Astronomer Airflow DAG design and development Python and SQL dbt Snowflake Azure Blob Storage Data pipeline orchestration Workflow testing and validation Secondary Skills: Good to Have Azure Data Factory (ADF) ADF to Airflow migration experience Data pipeline modernization experience Snowflake performance tuning dbt model and flow optimization Git / Azure DevOps CI/CD for data pipelines Monitoring and alerting for data workflows Key Responsibilities: 1) Requirement Understanding Understand existing ADF pipelines and orchestration flows. Analyze current workflow schedules, triggers, parameters, and dependencies. Understand ingestion steps, transformation logic, and downstream impact. Work with the Principal Data Engineer and team to clarify open points. Identify risks, blockers, assumptions, and dependencies early. 2) Solution Design Design the migration approach from ADF to Astronomer / Airflow . Convert existing orchestration logic into clean Airflow DAG design. Define DAG structure, task dependencies, retry logic, and schedule patterns. Design workflows that are easy to maintain and support. Suggest improvements instead of only doing one-to-one migration. 3) Development / Implementation Design and build DAGs in Apache Airflow / Astronomer . Develop and adapt data pipelines as per migration requirement. Modify existing ingestion steps in Astronomer where required. Modify existing dbt flows and transformations when needed. Use Python and SQL for workflow logic, validation, and automation. Follow coding standards and project guidelines. 4) Integration / Configuration Configure schedules and dependencies for Airflow DAGs. Integrate Airflow workflows with dbt , Snowflake , and Azure Blob Storage . Ensure dbt jobs are properly triggered and monitored through Airflow. Validate integration between ingestion, transformation, and consumption layers. Configure workflow parameters, environment settings, and required connections. 5) Testing & Validation Perform unit testing for DAGs and workflow components. Perform integration testing for end-to-end data workflows. Validate migrated workflows against existing ADF output wherever applicable. Check data accuracy, completeness, and processing status. Fix defects and retest before production readiness. Prepare test evidence and validation notes. 6) Performance Optimization Review existing workflows and identify improvement areas. Optimize Airflow DAG performance and execution time. Improve reliability using proper retries, failure handling, and dependency management. Suggest cost-efficient execution patterns. Improve maintainability by creating reusable workflow components. Proactively recommend better design wherever useful. 7) Security, Compliance & Governance Follow client security and data handling guidelines. Use secure connection and access patterns as per project standards. Avoid hardcoding secrets, passwords, or sensitive values. Ensure access and workflow configurations are controlled and traceable. Follow required governance process for data pipeline changes. 8) Deployment & Release Management Support deployment of Airflow / Astronomer workflows across environments. Prepare deployment steps and release notes. Support production readiness checks before go-live. Coordinate with the Data Engineering team during release activities. Validate workflows after deployment. Support rollback or quick fix approach if any deployment issue occurs. 9) Production Support & RCA Monitor workflow execution and identify failures. Debug failures using Airflow logs, task history, dbt logs, and Snowflake queries. Perform root cause analysis for recurring issues. Implement preventive fixes to improve workflow stability. Support production issues during migration and stabilization phase. 10) Documentation & Knowledge Transfer Prepare technical documentation for developed DAGs and workflows. Document schedule, dependency, parameter, and configuration details. Prepare development and testing documentation. Maintain handover notes and support instructions. Provide knowledge transfer to the Data Engineering team. Explain design decisions, known issues, and support steps clearly. 11) Agile Delivery & Collaboration Work closely with the Principal Data Engineer and Data Engineering team. Provide regular status updates on progress, risks, and blockers. Collaborate with team members in a structured and transparent way. Take ownership of assigned tasks and deliver on time. Work as a hands-on team reinforcement for the migration project. Tools / Technologies: Cloud / Platform: Azure Orchestration Tools: Apache Airflow, Astronomer, Azure Data Factory Data Transformation: dbt Storage: Azure Blob Storage Database / Warehouse: Snowflake Programming Languages: Python, SQL DevOps Tools: Git / Azure DevOps, if used in project Monitoring Tools: Airflow UI, Astronomer monitoring, dbt logs, Snowflake query history Documentation Tools: Confluence / SharePoint / Project documentation repository, as applicable Qualification: BE / BTech / MCA / MSc / BSc / BCA or equivalent practical experience Relevant Data Engineering, Cloud, Snowflake, dbt, or Airflow certification is good to have Soft Skills: Autonomy and ownership Curiosity to understand existing systems Proactive mindset Ability to suggest practical improvements Good communication Strong collaboration with team members Ability to integrate smoothly with existing team Good documentation discipline Preferred Candidate Profile: 3 to 5 years of Data Engineering experience. Strong hands-on experience in Airflow / Astronomer . Good working knowledge of dbt, Snowflake, Azure Blob Storage, Python, and SQL . Prior experience in orchestration migration or data pipeline modernization. Ability to understand ADF workflows and redesign them in Airflow. Good delivery ownership and clear communication. Able to work independently with minimum supervision. Relocation Supported: No Visa Sponsorship Approved: No