Senior AWS Data Engineer / Lead

Designation: Senior AWS Data Engineer / Lead
Location: Bangalore
Experience: 8 - 10 years
Opening: 08

Join us as a Senior AWS Data Engineer / Lead and spearhead our data initiatives and leverage your expertise in the AWS cloud to design, build, and maintain data-driven solutions.

Job Description:

  • Orchestration of Amazon MWAA: Expertise in managing Amazon Managed Workflows for Apache Airflow (MWAA) for data orchestration, including ingestion, transformation, and data integrity tasks.
  • Transition to ELT: Implementing ELT (Extract, Load, Transform) approach aligned with Gartner’s data transformation strategy, focusing on metadata-driven integration design and rapid onboarding of new systems.
  • DataOps Delivery: Adopting a DataOps approach with Informatica, pySpark, Python shell, and SQL programming skills to build data pipelines, along with CI/CD pipelines using GitHub Actions and Infrastructure as Code (IaC) with Terraform.
  • Data Lake House Architecture: Utilizing AWS services like S3 and Amazon Redshift to create a true Data Lake house architecture with native Role-Based Access Control (RBAC) and security controls, integrating Snowflake for enhanced RBAC and observability using AWS CloudWatch and Apache Airflow.
  • Event-Driven Architecture: Implementing event-based architecture with AWS EventBridge to monitor and audit AWS environments, enabling real-time response to operational changes and preventing infrastructure vulnerabilities.

Skills and Experience Required:

  • Bachelor’s or Master’s degree in Computer Science or related fields with 8-10 years of professional experience.
  • Proficiency in managing Amazon MWAA for Apache Airflow orchestration.
  • Experience transitioning from ETL to ELT approach with a focus on metadata-driven integration.
  • Strong programming skills in pySpark, Python shell, and SQL for building data pipelines.
  • Hands-on experience with Informatica, GitHub Actions for CI/CD pipelines, and Terraform for Infrastructure as Code.
  • Expertise in implementing Data Lake house architecture using AWS services like S3, Amazon Redshift, and Snowflake.
  • Familiarity with Role-Based Access Control (RBAC), observability using AWS CloudWatch and Apache Airflow, and event-based architecture with AWS EventBridge.