Key Responsibilities

• Manage, monitor, and troubleshoot cloud-based data pipelines.
• Optimize data pipelines to enhance performance, cost-efficiency, and reliability.
• Automate repetitive tasks in data processing and management.
• Perform SLA-oriented monitoring of critical pipelines and ensure adherence.
• Analyze, troubleshoot, and resolve complex pipeline issues and conduct post-incident reviews.
• Maintain infrastructure reliability for data pipelines and MDM jobs.
• Implement and improve monitoring, alerting, and testing mechanisms for pipeline reliability.
• Develop and maintain technical documentation for data pipeline systems and processes.
• Collaborate with stakeholders, ensuring effective communication and reporting.
• Open to working in a 24x7 shift environment.

---

Required Skills & Experience

• 5+ years of industry experience in Data Engineering support and enhancement.
• Proficiency in cloud platforms: GCP, Azure, or AWS (experience with BigQuery, Cloud Storage, GKE, Glue, DMS, Athena, Lake Formation preferred).
• Strong understanding of data pipeline architectures and ETL processes.
• Excellent programming skills in Python (Pandas, PyArrow, Ibis) for data processing and automation.
• Proficiency in SQL for data analysis with relational databases.
• Hands-on experience with Apache PySpark and/or Apache Flink (or Kafka Streams, Apache Storm).
• Practical knowledge of Docker & Kubernetes (containerization and orchestration).
• Experience with Git, CI/CD pipelines, and automated testing.
• Strong troubleshooting and problem-solving skills.
• Proven ability to analyze, optimize, and maintain high-reliability pipelines.
• Strong documentation and communication skills.

---

Qualifications

• Bachelor’s degree in Computer Science or a related technical field (or equivalent practical experience).
• Cloud Professional Data Engineer certification is an advantage.
• Excellent verbal and written communication skills.

---

Nice to Have (Preferred Skills)

• Knowledge of FHIR R4 standards and healthcare data interoperability.
• Experience with data visualization tools (Google Looker, Tableau, Power BI).
• Hands-on with PyFlink (Python API for Apache Flink) and Flink SQL.
• Familiarity with Pixi environment management and modern Python dependency management.
• Experience with Apache Iceberg or other modern data lake storage formats.
• Understanding of real-time vs. batch processing trade-offs, particularly in healthcare data systems.