DataOps Engineer

Sheikh Shakhbout Medical City

DataOps Engineer

Job Profile Definition

Key Tasks

We are seeking a skilled DataOps Engineer to design, automate, and operate CI/CD processes, testing frameworks, and monitoring for data pipelines across our hybrid environment. This role applies DevOps principles to data engineering, ensuring datasets are delivered reliably, securely, and in compliance with governance policies. The DataOps Engineer works closely with Data Engineering, DevOps, and Platform teams to embed automation, observability, and resilience in pipelines that power AI, BI, and operational use cases.

• Build and maintain CI/CD pipelines for data workloads, integrating with orchestration frameworks.
• Implement automated tests, schema validation, and data contracts to enforce quality and trust.
• Apply metadata, lineage, and governance rules programmatically across data assets.
• Configure and maintain dashboards, alerts, and telemetry for pipeline health and cost monitoring.
• Support incident response and remediation for pipeline failures and SLA breaches.
• Collaborate with Data Engineers, BI, and AI teams to ensure data is production-ready and compliant.

Key Responsibilities:

Architectural Alignment

• Apply DataOps principles within the lakehouse architecture with on-prem S3 compatible storage (e.g. MinIO, VAST)
• Implement CI/CD workflows e.g. Azure DevOps, GitHub Actions, or equivalent.
• Automate deployments for data pipelines e.g. ADF, Databricks Workflows, or Airflow.
• Ensure deployment processes are reproducible, traceable, and auditable.

Governance and Metadata Integration

• Enforce schema evolution policies, contracts, and stewardship rules in pipelines.
• Register data assets in catalogs with complete metadata and maintain lineage records.
• Apply masking, tokenisation, and role-based access controls for sensitive data (e.g. PHI, PII, FHIR, MDR).
• Monitor and report on quality KPIs such as freshness, validity, and completeness.

Platform and Operational Alignment

• Configure monitoring, alerting, and dashboards for data pipelines, latency, and cost metrics.
• Support runbook execution and incident management when jobs fail or SLAs are at risk.
• Optimise compute and storage performance with partitioning, caching, and cost-efficient design.
• Integrate with DevOps and Platform Engineering for identity, secrets, and security guardrails.

Cross Functional Collaboration

• Work with Data Engineers to embed DataOps practices into pipeline development.
• Partner with BI teams to ensure gold datasets are certified and published to the semantic layer.
• Support AI engineers by enabling automated delivery of features, embeddings, and ML-ready datasets.Collaborate with Delivery Governance to ensure releases meet intake and readiness standards.

Critical Datasets and Domains

• Maintain operational reliability for tier-one domains e.g. Patient, Encounter, Provider, Orders and Results, Claims and Billing, Master Data (MDM), Genomics (VCF and omics), and telemetry (identity and access logs).

Academic/Skill Requirements

• Bachelor s degree in Computer Science, Software Engineering, or related field.
• Strong knowledge of CI/CD practices, automation, and DevOps applied to data systems.
• Proficiency in Data Platforms e.g. Databricks, Spark, Delta Lake, Azure Data Factory, and dbt.
• Familiarity with orchestration frameworks (e.g. ADF, Airflow, Databricks Workflows).
• Experience with GitOps, Infrastructure as Code (e.g. Terraform, Ansible), and monitoring tools.
• Understanding of data governance and privacy requirements (e.g. Purview, OvalEdge, PHI/PII).

Experience Requirements

• 5+ years of professional experience in engineering roles, with at least 2 years focused on DataOps, DevOps, or automation.
• Hands-on track record implementing CI/CD and observability for data pipelines in enterprise or hybrid environments.
• Experience working with governance frameworks, metadata, and lineage tools.
• Demonstrated ability to troubleshoot, optimise, and improve pipeline reliability.

Apply now.