Capitole is growing fast, and we want you to be part of the journey We are looking for a Senior Data Platform Engineer with +6 years of experience to help drive the evolution of our Azure-based Data Platform and Lakehouse architecture. This role is focused on the platform itself. The professional will own and evolve the foundations that data product teams build on: Apache Airflow orchestration (migrating from ADF), Azure resource management (Key Vault, identity and access, networking, storage), Azure DevOps CI/CD and Infrastructure as Code, Databricks and Unity Catalog governance, observability and cost efficiency, while remaining hands‐on and close to the teams that use the platform.

Responsibilities:
Orchestration Platform (Apache Airflow & ADF)
* Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards.
* Work on the migration from Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework (DAG generation, reusable operators, dependency and scheduling patterns).
* Ensure Airflow is reliable and observable : SLAs, alerting, retries, backfills, health monitoring and clear failure ownership. Azure Resource Management
* Manage Azure resources with Infrastructure as Code (Terraform) : Databricks workspaces, Data Lake Storage, Data Factory, Key Vault, networking and supporting services.
* Own secrets and identity management : Key Vault, service principals and managed identities, secret scopes and rotation.
* Own access management : Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege, plus access requests and periodic reviews.
* Own Azure DevOps : repositories, CI/CD pipelines, branch policies and multi‐environment promotion, including deployment validation and Terraform state.




* Track cost and governance of Azure resources: tagging, budgets, right‐sizing and cost attribution (e.g. query tagging). Hands‐on Data Platform evolution
* Contribute to the implementation of the Data Catalog and Data Quality Framework and it's lineage across the platform.
* Drive observability : pipelines that are observable, debuggable and alert‐driven, with clear ownership of failures.
* Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads. Technical Leadership
* Participate in architecture reviews and technical decision‐making, stating trade offs (performance, cost, reliability, complexity) with data.
* Define and promote engineering standards : naming conventions, patterns and guidelines, through code reviews and design reviews.
* Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
* Build and maintain developer tooling and automation, including AI‐assisted engineering workflows (shared agent instructions, skills, automated PR review). Support & Operations (Run Team Collaboration)
* Provide second‐line support to the Run/Operations team for complex production topics.
* Troubleshoot and resolve platform incidents, failures and performance issues; lead root cause analysis and post‐mortems.
* Ensure proper handover,



documentation and operational readiness of platform changes.
* Continuously improve runbooks, monitoring and operational processes.

Requirements:
Technical Skills
* 6+ years of experience in data engineering, with significant time on platform or infrastructure topics
* Strong technical expertise with Python and SQL (advanced), including PySpark and Delta Lake semantics
* Deep hands‐on experience with Apache Airflow in production: DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads
* Hands‐on experience with Azure resource management : o Key Vault, managed identities and service principals, secret management o Access management : Azure RBAC, Entra ID, Unity Catalog permissions o Azure DevOps (repos, pipelines, service connections, environments) and CI/CD for data platforms o ADLS, networking and security basics, Azure Data Factory
* Infrastructure as Code (Terraform) and multi‐environment promotion
* Hands‐on experience with Databricks and Unity Catalog (compute, jobs, governance)
* Observability and monitoring, alerting and incident management
* Solid understanding of Data Lakehouse architecture, Medallion architecture and metadata‐driven architecture ; awareness of cost management and performance Soft Skills
* Strong problem‐solving and analytical thinking
* Ability to work in cross‐functional teams and to influence without authority
* Ownership mindset and attention to detail
* Good communication skills with both technical and business stakeholders: leads with the recommendation, quantifies, and is transparent about uncertainty What we're looking for in you:
* Led an Airflow implementation or migration (ideally from ADF), or run it at scale.
* Built metadata‐driven

📌 Hibrido BCN (Barcelona)
🏢 Capitole Consulting
📍 Barcelona