Keep production Databricks pipelines running, including ingestion, transformation, and delivery to downstream consumers

Diagnose and resolve pipeline failures and data quality issues

Reverse-engineer and document existing transformation logic and business rules

Migrate legacy tables from Hive Metastore to Unity Catalog

Maintain Iceberg-enabled table sharing between Databricks and Snowflake

Build and test dbt models, including incremental materializations and data tests

Develop and maintain Airflow DAGs for orchestration

Validate migrated pipelines against Databricks outputs

Contribute to Snowflake modeling, performance, and cost decisions

Work directly with client stakeholders on technical topics alongside the team lead

Requirements

3+ years operating production data pipelines

PySpark and SQL — able to read, debug, and modify existing pipelines

Delta Lake: MERGE/upsert patterns, table properties, OPTIMIZE, partitioning

Databricks Workflows, cluster configuration, job troubleshooting

Unity Catalog: catalogs, schemas, grants, lineage, and the metastore model

Snowflake warehouses, roles and grants, and general operating model

Snowflake query performance and awareness of compute cost behavior

dbt models, sources, tests, and incremental materializations

dbt project structure and deployment workflow

Airflow DAGs, operators, scheduling, and dependency management

Airflow retries, backfills, and idempotent task design

Strong SQL, including window functions,



complex joins, and reading transformation logic

Python for scripting, automation, and API integration

Incremental loading patterns, idempotency, late-arriving data, and reprocessing

AWS S3 and IAM basics

Basic working knowledge of Redshift and its role in wider architecture

Fluent English

Self-directed and able to progress on an unfamiliar codebase without structured onboarding

Able to explain production incidents to non-technical stakeholders and provide realistic ETAs

Core Competencies Demonstrates expertise in managing production data pipelines using Databricks, including ingestion, transformation, and delivery processes. Proficient in SQL, PySpark, and dbt for building and testing data models, with a strong understanding of Snowflake and Airflow for orchestration and performance optimization.

Highest-signal resume keywords

Databricks Pipeline Management

SQL Proficiency

PySpark Development

Airflow DAG Development

Dbt Model Building

Hard Skills

SQL

PySpark

Dbt

Airflow

Delta Lake

Unity Catalog

Snowflake

AWS S3

Incremental Loading Patterns

Data Quality Diagnosis

Soft Skills

Self-Directed

Effective Communication

Stakeholder Engagement

Industry Keywords

Data Pipeline

Data Transformation

Data Quality

Data Modeling

Orchestration

Tools & Technologies

Databricks

Snowflake

Airflow

Hive Metastore

Redshift

#J-18808-Ljbffr

📌 Mid Data Engineer (Madrid)
🏢 Jobtailor
📍 Madrid