Keep production Databricks pipelines running, including ingestion, transformation, and delivery to downstream consumers
Diagnose and resolve pipeline failures and data quality issues
Reverse-engineer and document existing transformation logic and business rules
Migrate legacy tables from Hive Metastore to Unity Catalog
Maintain Iceberg-enabled table sharing between Databricks and Snowflake
Build and test dbt models, including incremental materializations and data tests
Develop and maintain Airflow DAGs for orchestration
Validate migrated pipelines against Databricks outputs
Contribute to Snowflake modeling, performance, and cost decisions
Work directly with client stakeholders on technical topics alongside the team lead
Requirements
~3+ years operating production data pipelines
~ PySpark and SQL — able to read, debug, and modify existing pipelines
~ Delta Lake: MERGE/upsert patterns, table properties, OPTIMIZE, partitioning
~ Databricks Workflows, cluster configuration, job troubleshooting
~ Unity Catalog: catalogs, schemas, grants, lineage, and the metastore model




~ Snowflake warehouses, roles and grants, and general operating model
~ Snowflake query performance and awareness of compute cost behavior
~ dbt models, sources, tests, and incremental materializations
~ dbt project structure and deployment workflow
~ Airflow DAGs, operators, scheduling, and dependency management
~ Airflow retries, backfills, and idempotent task design
~ Strong SQL, including window functions, complex joins, and reading transformation logic
~ Python for scripting, automation, and API integration
~ Incremental loading patterns, idempotency, late-arriving data, and reprocessing
~ AWS S3 and IAM basics
~ Basic working knowledge of Redshift and its role in wider architecture
~ Fluent English
~ Self-directed and able to progress on an unfamiliar codebase without structured onboarding
~ Able to explain production incidents to non-technical stakeholders and provide realistic ETAs

Core Competenc

📌 Data/BI Engineer (Barcelona)
🏢 Jobtailor
📍 Barcelona