Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.
This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.
You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.
Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.
Responsibilities
• Ingestion Architecture & GCP Foundation:
• Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
• Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
• Enterprise ETL, CDC & Incremental Processing:
• Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
• Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
• Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
• Multi-Engine Analytics & Query Optimization:
• Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
• Depl
📌 Senior Data Engineer (España)
🏢 N-iX
📍 España