Our partner is looking for a Senior Data Engineer based in Spain. As a Senior Data Engineer, you will take ownership of the data platform, shaping its architecture and ensuring reliable performance from development through production. You will build scalable pipelines that transform high volumes of data from APIs, webhooks, files, and web scraping into trusted, product-ready information. You will establish robust data quality standards, resilient recovery mechanisms, and transparent data lineage to ensure accuracy and consistency at scale. Working with technologies such as Python, PostgreSQL, dbt, and Google Cloud Platform, you will integrate external data sources and AI-powered extraction capabilities. This fully remote opportunity offers broad technical ownership, the freedom to shape foundational data infrastructure, and the chance to make a direct impact in a collaborative, fast-moving engineering environment. Accountabilities
- Own Data Platform Architecture: Design, build, and operate a production-grade data platform from initial architecture through deployment, optimization, and ongoing maintenance.
- Develop concurrent ingestion systems for provider APIs, webhooks, files, and web scraping, incorporating rate limiting, error handling, and reliable processing at scale.
- Implement scheduled pipelines, historical backfills, and dependency management using orchestration tools such as Airflow, Dagster, or Prefect. Build event-driven processing with messaging and task queue services, including Pub/Sub or Cloud Tasks.
- Implement retries, dead-letter queues, controlled replay mechanisms,



and idempotent upserts to recover from failures without generating duplicate records or corrupting data.
- Establish Raw-to-Curated Data Architecture: Preserve original source payloads, including provider JSON, in Google Cloud Storage (GCS), then transform and standardize data using dbt. Maintain clearly defined raw, standardized, and curated data layers.
- Strengthen Data Quality and Governance: Define data contracts, automated validation rules, pipeline tests, retention policies, and monitoring systems to track data freshness, processing failures, and overall reliability.
- Optimize Database Performance: Design and maintain PostgreSQL schemas, queries, and incremental loading strategies to ensure efficient processing and dependable data delivery to downstream products.
- Integrate Enrichment and AI Extraction: Incorporate third-party enrichment providers and large language model (LLM)-based structured extraction from text into the data platform.
- Ensure Data Traceability: Maintain clear lineage between source data and curated records, enabling safe reprocessing, historical recovery, auditing, and consistent data management.
-

Requirements:
- End-to-End Data Engineering

Experience:
Proven experience building production data platforms from the ground up, including architecture, pipeline implementation, deployment, monitoring, and ongoing operations.
- Hands-on production experience with Airflow, Dagster, Prefect, or Temporal, including workflow dependencies, retries, scheduling, and failure recovery. Airflow experience is preferred.

📌 Senior Engineer, Data Engineering (España)
🏢 Jobgether
📍 España