Data Engineer – Databricks / Spark

Role Overview

We are looking for a Data Engineer with 6+ years of hands-on experience in data engineering, with strong expertise in Apache Spark and Databricks.

The idóneo candidate combines solid technical capabilities with a proactive and autonomous mindset. We are looking for someone who can take end-to-end ownership of data engineering projects, drive topics forward, identify issues and improvement opportunities, and collaborate effectively with both technical teams and business stakeholders.

This role requires someone comfortable working with limited supervision, making technical decisions within their area of responsibility, and being accountable for delivering reliable, scalable and maintainable data solutions.

Key Responsibilities

- Design, develop and maintain scalable data pipelines and ETL/ELT processes using Apache Spark and Databricks.
- Develop data transformations using PySpark and SQL, following engineering best practices.
- Build and maintain Delta Lake tables and data models across different layers of the data platform.
- Develop reliable ingestion and transformation pipelines for structured and semi-structured data.
- Implement appropriate approaches for incremental processing, CDC and data quality validation.
- Optimize Spark workloads in terms of performance, scalability and cost efficiency.
- Apply software engineering practices including Git, code reviews, testing, CI/CD and automated deployments.
- Monitor production pipelines, troubleshoot technical issues and perform root-cause analysis.
- Collaborate with Data Architects, Data Engineers,



Data Analysts and business stakeholders to translate requirements into technical solutions.
- Ensure solutions comply with established data architecture, security, governance and engineering standards.
- Produce appropriate technical documentation and contribute to reusable engineering patterns.

Ownership & Autonomy A key expectation for this position is the ability to take ownership of assigned projects and drive them from requirements through implementation and production. The successful candidate is expected to:

- Work autonomously, organizing and prioritizing their own work.
- Be proactive in identifying risks, dependencies, technical issues and improvement opportunities rather than waiting for instructions.
- Take responsibility for the quality, reliability and maintainability of the solutions delivered.
- Drive technical discussions and propose solutions when requirements are incomplete or challenges arise.
- Coordinate with other teams and stakeholders when dependencies need to be resolved.
- Communicate progress, risks and blockers clearly and escalate when appropriate.
- Follow topics through to completion and ensure that agreed outcomes are delivered.

Technical Skills

Required

- 3+ years of professional experience in Data Engineering.
- Strong hands-on experience with Apache Spark / PySpark.




- Practical experience developing data solutions in Databricks.
- Strong SQL skills and understanding of relational and analytical data modelling.
- Experience designing and developing ETL/ELT data pipelines.
- Experience working with Delta Lake or comparable modern lakehouse technologies.
- Understanding of distributed processing concepts and Spark performance optimization.
- Experience with Git-based development workflows.
- Experience with automated testing, deployment and CI/CD practices.
- Good understanding of data quality, observability and production support principles.

Nice to Have

- Experience with Azure Databricks and the broader Azure data ecosystem.
- Knowledge of Unity Catalog and data governance concepts.
- Experience with Databricks Workflows / Lakeflow Jobs.
- Experience with streaming or near-real-time processing.
- Knowledge of CDC and SCD patterns.
- Experience with infrastructure-as-code tools such as Terraform.
- Experience working with Azure DevOps or similar development platforms.

Soft Skills

- Strong sense of ownership and accountability.
- Proactive, autonomous and solution-oriented mindset.
- Ability to work effectively with limited supervision.
- Strong analytical and problem-solving skills.
- Ability to challenge existing approaches constructively and propose improvements.
- Good communication skills with both technical and non-technical stakeholders.
- Collaborative attitude and willingness to share knowledge within the engineering team.
- Ability to manage priorities and operate effectively in a multi-project environment.

📌 Data Engineer – Databricks / Spark (Madrid)
🏢 Response Informatics
📍 Madrid