29 ago
|
Lawrence Harvey
|
Madrid
29 ago
Lawrence Harvey
Madrid
Software Engineer - Data | Data Mesh & Lakehouse About the Role We are looking for a Software Engineer - Data to join a central Data Delivery team building the foundation of a integral Data Mesh platform based on a Lakehouse architecture . You will work on the engineering layer responsible for ingesting, processing and provisioning source-aligned data products into central data catalogs, enabling teams across the organisation to consume trusted data for analytics, reporting, machine learning and other data-driven applications. A key part of the platform is enabling scalable data consumption through zero-copy data sharing , while maintaining strong governance, security and data quality standards.
What You'll Be Working On Design and build scalable batch and streaming data pipelines . Develop ingestion solutions for source-aligned data products.
Work with Apache Spark and Databricks within a modern Lakehouse environment. Implement data ingestion strategies including Full Loads, Delta Loads and Change Data Capture (CDC) .
Build and maintain streaming pipelines using technologies such as Kafka, Flink or Confluent . Manage datasets stored across Google Cloud Storage and Azure Blob Storage . Enable secure zero-copy data sharing through technologies such as Databricks Unity Catalog and Big Query.
Implement data governance and access-control models including RBAC and attribute-based access control . Design solutions capable of handling complex schema evolution , including backward and forward compatibility.
Tech Environment Data & Processing: Apache Spark, Databricks, Big Query Streaming & Ingestion: Kafka, Flink, Confluent, Airbyte Cloud & Storage: GCP, Google Cloud Storage, Azure, Azure Blob Storage Orchestration & Platform: Airflow, Kubernetes Governance: Databricks Unity Catalog, RBAC, ABAC/CBAC, Data Contracts CI/CD: Git Lab, Azure Dev Ops, JFrog Artifactory Quality & Security: Sonar Qube, Snyk What We're Looking For Strong professional experience in Data Engineering or
📌 Software engineer (Madrid)
🏢 Lawrence Harvey
📍 Madrid