29 ago
|
Lawrence Harvey
|
Cataluña
29 ago
Lawrence Harvey
Cataluña
Software Engineer - Data | Data Mesh & Lakehouse About the Role
We are looking for a Software Engineer - Data to join a central Data Delivery team building the foundation of a integral Data Mesh platform based on a Lakehouse architecture.
You will work on the engineering layer responsible for ingesting, processing and provisioning source-aligned data products into central data catalogs, enabling teams across the organisation to consume trusted data for analytics, reporting, machine learning and other data-driven applications.
A key part of the platform is enabling scalable data consumption through zero-copy data sharing, while maintaining strong governance, security and data quality standards.
What You'll Be Working On
Design and build scalable batch and streaming data pipelines.
Develop ingestion solutions for source-aligned data products.
Work with Apache Spark and Databricks within a modern Lakehouse environment.
Implement data ingestion strategies including Full Loads, Delta Loads and Change Data Capture (CDC).
Build and maintain streaming pipelines using technologies such as Kafka,
Flink or Confluent.
Manage datasets stored across Google Cloud Storage and Azure Blob Storage.
Enable secure zero-copy data sharing through technologies such as Databricks Unity Catalog and BigQuery.
Implement data governance and access-control models including RBAC and attribute-based access control.
Design solutions capable of handling complex schema evolution, including backward and forward compatibility.
Tech Environment
Data & Processing: Apache Spark, Databricks, BigQuery
Streaming & Ingestion: Kafka, Flink, Confluent, Airbyte
Cloud & Storage: GCP, Google Cloud Storage, Azure, Azure Blob Storage
Orchestration & Platform: Airflow, Kubernetes
Governance: Databricks Unity Catalog, RBAC, ABAC/CBAC, Data Contracts
CI/CD: GitLab, Azure DevOps, JFrog Artifactory
Quality & Security: SonarQube, Snyk
What We're Looking For
Strong professional experience in Data Engineering or Software Engi
📌 Software Engineer (Cataluña)
🏢 Lawrence Harvey
📍 Cataluña