As a Databricks Developer, you will design and build the enterprise data pipelines that power analytics, reporting and AI initiatives for a leading company in the energy sector. Join a fully remote data engineering team working hands-on with cutting-edge Lakehouse technology.
HIGH-IMPACT DATA PROJECTS
LATEST LAKEHOUSE TECH
FULLY REMOTE
LEARNING & GROWTH
*Don't tick every box? If you meet around 70% of the requirements above, we'd still encourage you to apply.*
About The Role
We are looking for a highly skilled Data Engineer with 5+ years of experience to design, build and optimize enterprise data pipelines on the Databricks Lakehouse platform for a leading energy sector company. In this role, you will be the hands-on technical driver responsible for transforming raw data into high-quality, actionable datasets. You will build and maintain a Medallion architecture, optimize Spark workloads, and ensure the data infrastructure seamlessly supports advanced analytics, BI dashboards and emerging Generative AI applications.
Key Responsibilities
Data Pipeline Engineering
- Design,
build and maintain scalable, robust ETL/ELT pipelines using Python, SQL and Apache Spark within the Databricks environment.
- Implement and manage a robust Medallion architecture (Bronze, Silver, Gold layers) to process and refine data from diverse sources.
- Develop and maintain the Gold semantic layer specifically optimized for high-performance consumption by BI tools (e.g., Power BI).
Platform Optimization & Architecture
- Optimize Databricks workloads, cluster configurations and Spark queries to ensure high performance and cost efficiency.
- Work extensively with open table formats, specifically Delta Lake and Apache Iceberg, to ensure ACID compliance, time travel and efficient data storage.
- Execute complex data migrations, including transitioning legacy workloads from traditional cloud data warehouses (e.g., AWS Redshift) into the Databricks Lakehouse.