NVIDIA is looking for a Day 2 Site Reliability Engineer to join the EMEA NVIDIA Cloud Partner (NCP) Day 1 / Day 2 Operations team. EMEA NCP fleets are scaling towards a million deployed GPUs, and this role exists to keep those fleets running at maximum goodput once they are in production — owning fault detection and remediation, preventive maintenance, and proactive firmware and field-notice rollout across partner estates.

We are looking for a hands-on reliability engineer who thinks in SLOs, error budgets and blast radius rather than tickets. You will measure and drive MTBI and job goodput on multi-thousand-GPU clusters, automate the detection-to-remediation loop, and work directly with NCP operations teams, NVEX and TAMs to review incidents and escalations and turn each one into a permanent fix.

NCP estates are open-source-first, home-grown and heterogeneous — upstream Kubernetes, KubeVirt, Slurm, Prometheus/Grafana, Cumulus/SONiC and a long tail of ISV and in-house tooling — so this role is deliberately tool-agnostic: you will meet each partner on the stack they actually run rather than on a single proprietary product, and you will keep GPU capacity generating revenue instead of burning cost in reactive support.

What You’ll Be Doing:

- Own Day 2 production stability and reliability at fleet scale across EMEA NCP partners: define, measure and improve MTBI and job goodput on multi-thousand-GPU clusters, and hold the error budget for the platforms you support.

- Build and operate the fault detection to remediation loop—GPU, node, fabric and storage fault signatures, automated triage, node drain, cordon, reset and return-to-service workflows—so failures are caught and cleared before they cost training jobs.

- Run preventive maintenance and proactive firmware and field-notice rollout campaigns across partner fleets (e.g. CPLD and switch firmware levels), including risk assessment, staged rollout, maintenance-window planning and verification at scale.

- Own observability and reliability tooling for GPU fleets: Prometheus/Grafana/Loki, DCGM and XID diagnostics, node-level health agents such as GPUd and NVSentinel, and fleet-wide reliability intelligence—extending or integrating them into the partner’s own home-grown stack.

- Lead incident response and post-incident review in collaboration with NVEX, TAM and partner operations teams: drive root-cause analysis on escalations, track corrective actions to closure, and convert every recurring incident into automation, a health check or a runbook.

- Own Kubernetes node lifecycle and reliability engineering on heterogeneous open stacks—upstream Kubernetes, KubeVirt, Slurm and GPU-aware schedulers—covering node health policy, upgrade and reboot orchestration, and safe workload draining on GPU-accelerated nodes.

What We Need to See:

- BS/MS/PhD in Computer Science, Electrical/Computer Engineering, Physics, Mathematics, or related fields, or equivalent experience.

- 8+ years in SRE, production engineering or large-scale infrastructure operations roles.

- Reliability Engineering Practice: Demonstrated ownership of production reliability at scale—SLO and error-budget definition, on-call and incident command, blameless post-incident review, and measurable improvement of availability or mean time between interruptions.

- GPU & HPC Fleet Operations: Hands-on experience operating HPC/AI clusters and NVIDIA GPU-accelerated infrastructure—driver and CUDA toolkit lifecycle, firmware management, workload profiling, and troubleshooting across CPUs, GPUs and high-speed interconnects.

- Kubernetes at Scale: Extensive expertise in running Kubernetes in GPU-accelerated and HPC environments—node lifecycle, upgrades and draining, scheduler behaviour, batch schedulers such as Slurm, and mixed bare-metal/virtualised (e.g. KubeVirt) multi-tenant estates.

- Observability & Automation: Proficiency in Python and Bash, Infrastructure-as-Code and configuration management (e.g. Ansible, Terraform), GitOps-based fleet lifecycle and upgrade management, and observability stacks (Prometheus, Grafana, Loki) for monitoring, alerting and building fault-tolerant systems.

- Linux, Storage & Networking Fundamentals: Deep knowledge of Linux (RedHat, Ubuntu), OS-level tuning and security, data centre networking fundamentals, and parallel/enterprise storage (Lustre, GPFS, ZFS, XFS) as they affect cluster stability.

Ways to Stand Out from the Crowd:

- Experience defining and running proactive firmware, microcode or field-notice rollout campaigns across a large production fleet.

- Familiarity with GPU health and fleet telemetry tooling—DCGM and XID diagnostics, GPUd, NVSentinel, node-level health agents and fleet-w

📌 Senior Cloud Infrastructure and Reliability Solutions Architect (España)
🏢 NVIDIA
📍 España