Validate high-value emerging AI and automation technologies and de-risk their adoption across dLocal Own technology scouting, prototyping, and evaluation for dLocal Run instrumented spikes and benchmarks on LLMs, agentic systems, vector databases, orchestration frameworks, copilots, assistants, and other emerging technologies Compare vendor and open-source options across quality, cost, latency, security, and integration complexity Deliver decision memos with recommendations to adopt, watch, or avoid Design and maintain evaluation environments with datasets, prompts, scenarios, and telemetry Build automation and tooling to measure quality, robustness, latency, cost, and regressions Build focused prototypes to explore architecture, integration patterns, operational constraints, security boundaries, and failure modes Define readiness guidance, patterns, guardrails, limitations, operational considerations, and integration requirements Coordinate hand-offs to engineering teams responsible for productionization and support transitions as needed Track outcomes of Lab recommendations to improve evaluation methods Work with Security, Legal, Compliance, and AI teams on risk assessments and governance recommendations Maintain reusable checklists, decision templates, and standards Incorporate learnings from external copilots and the AWS AI suite into adoption guidelines Partner with AI and domain teams to ensure collaboration and clear boundaries Participate in hiring as a technical evaluator and culture champion Mentor engineers on evaluation methods, benchmarking, and experimental design Share knowledge through internal write-ups, tech talks, meetups, and conferences

Requirements





8+ years of software engineering experience, including significant experience operating at senior or Staff-level scope Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling Strong software engineering fundamentals and ability to rapidly build high-quality experimental systems Experience building agentic or multi-step AI systems involving tool use, orchestration, state, retrieval, or external integrations Strong knowledge of cloud infrastructure, preferably AWS, and ability to run experimental workloads securely and cost-consciously Experience with observability, telemetry, testing, and benchmarking of complex systems Ability to reason about system architecture, reliability, scalability, asynchronous workflows, and distributed components Track record of designing experiments or benchmarks that influenced meaningful technical decisions Experience constructing evaluation datasets, including task selection, labelling, and holdout discipline Working knowledge of LLM-as-judge methods, human evaluation, inter-annotator agreement, and their appropriate use Ability to reason about statistical significance on small samples Familiarity with regression tracking, telemetry,



and versioning Ability to turn ambiguous ideas into scoped evaluation plans with hypotheses and metrics Comfortable making trade-off calls across quality, latency, cost, and vendor lock-in Experience writing concise decision memos Ability to explain technical results to non-specialists Experience working with platform, product, and operations teams Ability to influence without authority and align teams around standards and guardrails Curious, experimentation-oriented mindset with disciplined measurement and risk awareness Comfortable in a small, high-leverage team without embedded PMs Builder attitude favoring reusable tools, templates, and playbooks

Core Competencies Demonstrates extensive experience in evaluating and adopting AI and automation technologies, with a strong focus on LLMs and cloud infrastructure, particularly AWS. Capable of designing experiments, building prototypes, and collaborating across teams to ensure effective integration and governance. Highest-signal resume keywords

LLM Evaluation Cloud Infrastructure (AWS) Experimental Design Benchmarking and Telemetry Decision Memo Writing

Hard Skills

Software Engineering AI Tooling System Architecture Observability Statistical Significance Reasoning Evaluation Dataset Construction Regression Tracking Integration Patterns Automation and Tooling Prototyping

Soft Skills

Influencing Without Authority Curiosity Collaboration Mentoring Communication

Industry Keywords

AI Governance Risk Assessment Compliance Experimental Workloads Vendor Evaluation

Tools & Technologies

LLMs Agentic Systems Vector Databases Orchestration Frameworks Telemetry Tools Decision Templates Reusable Checklists AWS AI Suite

#J-18808-Ljbffr

📌 Staff AI Engineer – AI Labs (Madrid)
🏢 Jobtailor
📍 Madrid