Validate high-value emerging AI and automation technologies and de-risk their adoption across dLocal

Own technology scouting, prototyping, and evaluation for dLocal

Run instrumented spikes and benchmarks on LLMs, agentic systems, vector databases, orchestration frameworks, copilots, assistants, and other emerging technologies

Compare vendor and open-source options across quality, cost, latency, security, and integration complexity

Deliver decision memos with recommendations to adopt, watch, or avoid

Design and maintain evaluation environments with datasets, prompts, scenarios, and telemetry

Build automation and tooling to measure quality, robustness, latency, cost, and regressions

Build focused prototypes to explore architecture, integration patterns, operational constraints, security boundaries, and failure modes

Define readiness guidance, patterns, guardrails, limitations, operational considerations, and integration requirements

Coordinate hand-offs to engineering teams responsible for productionization and support transitions as needed

Track outcomes of Lab recommendations to improve evaluation methods

Work with Security, Legal, Compliance, and AI teams on risk assessments and governance recommendations

Maintain reusable checklists, decision templates, and standards

Incorporate learnings from external copilots and the AWS AI suite into adoption guidelines

Partner with AI and domain teams to ensure collaboration and clear boundaries

Participate in hiring as a technical evaluator and culture champion

Mentor engineers on evaluation methods, benchmarking, and experimental design

Share knowledge through internal write-ups, tech talks, meetups, and conferences

Requirements

8+ years of software engineering experience,



including significant experience operating at senior or Staff-level scope

Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling

Strong software engineering fundamentals and ability to rapidly build high-quality experimental systems

Experience building agentic or multi-step AI systems involving tool use, orchestration, state, retrieval, or external integrations

Strong knowledge of cloud infrastructure, preferably AWS, and ability to run experimental workloads securely and cost-consciously

Experience with observability, telemetry, testing, and benchmarking of complex systems

Ability to reason about system architecture, reliability, scalability, asynchronous workflows, and distributed components

Track record of designing experiments or benchmarks that influenced meaningful technical decisions

Experience constructing evaluation datasets, including task selection, labelling, and holdout discipline

Working knowledge of LLM-as-judge methods, human evaluation, inter-annotator agreement, and their appropriate use

Ability to reason about statistical significance on small samples

Familiarity with regression tracking, telemetry, and versioning

Ability to turn ambiguous ideas into scoped evaluation plans with hypotheses and metrics

Comfortable making trade-off calls across quality, latency,



cost, and vendor lock-in

Experience writing concise decision memos

Ability to explain technical results to non-specialists

Experience working with platform, product, and operations teams

Ability to influence without authority and align teams around standards and guardrails

Curious, experimentation-oriented mindset with disciplined measurement and risk awareness

Comfortable in a small, high-leverage team without embedded PMs

Builder attitude favoring reusable tools, templates, and playbooks

Core Competencies Demonstrates extensive experience in evaluating and adopting AI and automation technologies, with a strong focus on LLMs and cloud infrastructure, particularly AWS. Capable of designing experiments, building prototypes, and collaborating across teams to ensure effective integration and governance.

Highest-signal resume keywords

LLM Evaluation

Cloud Infrastructure (AWS)

Experimental Design

Benchmarking and Telemetry

Decision Memo Writing

Hard Skills

Software Engineering

AI Tooling

System Architecture

Observability

Statistical Significance Reasoning

Evaluation Dataset Construction

Regression Tracking

Integration Patterns

Automation and Tooling

Prototyping

Soft Skills

Influencing Without Authority

Curiosity

Collaboration

Mentoring

Communication

Industry Keywords

AI Governance

Risk Assessment

Compliance

Experimental Workloads

Vendor Evaluation

Tools & Technologies

LLMs

Agentic Systems

Vector Databases

Orchestration Frameworks

Telemetry Tools

Decision Templates

Reusable Checklists

AWS AI Suite

#J-18808-Ljbffr

📌 Staff AI Engineer – AI Labs (Madrid)
🏢 Jobtailor
📍 Madrid