-
¿Tiene las habilidades necesarias para este puesto? Lea todos los detalles a continuación y presente su candidatura hoy mismo.

- Validate high-value emerging AI and automation technologies and de-risk their adoption across dLocal

- Own technology scouting, prototyping, and evaluation for dLocal

- Run instrumented spikes and benchmarks on LLMs, agentic systems, vector databases, orchestration frameworks, copilots, assistants, and other emerging technologies

- Compare vendor and open-source options across quality, cost, latency, security, and integration complexity

- Deliver decision memos with recommendations to adopt, watch, or avoid

- Design and maintain evaluation environments with datasets, prompts, scenarios, and telemetry

- Build automation and tooling to measure quality, robustness, latency, cost, and regressions

- Build focused prototypes to explore architecture, integration patterns, operational constraints, security boundaries, and failure modes

- Define readiness guidance, patterns, guardrails, limitations, operational considerations, and integration requirements

- Coordinate hand-offs to engineering teams responsible for productionization and support transitions as needed

- Track outcomes of Lab recommendations to improve evaluation methods

- Work with Security, Legal, Compliance, and AI teams on risk assessments and governance recommendations

- Maintain reusable checklists, decision templates, and standards

- Incorporate learnings from external copilots and the AWS AI suite into adoption guidelines

- Partner with AI and domain teams to ensure collaboration and clear boundaries

- Participate in hiring as a technical evaluator and culture champion

- Mentor engineers on evaluation methods, benchmarking, and experimental design

- Share knowledge through internal write-ups, tech talks, meetups,



and conferences

Requirements

-

- 8+ years of software engineering experience, including significant experience operating at senior or Staff-level scope

- Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling

- Strong software engineering fundamentals and ability to rapidly build high-quality experimental systems

- Experience building agentic or multi-step AI systems involving tool use, orchestration, state, retrieval, or external integrations

- Strong knowledge of cloud infrastructure, preferably AWS, and ability to run experimental workloads securely and cost-consciously

- Experience with observability, telemetry, testing, and benchmarking of complex systems

- Ability to reason about system architecture, reliability, scalability, asynchronous workflows, and distributed components

- Track record of designing experiments or benchmarks that influenced meaningful technical decisions

- Experience constructing evaluation datasets, including task selection, labelling, and holdout discipline

- Working knowledge of LLM-as-judge methods, human evaluation, inter-annotator agreement, and their appropriate use

- Ability to reason about statistical significance on small samples

- Familiarity with regression tracking, telemetry, and versioning

- Ability to turn ambiguous ideas into scoped evaluation plans with hypotheses and metrics

- Comfortable making trade-off calls across quality, latency, cost,



and vendor lock-in

- Experience writing concise decision memos

- Ability to explain technical results to non-specialists

- Experience working with platform, product, and operations teams

- Ability to influence without authority and align teams around standards and guardrails

- Curious, experimentation-oriented mindset with disciplined measurement and risk awareness

- Comfortable in a small, high-leverage team without embedded PMs

- Builder attitude favoring reusable tools, templates, and playbooks

Core Competencies

Demonstrates extensive experience in evaluating and adopting AI and automation technologies, with a strong focus on LLMs and cloud infrastructure, particularly xqbhyrx AWS. Capable of designing experiments, building prototypes, and collaborating across teams to ensure effective integration and governance.

Highest-signal resume keywords

-

- LLM Evaluation

- Cloud Infrastructure (AWS)

- Experimental Design

- Benchmarking and Telemetry

- Decision Memo Writing

Hard Skills

-

- Software Engineering

- AI Tooling

- System Architecture

- Observability

- Statistical Significance Reasoning

- Evaluation Dataset Construction

- Regression Tracking

- Integration Patterns

- Automation and Tooling

- Prototyping

Soft Skills

-

- Influencing Without Authority

- Curiosity

- Collaboration

- Mentoring

- Communication

Industry Keywords

-

- AI Governance

- Risk Assessment

- Compliance

- Experimental Workloads

- Vendor Evaluation

Tools & Technologies

-

- LLMs

- Agentic Systems

- Vector Databases

- Orchestration Frameworks

- Telemetry Tools

- Decision Templates

- Reusable Checklists

- AWS AI Suite

#J-18808-Ljbffr

📌 Staff AI Engineer – AI Labs (Madrid)
🏢 Jobtailor
📍 Madrid