- Validate high-value emerging AI and automation technologies and de-risk their adoption across dLocal- Own technology scouting, prototyping, and evaluation for dLocal- Run instrumented spikes and benchmarks on LLMs, agentic systems, vector databases, orchestration frameworks, copilots, assistants, and other emerging technologies- Compare vendor and open-source options across quality, cost, latency, security, and integration complexity- Deliver decision memos with recommendations to adopt, watch, or avoid- Design and maintain evaluation environments with datasets, prompts, scenarios, and telemetry- Build automation and tooling to measure quality, robustness, latency, cost, and regressions- Build focused prototypes to explore architecture, integration patterns, operational constraints, security boundaries, and failure modes- Define readiness guidance, patterns, guardrails, limitations, operational considerations, and integration requirements- Coordinate hand-offs to engineering teams responsible for productionization and support transitions as needed- Track outcomes of Lab recommendations to improve evaluation methods- Work with Security, Legal, Compliance, and AI teams on risk assessments and governance recommendations- Maintain reusable checklists, decision templates, and standards- Incorporate learnings from external copilots and the AWS AI suite into adoption guidelines- Partner with AI and domain teams to ensure collaboration and clear boundaries- Participate in hiring as a technical evaluator and culture champion- Mentor engineers on evaluation methods, benchmarking, and experimental design- Share knowledge through internal write-ups, tech talks, meetups,



and conferencesRequirements- 8+ years of software engineering experience, including significant experience operating at senior or Staff-level scope- Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling- Strong software engineering fundamentals and ability to rapidly build high-quality experimental systems- Experience building agentic or multi-step AI systems involving tool use, orchestration, state, retrieval, or external integrations- Strong knowledge of cloud infrastructure, preferably AWS, and ability to run experimental workloads securely and cost-consciously- Experience with observability, telemetry, testing, and benchmarking of complex systems- Ability to reason about system architecture, reliability, scalability, asynchronous workflows, and distributed components- Track record of designing experiments or benchmarks that influenced meaningful technical decisions- Experience constructing evaluation datasets, including task selection, labelling, and holdout discipline- Working knowledge of LLM-as-judge methods, human evaluation, inter-annotator agreement, and their appropriate use- Ability to reason about statistical significance on small samples- Familiarity with regression tracking, telemetry,



and versioning- Ability to turn ambiguous ideas into scoped evaluation plans with hypotheses and metrics- Comfortable making trade-off calls across quality, latency, cost, and vendor lock-in- Experience writing concise decision memos- Ability to explain technical results to non-specialists- Experience working with platform, product, and operations teams- Ability to influence without authority and align teams around standards and guardrails- Curious, experimentation-oriented mindset with disciplined measurement and risk awareness- Comfortable in a small, high-leverage team without embedded PMs- Builder attitude favoring reusable tools, templates, and playbooksCore CompetenciesDemonstrates extensive experience in evaluating and adopting AI and automation technologies, with a strong focus on LLMs and cloud infrastructure, particularly AWS. Capable of designing experiments, building prototypes, and collaborating across teams to ensure effective integration and governance.Highest-signal resume keywords- LLM Evaluation- Cloud Infrastructure (AWS)- Experimental Design- Benchmarking and Telemetry- Decision Memo WritingHard Skills- Software Engineering- AI Tooling- System Architecture- Observability- Statistical Significance Reasoning- Evaluation Dataset Construction- Regression Tracking- Integration Patterns- Automation and Tooling- PrototypingSoft Skills- Influencing Without Authority- Curiosity- Collaboration- Mentoring- CommunicationIndustry Keywords- AI Governance- Risk Assessment- Compliance- Experimental Workloads- Vendor EvaluationTools & Technologies- LLMs- Agentic Systems- Vector Databases- Orchestration Frameworks- Telemetry Tools- Decision Templates- Reusable Checklists- AWS AI Suite#J-18808-Ljbffr

📌 Staff Ai Engineer – Ai Labs (Madrid)
🏢 Jobtailor
📍 Madrid