Responsibilities

Design and implement automated and manual QA strategies for GenAI systems

Build regression tests, generate synthetic test data, build test harnesses and evaluation pipelines for accuracy, grounding, robustness, safety

Validate prompt logic, model behavior and evaluation metrics

Execute tests and debug pipelines

Support deployment of GenAI products

Lead system design reviews, advocating best practices for architecture and reliability of GenAI applications

Collaborate closely with data scientists and developers to embed testing and quality checkpoints

Establish testing standards and workflows for continuous integration and deployment

Conduct code reviews with a focus on design, testability, and maintainability

Document system design decisions, test cases, and best practices

Close collaboration with domain experts and stakeholders to validate developed solutions

Requirements

Strong software engineering background with QA ownership

Familiarity with RAG & Graph frameworks

3+ years of experience as a software engineer (preferably 5), or QA engineer in production systems,



preferably including at least 1 year experience in GenAI systems

Advanced Python skills

Experience working with LLM-based systems in real environments

Hands-on experience with: LLM APIs, AWS Bedrock (Anthropic & OSS), Azure Foundry (OpenAI), GCP (Gemini), Cloud platforms (AWS, Azure, GCP)

Proven ability to implement test frameworks for GenAI applications

Knowledge of API testing, data validation, and data pipeline testing

Experience with version control (Git)

Experience with building and deploying AI/ML products through the full product lifecycle

Good understanding of scientific computing and ML frameworks

Solid understanding of safety, bias, and failure modes of GenAI systems

Comfortable working in a relatively fast-moving environment where priorities may shift and initiative is valued

Proactive problem-solver who can anticipate needs and find solutions

Comfortable working independently, but also a strong team player.

📌 Senior Engineer – GenAI Quality Assurance (Madrid)
🏢 Johnson Johnson
📍 Madrid