Responsibilities - Design and implement automated and manual QA strategies for GenAI systems

- Build regression tests, generate synthetic test data, build test harnesses and evaluation pipelines for accuracy, grounding, robustness, safety
- Validate prompt logic, model behavior and evaluation metrics
- Execute tests and debug pipelines
- Support deployment of GenAI products
- Lead system design reviews, advocating best practices for architecture and reliability of GenAI applications
- Collaborate closely with data scientists and developers to embed testing and quality checkpoints
- Establish testing standards and workflows for continuous integration and deployment
- Conduct code reviews with a focus on design, testability, and maintainability
- Document system design decisions, test cases, and best practices
- Close collaboration with domain experts and stakeholders to validate developed solutions Requirements - Strong software engineering background with QA ownership
- Familiarity with RAG & Graph frameworks
- 3+ years of experience as a software engineer (preferably 5), or QA engineer in production systems,



preferably including at least 1 year experience in GenAI systems
- Advanced Python skills
- Experience working with LLM-based systems in real environments
- Hands-on experience with: LLM APIs, AWS Bedrock (Anthropic & OSS), Azure Foundry (OpenAI), GCP (Gemini), Cloud platforms (AWS, Azure, GCP)
- Proven ability to implement test frameworks for GenAI applications
- Knowledge of API testing, data validation, and data pipeline testing
- Experience with version control (Git)
- Experience with building and deploying AI/ML products through the full product lifecycle
- Good understanding of scientific computing and ML frameworks
- Solid understanding of safety, bias, and failure modes of GenAI systems
- Comfortable working in a relatively fast-moving environment where priorities may shift and initiative is valued
- Proactive problem-solver who can anticipate needs and find solutions
- Comfortable working independently, but also a strong team player.

📌 Senior Engineer – GenAI Quality Assurance (Madrid)
🏢 Jobtailor
📍 Madrid