More in detail, UST is a multinational company based in North America, certified as a Top Employer and Great Place to Work, with over 35,000 employees worldwide and presence in more than 35 countries. We are leaders in digital technology services, delivering large-scale technological solutions to top-tier companies.
We are looking for a Senior AI Test Automation Engineer to join the ML & AI ART, owning the end-to-end test automation strategy for an advanced AI/MLOps platform. Define and evolve the test automation architecture aligned with the Bank’s Test Strategy
Design scalable, reusable automation frameworks and patterns across teams
Analyse requirements and ensure testability during Agile ceremonies (PI Planning, Backlog Refinement, etc.)
Design and automate test cases (functional & non-functional) based on risk and coverage
Implement automation using Python frameworks (Playwright, pytest, Behave/BDD)
Develop AI/ML-specific testing approaches (LLM evaluation, non-deterministic behaviour, RAG workflows)
Build and maintain API contract tests and microservices validation
Integrate and run automated tests in CI/CD pipelines (GitLab)
Use Docker/Kubernetes for reproducible test environments
Manage test execution across environments (DEV, INT, UAT, pre-PROD)
Monitor test health, handle flaky tests, and ensure reliable reporting
Report defects in Jira with proper traceability and evidence
Produce audit-ready test artefacts and ensure compliance with governance standards
Collaborate with cross-functional teams (POs,



Engineers, MLOps, Data Engineers)
Act as Test Automation Ambassador within the ART
Ensure end-to-end traceability across requirements, tests, and results
Maintain and evolve test suites as systems (APIs, models, prompts) change
Strong experience in Python-based test automation (Playwright, pytest, BDD frameworks)
Proven ability to design and own test frameworks, not just test scripts
Experience integrating automation into CI/CD pipelines (GitLab preferred)
Knowledge of parallel execution, reporting tools, and flaky test management
Experience with Docker and basic knowledge of Kubernetes
Exposure to AI/ML testing (LLMs, RAG, non-deterministic systems) or strong motivation to learn
Familiarity with SAFe, Agile, and DevOps principles
Experience with Jira and test management tools (Xray, Agile Hive, etc.)
Awareness of data privacy, security, and compliance in regulated environments
Work Location Hybrid position in Madrid (3 days at client´s office, Sol area, next to the underground).
Numerous benefits (Health Care Plan, teleworking compensation, Life and Accident Insurances).
~`Retribución Versátil´ Program: (Meals, Kinder Garden, Transport, online English lessons, Health Care Plan…)
~ Free access to several training platforms
~ Real Work Life Balance measures (flexibility, WFH or remote work policy, compacted hours during summertime…)
~ Python
API Testing
pytest
GitLab CI
Test Data Management

📌 Senior AI Test Automation Engineer - Hybrid Madrid
🏢 Ust
📍 Madrid