ResponsibilitiesDesign and implement automated and manual QA strategies for GenAI systemsBuild regression tests, generate synthetic test data, build test harnesses and evaluation pipelines for accuracy, grounding, robustness, safetyValidate prompt logic, model behavior and evaluation metricsExecute tests and debug pipelinesSupport deployment of GenAI productsLead system design reviews, advocating best practices for architecture and reliability of GenAI applicationsCollaborate closely with data scientists and developers to embed testing and quality checkpointsEstablish testing standards and workflows for continuous integration and deploymentConduct code reviews with a focus on design, testability, and maintainabilityDocument system design decisions, test cases, and best practicesClose collaboration with domain experts and stakeholders to validate developed solutionsRequirementsStrong software engineering background with QA ownershipFamiliarity with RAG & Graph frameworks3+ years of experience as a software engineer (preferably 5), or QA engineer in production systems,
preferably including at least 1 year experience in GenAI systemsAdvanced Python skillsExperience working with LLM-based systems in real environmentsHands-on experience with: LLM APIs, AWS Bedrock (Anthropic & OSS), Azure Foundry (OpenAI), GCP (Gemini), Cloud platforms (AWS, Azure, GCP)Proven ability to implement test frameworks for GenAI applicationsKnowledge of API testing, data validation, and data pipeline testingExperience with version control (Git)Experience with building and deploying AI/ML products through the full product lifecycleGood understanding of scientific computing and ML frameworksSolid understanding of safety, bias, and failure modes of GenAI systemsComfortable working in a relatively fast-moving environment where priorities may shift and initiative is valuedProactive problem-solver who can anticipate needs and find solutions Comfortable working independently, but also a strong team pl