Remote (Spain)

Contract
Role Overview

We’re hiring an AI Quality Analyst to help us evaluate a personalization feature we’re building into Gemini. The idea behind it is pretty straightforward, the model should be able to use what it knows about you (past conversations, Gmail, Search, YouTube activity) to give you answers that actually feel relevant, not just technically correct.

Your job is to put that to the test. You’ll come up with prompts based on your own experiences, run them through the model, and then honestly assess whether the responses felt personalized in a meaningful way or just kind of generic with a personal detail tacked on. It’s equal parts creative and analytical, and the quality of your judgment really does matter here.
Responsibilities
- Design multi-turn conversational prompts (typically 1–5 turns) that require the AI to draw on real personal information and experiences.
- Evaluate whether the model applied personalization correctly based on what was actually being asked.
- Review responses for Grounding issues.



Flag anything that looks like a flawed inference or hallucination rather than evidence-backed reasoning.
- Assess Integration quality. Does the personal data feel naturally woven in, or does it come across as robotic and forced?
- Stack-rank two model responses side-by-side (SxS) based on helpfulness, ease of use, and overall quality.
- Write clear, well-structured rationales that reference specific turns in the conversation.
- Extract and verify “Debug Info” to confirm chat summaries and data sources were properly used.
- Clear evaluation conversations after each session to maintain clean data.
Required Qualifications
- Strong English reading and writing skills, the project is conducted entirely in English.
- Demonstrated ability to evaluate nuanced or ambiguous AI responses and explain your reasoning clearly.
- Comfortable working independently in a remote setup with minimal hand-holding.
- Reliable desktop or laptop with a stable i

📌 LLM - AI Quality Analyst (Personalization) - Spanish (España)
🏢 Careerflow
📍 España