AI Dialogue Quality Analyst

AI Dialogue Quality Analyst

About the Role

We're looking for experienced AI Dialogue Quality Analysts in London to evaluate and improve AI-powered chatbots and dialogue systems. You'll assess how well AI maintains context, coherence, and helpfulness across extended conversations — and your insights will directly shape the next generation of conversational products.

• Organization: Alignerr
• Type: Hourly Contract
• Compensation: $20–$40 /hour
• Location: Remote
• Commitment: 10–40 hours/week

What You'll Do

• Conduct structured evaluations of AI chatbot conversations across diverse topics and use cases
• Assess dialogue quality for context retention, state management, coherence, and natural flow
• Identify where and how conversations degrade over extended multi-turn interactions
• Design and execute test scenarios that stress-test conversational AI capabilities
• Document failure patterns, edge cases, and quality issues using structured feedback frameworks
• Rate chatbot responses for accuracy, helpfulness, safety, and tone consistency
• Provide actionable recommendations for improving dialogue system performance
• Collaborate asynchronously with research and product teams to refine evaluation criteria

Who You Are

• 2+ years of experience in conversational AI evaluation, chatbot QA, dialogue system testing, or UX research for conversational products
• Strong understanding of how context, state, and coherence degrade across extended conversations
• Analytical mindset with the ability to identify subtle quality issues in natural language interactions
• Excellent written communication skills — you can clearly articulate what went wrong and why
• Detail-oriented and systematic in your approach to testing
• Comfortable working independently and managing your own schedule
• Familiar with conversation design principles and common chatbot failure modes

Nice to Have

• Experience with NLP, LLMs, or machine learning concepts
• Background in linguistics, cognitive science, or human-computer interaction
• Familiarity with evaluation frameworks like BLEU, human preference ratings, or custom rubrics
• Experience writing test plans or QA documentation for conversational products
• Prior work with prompt engineering or red-teaming AI systems

Why Join Us

• Work on cutting-edge AI projects shaping the future of conversational technology
• Flexible, fully remote work — set your own schedule and work asynchronously
• Collaborate with top AI research labs and industry leaders
• Autonomy and independence in how you approach your work
• Potential for ongoing work and contract extension
• Build deep expertise in one of the fastest-growing areas of AI

Application Process (Takes 10–15 min)

• Submit your resume
• Complete a short screening
• Project matching and onboarding

PS: Our team reviews applications daily. Please complete your application steps to be considered for this opportunity.