LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. In this remote position you will analyze AI‑generated content for factual accuracy, coherence, safety, bias, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines for ambiguous or edge‑case scenarios.
• Participate in cross‑checking sessions to calibrate scoring standards and ensure inter‑rater reliability.
• Investigate underperforming outputs, hypothesizing root causes such as training data gaps or prompt misinterpretation.
• Document novel model behaviors and contribute to a living library of failure modes.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Strong understanding of prompt engineering and the ability to explain why a model output is good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection.
• Proven ability to monitor and improve consistency among evaluation teams.
• Familiarity with dataset sourcing, cleaning, annotation, and A/B testing concepts for AI.

Required skills

• Prompt engineering
• Reinforcement Learning from Human Feedback (RLHF) data collection
• Dataset sourcing, cleaning, and annotation
• A/B testing for AI model evaluation