This job is no longer available
This job expired on 06/10/2026. It no longer accepts applications.
LLM Evaluator (Model Response Analyst)
Odixcity Consulting
Job description
About the role
We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. In this remote position you will analyze AI‑generated content for factual accuracy, coherence, safety, bias, and alignment with our guidelines.
Key responsibilities
- Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
- Compare multiple model responses to the same prompt and justify the preferred output.
- Provide concise feedback to modeling and training teams about recurring failure patterns.
- Craft adversarial prompts to expose biased, harmful, or insecure model behavior.
- Collaborate with QA to refine evaluation guidelines for ambiguous or edge‑case scenarios.
- Participate in cross‑checking sessions to calibrate scoring standards and ensure inter‑rater reliability.
- Investigate underperforming outputs, hypothesizing root causes such as training data gaps or prompt misinterpretation.
- Document novel model behaviors and contribute to a living library of failure modes.
Required profile
- Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
- Bachelor’s degree in Computer Science or a related field.
- Strong understanding of prompt engineering and the ability to explain why a model output is good or bad.
- Experience with Reinforcement Learning from Human Feedback (RLHF) data collection.
- Proven ability to monitor and improve consistency among evaluation teams.
- Familiarity with dataset sourcing, cleaning, annotation, and A/B testing concepts for AI.
Required skills
- Prompt engineering
- Reinforcement Learning from Human Feedback (RLHF) data collection
- Dataset sourcing, cleaning, and annotation
- A/B testing for AI model evaluation
Questions fréquentes
Why are you reporting this job?
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Odixcity Consulting
Related job offers
-
Senior Software Engineer, Native Mobile (International)
Coinbase -
Forward Deployed Engineer - ProcessOS
Camunda -
Programmeur·euse compétitif·ve – Benchmarks IA (Remote)
Vettedlance -
Enregistreur de flux de travail pour outils de développement (éditeurs de code, IDE, clients API)
Vettedlance -
Ingénieur DevOps, Cloud & Linux – Environnement d'Entraînement IA (Remote)
Vettedlance