Round 1: Head to Head Comparison

View Qualitative Analysis

Dimagi vs Vanilla Bot Comparison

Total Annotations

100

Average Agreement

34.51%

Dimagi Preferred

62%

Vanilla Preferred

38%

Overall Preference Distribution

Lead Time Distribution (Box Plot)

Dimagi Bot: Median 146s (Q1: 79s, Q3: 518s) | Vanilla Bot: Median 122s (Q1: 76s, Q3: 231s)

Preferences by Annotator

Preferences by Category

Reasoning Theme Analysis

Distinctive Keywords - Dimagi Bot

Distinctive Keywords - Vanilla Bot

Annotator Disagreements - Sample Cases

Input query: Salut vous allez bien çv je voulais vous demander j'ai eu à faire un rapport le ...

Annotator 1 chose: Vanilla Bot

Reasoning: "message bienveillant et sécurisant"

Translation: "reassuring and benevolent message"

Annotator 2 chose: Dimagi Bot

Reasoning: "empathie+explications sans dramatisation+ solution concrète"

Translation: "empathy+explanations without dramatization+ concrete solution"

Input query: Bonjour ici, D'accord je voulais me renseigner sur les méthodes de contraception...

Annotator 1 chose: Dimagi Bot

Reasoning: "c'est plus méthodique, mais attention l'abstinence n'est pas une contraception"

Translation: "it's more methodical, but note that abstinence is not a contraceptive method"

Annotator 2 chose: Vanilla Bot

Reasoning: "le professionnel de santé est impliqué"

Translation: "healthcare professional is involved"

Key Findings

Overall Preference: Dimagi Bot was preferred in 62% of evaluations while Vanilla Bot was preferred in 38%, with consistent preferences across both annotators.

Category Performance: Dimagi Bot performed consistently well across all categories (Easy: 64%, Medium: 57%, Critical: 65%), showing particular strength in critical cases.

Lead Time Analysis: Vanilla Bot evaluations showed more consistency (Q1-Q3 range: 76-231s) compared to Dimagi Bot (Q1-Q3 range: 79-518s), suggesting more straightforward decision-making.

Distinctive Strengths: Dimagi Bot excelled in solution-oriented responses (17 vs 2 mentions) while Vanilla Bot led in empathy/tone (18 vs 15 mentions). Keywords reflect this: "solution/concrète" vs "chaleureux/bienveillant".

Disagreement Patterns: In 24% of cases, annotators disagreed, often prioritizing different aspects: solution quality vs emotional support. This suggests the need for clearer evaluation criteria based on query context.