Home / Chatbot Accuracy Evaluation / Sample report

This is a sample report published by GlobeZenith. The full evaluation service is described here.

AI Chatbot Accuracy Evaluation — Sample Report

Sample report — this sample evaluates a demo support bot built in-house for illustration. All scores are from an actual evaluation run.

The store, its FAQ corpus, and the test questions were created for this sample; this is not an evaluation of a real client's system. The bot's answers and the scores below are the real output of running that test set against that demo bot.

Client (sample): Northwind Supply Co. (fictional demo store)
System evaluated: Retrieval-based help-center support bot (demo build)
Knowledge base: 15-entry FAQ corpus (shipping, returns, payment, orders, account, products, discounts)
Test set: 20 questions — 12 straightforward, 4 paraphrases, 2 out-of-scope, 2 unanswerable-detail
Evaluation date: 2026-11-13
Evaluator: GlobeZenith OÜ (Estonian registry code 16979948 — company overview)

---

1. Executive summary

The bot was run once against all 20 test questions, and every response was graded against the facts in its own FAQ corpus. No scores were estimated or adjusted.

Verdict distribution: Correct 13 · Wrong 7 · Partially correct 0 · Correctly declined 0 · Fabricated 0

The headline finding: on this test set, the bot is accurate when questions are phrased the way the FAQ is written (12/12 on straightforward questions), but it breaks down in two specific ways:

1. It does not know what it does not know. All 4 questions that should have been refused were answered anyway — with real FAQ text that did not address the question. The most damaging case quoted domestic US shipping rates ($4.99) for a shipment to Germany — a destination the corpus does not cover at all (its international entry lists Canada, the UK and Australia, with costs calculated at checkout).
2. It is brittle to paraphrasing. Only 1 of 4 reworded questions was answered correctly. Two answerable questions were refused because their wording scored just below the matching threshold, and one was matched to a completely wrong FAQ entry (a returns question received password-reset instructions).

Both failure modes are measurable and reproducible, and the analysis points to configuration and corpus changes rather than a bot replacement — see §7 for the remediation plan and §6 for a threshold simulation computed from the measured scores. The recommended fixes were not re-tested in this run.

---

2. System under test

The demo bot is a first-generation retrieval bot of the kind many stores deploy as a first help-center automation step:

• Retrieval only. The bot can only reply with the verbatim text of one FAQ entry. It has no generative capability and cannot compose new text.
• Matching. Each FAQ entry (question + answer text) is indexed with TF-IDF (word unigrams). A customer question is scored against every entry by cosine similarity, and the top entry is returned.
• Safety threshold. If the best similarity score is below 0.20, the bot replies with a fixed fallback ("I'm sorry, I don't have information about that…") instead of guessing.
• Corpus. 15 FAQ entries covering shipping (3), returns (3), payment (2), orders (3), account (2), products (1), and discounts (1).

Because the bot is retrieval-only, it cannot invent facts. Its risk profile is different from a generative (LLM) chatbot: the danger is not hallucinated text but confidently returning the wrong policy — which is exactly what this evaluation found. The same method — a frozen test set, pre-registered expected behaviours, per-question grading — applies to generative bots, where the Fabricated category becomes the primary risk; the harness itself is adapted to capture free-text answers.

---

3. Method

1. Test design. 20 questions were written before the run and frozen. Each carries an expected behaviour — answer (with the specific facts a correct answer must contain) or decline (the corpus contains no answer, so refusal is the only correct behaviour):
   • Straightforward (12): phrased close to the FAQ wording; the everyday happy path.
   • Paraphrase (4): the same facts requested in everyday wording ("send it back", "coupon", "guarantee") that shares few keywords with the FAQ.
   • Out of scope (2): topics absent from the corpus (physical store hours; a competitor price-match policy).
   • Unanswerable detail (2): questions that sound answerable and share vocabulary with real entries, but ask for specifics the corpus never states (an exact discount-code expiry date; an exact shipping price to Germany).
2. Execution. The bot answered all 20 questions in a single run. Raw outputs — the exact answer text, matched FAQ entry, similarity score, and whether the bot declined — were recorded to a results file.
3. Grading. A human evaluator graded each answer against the frozen expected facts, using five categories:
   • Correct — the right FAQ content, containing all expected facts.
   • Partially correct — the right topic, but material facts missing or incomplete.
   • Wrong — a wrong/irrelevant answer, or a refusal of a question the corpus can answer.
   • Correctly declined — refusal of a question the corpus cannot answer.
   • Fabricated — facts stated that appear nowhere in the corpus. (Not observed in this run; see §2 on why this architecture cannot produce it, and why it is graded anyway.)

---

4. Results by category

---

5. Full results — all 20 questions

Retrieval score is the bot's cosine similarity for its top match (threshold 0.20). "Matched" is the FAQ entry the bot returned, or — if it declined.

---

6. Failure analysis

All 7 failures come from three measured patterns:

Pattern A — No effective "I don't know" (4 cases: Q17, Q18, Q19, Q20).
The decline threshold (0.20) is too permissive for questions that share vocabulary with real entries. Three of the four traps cleared it comfortably (0.2180–0.6338). A retrieval score measures word overlap, not whether the entry answers the question — Q19 scored the highest similarity of the entire run while being unanswerable.

Pattern B — Paraphrase fragility (2 cases: Q13, Q16).
Everyday synonyms ("parcel", "normal delivery option", "guarantee") are invisible to a unigram keyword index, so answerable questions scored 0.1802 and 0.1339 and were refused. Customers who phrase questions in their own words hit this.

Pattern C — Wrong-entry retrieval just above threshold (1 case: Q14).
The wrong entry (password reset) scored 0.2038 against the correct entry (returns). A match this weak should never be served as a confident answer.

Threshold simulation (computed from the measured scores — not a re-run)

Because retrieval rankings do not depend on the threshold, the effect of other threshold values can be computed exactly from the recorded scores:

On this recorded test set, a threshold of 0.25 yields +2 outcomes (13 → 15). That is a simulation on the same set used to choose the value, not a measured improvement: whether it generalises to unseen questions can only be shown on a separate validation set. It is also a trade-off curve, not a fix: Q19 (0.6338) stays wrong at every threshold that keeps any correct answer, and Q13/Q14/Q16 are unaffected or worse. Threshold tuning must be paired with the structural fixes below.

---

7. Recommendations

In priority order, based on the failures above:

1. Raise the decline threshold to 0.25 and add a margin rule (immediate, configuration only). Accept a match only if it scores ≥ 0.25 and leads the runner-up entry by a clear margin. The threshold change alone accounts for the simulated 13 → 15 on this test set (Q18 and Q20 become correct refusals; subject to validation on unseen questions, per §6). The margin rule addresses Pattern C differently: Q14's wrong match (0.2038) would become a refusal instead of password-reset instructions. That is still graded Wrong — the question was answerable — but a refusal invites the customer to contact support, while wrong instructions actively mislead them. The margin rule's benefit is harm reduction, not a score gain, and its exact margin value needs the runner-up scores from a validation run.
2. Add a coverage check for "detail trap" questions (Pattern A). When a question asks for a specific value — a date, a price, a named code — verify that value actually appears in the matched entry before answering. Q19 asked for an expiry date that no entry contains; Q20 asked for a Germany price that no entry contains. Both would be refused by this check at any threshold.
3. Close the vocabulary gap (Pattern B). Add synonym/alias terms to the index (parcel → shipping, guarantee → warranty, coupon → discount code, send back → return), or move matching to embeddings, which handle paraphrases natively. Re-run this same 20-question set afterwards and require the paraphrase category to reach at least 3/4 before shipping.
4. Expand the corpus where refusals cluster. False refusals (Q13, Q16) and traps (Q17, Q20) point at real customer intents the FAQ does not cover explicitly — delivery-time phrasings, warranty scope, and international pricing. Add entries for the intents, and add explicit "we do not operate physical stores / we do not price-match" entries so those questions have a correct, answerable home instead of a trap.
5. Re-evaluate on a fixed cadence with a frozen test set. This 20-question set, plus every new failure found in production logs, should be re-run after each change to the corpus, threshold, or matching engine. Accuracy claims should always cite the run, not an impression.

---

8. Limitations and reproducibility

• The corpus (15 entries) and test set (20 questions) are small by design: this is a sample engagement demonstrating method and reporting format. A production evaluation should use 100+ questions drawn from real support tickets, including multi-turn conversations, which this sample does not test.
• The system evaluated is a deliberately simple demo bot built for this sample. Scores describe this demo system only and are not a benchmark for any other product.
• Grading was performed by a single human evaluator against pre-registered expected facts; borderline calls were graded against the corpus text only. The grading file is preserved with the run data.
• The threshold analysis in §6 is computed from the recorded scores, not from additional runs; the bot was executed once.
• All artefacts of this run are preserved together: the FAQ corpus (faq_corpus.json), the bot source (bot.py), the test set (test_questions.json), the raw run output (results.json), the grading (scores.json), and the aggregation script (analyze.py, producing analysis.json). Re-running python3 run_evaluation.py && python3 analyze.py reproduces every number in this report.

Prepared by GlobeZenith OÜ — AI Chatbot Accuracy Evaluation (fixed-scope engagement). This sample uses a fictional store and a demo bot; in a client engagement the same method is applied to the client's live bot, corpus, and real customer questions.