iRCFA-data/benchmark/50_queries/BENCHMARK.md

RCFA LLM Benchmark Report

Procedure Reference: KPJB-0953-15-WI-05
Dataset: hph_inference_test_set_50.jsonl (50 cases — 25 High, 15 Medium, 10 Low risk)
Hardware: Local inference via Ollama (http://localhost:11434)
Date: 2026-04-28
Evaluated Models: qwen3:4b, qwen3:8b, gemma4:e2b, gemma4:e4b

---

1. Performance Summary

Model	Cases	Avg Speed (t/s)	Min Speed	Max Speed	Avg Time/Case	Total Tokens	Avg Tokens/Case
qwen3:4b	50	59.4	55.5	65.2	51.2 s	148,448	2,969
qwen3:8b	50	41.0	40.1	43.8	34.3 s	68,320	1,366
gemma4:e2b	50	41.6	39.5	43.7	31.8 s	62,526	1,251
gemma4:e4b	50	28.3	27.2	29.7	60.2 s	82,364	1,647

Note: qwen3:4b is the fastest model by a significant margin at 59.4 t/s, nearly 45% faster than the Gemma variants. However, it generates ~2.4× more tokens per case than gemma4:e2b, which drives up total latency despite the higher token throughput.

---

2. SOP Compliance (KPJB-0953-15-WI-05)

All five required RCFA sections evaluated: Gejala, Verifikasi Lapangan, Akar Masalah, Dampak Sistem, Rekomendasi FDT.

Model	5/5 Cases	SOP Avg	Gejala	Verifikasi	Akar Masalah	Dampak	FDT
qwen3:4b	50/50 (100%)	5.00/5	50	50	50	50	50
qwen3:8b	50/50 (100%)	5.00/5	50	50	50	50	50
gemma4:e2b	50/50 (100%)	5.00/5	50	50	50	50	50
gemma4:e4b	50/50 (100%)	5.00/5	50	50	50	50	50

Important: All four models achieved perfect SOP compliance (100%) across all 50 test cases. This confirms the prompt template is effective and all models can reliably follow the RCFA SOP structure on this domain. SOP adherence is therefore not a differentiator in this benchmark — harder adversarial prompts are needed to stress-test compliance.

---

3. Technical Terminology Density

Average count of recognized TJB engineering acronyms per response (HHWL, BFP, DCV, DCA, HPH, RCFA, ECT, DCS, PLC, TTD, PAUT, NDT, PID, FDT, NWL, SSC, MOV).

Model	Avg Domain Terms / Response	Delta vs. qwen3:4b
qwen3:4b	5.2	baseline
qwen3:8b	7.8	+50%
gemma4:e2b	6.8	+31%
gemma4:e4b	8.3	+60%

Takeaway: gemma4:e4b uses the richest engineering vocabulary despite generating fewer total tokens than qwen3:4b. qwen3:4b writes longer, more prose-heavy explanations that dilute the domain signal — its low terminology density relative to its token count suggests more generic/padded language.

---

4. Tag Integrity (Equipment Tag Cross-Contamination)

Checks whether a response accidentally references the wrong equipment tag (e.g., mentioning 3FW-H060 in a case about 3FW-H050).

Model	Tag Mismatches	Pass Rate
qwen3:4b	0	100%
qwen3:8b	1	98%
gemma4:e2b	0	100%
gemma4:e4b	0	100%

Warning: qwen3:8b produced 1 tag mismatch across 50 cases. In a real plant maintenance context, an incorrect equipment tag in an RCFA report could direct a crew to the wrong asset — a meaningful safety risk. This should be monitored closely at larger dataset scales.

---

5. Verbosity vs. Efficiency

Model	Avg Tokens/Case	Relative Verbosity
gemma4:e2b	1,251	most concise (baseline)
qwen3:8b	1,366	+9%
gemma4:e4b	1,647	+32%
qwen3:4b	2,969	+137%

qwen3:4b generates 2.4× more tokens than gemma4:e2b for the same SOP compliance score. The extra tokens consist of preambles, repetition, and hedging language that add no measurable structure. Verbosity should be treated as a cost, not a quality signal.

---

6. Speed vs. Efficiency Trade-off Matrix

Throughput (t/s)
    60 |  [qwen3:4b]  ← fastest clock speed, most tokens
       |
    42 |      [gemma4:e2b]  [qwen3:8b]  ← fastest wall-clock / case
       |
    28 |                         [gemma4:e4b]  ← richest terminology, slowest
       +------------------------------------------------→ Avg Tokens / Response
          1,250          1,650             3,000

• Best for real-time / low latency: gemma4:e2b — 31.8 s/case avg, zero errors
• Best raw throughput (t/s): qwen3:4b — but penalized by verbosity
• Best terminology quality: gemma4:e4b — limited by hardware, not model quality
• Best balance: qwen3:8b — concise, fast, high domain density (minor tag caveat)

---

7. Model Recommendations by Use Case

Use Case	Recommended Model	Rationale
Real-time operator advisory	gemma4:e2b	Lowest avg time (31.8 s), 100% SOP, zero tag errors
Offline batch annotation / RLHF data generation	qwen3:4b	Highest raw throughput; verbose outputs good for preference labeling
High-quality RCFA report drafting	gemma4:e4b	Richest domain vocabulary density; most precise phrasing
SFT/DPO training data	qwen3:8b	Balanced length + terminology; only minor safety flag

---

8. Key Takeaways

1. SOP compliance is fully saturated — all models score 100% on the 5-section check. The prompt engineering is the primary driver. To truly differentiate models, future benchmarks need adversarial prompts and expert-authored ground truth for semantic scoring (BLEU, BERTScore, or domain-expert review).

2. Gemma models are more token-efficient. Both Gemma variants achieve the same SOP coverage with ~55% fewer tokens than qwen3:4b. This matters enormously at inference cost for large-scale annotation runs.

3. qwen3:4b verbosity is a hidden cost. Despite having the highest t/s, running 500 cases with qwen3:4b (~7.1 hours) costs ~60% more compute time than gemma4:e2b (~4.4 hours) solely due to token volume.

4. gemma4:e4b is hardware-constrained, not model-constrained. Its 28.3 t/s and 60.2 s/case latency suggest memory-bandwidth saturation on the current machine. This model would likely outperform all others on better hardware given its terminology density.

5. qwen3:8b has a latent tag-hallucination risk. One cross-equipment tag reference in 50 cases is small but not negligible in safety-critical plant contexts. Monitor carefully and consider tag-anchoring instructions in the prompt.

6. No model exhibited a "weak section" in SOP. All sections (including Verifikasi Lapangan, historically the hardest to elicit) are consistently present. This suggests the FMEA context injection in the prompt is sufficient scaffolding for all models.

---

9. Next Steps

• Expand to 200+ cases with out-of-distribution failure modes (multi-equipment incidents, ambiguous indications, missing FMEA fields)
• Add semantic scoring (BERTScore / cosine similarity) against expert-written ground truth RCFA reports
• Run adversarial prompts to find SOP compliance failure thresholds for each model
• Use the ⭐ Best Response RLHF export in inference_results_viewer.html to collect preference labels for DPO fine-tuning
• Re-benchmark gemma4:e4b on upgraded hardware (expected significant throughput improvement)
• Evaluate num_ctx sensitivity: test 4096 vs. 8192 and measure whether longer context improves FDT specificity

---

Generated by the RCFA Benchmarking Pipeline — ollama_benchmark.py + inference_results_viewer.html
Analysis script: analyze_bench.py