iRCFA-data/benchmark/100_queries/BENCHMARK.md

RCFA LLM Benchmark Report — 100 Queries

Procedure Reference: KPJB-0953-15-WI-05
Dataset: hph_inference_test_set_100.jsonl (100 cases — 50 High, 30 Medium, 20 Low risk)
Hardware: Local inference via Ollama (http://localhost:11434)
Date: 2026-04-28 / 2026-04-29
Evaluated Models: qwen3:4b, qwen3:8b, gemma4:e2b, gemma4:e4b

---

1. Performance Summary

Model	Cases	Avg Speed (t/s)	Min Speed	Max Speed	Avg Time/Case	Total Tokens	Avg Tokens/Case
gemma4:e2b	100	78.6	53.7	91.2	18.2 s	126,262	1,263
qwen3:4b	100	59.9	43.2	66.7	49.8 s	287,191	2,872
qwen3:8b	100	41.7	39.3	44.6	32.6 s	132,632	1,326
gemma4:e4b	100	27.8	26.5	30.5	61.9 s	166,522	1,665

Headline finding: gemma4:e2b is the decisive throughput winner at 78.6 t/s — 31% faster than qwen3:4b and nearly 3× faster than gemma4:e4b. More importantly, it achieves this speed with only 1,263 tokens/case (the most concise model) and 18.2 s/case average wall-clock time. In a real-time operator advisory context, completing a full RCFA analysis in under 20 seconds is a strong operational advantage.

qwen3:4b appears fast at 59.9 t/s but generates 2.27× more tokens than gemma4:e2b, making it 2.7× slower per case at the wall-clock level.

---

2. SOP Compliance (KPJB-0953-15-WI-05)

All five required RCFA sections evaluated: Gejala, Verifikasi Lapangan, Akar Masalah, Dampak Sistem, Rekomendasi FDT.

Model	5/5 Cases	SOP Avg	Gejala	Verifikasi	Akar Masalah	Dampak	FDT
qwen3:4b	91/100 (91%)	4.68/5	92	93	92	91	100
qwen3:8b	100/100 (100%)	5.00/5	100	100	100	100	100
gemma4:e2b	100/100 (100%)	5.00/5	100	100	100	100	100
gemma4:e4b	100/100 (100%)	5.00/5	100	100	100	100	100

⚠️ Critical Finding: qwen3:4b SOP Regression at 100-Query Scale

qwen3:4b achieved 100% SOP compliance in the 50-query test but dropped to 91% at 100 queries. Nine cases failed to include all five mandatory sections:

Case	Tag	Risk Level	Missing Sections
#11	3FW-H070	Low	Gejala, Verifikasi, Akar Masalah, Dampak
#12	3FW-H060	High	Gejala, Verifikasi, Akar Masalah, Dampak
#24	3FW-H070	High	Gejala, Verifikasi, Akar Masalah, Dampak
#40	3FW-H050	High	Gejala, Verifikasi, Akar Masalah, Dampak
#57	3FW-H060	High	Gejala, Verifikasi, Akar Masalah, Dampak
#61	3FW-H050	Medium	Gejala, Verifikasi, Akar Masalah, Dampak
#89	3FW-H050	Medium	Gejala, Verifikasi, Akar Masalah, Dampak
#96	3FW-H070	Low	Gejala, Akar Masalah, Dampak
#99	3FW-H060	Medium	Dampak only

Safety Alert: 4 of the 9 SOP failures occurred on High-risk cases. The pattern of missing Gejala, Verifikasi, Akar Masalah, and Dampak simultaneously suggests qwen3:4b occasionally abandons the structured 5-section RCFA format entirely and reverts to a prose-style response. In a real RCFA audit, any response missing Akar Masalah (Root Cause) or Dampak Sistem on a High-risk asset is a disqualifying failure.

Note: qwen3:8b, gemma4:e2b, and gemma4:e4b all achieved zero SOP failures across all 100 cases and all three risk levels.

---

3. Technical Terminology Density

Average count of recognized TJB engineering acronyms per response (HHWL, BFP, DCV, DCA, HPH, RCFA, ECT, DCS, PLC, TTD, PAUT, NDT, PID, FDT, NWL, SSC, MOV, FMEA).

Model	Avg Domain Terms / Response	Delta vs. qwen3:4b
qwen3:4b	3.67	baseline
qwen3:8b	4.34	+18%
gemma4:e2b	4.88	+33%
gemma4:e4b	5.08	+38%

Takeaway: gemma4:e4b retains the highest terminology density (5.08). Notably, gemma4:e2b (4.88) achieves near-equivalent domain precision to the much larger e4b variant, while being 4.3× faster. qwen3:4b remains last in terminology density despite generating more than twice as many tokens as any other model — confirming that its extra output is generic prose filler.

---

4. Tag Integrity (Equipment Tag Cross-Contamination)

Checks whether a response accidentally references the wrong equipment tag (e.g., mentioning 3FW-H060 in a case about 3FW-H050).

Model	Tag Mismatches	Pass Rate
qwen3:4b	0	100%
qwen3:8b	0	100%
gemma4:e2b	0	100%
gemma4:e4b	0	100%

All four models achieved perfect tag integrity across all 100 cases. The prompt's tag-anchoring mechanism is working reliably at this scale for all model families.

---

5. Verbosity vs. Efficiency

Model	Avg Tokens/Case	Relative Verbosity
gemma4:e2b	1,263	most concise (baseline)
qwen3:8b	1,326	+5%
gemma4:e4b	1,665	+32%
qwen3:4b	2,872	+127%

gemma4:e2b is now the most token-efficient model overall — narrowly edging qwen3:8b by 5%. Both Gemma and Qwen-8b occupy a similarly efficient tier, while gemma4:e4b trades additional tokens for richer vocabulary. Running the full 672-case dataset with qwen3:4b would generate ~1.93M tokens versus gemma4:e2b's projected ~849K — a 2.3× overhead for demonstrably worse SOP compliance.

---

6. Speed vs. Efficiency Trade-off Matrix

Throughput (t/s)
    79 |  [gemma4:e2b]  ← fastest clock + wall-clock, 100% SOP, most concise
       |
    60 |      [qwen3:4b]  ← fast clock, slow wall-clock, SOP regression
       |
    42 |           [qwen3:8b]  ← balanced speed, 100% SOP
       |
    28 |                  [gemma4:e4b]  ← richest terminology, slowest
       +------------------------------------------------→ Avg Tokens / Response
           1,263         1,326         1,665           2,872

• Best overall (speed + SOP + conciseness): gemma4:e2b — 18.2 s/case, 100% SOP, zero tag errors
• Best balance for batch runs: qwen3:8b — consistent 32.6 s/case, rock-solid reliability
• Best terminology quality: gemma4:e4b — hardware-constrained; most precise phrasing
• Disqualified for production use: qwen3:4b — 9% SOP failure rate on structured reports at scale

---

7. Risk-Stratified SOP Compliance

Risk Level	Cases	qwen3:4b	qwen3:8b	gemma4:e2b	gemma4:e4b
High	50	46/50 (8% failure)	50/50 ✅	50/50 ✅	50/50 ✅
Medium	30	27/30 (10% failure)	30/30 ✅	30/30 ✅	30/30 ✅
Low	20	18/20 (10% failure)	20/20 ✅	20/20 ✅	20/20 ✅

qwen3:4b's failure rate is consistent across all risk levels — it does not apply additional structural discipline on High-risk cases. qwen3:8b, gemma4:e2b, and gemma4:e4b all scored zero failures across every risk stratum.

---

8. Model Recommendations by Use Case

Use Case	Recommended Model	Rationale
Real-time operator advisory	gemma4:e2b	Fastest wall-clock (18.2 s), 100% SOP, most concise, zero errors
Offline batch annotation / RLHF data	gemma4:e2b	Best throughput for volume runs; structured output at scale
High-quality RCFA report drafting	gemma4:e4b	Richest domain vocabulary (5.08 terms/resp); 100% SOP
SFT/DPO chosen responses	gemma4:e4b	Highest terminology precision; ideal for quality preference pairs
Reliable fallback / balanced option	qwen3:8b	Proven stability, 100% SOP, predictable latency
⛔ Production use	qwen3:4b	Disqualified — 9% SOP failure rate including High-risk cases

---

9. Key Takeaways

1. gemma4:e2b is the surprise leader. At 78.6 t/s and 18.2 s/case, it is the fastest model by a large margin while maintaining 100% SOP compliance and the second-highest terminology density. It achieves more with fewer tokens than any competitor.

2. qwen3:4b SOP compliance is not stable at scale. The 9% failure rate exposed at 100 queries was invisible at 50. Its failures concentrate on complete format collapse — skipping 4 of 5 mandatory RCFA sections while always retaining the FDT recommendation. A recommendation without root cause analysis is operationally dangerous.

3. gemma4:e4b is hardware-constrained, not model-constrained. Its 100% SOP compliance and leading terminology density (5.08) confirm highest qualitative output. The 61.9 s/case latency is a local hardware limitation.

4. All four models passed tag integrity at 100 queries. The qwen3:8b mismatch from the 50-query run did not recur — confirming it was a stochastic edge case.

5. The Gemma model family dominates on efficiency. Both gemma4:e2b and gemma4:e4b outperform Qwen models on token efficiency and domain terminology density. The e2b variant in particular offers an exceptional quality-to-latency ratio for operational deployment.

---

10. Next Steps

• gemma4:e2b 100-query results added — 4-model comparison now complete
• Root cause qwen3:4b failures — inspect the 9 failing cases to determine if a specific FMEA failure mode or prompt variant triggers the format collapse
• Expand to 200+ cases with out-of-distribution failure modes (multi-equipment incidents, ambiguous indications, missing FMEA fields)
• Add semantic scoring (BERTScore / cosine similarity) against expert-written ground truth RCFA reports
• Run adversarial prompts to find SOP compliance failure thresholds for qwen3:8b, gemma4:e2b, and gemma4:e4b
• Re-benchmark gemma4:e4b on upgraded hardware (expected significant throughput improvement)
• Use the ⭐ Best Response RLHF export in inference_results_viewer.html to collect preference labels for DPO fine-tuning

---

Generated by the RCFA Benchmarking Pipeline — ollama_benchmark.py + inference_results_viewer.html
Analysis script: analyze_100.py