iRCFA-data/benchmark/100_queries/BENCHMARK.md

# RCFA LLM Benchmark Report — 100 Queries
**Procedure Reference:** KPJB-0953-15-WI-05  
**Dataset:** `hph_inference_test_set_100.jsonl` (100 cases — 50 High, 30 Medium, 20 Low risk)  
**Hardware:** Local inference via Ollama (`http://localhost:11434`)  
**Date:** 2026-04-28 / 2026-04-29  
**Evaluated Models:** `qwen3:4b`, `qwen3:8b`, `gemma4:e2b`, `gemma4:e4b`

---

## 1. Performance Summary

| Model | Cases | Avg Speed (t/s) | Min Speed | Max Speed | Avg Time/Case | Total Tokens | Avg Tokens/Case |
|---|---|---|---|---|---|---|---|
| **gemma4:e2b** | 100 | **78.6** | 53.7 | 91.2 | **18.2 s** | 126,262 | 1,263 |
| qwen3:4b | 100 | 59.9 | 43.2 | 66.7 | 49.8 s | 287,191 | 2,872 |
| qwen3:8b | 100 | 41.7 | 39.3 | 44.6 | 32.6 s | 132,632 | 1,326 |
| gemma4:e4b | 100 | 27.8 | 26.5 | 30.5 | 61.9 s | 166,522 | 1,665 |

> **Headline finding:** `gemma4:e2b` is the decisive throughput winner at **78.6 t/s** — 31% faster than `qwen3:4b` and nearly **3× faster than `gemma4:e4b`**. More importantly, it achieves this speed with only **1,263 tokens/case** (the most concise model) and **18.2 s/case** average wall-clock time. In a real-time operator advisory context, completing a full RCFA analysis in under 20 seconds is a strong operational advantage.

> `qwen3:4b` appears fast at 59.9 t/s but generates 2.27× more tokens than `gemma4:e2b`, making it **2.7× slower per case** at the wall-clock level.

---

## 2. SOP Compliance (KPJB-0953-15-WI-05)

All five required RCFA sections evaluated: **Gejala**, **Verifikasi Lapangan**, **Akar Masalah**, **Dampak Sistem**, **Rekomendasi FDT**.

| Model | 5/5 Cases | SOP Avg | Gejala | Verifikasi | Akar Masalah | Dampak | FDT |
|---|---|---|---|---|---|---|---|
| qwen3:4b | **91/100 (91%)** | 4.68/5 | 92 | 93 | 92 | 91 | 100 |
| **qwen3:8b** | **100/100 (100%)** | **5.00/5** | 100 | 100 | 100 | 100 | 100 |
| **gemma4:e2b** | **100/100 (100%)** | **5.00/5** | 100 | 100 | 100 | 100 | 100 |
| **gemma4:e4b** | **100/100 (100%)** | **5.00/5** | 100 | 100 | 100 | 100 | 100 |

### ⚠️ Critical Finding: qwen3:4b SOP Regression at 100-Query Scale

`qwen3:4b` achieved 100% SOP compliance in the 50-query test but **dropped to 91%** at 100 queries. Nine cases failed to include all five mandatory sections:

| Case | Tag | Risk Level | Missing Sections |
|---|---|---|---|
| #11 | 3FW-H070 | Low | Gejala, Verifikasi, Akar Masalah, Dampak |
| #12 | 3FW-H060 | **High** | Gejala, Verifikasi, Akar Masalah, Dampak |
| #24 | 3FW-H070 | **High** | Gejala, Verifikasi, Akar Masalah, Dampak |
| #40 | 3FW-H050 | **High** | Gejala, Verifikasi, Akar Masalah, Dampak |
| #57 | 3FW-H060 | **High** | Gejala, Verifikasi, Akar Masalah, Dampak |
| #61 | 3FW-H050 | Medium | Gejala, Verifikasi, Akar Masalah, Dampak |
| #89 | 3FW-H050 | Medium | Gejala, Verifikasi, Akar Masalah, Dampak |
| #96 | 3FW-H070 | Low | Gejala, Akar Masalah, Dampak |
| #99 | 3FW-H060 | Medium | Dampak only |

> **Safety Alert:** 4 of the 9 SOP failures occurred on **High-risk cases**. The pattern of missing *Gejala, Verifikasi, Akar Masalah, and Dampak* simultaneously suggests `qwen3:4b` occasionally abandons the structured 5-section RCFA format entirely and reverts to a prose-style response. In a real RCFA audit, any response missing `Akar Masalah` (Root Cause) or `Dampak Sistem` on a High-risk asset is a disqualifying failure.

> **Note:** `qwen3:8b`, `gemma4:e2b`, and `gemma4:e4b` all achieved **zero SOP failures across all 100 cases and all three risk levels**.

---

## 3. Technical Terminology Density

Average count of recognized TJB engineering acronyms per response (HHWL, BFP, DCV, DCA, HPH, RCFA, ECT, DCS, PLC, TTD, PAUT, NDT, PID, FDT, NWL, SSC, MOV, FMEA).

| Model | Avg Domain Terms / Response | Delta vs. qwen3:4b |
|---|---|---|
| qwen3:4b | 3.67 | baseline |
| qwen3:8b | 4.34 | +18% |
| gemma4:e2b | 4.88 | +33% |
| **gemma4:e4b** | **5.08** | **+38%** |

> **Takeaway:** `gemma4:e4b` retains the highest terminology density (5.08). Notably, `gemma4:e2b` (4.88) achieves near-equivalent domain precision to the much larger `e4b` variant, while being 4.3× faster. `qwen3:4b` remains last in terminology density despite generating more than twice as many tokens as any other model — confirming that its extra output is generic prose filler.

---

## 4. Tag Integrity (Equipment Tag Cross-Contamination)

Checks whether a response accidentally references the **wrong equipment tag** (e.g., mentioning `3FW-H060` in a case about `3FW-H050`).

| Model | Tag Mismatches | Pass Rate |
|---|---|---|
| qwen3:4b | 0 | **100%** |
| qwen3:8b | 0 | **100%** |
| **gemma4:e2b** | **0** | **100%** |
| gemma4:e4b | 0 | **100%** |

> All four models achieved perfect tag integrity across all 100 cases. The prompt's tag-anchoring mechanism is working reliably at this scale for all model families.

---

## 5. Verbosity vs. Efficiency

| Model | Avg Tokens/Case | Relative Verbosity |
|---|---|---|
| **gemma4:e2b** | **1,263** | most concise (baseline) |
| qwen3:8b | 1,326 | +5% |
| gemma4:e4b | 1,665 | +32% |
| qwen3:4b | 2,872 | **+127%** |

> `gemma4:e2b` is now the most token-efficient model overall — narrowly edging `qwen3:8b` by 5%. Both Gemma and Qwen-8b occupy a similarly efficient tier, while `gemma4:e4b` trades additional tokens for richer vocabulary. Running the full 672-case dataset with `qwen3:4b` would generate ~1.93M tokens versus `gemma4:e2b`'s projected ~849K — a 2.3× overhead for demonstrably worse SOP compliance.

---

## 6. Speed vs. Efficiency Trade-off Matrix

```
Throughput (t/s)
    79 |  [gemma4:e2b]  ← fastest clock + wall-clock, 100% SOP, most concise
       |
    60 |      [qwen3:4b]  ← fast clock, slow wall-clock, SOP regression
       |
    42 |           [qwen3:8b]  ← balanced speed, 100% SOP
       |
    28 |                  [gemma4:e4b]  ← richest terminology, slowest
       +------------------------------------------------→ Avg Tokens / Response
           1,263         1,326         1,665           2,872
```

- **Best overall (speed + SOP + conciseness):** `gemma4:e2b` — 18.2 s/case, 100% SOP, zero tag errors
- **Best balance for batch runs:** `qwen3:8b` — consistent 32.6 s/case, rock-solid reliability
- **Best terminology quality:** `gemma4:e4b` — hardware-constrained; most precise phrasing
- **Disqualified for production use:** `qwen3:4b` — 9% SOP failure rate on structured reports at scale

---

## 7. Risk-Stratified SOP Compliance

| Risk Level | Cases | qwen3:4b | qwen3:8b | gemma4:e2b | gemma4:e4b |
|---|---|---|---|---|---|
| **High** | 50 | 46/50 (**8% failure**) | 50/50 ✅ | 50/50 ✅ | 50/50 ✅ |
| **Medium** | 30 | 27/30 (**10% failure**) | 30/30 ✅ | 30/30 ✅ | 30/30 ✅ |
| **Low** | 20 | 18/20 (**10% failure**) | 20/20 ✅ | 20/20 ✅ | 20/20 ✅ |

> `qwen3:4b`'s failure rate is consistent across all risk levels — it does **not** apply additional structural discipline on High-risk cases. `qwen3:8b`, `gemma4:e2b`, and `gemma4:e4b` all scored **zero failures across every risk stratum**.

---

## 8. Model Recommendations by Use Case

| Use Case | Recommended Model | Rationale |
|---|---|---|
| **Real-time operator advisory** | **gemma4:e2b** | Fastest wall-clock (18.2 s), 100% SOP, most concise, zero errors |
| **Offline batch annotation / RLHF data** | **gemma4:e2b** | Best throughput for volume runs; structured output at scale |
| **High-quality RCFA report drafting** | **gemma4:e4b** | Richest domain vocabulary (5.08 terms/resp); 100% SOP |
| **SFT/DPO chosen responses** | **gemma4:e4b** | Highest terminology precision; ideal for quality preference pairs |
| **Reliable fallback / balanced option** | **qwen3:8b** | Proven stability, 100% SOP, predictable latency |
| ⛔ **Production use** | ~~qwen3:4b~~ | **Disqualified** — 9% SOP failure rate including High-risk cases |

---

## 9. Key Takeaways

1. **`gemma4:e2b` is the surprise leader.** At 78.6 t/s and 18.2 s/case, it is the fastest model by a large margin while maintaining 100% SOP compliance and the second-highest terminology density. It achieves more with fewer tokens than any competitor.

2. **`qwen3:4b` SOP compliance is not stable at scale.** The 9% failure rate exposed at 100 queries was invisible at 50. Its failures concentrate on complete format collapse — skipping 4 of 5 mandatory RCFA sections while always retaining the FDT recommendation. A recommendation without root cause analysis is operationally dangerous.

3. **`gemma4:e4b` is hardware-constrained, not model-constrained.** Its 100% SOP compliance and leading terminology density (5.08) confirm highest qualitative output. The 61.9 s/case latency is a local hardware limitation.

4. **All four models passed tag integrity at 100 queries.** The `qwen3:8b` mismatch from the 50-query run did not recur — confirming it was a stochastic edge case.

5. **The Gemma model family dominates on efficiency.** Both `gemma4:e2b` and `gemma4:e4b` outperform Qwen models on token efficiency and domain terminology density. The e2b variant in particular offers an exceptional quality-to-latency ratio for operational deployment.

---

## 10. Next Steps

- [x] **`gemma4:e2b` 100-query results added** — 4-model comparison now complete
- [ ] **Root cause `qwen3:4b` failures** — inspect the 9 failing cases to determine if a specific FMEA failure mode or prompt variant triggers the format collapse
- [ ] Expand to 200+ cases with out-of-distribution failure modes (multi-equipment incidents, ambiguous indications, missing FMEA fields)
- [ ] Add semantic scoring (BERTScore / cosine similarity) against expert-written ground truth RCFA reports
- [ ] Run adversarial prompts to find SOP compliance failure thresholds for `qwen3:8b`, `gemma4:e2b`, and `gemma4:e4b`
- [ ] Re-benchmark `gemma4:e4b` on upgraded hardware (expected significant throughput improvement)
- [ ] Use the `⭐ Best Response` RLHF export in `inference_results_viewer.html` to collect preference labels for DPO fine-tuning

---

*Generated by the RCFA Benchmarking Pipeline — `ollama_benchmark.py` + `inference_results_viewer.html`*  
*Analysis script: `analyze_100.py`*