iRCFA-data/benchmark/50_queries/BENCHMARK.md

# RCFA LLM Benchmark Report
**Procedure Reference:** KPJB-0953-15-WI-05  
**Dataset:** `hph_inference_test_set_50.jsonl` (50 cases — 25 High, 15 Medium, 10 Low risk)  
**Hardware:** Local inference via Ollama (`http://localhost:11434`)  
**Date:** 2026-04-28  
**Evaluated Models:** `qwen3:4b`, `qwen3:8b`, `gemma4:e2b`, `gemma4:e4b`

---

## 1. Performance Summary

| Model | Cases | Avg Speed (t/s) | Min Speed | Max Speed | Avg Time/Case | Total Tokens | Avg Tokens/Case |
|---|---|---|---|---|---|---|---|
| **qwen3:4b** | 50 | **59.4** | 55.5 | 65.2 | 51.2 s | 148,448 | 2,969 |
| qwen3:8b | 50 | 41.0 | 40.1 | 43.8 | 34.3 s | 68,320 | 1,366 |
| gemma4:e2b | 50 | 41.6 | 39.5 | 43.7 | 31.8 s | 62,526 | 1,251 |
| gemma4:e4b | 50 | 28.3 | 27.2 | 29.7 | **60.2 s** | 82,364 | 1,647 |

> **Note:** `qwen3:4b` is the fastest model by a significant margin at **59.4 t/s**, nearly 45% faster than the Gemma variants. However, it generates ~2.4× more tokens per case than `gemma4:e2b`, which drives up total latency despite the higher token throughput.

---

## 2. SOP Compliance (KPJB-0953-15-WI-05)

All five required RCFA sections evaluated: **Gejala**, **Verifikasi Lapangan**, **Akar Masalah**, **Dampak Sistem**, **Rekomendasi FDT**.

| Model | 5/5 Cases | SOP Avg | Gejala | Verifikasi | Akar Masalah | Dampak | FDT |
|---|---|---|---|---|---|---|---|
| qwen3:4b | **50/50 (100%)** | 5.00/5 | 50 | 50 | 50 | 50 | 50 |
| qwen3:8b | **50/50 (100%)** | 5.00/5 | 50 | 50 | 50 | 50 | 50 |
| gemma4:e2b | **50/50 (100%)** | 5.00/5 | 50 | 50 | 50 | 50 | 50 |
| gemma4:e4b | **50/50 (100%)** | 5.00/5 | 50 | 50 | 50 | 50 | 50 |

> **Important:** All four models achieved **perfect SOP compliance (100%)** across all 50 test cases. This confirms the prompt template is effective and all models can reliably follow the RCFA SOP structure on this domain. SOP adherence is therefore **not a differentiator** in this benchmark — harder adversarial prompts are needed to stress-test compliance.

---

## 3. Technical Terminology Density

Average count of recognized TJB engineering acronyms per response (HHWL, BFP, DCV, DCA, HPH, RCFA, ECT, DCS, PLC, TTD, PAUT, NDT, PID, FDT, NWL, SSC, MOV).

| Model | Avg Domain Terms / Response | Delta vs. qwen3:4b |
|---|---|---|
| qwen3:4b | 5.2 | baseline |
| qwen3:8b | 7.8 | +50% |
| gemma4:e2b | 6.8 | +31% |
| **gemma4:e4b** | **8.3** | **+60%** |

**Takeaway:** `gemma4:e4b` uses the richest engineering vocabulary despite generating fewer total tokens than `qwen3:4b`. `qwen3:4b` writes longer, more prose-heavy explanations that dilute the domain signal — its low terminology density relative to its token count suggests more generic/padded language.

---

## 4. Tag Integrity (Equipment Tag Cross-Contamination)

Checks whether a response accidentally references the **wrong equipment tag** (e.g., mentioning `3FW-H060` in a case about `3FW-H050`).

| Model | Tag Mismatches | Pass Rate |
|---|---|---|
| qwen3:4b | 0 | **100%** |
| **qwen3:8b** | **1** | 98% |
| gemma4:e2b | 0 | **100%** |
| gemma4:e4b | 0 | **100%** |

> **Warning:** `qwen3:8b` produced **1 tag mismatch** across 50 cases. In a real plant maintenance context, an incorrect equipment tag in an RCFA report could direct a crew to the wrong asset — a meaningful safety risk. This should be monitored closely at larger dataset scales.

---

## 5. Verbosity vs. Efficiency

| Model | Avg Tokens/Case | Relative Verbosity |
|---|---|---|
| **gemma4:e2b** | **1,251** | most concise (baseline) |
| qwen3:8b | 1,366 | +9% |
| gemma4:e4b | 1,647 | +32% |
| qwen3:4b | 2,969 | **+137%** |

`qwen3:4b` generates **2.4× more tokens** than `gemma4:e2b` for the **same SOP compliance score**. The extra tokens consist of preambles, repetition, and hedging language that add no measurable structure. Verbosity should be treated as a cost, not a quality signal.

---

## 6. Speed vs. Efficiency Trade-off Matrix

```
Throughput (t/s)
    60 |  [qwen3:4b]  ← fastest clock speed, most tokens
       |
    42 |      [gemma4:e2b]  [qwen3:8b]  ← fastest wall-clock / case
       |
    28 |                         [gemma4:e4b]  ← richest terminology, slowest
       +------------------------------------------------→ Avg Tokens / Response
          1,250          1,650             3,000
```

- **Best for real-time / low latency:** `gemma4:e2b` — 31.8 s/case avg, zero errors
- **Best raw throughput (t/s):** `qwen3:4b` — but penalized by verbosity
- **Best terminology quality:** `gemma4:e4b` — limited by hardware, not model quality
- **Best balance:** `qwen3:8b` — concise, fast, high domain density (minor tag caveat)

---

## 7. Model Recommendations by Use Case

| Use Case | Recommended Model | Rationale |
|---|---|---|
| Real-time operator advisory | **gemma4:e2b** | Lowest avg time (31.8 s), 100% SOP, zero tag errors |
| Offline batch annotation / RLHF data generation | **qwen3:4b** | Highest raw throughput; verbose outputs good for preference labeling |
| High-quality RCFA report drafting | **gemma4:e4b** | Richest domain vocabulary density; most precise phrasing |
| SFT/DPO training data | **qwen3:8b** | Balanced length + terminology; only minor safety flag |

---

## 8. Key Takeaways

1. **SOP compliance is fully saturated** — all models score 100% on the 5-section check. The prompt engineering is the primary driver. To truly differentiate models, future benchmarks need **adversarial prompts** and **expert-authored ground truth** for semantic scoring (BLEU, BERTScore, or domain-expert review).

2. **Gemma models are more token-efficient.** Both Gemma variants achieve the same SOP coverage with ~55% fewer tokens than qwen3:4b. This matters enormously at inference cost for large-scale annotation runs.

3. **qwen3:4b verbosity is a hidden cost.** Despite having the highest t/s, running 500 cases with qwen3:4b (~7.1 hours) costs ~60% more compute time than gemma4:e2b (~4.4 hours) solely due to token volume.

4. **gemma4:e4b is hardware-constrained, not model-constrained.** Its 28.3 t/s and 60.2 s/case latency suggest memory-bandwidth saturation on the current machine. This model would likely outperform all others on better hardware given its terminology density.

5. **qwen3:8b has a latent tag-hallucination risk.** One cross-equipment tag reference in 50 cases is small but not negligible in safety-critical plant contexts. Monitor carefully and consider tag-anchoring instructions in the prompt.

6. **No model exhibited a "weak section" in SOP.** All sections (including Verifikasi Lapangan, historically the hardest to elicit) are consistently present. This suggests the FMEA context injection in the prompt is sufficient scaffolding for all models.

---

## 9. Next Steps

- [ ] Expand to 200+ cases with out-of-distribution failure modes (multi-equipment incidents, ambiguous indications, missing FMEA fields)
- [ ] Add semantic scoring (BERTScore / cosine similarity) against expert-written ground truth RCFA reports
- [ ] Run adversarial prompts to find SOP compliance failure thresholds for each model
- [ ] Use the `⭐ Best Response` RLHF export in `inference_results_viewer.html` to collect preference labels for DPO fine-tuning
- [ ] Re-benchmark `gemma4:e4b` on upgraded hardware (expected significant throughput improvement)
- [ ] Evaluate `num_ctx` sensitivity: test 4096 vs. 8192 and measure whether longer context improves FDT specificity

---

*Generated by the RCFA Benchmarking Pipeline — `ollama_benchmark.py` + `inference_results_viewer.html`*  
*Analysis script: `analyze_bench.py`*