# Multimodal LLM Improvement Analysis — stream2.py
## iRCFA Application: Tanjung Jati B Unit 3 Power Plant

**Date:** 2026-06-17  
**Author:** aimo (AI-Assisted Analysis)  
**Scope:** `src/ircfa_interaction/stream2.py` and related multimodal data pipelines  
**Target:** Maximum potential extraction for Multi-modal LLM in RCFA workflow  

---

## TL;DR

**Current State:** stream2.py already supports basic multimodal — images go to Gemma 4 via base64 encoding, documents go through Docling OCR → text extraction → RAG vector store. However, the system treats all images equally and all documents as flat text, losing critical visual information from P&ID diagrams, thermal images, vibration plots, and equipment photos that are vital for RCFA at a coal-fired power plant.

**Key Opportunities (Ranked by Impact):**

1. **P&ID Diagram Understanding** — Currently treated as flat text via OCR. A multimodal LLM can read valve tags, trace piping, identify flow direction, and locate equipment in the plant layout. This is the single highest-value improvement.
2. **Equipment Photo Analysis** — Failure evidence photos (corrosion, cracks, leaks) are passed to the LLM without preprocessing or structured prompting. Adding visual inspection context and failure taxonomy alignment would dramatically improve root cause identification.
3. **Table & Chart Extraction from PDFs** — Maintenance reports, vibration spectra, and thermal scans contain tabular/visual data that Docling flattens into text. Native multimodal processing preserves structure.
4. **Document-Type Classification** — All attachments processed identically. Classifying documents (P&ID, maintenance report, FMEA sheet, equipment photo) before processing allows type-specific prompt engineering and tool routing.
5. **Image Preprocessing Pipeline** — No resize, enhancement, or quality normalization before base64 encoding. Scanned field photos with poor lighting or blurry text degrade LLM performance.
6. **Cross-Modal Context Fusion** — Images and tool data (sensor readings, reliability metrics) are processed in isolation. Fusing visual evidence with quantitative data in a single prompt gives the LLM richer context for root cause analysis.

**Estimated Impact:** These improvements are expected to reduce repeated analysis turns and improve evidence grounding for visual-evidence-dependent failure modes (corrosion, erosion, thermal damage, mechanical wear). Exact gains in RCFA accuracy and response time must be validated through benchmark cases before being used as delivery targets.

---

## 1. Current Architecture Analysis

### 1.1 Image Processing Flow (stream2.py lines 401-424)

```
User Upload → stream2.py
  ├── Image (.png/.jpg/.jpeg/.gif/.webp)
  │   └── saved_images[] → GemmaLlamaCppLLM.current_attachments
  │       └── _build_payload() → base64 encode → OpenAI image_url format
  │           └── Injected into LAST user message as multimodal content
  │
  └── Document (.pdf/.doc/.txt)
      └── saved_docs[] → create_docling_rag()
          ├── Docling DocumentConverter (primary) → Markdown
          └── pypdf fallback → Plain text
              └── accumulated_docs → text injection into user message
```

### 1.2 GemmaLlamaCppLLM Multimodal Support (gemma_llamacpp.py lines 332-370)

- Encodes up to **3 images** as base64 data URIs
- Injects as `image_url` parts into the **last user message** (OpenAI multimodal format)
- Graceful fallback: if server returns "image input not supported", retries without images
- No image preprocessing (resize, compress, quality check)
- No image metadata extraction (EXIF, GPS, timestamp)

### 1.3 Document Processing (stream2.py lines 164-226)

Three fallback levels:
1. **Docling** (best) — OCR, table extraction, layout preservation → Markdown
2. **Unstructured.io** — OCR-only strategy with chunking by title
3. **pypdf** (fallback) — Basic text extraction, no OCR

### 1.4 Tool Ecosystem (lines 488-500)

Available tools that provide quantitative context:
- `RCFATool` — Equipment predicted to fail in 7 days
- `CurrentReliabilityTool` — Reliability metrics (MTTR, MTBF, failure rate)
- `CurrentSensorTool` — Real-time sensor readings
- `KondisiSensorHistorisTool` — Historical sensor data
- `NextFailureRPTool/PFITool` — Predicted failure timing
- `RootCauseSPLNTool` — SPLN root cause database + FMEA records
- `StandardMetricsTool` — Equipment standard metrics
- `PlantStatusTool` — Plant running/shutdown status

**Critical Gap:** None of these tools receive visual context from attached images. The LLM sees images and tool data in separate contexts.

---

## 2. Identified Improvement Areas

### 2.1 [HIGH IMPACT] P&ID Diagram Understanding

**Current Problem:** When a user uploads a P&ID (Piping & Instrumentation Diagram), it goes through Docling OCR which converts it to flat text. The visual structure — pipe routing, valve positions, sensor locations, instrument tags — is completely lost.

**Tanjung Jati B Context:** Unit 3 has thousands of P&ID sheets covering:
- Boiler water/steam circuit
- Feedwater heating system (HPH/LPH)
- Condenser and cooling water system
- Turbine lube oil system
- Coal handling and ash handling
- DCS/instrumentation loops

**Proposed Solution:**

```python
# NEW: Document classifier (pre-processing step)
DOCUMENT_TYPES = {
    "pid": {
        "keywords": ["P&ID", "Piping and Instrumentation", "Process Flow", "Line List"],
        "processing": "multimodal_vision",  # Send as image to LLM
        "prompt_prefix": "Analisis diagram P&ID ini. Identifikasi: (1) equipment yang ditunjuk, (2) valve dan instrumen, (3) aliran fluida, (4) lokasi sensor. Fokus pada location tag yang relevan."
    },
    "equipment_photo": {
        "keywords": [],  # Detected by image metadata or user description
        "processing": "multimodal_vision",
        "prompt_prefix": "Analisis foto equipment ini untuk RCFA. Identifikasi: (1) jenis kerusakan yang terlihat, (2) lokasi kerusakan, (3) severity estimasi, (4) kemungkinan failure mode berdasarkan penampakan."
    },
    "maintenance_report": {
        "keywords": ["Maintenance", "Inspection", "Work Order", "PM Report"],
        "processing": "text_extraction_enhanced",  # Docling with table preservation
        "prompt_prefix": "Ekstrak data maintenance dari dokumen ini: equipment, failure history, tindakan yang dilakukan, dan rekomendasi."
    },
    "vibration_data": {
        "keywords": ["Vibration", "Spectrum", "FFT", "Acceleration"],
        "processing": "multimodal_vision",  # Charts need visual analysis
        "prompt_prefix": "Analisis plot vibrasi ini. Identifikasi: (1) dominan frequency, (2) defect signature (imbalance, misalignment, bearing defect), (3) severity vs threshold."
    }
}
```

### 2.2 [HIGH IMPACT] Equipment Failure Photo Analysis

**Current Problem:** Photos of equipment damage (corrosion, cracks, leaks, burn marks) are sent to Gemma 4 with generic instructions. The system prompt only says "fokuskan analisis utama pada gambar/dokumen terbaru ini" (line 693).

**Proposed Solution — Failure Taxonomy-Aligned Prompting:**

```python
FAILURE_VISUAL_TAXONOMY = {
    "corrosion": {
        "visual_signs": ["rust color", "pitting", "uniform thinning", "crevice patterns"],
        "indicators": ["scale buildup", "color change", "surface roughness"],
        "tjb_relevant_areas": ["boiler tubes", "condenser tubes", "feedwater piping", "cooling water system"]
    },
    "erosion": {
        "visual_signs": ["material loss", "smooth worn surfaces", "directional grooves"],
        "indicators": ["wall thinning patterns", "flow-aligned damage"],
        "tjb_relevant_areas": ["coal pipes", "ash handling", "soot blower areas", "economizer tubes"]
    },
    "thermal_damage": {
        "visual_signs": ["discoloration", "warping", "scaling", "blistering"],
        "indicators": ["heat tint colors", "deformation patterns"],
        "tjb_relevant_areas": ["superheater", "reheater", "burner zone", "turbine casing"]
    },
    "mechanical_wear": {
        "visual_signs": ["scratch marks", "grooving", "brinelling", "spalling"],
        "indicators": ["bearing surface damage", "seal wear patterns"],
        "tjb_relevant_areas": ["pump bearings", "turbine bearings", "gearbox", "coupling"]
    },
    "leakage": {
        "visual_signs": ["wet spots", "staining", "mineral deposits", "steam plumes"],
        "indicators": ["joint failures", "gasket degradation", "tube rupture"],
        "tjb_relevant_areas": ["heat exchanger joints", "flange connections", "valve stems"]
    }
}
```

### 2.3 [MEDIUM IMPACT] Image Preprocessing Pipeline

**Current Problem:** Raw images from field technicians are sent directly as base64 without any preprocessing. Field photos often have:
- Poor lighting (dark boiler areas, backlit photos)
- Blurry/damaged text on nameplates and tags
- Very large file sizes (20+ MB phone photos)
- Rotation issues (EXIF orientation not always honored)

**Proposed Addition (new file: `src/ircfa_interaction/lib/image_preprocessor.py`):**

```python
class ImagePreprocessor:
    MAX_DIMENSION = 2048  # Max width/height in pixels
    JPEG_QUALITY = 85
    TARGET_FORMAT = "JPEG"
    
    @staticmethod
    def prepare_for_llm(image_path: str) -> dict:
        """
        Prepare an image for multimodal LLM consumption.
        Returns: {"path": processed_path, "metadata": {...}, "classification": "..."}
        """
        from PIL import Image, ExifTags
        import hashlib
        
        img = Image.open(image_path)
        
        # 1. Auto-rotate based on EXIF
        img = ImagePreprocessor._auto_rotate(img)
        
        # 2. Resize if too large (preserving aspect ratio)
        w, h = img.size
        if max(w, h) > ImagePreprocessor.MAX_DIMENSION:
            ratio = ImagePreprocessor.MAX_DIMENSION / max(w, h)
            img = img.resize((int(w*ratio), int(h*ratio)), Image.LANCZOS)
        
        # 3. Enhance contrast for dark photos
        img = ImagePreprocessor._enhance_if_dark(img)
        
        # 4. Save as optimized JPEG
        output_path = image_path + ".processed.jpg"
        img.save(output_path, "JPEG", quality=ImagePreprocessor.JPEG_QUALITY)
        
        # 5. Extract metadata
        metadata = {
            "original_size": os.path.getsize(image_path),
            "processed_size": os.path.getsize(output_path),
            "dimensions": img.size,
            "content_hash": hashlib.md5(open(image_path, 'rb').read()).hexdigest()
        }
        
        # 6. Classify image type
        classification = ImagePreprocessor._classify_image(img, metadata)
        
        return {"path": output_path, "metadata": metadata, "classification": classification}
    
    @staticmethod
    def _enhance_if_dark(img):
        """Auto-brighten dark images (common in plant interiors)."""
        import numpy as np
        arr = np.array(img)
        mean_brightness = arr.mean()
        if mean_brightness < 80:  # Dark threshold
            from PIL import ImageEnhance
            factor = min(2.0, 120.0 / mean_brightness)
            return ImageEnhance.Brightness(img).enhance(factor)
        return img
```

### 2.4 [MEDIUM IMPACT] Document-Type-Aware Processing

**Current Problem:** All documents go through the same `create_docling_rag()` pipeline regardless of type.

**Proposed Enhancement:**

```python
def process_attachment_smart(file_path: str, user_context: str = "") -> dict:
    """
    Smart document processing with type classification.
    Returns structured data for multimodal or text pipelines.
    """
    suffix = os.path.splitext(file_path)[1].lower()
    
    if suffix in [".png", ".jpg", ".jpeg", ".gif", ".webp"]:
        return _process_image(file_path, user_context)
    elif suffix == ".pdf":
        return _process_pdf_smart(file_path, user_context)
    elif suffix in [".xlsx", ".xls", ".csv"]:
        return _process_tabular(file_path, user_context)
    else:
        return {"type": "text", "content": create_docling_rag(file_path), "pipeline": "text"}

def _process_pdf_smart(pdf_path: str, user_context: str) -> dict:
    """Classify PDF content and choose optimal processing pipeline."""
    # Quick classification via first page content
    from pypdf import PdfReader
    reader = PdfReader(pdf_path)
    first_page_text = reader.pages[0].extract_text()[:2000] if reader.pages else ""
    
    # Check for P&ID characteristics
    pid_indicators = ["P&ID", "Piping", "Instrumentation", "Line No", "Tag No", "P&I"]
    if any(ind.lower() in first_page_text.lower() for ind in pid_indicators):
        return {
            "type": "pid_diagram",
            "content": create_docling_rag(pdf_path),  # Text extraction for context
            "image_path": pdf_path,  # Also pass PDF for multimodal analysis
            "pipeline": "multimodal",
            "prompt_context": "Dokumen ini adalah P&ID (Piping & Instrumentation Diagram). Analisis struktur visual diagram."
        }
    
    # Check for maintenance/inspection report
    maintenance_indicators = ["Maintenance", "Inspection", "Work Order", "Corrective", "Preventive"]
    if any(ind.lower() in first_page_text.lower() for ind in maintenance_indicators):
        return {
            "type": "maintenance_report",
            "content": create_docling_rag(pdf_path),
            "pipeline": "text_enhanced",
            "prompt_context": "Dokumen ini adalah laporan maintenance. Ekstrak data equipment, failure history, dan tindakan."
        }
    
    # Default: standard RAG processing
    return {
        "type": "generic_document",
        "content": create_docling_rag(pdf_path),
        "pipeline": "text"
    }
```

### 2.5 [MEDIUM IMPACT] Cross-Modal Context Fusion

**Current Problem:** Images and tool data are processed in separate contexts. The LLM sees the image but doesn't have sensor data or reliability metrics overlaid.

**Proposed Solution — Enhanced Prompt Injection:**

```python
# In stream2.py, after tool results are collected and before final LLM call:

def build_multimodal_context(
    saved_images: list,
    function_results: list,
    tool_data: dict,
    location_tag: str
) -> str:
    """
    Build rich context that fuses visual evidence with quantitative data.
    """
    context_parts = []
    
    if saved_images:
        context_parts.append(
            "📷 **Bukti Visual yang Dilampirkan:**\n"
            "Gambar di atas menunjukkan kondisi equipment. Analisis berdasarkan:\n"
            "- Jenis kerusakan visual (korosi, erosi, retak, kebocoran, deformasi)\n"
            "- Lokasi kerusakan relatif terhadap equipment\n"
            "- Severity estimasi (ringan/sedang/berat)\n"
            "- Korelasi dengan data sensor dan reliability di bawah"
        )
    
    if function_results:
        sensor_data = next((r for r in function_results if r["name"] == "get_current_sensor"), None)
        reliability_data = next((r for r in function_results if r["name"] == "get_current_reliability"), None)
        
        if sensor_data:
            context_parts.append(f"\n📊 **Data Sensor Terkini ({location_tag}):**\n{sensor_data['content'][:1500]}")
        if reliability_data:
            context_parts.append(f"\n📈 **Data Reliability ({location_tag}):**\n{reliability_data['content'][:1500]}")
    
    return "\n---\n".join(context_parts)
```

### 2.6 [LOW-MEDIUM IMPACT] Attachment Limit & Token Budget

**Current Limits:**
- Max 3 attachments (line 403)
- MAX_DOC_TOKENS = 32,764 (config.py line 12)
- No separate token budget for images vs text

**Proposed Changes:**

```python
# config.py additions
MAX_ATTACHMENTS = 5  # Increased from 3
MAX_IMAGE_TOKENS = 4096  # Separate budget for image content
MAX_TEXT_DOC_TOKENS = 28672  # Remaining for text documents

# stream2.py — smarter attachment management
MAX_ATTACHMENTS = 5
if len(request.attachments) > MAX_ATTACHMENTS:
    # Prioritize: images first (visual evidence), then by file size (larger = more content)
    sorted_attachments = _prioritize_attachments(request.attachments[:8])
    attachments_to_process = sorted_attachments[:MAX_ATTACHMENTS]
```

### 2.7 [LOW IMPACT] Minor Code Quality Issues

| Location | Issue | Fix |
|---|---|---|
| Line 397 | `print(request.attachments)` — debug statement in production | Remove or convert to `logger.debug()` |
| Line 436 | `UploadedFile` referenced but not imported | Add `from src.rag.model import UploadedFile` or guard with try/except |
| Line 357 | `print("Extracted location tag...")` — should use logger | Change to `logger.info()` |
| Line 641 | `print(len(message), ...)` — production noise | Convert to `logger.debug()` |
| Line 784 | `print("Files passed to Assistant:", ...)` — production noise | Convert to `logger.debug()` |
| Line 789 | `print("Streaming started at:", ...)` — production noise | Convert to `logger.debug()` |
| Lines 46-52 | Duplicate imports (`ChatPromptTemplate` imported twice, lines 25 and 66) | Remove duplicate |
| Lines 8, 74-75 | Duplicate imports (`requests`, `json`, `asyncio` imported multiple times) | Deduplicate |
| Line 230 | `equipment_keywords` list defined inline and also in schema.py | Import from schema.py to avoid maintenance burden |

---

## 3. Tanjung Jati B Unit 3 — Specific Data Opportunities

### 3.1 Power Plant Document Types Available for Multimodal

| Document Type | Source | Current Processing | Multimodal Opportunity |
|---|---|---|---|
| **P&ID Diagrams** | Engineering dept | Text OCR (loses structure) | P&ID-assisted tag, equipment, and relationship extraction requiring human verification |
| **Equipment Photos** | Field technicians | Generic image → LLM | Failure taxonomy-aligned visual inspection |
| **Thermal Images** | IR camera surveys | Not processed | Temperature gradient analysis, hot spot identification |
| **Vibration Spectra** | Online/offline monitoring | Text summary only | FFT pattern recognition, defect signature matching |
| **Maintenance Reports** | CMMS/Maximo | Docling text extraction | Table extraction for failure history, MTTR trends |
| **FMEA Sheets** | Reliability dept | Database query only | Cross-reference with visual evidence |
| **Boiler Tube Maps** | Inspection team | Not processed | Map failure locations, identify patterns |
| **Turbine Cross-Sections** | OEM manuals | Not processed | Structural analysis reference |

### 3.2 Equipment Categories with Highest Multimodal Value

Ranked by frequency of visual-evidence-dependent RCFA:

1. **Boiler Tubes** (Superheater, Reheater, Economizer, Waterwall)
   - Failure modes: creep, erosion, corrosion, hydrogen damage
   - Visual evidence: tube color, deformation, wall thinning patterns
   
2. **Heat Exchangers** (HPH, LPH, Condenser)
   - Failure modes: tube leakage, fouling, baffle erosion
   - Visual evidence: tube-end condition, scaling patterns, leak staining

3. **Turbine Components** (HP/IP/LP sections)
   - Failure modes: blade erosion, foreign object damage, creep
   - Visual evidence: blade surface condition, deposits, discoloration

4. **Pumps & Valves** (Feedwater, Cooling Water, Condensate)
   - Failure modes: impeller erosion, seal leakage, seat damage
   - Visual evidence: wear patterns, leakage paths, vibration evidence

5. **Coal & Ash Handling**
   - Failure modes: abrasive wear, blockage, corrosion
   - Visual evidence: wear groove patterns, buildup locations

### 3.3 Recommended Multimodal Use Cases per RCFA Phase

| RCFA Phase | Multimodal Input | Tool Integration |
|---|---|---|
| **1. Failure Prediction** | Equipment health photos, trend charts | NextFailurePFITool + image |
| **2. Sensor Data Mapping** | Vibration spectra, thermal images | CurrentSensorTool + visual overlay |
| **3. Potential Root Causes** | P&ID for pipe tracing, FMEA sheets | RootCauseSPLNTool + diagram |
| **4. Diagnosis** | Equipment damage photos, inspection reports | Cross-reference with sensor data |
| **5. Final Reporting** | All images embedded in report | Export with annotated images |

---

## 4. Revision Notes And Implementation Judgment

This revision tightens the original analysis so it can be used safely as an implementation reference for iRCFA on a specific coal power plant unit. The core direction remains valid: the app already has a basic multimodal path, but the highest-value improvement is not simply "send more images to the LLM." The highest-value improvement is structured evidence fusion across attachment content, unit context, location tag, plant data, reliability predictions, and RCFA phase.

### 4.1 Confirmed Current Behavior

- `stream2.py` supports basic multimodal image routing by separating image attachments from document attachments.
- `GemmaLlamaCppLLM` injects up to 3 image files as base64 `image_url` parts into the final OpenAI-style chat payload.
- Documents are currently reduced to extracted text or markdown before being appended to the reasoning context.
- Docling may preserve some table or layout information in markdown, but the current app does not route documents by document type or preserve visual structure as first-class RCFA evidence.
- Tool outputs and visual evidence are not explicitly fused into a structured RCFA evidence packet before final reasoning.

### 4.2 Corrections To Original Claims

- Accuracy improvement numbers such as `~70%` to `~90%+` should be treated as target hypotheses, not confirmed outcomes.
- Time reduction numbers such as `40-60%` should be treated as expected directionality until measured against real or representative RCFA cases.
- P&ID support should be framed as P&ID-assisted tag, equipment, and relationship extraction requiring human verification, not autonomous full structural analysis.
- Multimodal output must be treated as engineering decision support. It should help technicians and reliability engineers inspect evidence, compare hypotheses, and prepare RCFA reasoning, but it should not be treated as authoritative diagnosis without human review.

### 4.3 Revised Priority

The implementation priority should shift from generic image understanding to unit-bound evidence fusion:

1. Attachment classification: identify whether the upload is an equipment photo, P&ID, thermal image, vibration chart, DCS screenshot, maintenance report, FMEA sheet, or generic document.
2. Unit/location-tag binding: connect each attachment to an explicit unit, equipment, system, and location tag where available.
3. Structured RCFA evidence packet: combine image/document evidence with current sensor data, historical sensor data, reliability metrics, PFI/RP predictions, SPLN/FMEA records, and current RCFA phase.
4. Type-specific prompting: use different prompts for photos, diagrams, DCS screenshots, vibration plots, thermal images, and maintenance reports.
5. Advanced analytics: add P&ID page rendering, thermal interpretation, vibration spectrum recognition, and annotated reporting only after baseline validation.

### 4.4 Validation Requirement

Before claiming accuracy or time improvements, create a benchmark set of 20-50 real or representative Tanjung Jati B Unit 3 RCFA cases. Each case should include the expected failure mode, available visual/document evidence, relevant location tag, tool data snapshot, and accepted engineering conclusion.

Measure:

- Diagnosis quality and root-cause ranking.
- Evidence grounding and citation of observed facts.
- Hallucination rate, especially for visual details not present in attachments.
- Correctness of tool usage and location-tag binding.
- Time-to-useful-answer and number of turns required.

All benefit numbers in this document should be treated as hypotheses until this benchmark exists.

---

## 5. Implementation Roadmap

### Phase 1: Safety And Baseline Cleanup (1-2 weeks)
- [x] Fix debug print statements → logger
- [x] Remove duplicate imports
- [x] Import `UploadedFile` properly or add guard
- [x] Keep image limit at 3 initially unless benchmarking shows safe latency and memory headroom
- [x] Add lightweight attachment routing tests: image-only, document-only, mixed attachments, and over-limit attachments

### Phase 2: Attachment Classification (2-3 weeks)
- [x] Add a small attachment classifier module under `src/ircfa_interaction/lib/`
- [x] Classify attachments into `equipment_photo`, `pid_diagram`, `thermal_image`, `vibration_chart`, `dcs_screenshot`, `maintenance_report`, `fmea_sheet`, or `generic_document`
- [x] Use filename, extension, content type, first-page text, and user message hints
- [x] Return structured metadata: `attachment_type`, `processing_mode`, `confidence`, and `prompt_context`

### Phase 3: Unit/Tag-Bound Multimodal Context (3-4 weeks)
- [x] Build a multimodal context builder that produces a structured RCFA evidence packet
- [x] Include explicit unit, location tag, attachment summary, visual prompt context, relevant sensor/reliability/PFI/RP guidance, SPLN/FMEA guidance, and existing RCFA state evidence
- [x] Preserve the current safety behavior: only bind sensor and reliability tool calls to location tags explicitly provided by the user or frontend
- [x] Add type-specific prompt snippets for equipment photos, P&IDs, vibration charts, DCS screenshots, and maintenance reports

Phase 3 implementation note: the evidence packet is injected internally into the LLM message context and exposed in `context_info` for debugging; it does not change the frontend upload or streaming API.

### Phase 4: Validation And Advanced Analytics (4-6 weeks)
- [ ] Create a benchmark dataset from known or representative coal plant RCFA cases
- [ ] Add evaluation scripts for multimodal answers versus expected evidence and root-cause reasoning
- [ ] Only after validation, add thermal image interpretation, vibration spectrum pattern recognition, P&ID page rendering, boiler tube map visualization, and annotated final reports

### 5.1 Public Interfaces

- Add internal structured metadata for attachments; no frontend API change is required for the first implementation.
- Preserve existing `ChatRequest.attachments` behavior for compatibility with current upload flows.
- Optional future frontend enhancement: allow users to provide or confirm `unit`, `system`, `equipment`, and `location_tag` per attachment.

### 5.2 Test Plan

- Unit test attachment classification for representative filenames, extensions, content types, first-page text, and user message hints.
- Unit test image/document routing so current Gemma image payload behavior does not regress.
- Unit test document text extraction fallback so Docling and pypdf behavior remains compatible.
- Integration test a mixed request with one equipment image, one report PDF, and an explicit location tag.
- Regression test that inherited historical tags are not auto-injected into sensor or reliability tools.
- Benchmark test against 20-50 curated RCFA cases before claiming accuracy or response-time improvements.

### 5.3 Assumptions

- The backend remains centered on `stream2.py` and `GemmaLlamaCppLLM`.
- Multimodal analysis is treated as engineering decision support with human verification.
- Claimed benefit metrics must remain targets until benchmarked.
- The first implementation should add internal metadata and prompting improvements without requiring frontend schema changes.

---

## 6. Expected Benefits Summary

| Metric | Current | Target After Improvements |
|---|---|---|
| RCFA accuracy (visual failures) | Baseline required | Improvement target to be validated through benchmark cases |
| Average analysis time per failure | Baseline required | Fewer repeated turns and faster time-to-useful-answer, validated by benchmark |
| P&ID understanding | Flat text or markdown extraction only | P&ID-assisted tag, equipment, and relationship extraction with human verification |
| Equipment damage assessment | Generic description | Taxonomy-aligned classification |
| Cross-reference capability | Separate contexts | Structured RCFA evidence packet |
| Document processing variety | Generic document text extraction | Type-aware handling for PDFs, images, tables, charts, and DCS screenshots |

---

## Appendix A: Code Locations Reference

| File | Lines | Relevance |
|---|---|---|
| `stream2.py` | 401-424 | Image/document attachment routing |
| `stream2.py` | 410-418 | Image detection and queue |
| `stream2.py` | 420-424 | Document RAG processing |
| `stream2.py` | 689-695 | Multimodal system prompt injection |
| `stream2.py` | 829-832 | Image attachment handoff to GemmaLlamaCppLLM |
| `gemma_llamacpp.py` | 130-132 | current_attachments initialization |
| `gemma_llamacpp.py` | 332-370 | Base64 image encoding and injection |
| `gemma_llamacpp.py` | 463-474 | Image-not-supported graceful fallback |
| `config.py` | 12 | MAX_DOC_TOKENS constant |
| `config.py` | 14 | MAX_FILE_SIZE constant |
| `schema.py` | 45-49 | FileAttachment schema |
| `schema.py` | 56-66 | ChatRequest with attachments |

---

## Appendix B: Gemma 4 26B Multimodal Capabilities

The `gemma4-26b-llamacpp` model (currently enabled) supports:
- **Native image understanding** — No VDA (Visual Document Analysis) fallback needed
- **Max images per request:** 3 (enforced in code)
- **Image encoding:** base64 data URI in OpenAI multimodal format
- **Resolution:** Model accepts variable resolution; larger images provide more detail
- **Context window:** 192K tokens — sufficient for combined image + text + tool data
- **Recommended:** Preprocess images to 1536x1536 for optimal quality/speed balance

---

*Document generated from source analysis of be-ircfa repository, commit HEAD.*