dev-rcfa-memory/be-ircfa/docs/multimodal_llm_improvement_analysis_stream2.md

Multimodal LLM Improvement Analysis — stream2.py

iRCFA Application: Tanjung Jati B Unit 3 Power Plant

Date: 2026-06-17
Author: aimo (AI-Assisted Analysis)
Scope: src/ircfa_interaction/stream2.py and related multimodal data pipelines
Target: Maximum potential extraction for Multi-modal LLM in RCFA workflow

---

TL;DR

Current State: stream2.py already supports basic multimodal — images go to Gemma 4 via base64 encoding, documents go through Docling OCR → text extraction → RAG vector store. However, the system treats all images equally and all documents as flat text, losing critical visual information from P&ID diagrams, thermal images, vibration plots, and equipment photos that are vital for RCFA at a coal-fired power plant.

Key Opportunities (Ranked by Impact):

1. P&ID Diagram Understanding — Currently treated as flat text via OCR. A multimodal LLM can read valve tags, trace piping, identify flow direction, and locate equipment in the plant layout. This is the single highest-value improvement.
2. Equipment Photo Analysis — Failure evidence photos (corrosion, cracks, leaks) are passed to the LLM without preprocessing or structured prompting. Adding visual inspection context and failure taxonomy alignment would dramatically improve root cause identification.
3. Table & Chart Extraction from PDFs — Maintenance reports, vibration spectra, and thermal scans contain tabular/visual data that Docling flattens into text. Native multimodal processing preserves structure.
4. Document-Type Classification — All attachments processed identically. Classifying documents (P&ID, maintenance report, FMEA sheet, equipment photo) before processing allows type-specific prompt engineering and tool routing.
5. Image Preprocessing Pipeline — No resize, enhancement, or quality normalization before base64 encoding. Scanned field photos with poor lighting or blurry text degrade LLM performance.
6. Cross-Modal Context Fusion — Images and tool data (sensor readings, reliability metrics) are processed in isolation. Fusing visual evidence with quantitative data in a single prompt gives the LLM richer context for root cause analysis.

Estimated Impact: These improvements are expected to reduce repeated analysis turns and improve evidence grounding for visual-evidence-dependent failure modes (corrosion, erosion, thermal damage, mechanical wear). Exact gains in RCFA accuracy and response time must be validated through benchmark cases before being used as delivery targets.

---

1. Current Architecture Analysis

1.1 Image Processing Flow (stream2.py lines 401-424)

User Upload → stream2.py
  ├── Image (.png/.jpg/.jpeg/.gif/.webp)
  │   └── saved_images[] → GemmaLlamaCppLLM.current_attachments
  │       └── _build_payload() → base64 encode → OpenAI image_url format
  │           └── Injected into LAST user message as multimodal content
  │
  └── Document (.pdf/.doc/.txt)
      └── saved_docs[] → create_docling_rag()
          ├── Docling DocumentConverter (primary) → Markdown
          └── pypdf fallback → Plain text
              └── accumulated_docs → text injection into user message

1.2 GemmaLlamaCppLLM Multimodal Support (gemma_llamacpp.py lines 332-370)

• Encodes up to 3 images as base64 data URIs
• Injects as image_url parts into the last user message (OpenAI multimodal format)
• Graceful fallback: if server returns "image input not supported", retries without images
• No image preprocessing (resize, compress, quality check)
• No image metadata extraction (EXIF, GPS, timestamp)

1.3 Document Processing (stream2.py lines 164-226)

Three fallback levels:

1. Docling (best) — OCR, table extraction, layout preservation → Markdown
2. Unstructured.io — OCR-only strategy with chunking by title
3. pypdf (fallback) — Basic text extraction, no OCR

1.4 Tool Ecosystem (lines 488-500)

Available tools that provide quantitative context:

• RCFATool — Equipment predicted to fail in 7 days
• CurrentReliabilityTool — Reliability metrics (MTTR, MTBF, failure rate)
• CurrentSensorTool — Real-time sensor readings
• KondisiSensorHistorisTool — Historical sensor data
• NextFailureRPTool/PFITool — Predicted failure timing
• RootCauseSPLNTool — SPLN root cause database + FMEA records
• StandardMetricsTool — Equipment standard metrics
• PlantStatusTool — Plant running/shutdown status

Critical Gap: None of these tools receive visual context from attached images. The LLM sees images and tool data in separate contexts.

---

2. Identified Improvement Areas

2.1 [HIGH IMPACT] P&ID Diagram Understanding

Current Problem: When a user uploads a P&ID (Piping & Instrumentation Diagram), it goes through Docling OCR which converts it to flat text. The visual structure — pipe routing, valve positions, sensor locations, instrument tags — is completely lost.

Tanjung Jati B Context: Unit 3 has thousands of P&ID sheets covering:

• Boiler water/steam circuit
• Feedwater heating system (HPH/LPH)
• Condenser and cooling water system
• Turbine lube oil system
• Coal handling and ash handling
• DCS/instrumentation loops

Proposed Solution:

# NEW: Document classifier (pre-processing step)
DOCUMENT_TYPES = {
    "pid": {
        "keywords": ["P&ID", "Piping and Instrumentation", "Process Flow", "Line List"],
        "processing": "multimodal_vision",  # Send as image to LLM
        "prompt_prefix": "Analisis diagram P&ID ini. Identifikasi: (1) equipment yang ditunjuk, (2) valve dan instrumen, (3) aliran fluida, (4) lokasi sensor. Fokus pada location tag yang relevan."
    },
    "equipment_photo": {
        "keywords": [],  # Detected by image metadata or user description
        "processing": "multimodal_vision",
        "prompt_prefix": "Analisis foto equipment ini untuk RCFA. Identifikasi: (1) jenis kerusakan yang terlihat, (2) lokasi kerusakan, (3) severity estimasi, (4) kemungkinan failure mode berdasarkan penampakan."
    },
    "maintenance_report": {
        "keywords": ["Maintenance", "Inspection", "Work Order", "PM Report"],
        "processing": "text_extraction_enhanced",  # Docling with table preservation
        "prompt_prefix": "Ekstrak data maintenance dari dokumen ini: equipment, failure history, tindakan yang dilakukan, dan rekomendasi."
    },
    "vibration_data": {
        "keywords": ["Vibration", "Spectrum", "FFT", "Acceleration"],
        "processing": "multimodal_vision",  # Charts need visual analysis
        "prompt_prefix": "Analisis plot vibrasi ini. Identifikasi: (1) dominan frequency, (2) defect signature (imbalance, misalignment, bearing defect), (3) severity vs threshold."
    }
}

2.2 [HIGH IMPACT] Equipment Failure Photo Analysis

Current Problem: Photos of equipment damage (corrosion, cracks, leaks, burn marks) are sent to Gemma 4 with generic instructions. The system prompt only says "fokuskan analisis utama pada gambar/dokumen terbaru ini" (line 693).

Proposed Solution — Failure Taxonomy-Aligned Prompting:

FAILURE_VISUAL_TAXONOMY = {
    "corrosion": {
        "visual_signs": ["rust color", "pitting", "uniform thinning", "crevice patterns"],
        "indicators": ["scale buildup", "color change", "surface roughness"],
        "tjb_relevant_areas": ["boiler tubes", "condenser tubes", "feedwater piping", "cooling water system"]
    },
    "erosion": {
        "visual_signs": ["material loss", "smooth worn surfaces", "directional grooves"],
        "indicators": ["wall thinning patterns", "flow-aligned damage"],
        "tjb_relevant_areas": ["coal pipes", "ash handling", "soot blower areas", "economizer tubes"]
    },
    "thermal_damage": {
        "visual_signs": ["discoloration", "warping", "scaling", "blistering"],
        "indicators": ["heat tint colors", "deformation patterns"],
        "tjb_relevant_areas": ["superheater", "reheater", "burner zone", "turbine casing"]
    },
    "mechanical_wear": {
        "visual_signs": ["scratch marks", "grooving", "brinelling", "spalling"],
        "indicators": ["bearing surface damage", "seal wear patterns"],
        "tjb_relevant_areas": ["pump bearings", "turbine bearings", "gearbox", "coupling"]
    },
    "leakage": {
        "visual_signs": ["wet spots", "staining", "mineral deposits", "steam plumes"],
        "indicators": ["joint failures", "gasket degradation", "tube rupture"],
        "tjb_relevant_areas": ["heat exchanger joints", "flange connections", "valve stems"]
    }
}

2.3 [MEDIUM IMPACT] Image Preprocessing Pipeline

Current Problem: Raw images from field technicians are sent directly as base64 without any preprocessing. Field photos often have:

• Poor lighting (dark boiler areas, backlit photos)
• Blurry/damaged text on nameplates and tags
• Very large file sizes (20+ MB phone photos)
• Rotation issues (EXIF orientation not always honored)

Proposed Addition (new file: src/ircfa_interaction/lib/image_preprocessor.py):

class ImagePreprocessor:
    MAX_DIMENSION = 2048  # Max width/height in pixels
    JPEG_QUALITY = 85
    TARGET_FORMAT = "JPEG"
    
    @staticmethod
    def prepare_for_llm(image_path: str) -> dict:
        """
        Prepare an image for multimodal LLM consumption.
        Returns: {"path": processed_path, "metadata": {...}, "classification": "..."}
        """
        from PIL import Image, ExifTags
        import hashlib
        
        img = Image.open(image_path)
        
        # 1. Auto-rotate based on EXIF
        img = ImagePreprocessor._auto_rotate(img)
        
        # 2. Resize if too large (preserving aspect ratio)
        w, h = img.size
        if max(w, h) > ImagePreprocessor.MAX_DIMENSION:
            ratio = ImagePreprocessor.MAX_DIMENSION / max(w, h)
            img = img.resize((int(w*ratio), int(h*ratio)), Image.LANCZOS)
        
        # 3. Enhance contrast for dark photos
        img = ImagePreprocessor._enhance_if_dark(img)
        
        # 4. Save as optimized JPEG
        output_path = image_path + ".processed.jpg"
        img.save(output_path, "JPEG", quality=ImagePreprocessor.JPEG_QUALITY)
        
        # 5. Extract metadata
        metadata = {
            "original_size": os.path.getsize(image_path),
            "processed_size": os.path.getsize(output_path),
            "dimensions": img.size,
            "content_hash": hashlib.md5(open(image_path, 'rb').read()).hexdigest()
        }
        
        # 6. Classify image type
        classification = ImagePreprocessor._classify_image(img, metadata)
        
        return {"path": output_path, "metadata": metadata, "classification": classification}
    
    @staticmethod
    def _enhance_if_dark(img):
        """Auto-brighten dark images (common in plant interiors)."""
        import numpy as np
        arr = np.array(img)
        mean_brightness = arr.mean()
        if mean_brightness < 80:  # Dark threshold
            from PIL import ImageEnhance
            factor = min(2.0, 120.0 / mean_brightness)
            return ImageEnhance.Brightness(img).enhance(factor)
        return img

2.4 [MEDIUM IMPACT] Document-Type-Aware Processing

Current Problem: All documents go through the same create_docling_rag() pipeline regardless of type.

Proposed Enhancement:

def process_attachment_smart(file_path: str, user_context: str = "") -> dict:
    """
    Smart document processing with type classification.
    Returns structured data for multimodal or text pipelines.
    """
    suffix = os.path.splitext(file_path)[1].lower()
    
    if suffix in [".png", ".jpg", ".jpeg", ".gif", ".webp"]:
        return _process_image(file_path, user_context)
    elif suffix == ".pdf":
        return _process_pdf_smart(file_path, user_context)
    elif suffix in [".xlsx", ".xls", ".csv"]:
        return _process_tabular(file_path, user_context)
    else:
        return {"type": "text", "content": create_docling_rag(file_path), "pipeline": "text"}

def _process_pdf_smart(pdf_path: str, user_context: str) -> dict:
    """Classify PDF content and choose optimal processing pipeline."""
    # Quick classification via first page content
    from pypdf import PdfReader
    reader = PdfReader(pdf_path)
    first_page_text = reader.pages[0].extract_text()[:2000] if reader.pages else ""
    
    # Check for P&ID characteristics
    pid_indicators = ["P&ID", "Piping", "Instrumentation", "Line No", "Tag No", "P&I"]
    if any(ind.lower() in first_page_text.lower() for ind in pid_indicators):
        return {
            "type": "pid_diagram",
            "content": create_docling_rag(pdf_path),  # Text extraction for context
            "image_path": pdf_path,  # Also pass PDF for multimodal analysis
            "pipeline": "multimodal",
            "prompt_context": "Dokumen ini adalah P&ID (Piping & Instrumentation Diagram). Analisis struktur visual diagram."
        }
    
    # Check for maintenance/inspection report
    maintenance_indicators = ["Maintenance", "Inspection", "Work Order", "Corrective", "Preventive"]
    if any(ind.lower() in first_page_text.lower() for ind in maintenance_indicators):
        return {
            "type": "maintenance_report",
            "content": create_docling_rag(pdf_path),
            "pipeline": "text_enhanced",
            "prompt_context": "Dokumen ini adalah laporan maintenance. Ekstrak data equipment, failure history, dan tindakan."
        }
    
    # Default: standard RAG processing
    return {
        "type": "generic_document",
        "content": create_docling_rag(pdf_path),
        "pipeline": "text"
    }

2.5 [MEDIUM IMPACT] Cross-Modal Context Fusion

Current Problem: Images and tool data are processed in separate contexts. The LLM sees the image but doesn't have sensor data or reliability metrics overlaid.

Proposed Solution — Enhanced Prompt Injection:

# In stream2.py, after tool results are collected and before final LLM call:

def build_multimodal_context(
    saved_images: list,
    function_results: list,
    tool_data: dict,
    location_tag: str
) -> str:
    """
    Build rich context that fuses visual evidence with quantitative data.
    """
    context_parts = []
    
    if saved_images:
        context_parts.append(
            "📷 **Bukti Visual yang Dilampirkan:**\n"
            "Gambar di atas menunjukkan kondisi equipment. Analisis berdasarkan:\n"
            "- Jenis kerusakan visual (korosi, erosi, retak, kebocoran, deformasi)\n"
            "- Lokasi kerusakan relatif terhadap equipment\n"
            "- Severity estimasi (ringan/sedang/berat)\n"
            "- Korelasi dengan data sensor dan reliability di bawah"
        )
    
    if function_results:
        sensor_data = next((r for r in function_results if r["name"] == "get_current_sensor"), None)
        reliability_data = next((r for r in function_results if r["name"] == "get_current_reliability"), None)
        
        if sensor_data:
            context_parts.append(f"\n📊 **Data Sensor Terkini ({location_tag}):**\n{sensor_data['content'][:1500]}")
        if reliability_data:
            context_parts.append(f"\n📈 **Data Reliability ({location_tag}):**\n{reliability_data['content'][:1500]}")
    
    return "\n---\n".join(context_parts)

2.6 [LOW-MEDIUM IMPACT] Attachment Limit & Token Budget

Current Limits:

• Max 3 attachments (line 403)
• MAX_DOC_TOKENS = 32,764 (config.py line 12)
• No separate token budget for images vs text

Proposed Changes:

# config.py additions
MAX_ATTACHMENTS = 5  # Increased from 3
MAX_IMAGE_TOKENS = 4096  # Separate budget for image content
MAX_TEXT_DOC_TOKENS = 28672  # Remaining for text documents

# stream2.py — smarter attachment management
MAX_ATTACHMENTS = 5
if len(request.attachments) > MAX_ATTACHMENTS:
    # Prioritize: images first (visual evidence), then by file size (larger = more content)
    sorted_attachments = _prioritize_attachments(request.attachments[:8])
    attachments_to_process = sorted_attachments[:MAX_ATTACHMENTS]

2.7 [LOW IMPACT] Minor Code Quality Issues

Location	Issue	Fix
Line 397	print(request.attachments) — debug statement in production	Remove or convert to logger.debug()
Line 436	UploadedFile referenced but not imported	Add from src.rag.model import UploadedFile or guard with try/except
Line 357	print("Extracted location tag...") — should use logger	Change to logger.info()
Line 641	print(len(message), ...) — production noise	Convert to logger.debug()
Line 784	print("Files passed to Assistant:", ...) — production noise	Convert to logger.debug()
Line 789	print("Streaming started at:", ...) — production noise	Convert to logger.debug()
Lines 46-52	Duplicate imports (ChatPromptTemplate imported twice, lines 25 and 66)	Remove duplicate
Lines 8, 74-75	Duplicate imports (requests, json, asyncio imported multiple times)	Deduplicate
Line 230	equipment_keywords list defined inline and also in schema.py	Import from schema.py to avoid maintenance burden

---

3. Tanjung Jati B Unit 3 — Specific Data Opportunities

3.1 Power Plant Document Types Available for Multimodal

Document Type	Source	Current Processing	Multimodal Opportunity
P&ID Diagrams	Engineering dept	Text OCR (loses structure)	P&ID-assisted tag, equipment, and relationship extraction requiring human verification
Equipment Photos	Field technicians	Generic image → LLM	Failure taxonomy-aligned visual inspection
Thermal Images	IR camera surveys	Not processed	Temperature gradient analysis, hot spot identification
Vibration Spectra	Online/offline monitoring	Text summary only	FFT pattern recognition, defect signature matching
Maintenance Reports	CMMS/Maximo	Docling text extraction	Table extraction for failure history, MTTR trends
FMEA Sheets	Reliability dept	Database query only	Cross-reference with visual evidence
Boiler Tube Maps	Inspection team	Not processed	Map failure locations, identify patterns
Turbine Cross-Sections	OEM manuals	Not processed	Structural analysis reference

3.2 Equipment Categories with Highest Multimodal Value

Ranked by frequency of visual-evidence-dependent RCFA:

1. Boiler Tubes (Superheater, Reheater, Economizer, Waterwall)

   • Failure modes: creep, erosion, corrosion, hydrogen damage
   • Visual evidence: tube color, deformation, wall thinning patterns

2. Heat Exchangers (HPH, LPH, Condenser)

   • Failure modes: tube leakage, fouling, baffle erosion
   • Visual evidence: tube-end condition, scaling patterns, leak staining

3. Turbine Components (HP/IP/LP sections)

   • Failure modes: blade erosion, foreign object damage, creep
   • Visual evidence: blade surface condition, deposits, discoloration

4. Pumps & Valves (Feedwater, Cooling Water, Condensate)

   • Failure modes: impeller erosion, seal leakage, seat damage
   • Visual evidence: wear patterns, leakage paths, vibration evidence

5. Coal & Ash Handling

   • Failure modes: abrasive wear, blockage, corrosion
   • Visual evidence: wear groove patterns, buildup locations

3.3 Recommended Multimodal Use Cases per RCFA Phase

RCFA Phase	Multimodal Input	Tool Integration
1. Failure Prediction	Equipment health photos, trend charts	NextFailurePFITool + image
2. Sensor Data Mapping	Vibration spectra, thermal images	CurrentSensorTool + visual overlay
3. Potential Root Causes	P&ID for pipe tracing, FMEA sheets	RootCauseSPLNTool + diagram
4. Diagnosis	Equipment damage photos, inspection reports	Cross-reference with sensor data
5. Final Reporting	All images embedded in report	Export with annotated images

---

4. Revision Notes And Implementation Judgment

This revision tightens the original analysis so it can be used safely as an implementation reference for iRCFA on a specific coal power plant unit. The core direction remains valid: the app already has a basic multimodal path, but the highest-value improvement is not simply "send more images to the LLM." The highest-value improvement is structured evidence fusion across attachment content, unit context, location tag, plant data, reliability predictions, and RCFA phase.

4.1 Confirmed Current Behavior

• stream2.py supports basic multimodal image routing by separating image attachments from document attachments.
• GemmaLlamaCppLLM injects up to 3 image files as base64 image_url parts into the final OpenAI-style chat payload.
• Documents are currently reduced to extracted text or markdown before being appended to the reasoning context.
• Docling may preserve some table or layout information in markdown, but the current app does not route documents by document type or preserve visual structure as first-class RCFA evidence.
• Tool outputs and visual evidence are not explicitly fused into a structured RCFA evidence packet before final reasoning.

4.2 Corrections To Original Claims

• Accuracy improvement numbers such as ~70% to ~90%+ should be treated as target hypotheses, not confirmed outcomes.
• Time reduction numbers such as 40-60% should be treated as expected directionality until measured against real or representative RCFA cases.
• P&ID support should be framed as P&ID-assisted tag, equipment, and relationship extraction requiring human verification, not autonomous full structural analysis.
• Multimodal output must be treated as engineering decision support. It should help technicians and reliability engineers inspect evidence, compare hypotheses, and prepare RCFA reasoning, but it should not be treated as authoritative diagnosis without human review.

4.3 Revised Priority

The implementation priority should shift from generic image understanding to unit-bound evidence fusion:

1. Attachment classification: identify whether the upload is an equipment photo, P&ID, thermal image, vibration chart, DCS screenshot, maintenance report, FMEA sheet, or generic document.
2. Unit/location-tag binding: connect each attachment to an explicit unit, equipment, system, and location tag where available.
3. Structured RCFA evidence packet: combine image/document evidence with current sensor data, historical sensor data, reliability metrics, PFI/RP predictions, SPLN/FMEA records, and current RCFA phase.
4. Type-specific prompting: use different prompts for photos, diagrams, DCS screenshots, vibration plots, thermal images, and maintenance reports.
5. Advanced analytics: add P&ID page rendering, thermal interpretation, vibration spectrum recognition, and annotated reporting only after baseline validation.

4.4 Validation Requirement

Before claiming accuracy or time improvements, create a benchmark set of 20-50 real or representative Tanjung Jati B Unit 3 RCFA cases. Each case should include the expected failure mode, available visual/document evidence, relevant location tag, tool data snapshot, and accepted engineering conclusion.

Measure:

• Diagnosis quality and root-cause ranking.
• Evidence grounding and citation of observed facts.
• Hallucination rate, especially for visual details not present in attachments.
• Correctness of tool usage and location-tag binding.
• Time-to-useful-answer and number of turns required.

All benefit numbers in this document should be treated as hypotheses until this benchmark exists.

---

5. Implementation Roadmap

Phase 1: Safety And Baseline Cleanup (1-2 weeks)

• Fix debug print statements → logger
• Remove duplicate imports
• Import UploadedFile properly or add guard
• Keep image limit at 3 initially unless benchmarking shows safe latency and memory headroom
• Add lightweight attachment routing tests: image-only, document-only, mixed attachments, and over-limit attachments

Phase 2: Attachment Classification (2-3 weeks)

• Add a small attachment classifier module under src/ircfa_interaction/lib/
• Classify attachments into equipment_photo, pid_diagram, thermal_image, vibration_chart, dcs_screenshot, maintenance_report, fmea_sheet, or generic_document
• Use filename, extension, content type, first-page text, and user message hints
• Return structured metadata: attachment_type, processing_mode, confidence, and prompt_context

Phase 3: Unit/Tag-Bound Multimodal Context (3-4 weeks)

• Build a multimodal context builder that produces a structured RCFA evidence packet
• Include explicit unit, location tag, attachment summary, visual prompt context, relevant sensor/reliability/PFI/RP guidance, SPLN/FMEA guidance, and existing RCFA state evidence
• Preserve the current safety behavior: only bind sensor and reliability tool calls to location tags explicitly provided by the user or frontend
• Add type-specific prompt snippets for equipment photos, P&IDs, vibration charts, DCS screenshots, and maintenance reports

Phase 3 implementation note: the evidence packet is injected internally into the LLM message context and exposed in context_info for debugging; it does not change the frontend upload or streaming API.

Phase 4: Validation And Advanced Analytics (4-6 weeks)

• Create a benchmark dataset from known or representative coal plant RCFA cases
• Add evaluation scripts for multimodal answers versus expected evidence and root-cause reasoning
• Only after validation, add thermal image interpretation, vibration spectrum pattern recognition, P&ID page rendering, boiler tube map visualization, and annotated final reports

5.1 Public Interfaces

• Add internal structured metadata for attachments; no frontend API change is required for the first implementation.
• Preserve existing ChatRequest.attachments behavior for compatibility with current upload flows.
• Optional future frontend enhancement: allow users to provide or confirm unit, system, equipment, and location_tag per attachment.

5.2 Test Plan

• Unit test attachment classification for representative filenames, extensions, content types, first-page text, and user message hints.
• Unit test image/document routing so current Gemma image payload behavior does not regress.
• Unit test document text extraction fallback so Docling and pypdf behavior remains compatible.
• Integration test a mixed request with one equipment image, one report PDF, and an explicit location tag.
• Regression test that inherited historical tags are not auto-injected into sensor or reliability tools.
• Benchmark test against 20-50 curated RCFA cases before claiming accuracy or response-time improvements.

5.3 Assumptions

• The backend remains centered on stream2.py and GemmaLlamaCppLLM.
• Multimodal analysis is treated as engineering decision support with human verification.
• Claimed benefit metrics must remain targets until benchmarked.
• The first implementation should add internal metadata and prompting improvements without requiring frontend schema changes.

---

6. Expected Benefits Summary

Metric	Current	Target After Improvements
RCFA accuracy (visual failures)	Baseline required	Improvement target to be validated through benchmark cases
Average analysis time per failure	Baseline required	Fewer repeated turns and faster time-to-useful-answer, validated by benchmark
P&ID understanding	Flat text or markdown extraction only	P&ID-assisted tag, equipment, and relationship extraction with human verification
Equipment damage assessment	Generic description	Taxonomy-aligned classification
Cross-reference capability	Separate contexts	Structured RCFA evidence packet
Document processing variety	Generic document text extraction	Type-aware handling for PDFs, images, tables, charts, and DCS screenshots

---

Appendix A: Code Locations Reference

File	Lines	Relevance
stream2.py	401-424	Image/document attachment routing
stream2.py	410-418	Image detection and queue
stream2.py	420-424	Document RAG processing
stream2.py	689-695	Multimodal system prompt injection
stream2.py	829-832	Image attachment handoff to GemmaLlamaCppLLM
gemma_llamacpp.py	130-132	current_attachments initialization
gemma_llamacpp.py	332-370	Base64 image encoding and injection
gemma_llamacpp.py	463-474	Image-not-supported graceful fallback
config.py	12	MAX_DOC_TOKENS constant
config.py	14	MAX_FILE_SIZE constant
schema.py	45-49	FileAttachment schema
schema.py	56-66	ChatRequest with attachments

---

Appendix B: Gemma 4 26B Multimodal Capabilities

The gemma4-26b-llamacpp model (currently enabled) supports:

• Native image understanding — No VDA (Visual Document Analysis) fallback needed
• Max images per request: 3 (enforced in code)
• Image encoding: base64 data URI in OpenAI multimodal format
• Resolution: Model accepts variable resolution; larger images provide more detail
• Context window: 192K tokens — sufficient for combined image + text + tool data
• Recommended: Preprocess images to 1536x1536 for optimal quality/speed balance

---

Document generated from source analysis of be-ircfa repository, commit HEAD.