dev-rcfa-memory/BE_IRCFA.md

# `be-ircfa` Backend Architecture

This document explains the backend of the memory-enabled i-RCFA application in beginner-friendly terms.

## The backend's shape

```text
be-ircfa
│
├── run.py                  Local launcher; starts src.main:app
├── pyproject.toml          Python/Poetry project configuration
├── requirements/           Dependency groups
├── Dockerfile              Backend container definition
├── docker-compose.yaml     Local/container orchestration
├── docs/                   Design notes and reports
├── scratch/                Experiments and diagnostic scripts
├── tests/                  Automated and manual tests
├── workspace/              Workspace/tool files
└── src/                    Actual Python application package
    ├── main.py             Creates FastAPI and middleware
    ├── api.py              Connects route groups
    ├── config.py           Environment and service configuration
    ├── auth/               Checks who the user is
    ├── database/           Creates PostgreSQL sessions
    ├── ircfa_interaction/  Main chat and AI orchestration
    │   ├── router.py       HTTP endpoints
    │   ├── stream2.py      Main AI workflow
    │   ├── tools/          Sensor, PFI, reliability tools
    │   ├── llm/            Remote model adapters
    │   ├── lib/            Context and attachment helpers
    │   ├── system_templates/ Instructions and tool guides
    │   ├── model.py        Database table definitions
    │   ├── schema.py       Request/response shapes
    │   └── service.py      Reusable database/business operations
    ├── rag/                Document retrieval
    ├── root_cause/         Root-cause datasets and routes
    ├── failure_mode/       Failure-mode data and routes
    ├── equipments/         Equipment data and routes
    ├── fmea/               FMEA data and routes
    └── ircfa_alert/        Alert data and routes
```

An easy analogy:

- `run.py` is the key that starts the building.
- `src/main.py` is the building entrance.
- `src/api.py` is the directory showing where each department is.
- `router.py` is the receptionist.
- `stream2.py` is the supervisor.
- `tools/` are specialist workers.
- `llm/` is the phone connection to Gemma.
- `model.py` describes database tables.
- `service.py` contains reusable business actions.

## 1. The application starts in `run.py`

The normal local launcher is [`run.py`](be-ircfa/run.py). It starts Uvicorn and points it to the FastAPI application in [`src/main.py`](be-ircfa/src/main.py):

```text
run.py -> src.main:app
```

The FastAPI application itself is created in [`src/main.py`](be-ircfa/src/main.py).

At startup it:

1. creates the FastAPI application;
2. creates database tables if needed;
3. loads model configurations;
4. installs error handling;
5. installs rate limiting;
6. installs database-session middleware;
7. registers all API routes.

The startup function is in [`main.py`](be-ircfa/src/main.py:52).

Your memory-copy backend runs on port `8005`.

## 2. `api.py` connects the departments

[`api.py`](be-ircfa/src/api.py) creates the main route collection.

It registers:

```text
/chat
/rag
/equipments
/failure-mode
/fmea
/root-cause
/ircfa-alert
```

The chat router is connected under `/chat` and protected by a JWT dependency:

```python
authenticated_api_router.include_router(
    rcfarouter,
    prefix="/chat",
    tags=["chat"],
)
```

## 3. Authentication happens before chat logic

Before the chat code runs, the backend checks the login token using [`auth/service.py`](be-ircfa/src/auth/service.py).

The token identifies:

- which user is making the request;
- whether the token is valid;
- which private conversations and memories belong to that user.

The basic flow is:

```text
Login token -> JWTBearer -> CurrentUser -> chat request
```

This is important for Memory Lab because the memory query uses the authenticated user's ID.

## 4. Database sessions are created for each request

The backend creates database sessions in middleware.

There are two database connections:

```text
Main application database
  - conversations
  - chat history
  - memories
  - model configuration
  - application data

Collector database
  - collector/plant data
```

These connections are configured in [`database/core.py`](be-ircfa/src/database/core.py).

The request lifecycle is:

```text
Request starts
  ↓
Open main DB session
  ↓
Open collector DB session
  ↓
Run endpoint
  ↓
Commit or rollback
  ↓
Close sessions
```

The database URLs come from environment variables in [`config.py`](be-ircfa/src/config.py):

- `SQLALCHEMY_DATABASE_URI` is the main application database.
- `COLLECTOR_URI` is the collector database.
- `PFI_API_URL` is the PFI HTTP API.
- `RP_API_URL` is the reliability HTTP API.

## 5. Database models and API schemas

Database table definitions are in [`ircfa_interaction/model.py`](be-ircfa/src/ircfa_interaction/model.py).

Important tables include:

### Conversation: `rcfa_conversation`

Stores the larger chat container:

- user ID;
- conversation ID;
- conversation title;
- RCFA phase;
- RCFA state.

### Chat history: `rcfa_chat_history`

Stores individual messages and answers:

- user message;
- assistant response;
- timestamps;
- token usage;
- model name;
- location tag;
- references and context information.

### Memory: `rcfa_memory`

Stores private long-term memories:

- user ID;
- conversation ID;
- memory summary;
- source type;
- equipment;
- location tag;
- metadata.

### Model configuration: `rcfa_model_config`

Stores model connection details:

- model name;
- remote server URL;
- provider;
- API key;
- streaming setting;
- default model flag;
- generation settings.

API request and response shapes are in [`schema.py`](be-ircfa/src/ircfa_interaction/schema.py).

The difference is:

```text
model.py  = database shape
schema.py = API message shape
```

## 6. `router.py` receives the chat request

The main streaming endpoint is in [`router.py`](be-ircfa/src/ircfa_interaction/router.py):

```text
POST /chat/stream
```

It:

1. reads the JSON request;
2. finds the last user message;
3. extracts attachments;
4. reads the conversation ID;
5. reads the location tag;
6. reads the selected model;
7. builds a `ChatRequest`;
8. calls `stream_create_v2` from `stream2.py`.

The important handoff is:

```text
router.py -> stream2.py
```

`router.py` mainly receives, validates, and forwards. Most AI orchestration happens elsewhere.

## 7. `stream2.py` is the main supervisor

The main orchestration function is [`stream2.py`](be-ircfa/src/ircfa_interaction/stream2.py:603):

```python
async def stream_create(...)
```

It coordinates:

- conversation history;
- RCFA state;
- memory retrieval;
- document retrieval;
- model configuration;
- tools;
- streaming;
- database saving.

The flow is:

```text
Receive ChatRequest
  ↓
Find equipment/location
  ↓
Load recent conversation
  ↓
Load relevant memories
  ↓
Load document context
  ↓
Prepare system instructions
  ↓
Create model adapter
  ↓
Create tool list
  ↓
Run AI agent
  ↓
Execute tools when requested
  ↓
Stream answer
  ↓
Save chat and memory
```

## 8. Context is prepared before Gemma is called

Before Gemma receives the question, `stream2.py` builds the complete context. It can include:

```text
System instructions
RCFA workflow instructions
Memory privacy instructions
Conversation history
RCFA state
Relevant memories
Document text
Equipment tag
Current question
```

Gemma therefore receives more than the user's one sentence. The backend prepares the model's working desk first.

## 9. Technical tools are controlled backend functions

Tools are located under [`ircfa_interaction/tools/`](be-ircfa/src/ircfa_interaction/tools).

Examples include:

```text
current_sensor_tool.py
kondisi_sensor_historis.py
current_reliability_tool.py
history_reliability_tool.py
per_equipment_pfi_tool.py
next_failure_pfi_tool.py
root_cause_spln_tool.py
failure_mode_tool.py
plant_status_tool.py
```

Each tool generally:

1. receives arguments from the AI;
2. calls another API or reads data;
3. formats the result;
4. returns the result to the AI.

The tool loop looks like this:

```text
User asks about sensor data
  ↓
Gemma decides it needs the sensor tool
  ↓
i-RCFA backend runs CurrentSensorTool
  ↓
The tool calls the PFI API
  ↓
The tool returns sensor data
  ↓
Gemma explains the result
```

Gemma does not freely open database connections. It requests a tool, and the backend decides what code runs.

## 10. PFI integration is usually an API call

The backend configuration contains `PFI_API_URL` and `RP_API_URL` in [`config.py`](be-ircfa/src/config.py:55).

For PFI sensor information, the normal relationship is:

```text
Gemma
  ↓
i-RCFA PFI tool
  ↓
PFI HTTP API
  ↓
PFI data source
```

The i-RCFA backend does not normally query PFI database tables directly.

## 11. The remote model adapter

The remote model adapter is [`gemma_llamacpp.py`](be-ircfa/src/ircfa_interaction/llm/gemma_llamacpp.py).

It translates the application's internal request into an OpenAI-compatible request:

```json
{
  "model": "...",
  "messages": [],
  "tools": [],
  "tool_choice": "auto",
  "stream": true
}
```

The remote completion URL is built from the configured model server:

```text
https://gemma.digitaltwin.my.id/v1/chat/completions
```

The adapter also:

- adds authentication headers;
- converts messages into OpenAI-compatible format;
- includes tools;
- handles streaming;
- reads tool-call fragments;
- handles multimodal image data;
- logs remote errors.

## 12. The agent loop

Around the middle of `stream2.py`, the backend creates a `qwen_agent` assistant with:

- a system message;
- the selected LLM adapter;
- the allowed tool list.

Conceptually:

```python
bot = Assistant(
    system_message=system_prompt,
    llm=llm,
    function_list=tools,
)
```

Then it runs the assistant with the prepared messages.

One request can contain multiple rounds:

```text
Round 1: Gemma requests a tool
Round 2: backend executes the tool and returns the result
Round 3: Gemma produces the final answer
```

## 13. What happens when Gemma calls a tool

Inside the streaming loop, `stream2.py` handles:

- assistant messages;
- function/tool messages;
- reasoning content;
- final answer content.

When it sees a tool result, it can update RCFA state.

For example:

```text
PFI tool result
  -> save failure prediction state
  -> move RCFA phase forward

Current sensor tool result
  -> save sensor state
  -> move RCFA phase forward

Root-cause tool result
  -> save diagnosis state
```

This is why the UI can show RCFA progress such as:

```text
1. Failure Prediction
2. Sensor Data Mapping
3. Diagnosis
```

The backend is not only generating text. It is also maintaining an investigation state.

## 14. Streaming uses Server-Sent Events

`stream2.py` yields event strings such as:

```text
data: answer text

event: phase
data: 2. Sensor Data Mapping

event: usage_metadata
data: {...}

event: error
data: {...}
```

The full path is:

```text
Gemma token
  ↓
Gemma adapter
  ↓
stream2.py
  ↓
StreamingResponse
  ↓
nginx
  ↓
React chat screen
```

This is why the answer appears gradually.

## 15. Saving happens after the answer

After the agent finishes, the backend creates a chat-history record.

It saves things such as:

- user message;
- assistant response;
- timestamps;
- model;
- token counts;
- RCFA state;
- references;
- location tag.

Then it calls the memory-save logic if the user explicitly asked to remember something.

The normal save sequence is:

```text
Save chat history
  ↓
Check/save explicit memory
  ↓
Commit transaction
```

If saving fails, the backend rolls back and logs the error.

## 16. Memory is a separate backend layer

Memory retrieval happens before Gemma is called:

```text
Database -> memory filter -> relevant memories -> prompt
```

Memory saving happens after the answer:

```text
Explicit user request
  -> policy check
  -> lifecycle check
  -> database
```

The memory privacy rules are in [`memory_policy.py`](be-ircfa/src/ircfa_interaction/memory_policy.py).

The memory lifecycle rules are in [`memory_lifecycle.py`](be-ircfa/src/ircfa_interaction/memory_lifecycle.py).

The important separation is:

```text
Gemma decides how to answer.
Backend decides what memory is allowed.
Database stores the memory.
```

## 17. RAG is another AI-related path

The backend contains a RAG system under [`rag/`](be-ircfa/src/rag).

RAG means:

```text
Retrieve useful documents
  ↓
Give those documents to the AI
  ↓
Ask the AI to answer using them
```

This is different from text generation:

```text
Generation:
  Gemma writes the answer.

Retrieval:
  Embeddings/vector search find relevant documents.
```

The text-generation model can work while document retrieval has a separate problem.

## 18. Legacy model paths

Some older classes in [`model.py`](be-ircfa/src/ircfa_interaction/model.py) still define Ollama or vLLM-based configurations for older workflows and auxiliary classifiers.

That does not mean the normal i-RCFA chat has switched to local Ollama.

The active chat path is controlled by the model configuration in `rcfa_model_config` and the selected configuration passed into `stream2.py`.

Think of it as:

```text
Main chat path:
  router.py -> stream2.py -> configured remote model adapter

Other/legacy paths:
  may contain older Ollama, vLLM, or auxiliary configurations
```

## 19. Complete example

Suppose the user asks:

```text
What is the possible root cause of high vibration on 3DP-FDR711F?
```

The backend flow is:

```text
1. JWTBearer identifies the user.

2. router.py receives /chat/stream.

3. router.py creates ChatRequest.

4. stream2.py loads:
   - recent chat history;
   - private memories;
   - RCFA state;
   - location tag 3DP-FDR711F.

5. stream2.py creates the available tool list.

6. Gemma receives the question and tools.

7. Gemma requests current sensor data.

8. CurrentSensorTool calls the PFI API.

9. The sensor result returns to Gemma.

10. Gemma may request root-cause information.

11. RootCauseSPLNTool searches the root-cause data.

12. Gemma writes the explanation.

13. stream2.py sends answer pieces to the browser.

14. The backend updates RCFA state.

15. The backend saves chat history.

16. If the user explicitly asked to remember something, memory logic handles it.
```

## The most important relationship

```text
router.py
  -> stream2.py
      -> tools/
      -> llm/gemma_llamacpp.py
      -> database models/services
      -> StreamingResponse
```

That is the backbone of the backend.

## Recommended learning order

Read the backend in this order:

1. [`api.py`](be-ircfa/src/api.py) — where routes are connected.
2. [`router.py`](be-ircfa/src/ircfa_interaction/router.py) — how HTTP chat requests enter.
3. [`stream2.py`](be-ircfa/src/ircfa_interaction/stream2.py) — the main AI workflow.
4. One simple tool, such as [`current_sensor_tool.py`](be-ircfa/src/ircfa_interaction/tools/current_sensor_tool.py).
5. [`gemma_llamacpp.py`](be-ircfa/src/ircfa_interaction/llm/gemma_llamacpp.py) — how the backend talks to Gemma.
6. [`model.py`](be-ircfa/src/ircfa_interaction/model.py) — how chat, memory, and model settings are stored.
7. [`database/core.py`](be-ircfa/src/database/core.py) — how database connections are created.

## 20. Files not directly involved in the running backend

The normal backend started with `python run.py` follows this path:

```text
run.py -> src/main.py -> src/api.py -> route modules -> stream2.py -> tools / database / remote Gemma
```

The items below are **not directly used by that normal running path**. This does not automatically mean they are safe to delete: some are useful for maintenance, testing, or an older deployment approach.

| Item | Current role | Keep or remove? |
|---|---|---|
| `be-ircfa/src/metrics.py` | Old metrics code. The entire file and its import in `main.py` are commented out. | A clear cleanup candidate after a normal review. |
| `be-ircfa/src/ircfa_interaction/schema.py.orig` and `router.py.orig` | Editor backup copies; Python does not load `.orig` files. | Review their differences, then they can normally be removed. |
| `be-ircfa/scratch/` | Manual experiments, benchmarks, and diagnostic scripts. | Keep for learning/debugging; it is not part of the live backend. |
| `be-ircfa/docs/`, `.vscode/`, and `tests/` | Documentation, editor settings, and verification files. | Keep; they do not run as part of the backend service. |
| `be-ircfa/docker-compose.yaml`, `docker-kong.yml`, `kong.yml`, `config.yaml`, `docker_build_push.sh`, and `Jenkinsfile` | Older container, Kong, local-model, build, and CI deployment material. | Not used by the current manual port-8005 setup. Keep until the old deployment is formally retired. |
| `be-ircfa/requirements/requirements-full.txt`, `requirements-heavy.txt`, and `requirements-light.txt` | Alternative dependency lists with no repository references. Docker uses `requirements.txt` and `requirements/requirements-base.txt` instead. | Archive or remove only after confirming no external deployment uses them. |
| `be-ircfa/src/root_cause/sync.py` and `cleanup_data.py` | Manual database loading and cleanup scripts. They are not called when the app starts. | Keep if you may need to rebuild or clean root-cause data. |
| `be-ircfa/src/root_cause/*.json` and `fmea_data/*.tsv` | Source datasets used by manual data-loading work, not read during ordinary chat requests. | Do not remove unless the data is safely stored elsewhere and no future rebuild is needed. |
| `be-ircfa/src/ircfa_interaction/stream.py` | Older chat-orchestration implementation. All live chat endpoints call `stream2.py`. `router.py` still imports it, however. | Do not remove it alone; first remove the stale import and test the chat flow. |
| `be-ircfa/src/ircfa_interaction/llm/n8n.py` | An optional N8N LLM adapter. It is not selected by the current remote Gemma configuration. | Keep as a conditional fallback unless you decide to retire N8N support. |

In simple terms: `stream2.py` is the current engine. The other items above are mostly old tools, spare parts, instruction books, or data boxes. They do not make the normal chat answer a user request today.

### Important caution

The legacy Docker and deployment files should be treated as sensitive configuration. Before sharing, committing, or reusing them, move any credentials and connection values into protected environment variables.

## 21. Visual code maps

These diagrams show the **important driver files**, rather than every small helper or data class. Solid arrows show the normal running path. Dashed arrows show optional, legacy, or conditional paths.

### 21.1 Folder map: startup and HTTP routes

```mermaid
flowchart TD
    subgraph ROOT[be-ircfa root]
        RUN[run.py]
        ENV[Environment variables / .env]
    end

    subgraph CORE[src/ core files]
        MAIN[src/main.py]
        API[src/api.py]
        CFG[src/config.py]
        MID[src/middleware.py]
        LOG[src/logging.py]
    end

    subgraph AUTH[src/auth/]
        JWT[service.py: JWTBearer / CurrentUser]
        AUTHMODEL[model.py]
    end

    subgraph INTERACTION[src/ircfa_interaction/]
        CHATROUTER[router.py]
        STREAM2[stream2.py]
    end

    subgraph DOMAINS[Other routed feature folders]
        RAGROUTER[src/rag/router.py]
        ALERTROUTER[src/ircfa_alert/router.py]
        EQUIPROUTER[src/equipments/router.py]
        FMROUTER[src/failure_mode/router.py]
        FMEAROUTER[src/fmea/router.py]
        RCROUTER[src/root_cause/router.py]
    end

    RUN --> MAIN
    ENV --> CFG
    CFG --> MAIN
    MAIN --> LOG
    MAIN --> MID
    MAIN --> API
    API --> JWT
    API --> CHATROUTER
    API --> RAGROUTER
    API --> ALERTROUTER
    API --> EQUIPROUTER
    API --> FMROUTER
    API --> FMEAROUTER
    API --> RCROUTER
    CHATROUTER --> STREAM2
```

**What this means:** `run.py` turns on FastAPI. `main.py` prepares the application. `api.py` is the receptionist that sends each URL request to the correct folder. The chat requests go to `ircfa_interaction/router.py`, then to `stream2.py`.

### 21.2 Folder map: the i-RCFA interaction engine

```mermaid
flowchart LR
    subgraph INTERACTION[src/ircfa_interaction/]
        ROUTER[router.py\nchat, conversations, Memory Lab]
        STREAM[stream2.py\nmain orchestration]
        SCHEMA[schema.py\nrequest/response shapes]
        SERVICE[service.py\nchat/model persistence]
        MODEL[model.py\nchat, memory, LLM tables]
        MEMORY[memory_policy.py + memory_lifecycle.py]
        LEGACY[stream.py\nolder orchestration]
    end

    subgraph LLM[src/ircfa_interaction/llm/]
        GEMMA[gemma_llamacpp.py\nOpenAI-compatible Gemma adapter]
        N8N[n8n.py\noptional adapter]
    end

    subgraph TOOLS[src/ircfa_interaction/tools/]
        SENSOR[Sensor / PFI tools]
        RELIABILITY[Reliability tools]
        ROOTCAUSE[RootCauseSPLNTool]
        VECTOR[VectorSearchTool]
        OTHERTOOLS[Other RCFA tools]
    end

    subgraph SUPPORT[Supporting folders]
        RAG[src/rag/\nfiles, vector search, RAG history]
        DB[src/database/\nmain and collector sessions]
        EQUIP[src/equipments/]
        ROOTDATA[src/root_cause/]
    end

    ROUTER --> SCHEMA
    ROUTER --> STREAM
    ROUTER --> SERVICE
    STREAM --> MEMORY
    MEMORY --> MODEL
    STREAM --> MODEL
    SERVICE --> MODEL
    STREAM --> GEMMA
    STREAM -. optional configured path .-> N8N
    STREAM --> SENSOR
    STREAM --> RELIABILITY
    STREAM --> ROOTCAUSE
    STREAM --> VECTOR
    STREAM --> OTHERTOOLS
    STREAM --> RAG
    SERVICE --> DB
    MODEL --> DB
    SENSOR --> EQUIP
    ROOTCAUSE --> ROOTDATA
    VECTOR --> RAG
    ROUTER -. stale import; no active endpoint call .-> LEGACY
```

**What this means:** `stream2.py` is the conductor. It gathers chat context, private memory, RCFA data, attachments, and available tools; it then asks the configured LLM to answer. The LLM can request a tool, and `stream2.py` returns that tool result to the LLM before streaming the final response back to the browser.

### 21.3 Cross-folder runtime map: one chat request

```mermaid
sequenceDiagram
    participant FE as Frontend / nginx
    participant MAIN as src/main.py
    participant API as src/api.py
    participant AUTH as src/auth/service.py
    participant ROUTER as ircfa_interaction/router.py
    participant STREAM as ircfa_interaction/stream2.py
    participant DB as database/ + PostgreSQL
    participant TOOLS as tools/ + domain services
    participant EXT as PFI / Reliability APIs
    participant LLM as Remote Gemma

    FE->>MAIN: POST /ircfa/chat/stream
    MAIN->>API: send request to registered router
    API->>AUTH: verify JWT and identify user
    AUTH-->>ROUTER: authenticated user
    ROUTER->>STREAM: ChatRequest + database sessions
    STREAM->>DB: load chat, RCFA state, model config, private memories
    DB-->>STREAM: saved context
    STREAM->>LLM: question + system instructions + available tools
    opt LLM requests a technical tool
        LLM-->>STREAM: tool request
        STREAM->>TOOLS: run selected tool
        TOOLS->>EXT: request sensor, PFI, or reliability data
        EXT-->>TOOLS: technical data
        TOOLS-->>STREAM: tool result
        STREAM->>LLM: tool result
    end
    LLM-->>STREAM: streamed answer pieces
    STREAM->>DB: save chat, RCFA state, and explicit memory
    STREAM-->>FE: streamed answer events
```

This third diagram is deliberately cross-folder: a real answer passes through authentication, the chat router, orchestration, database storage, tools, external operational APIs, and the remote Gemma service.

### 21.4 Folder responsibilities at a glance

```mermaid
flowchart TB
    API[src/api.py\nconnects routes] --> AUTH[src/auth/\nwho is this user?]
    API --> CHAT[src/ircfa_interaction/\nchat, memory, LLM orchestration]
    API --> RAG[src/rag/\ndocuments and retrieval]
    API --> EQUIP[src/equipments/\nequipment and maintenance data]
    API --> FM[src/failure_mode/\nfailure-mode data]
    API --> FMEA[src/fmea/\nFMEA records]
    API --> RC[src/root_cause/\nroot-cause matrix data]
    API --> ALERT[src/ircfa_alert/\nalerts]
    CHAT --> DATABASE[src/database/\nPostgreSQL sessions]
    RAG --> DATABASE
    EQUIP --> DATABASE
    FM --> DATABASE
    FMEA --> DATABASE
    RC --> DATABASE
```

Use the diagrams together:

1. **21.1** answers: “Which folder receives my request?”
2. **21.2** answers: “What happens inside the AI/chat engine?”
3. **21.3** answers: “How do folders and outside services cooperate for one answer?”
4. **21.4** answers: “What is each feature folder responsible for?”