dev-rcfa-memory/be-ircfa/README.md

# i-RCFA Backend (`be-ircfa`)

`be-ircfa` is the Python FastAPI backend for the i-RCFA (Root Cause Failure Analysis) chat application. It receives authenticated chat requests, prepares conversation and RCFA context, retrieves sensor/reliability/PFI information through controlled tools, calls the configured LLM, streams the response, and saves the conversation.

The current memory-copy application is intended to keep using the established remote Gemma service. Do not start local Ollama, llama.cpp, or vLLM services for the normal i-RCFA chat flow.

## What the backend does

```text
Browser
  -> frontend nginx proxy
  -> FastAPI backend
      -> PostgreSQL chat/memory database
      -> collector database session
      -> PFI/reliability HTTP APIs
      -> remote Gemma service
  <- streamed answer
```

The main AI workflow is in [`src/ircfa_interaction/stream2.py`](src/ircfa_interaction/stream2.py). The complete architecture walkthrough is in [`../BE_IRCFA.md`](../BE_IRCFA.md).

## Project layout

```text
be-ircfa/
├── run.py                 Local launcher
├── pyproject.toml         Poetry project/dependencies
├── requirements/          Dependency groups
├── Dockerfile             Backend image definition
├── docker-compose.yaml    Legacy/all-in-one container setup
├── docs/                  Design notes and reports
├── scratch/               Experiments and diagnostics
├── tests/                 Tests
└── src/
    ├── main.py            FastAPI application and middleware
    ├── api.py             API route registration
    ├── auth/              Authentication
    ├── database/          Async PostgreSQL sessions
    ├── ircfa_interaction/ Chat, AI, tools, memory and model adapters
    ├── rag/               Document retrieval
    ├── root_cause/        Root-cause data and routes
    ├── failure_mode/      Failure-mode data and routes
    ├── equipments/        Equipment routes and services
    ├── fmea/              FMEA routes and models
    └── ircfa_alert/       Alert routes and services
```

## Prerequisites

You need:

- Python `>=3.11,<3.13`;
- a working virtual environment with the backend dependencies;
- access to the main PostgreSQL database;
- access to the collector database, if collector-backed features are used;
- access to the PFI and reliability APIs;
- the configured remote Gemma endpoint;
- a backend `.env` file or equivalent exported environment variables.

The backend reads environment variables in [`src/config.py`](src/config.py). Do not commit `.env` files or credentials.

## Recommended current setup: run the memory-copy backend manually

This is the setup currently used for the memory-enabled comparison copy.

### 1. Start the backend

Open a terminal and run:

```bash
cd /home/nvidia/dev-rcfa-memory/be-ircfa
source /home/nvidia/dev-rcfa/be-ircfa/venv/bin/activate

set -a
source /home/nvidia/dev-rcfa/be-ircfa/.env
set +a

export HOST=0.0.0.0
export PORT=8005
python run.py
```

Why is the `.env` path from `/home/nvidia/dev-rcfa`? The current memory-copy backend setup intentionally reuses the established environment while it is being tested. This also means it may point at the same PostgreSQL database as the baseline application. Isolate the database before serious memory experiments.

`run.py` starts Uvicorn with:

```text
run.py -> src.main:app
```

The backend should listen on:

```text
http://0.0.0.0:8005
```

### 2. Check backend health

In another terminal:

```bash
curl http://127.0.0.1:8005/healthcheck
```

Expected response:

```json
{"status":"ok"}
```

### 3. Run the matching frontend

The memory-copy frontend normally runs in Docker and forwards `/ircfa` requests to this backend:

```bash
cd /home/nvidia/dev-rcfa-memory/fe-digitaltwin-v2

FRONTEND_PORT=8083 \
BACKEND_URL_IRCFA=http://host.docker.internal:8005 \
docker compose up --build
```

Open the real chat route:

```text
http://127.0.0.1:8083/i-rcfa-app/chat
```

The request path is:

```text
Browser -> frontend nginx /ircfa proxy -> backend :8005
```

## Alternative local commands

### Run with Make

From `be-ircfa/`:

```bash
make install
make run
```

`make install` runs `poetry install`. `make run` runs `python run.py`.

The Makefile also defines:

```bash
make test
```

which runs the unit tests under `tests/unit/` through Poetry. The environment must have the test dependencies installed.

### Run with Poetry directly

```bash
cd be-ircfa
poetry install
poetry run python run.py
```

### Run Uvicorn directly

```bash
cd be-ircfa
poetry run uvicorn src.main:app --host 0.0.0.0 --port 8005 --reload
```

Do not run multiple backend processes against the same port.

## Docker Compose note

`docker-compose.yaml` contains an older all-in-one deployment definition. It includes:

- the i-RCFA backend;
- Kong;
- vLLM services;
- llama.cpp services;
- optional local Gemma services.

That file is useful as a deployment/reference artifact, but it is not the recommended command for the current memory-copy learning setup. In particular, do not start the local model services just to run i-RCFA. Keep the backend pointed at the established remote Gemma configuration.

If you intentionally need to inspect the legacy Compose setup, review the file first and verify every environment variable and service before running:

```bash
cd be-ircfa
docker compose config
docker compose up --build
```

## Important environment variables

The names below are read by [`src/config.py`](src/config.py):

| Variable | Purpose |
|---|---|
| `HOST` | Backend bind address, usually `0.0.0.0` when serving containers |
| `PORT` | Backend port, `8005` for the current memory-copy setup |
| `DATABASE_HOSTNAME` | Main PostgreSQL host |
| `DATABASE_PORT` | Main PostgreSQL port |
| `DATABASE_CREDENTIAL_USER` | Main PostgreSQL user |
| `DATABASE_CREDENTIAL_PASSWORD` | Main PostgreSQL password; keep secret |
| `DATABASE_NAME` | Main application database |
| `COLLECTOR_HOSTNAME` | Collector database host |
| `COLLECTOR_PORT` | Collector database port |
| `COLLECTOR_CREDENTIAL_USER` | Collector database user |
| `COLLECTOR_CREDENTIAL_PASSWORD` | Collector database password; keep secret |
| `COLLECTOR_NAME` | Collector database name |
| `PFI_API_URL` | PFI HTTP API base URL |
| `RP_API_URL` | Reliability HTTP API base URL |
| `MODEL_SERVER_URL` | OpenAI-compatible model server base URL |
| `MODEL_SERVER_API_KEY` | Model-service credential |
| `PLANT_STATUS_URL` | Plant-status endpoint |
| `ENABLE_BACKGROUND_ANALYSIS` | Enables background RCFA analysis behavior |
| `ENABLE_DEBUG_EXPORT` | Enables local debug snapshots; keep off in production |

The active/default model is stored in the `rcfa_model_config` table. Do not assume a model is selected from Python defaults alone; verify the database configuration used by the running backend.

## Main chat request flow

```text
1. Frontend sends POST /chat/stream.
2. JWT authentication identifies the user.
3. router.py validates and converts the request into ChatRequest.
4. stream2.py loads history, RCFA state, memories and documents.
5. stream2.py creates the available technical tools.
6. The configured LLM adapter calls remote Gemma.
7. Gemma may request a tool.
8. The backend executes the tool and returns its result to Gemma.
9. The final answer is streamed back as events.
10. Chat history and approved explicit memories are saved.
```

The main files are:

- [`src/ircfa_interaction/router.py`](src/ircfa_interaction/router.py) — HTTP chat routes;
- [`src/ircfa_interaction/stream2.py`](src/ircfa_interaction/stream2.py) — orchestration and streaming;
- [`src/ircfa_interaction/llm/gemma_llamacpp.py`](src/ircfa_interaction/llm/gemma_llamacpp.py) — remote model adapter;
- [`src/ircfa_interaction/tools/`](src/ircfa_interaction/tools) — controlled technical tools;
- [`src/ircfa_interaction/memory_policy.py`](src/ircfa_interaction/memory_policy.py) — explicit-only memory policy;
- [`src/ircfa_interaction/memory_lifecycle.py`](src/ircfa_interaction/memory_lifecycle.py) — memory expiry, archive, pinning and limits.

## Debugging and logs

The backend logs incoming requests, completion status, model activity, tool calls and failures. For the running manual backend, keep the terminal open and watch the logs while reproducing a problem.

To enable detailed debug snapshots in a development environment:

```bash
export ENABLE_DEBUG_EXPORT=true
```

Debug snapshots are written under the backend's debug output directory and may include:

- conversation metadata;
- LLM configuration;
- input/output messages;
- function-call results;
- retrieved context documents.

Treat these snapshots as sensitive because they may contain user questions, equipment information, documents, or model context. Keep `ENABLE_DEBUG_EXPORT=false` in production.

For a `503 Service Unavailable`, inspect the request in this order:

```text
1. Browser: did the request leave the chat screen?
2. Nginx: did /ircfa forward to the backend?
3. Backend: did /healthcheck and /chat/stream respond?
4. Tool/API: did PFI or reliability calls respond?
5. Remote Gemma: did /v1/chat/completions respond?
6. Database: did chat/memory saving commit?
```

A remote Gemma 503 usually means the remote model service was overloaded, restarting, timing out, or temporarily unavailable. It does not automatically mean the PFI database or memory system failed.

## Testing and validation

Useful checks:

```bash
cd be-ircfa

# Syntax check selected backend files
poetry run python -m py_compile \
  src/main.py \
  src/api.py \
  src/ircfa_interaction/router.py \
  src/ircfa_interaction/stream2.py

# Unit tests, when pytest is installed in the Poetry environment
make test
```

The repository also contains integration and manual tests under `tests/`, plus exploratory scripts under `scratch/`.

## Safety notes for this learning setup

- Keep using the established remote Gemma service for i-RCFA.
- Do not start local Ollama, llama.cpp or vLLM for the normal chat path.
- Do not modify `/home/nvidia/dev-rcfa`; it is the baseline application.
- Do not use `/home/nvidia/hermes-webui/pltu_agent_zero` for i-RCFA changes.
- The memory-copy backend currently may reuse the baseline `.env` and database. Create an isolated database before serious memory testing.
- The RAG path expects the embedding model `dengcao/Qwen3-Embedding-8B:Q8_0`; normal remote-LLM chat can work even when document retrieval is unavailable.
- Never commit `.env` files, API keys, passwords, tokens, or generated debug exports.