dev-rcfa-memory/be-ircfa/README.md

i-RCFA Backend (be-ircfa)

be-ircfa is the Python FastAPI backend for the i-RCFA (Root Cause Failure Analysis) chat application. It receives authenticated chat requests, prepares conversation and RCFA context, retrieves sensor/reliability/PFI information through controlled tools, calls the configured LLM, streams the response, and saves the conversation.

The current memory-copy application is intended to keep using the established remote Gemma service. Do not start local Ollama, llama.cpp, or vLLM services for the normal i-RCFA chat flow.

What the backend does

Browser
  -> frontend nginx proxy
  -> FastAPI backend
      -> PostgreSQL chat/memory database
      -> collector database session
      -> PFI/reliability HTTP APIs
      -> remote Gemma service
  <- streamed answer

The main AI workflow is in src/ircfa_interaction/stream2.py. The complete architecture walkthrough is in ../BE_IRCFA.md.

Project layout

be-ircfa/
├── run.py                 Local launcher
├── pyproject.toml         Poetry project/dependencies
├── requirements/          Dependency groups
├── Dockerfile             Backend image definition
├── docker-compose.yaml    Legacy/all-in-one container setup
├── docs/                  Design notes and reports
├── scratch/               Experiments and diagnostics
├── tests/                 Tests
└── src/
    ├── main.py            FastAPI application and middleware
    ├── api.py             API route registration
    ├── auth/              Authentication
    ├── database/          Async PostgreSQL sessions
    ├── ircfa_interaction/ Chat, AI, tools, memory and model adapters
    ├── rag/               Document retrieval
    ├── root_cause/        Root-cause data and routes
    ├── failure_mode/      Failure-mode data and routes
    ├── equipments/        Equipment routes and services
    ├── fmea/              FMEA routes and models
    └── ircfa_alert/       Alert routes and services

Prerequisites

You need:

• Python >=3.11,<3.13;
• a working virtual environment with the backend dependencies;
• access to the main PostgreSQL database;
• access to the collector database, if collector-backed features are used;
• access to the PFI and reliability APIs;
• the configured remote Gemma endpoint;
• a backend .env file or equivalent exported environment variables.

The backend reads environment variables in src/config.py. Do not commit .env files or credentials.

Recommended current setup: run the memory-copy backend manually

This is the setup currently used for the memory-enabled comparison copy.

1. Start the backend

Open a terminal and run:

cd /home/nvidia/dev-rcfa-memory/be-ircfa
source /home/nvidia/dev-rcfa/be-ircfa/venv/bin/activate

set -a
source /home/nvidia/dev-rcfa/be-ircfa/.env
set +a

export HOST=0.0.0.0
export PORT=8005
python run.py

Why is the .env path from /home/nvidia/dev-rcfa? The current memory-copy backend setup intentionally reuses the established environment while it is being tested. This also means it may point at the same PostgreSQL database as the baseline application. Isolate the database before serious memory experiments.

run.py starts Uvicorn with:

run.py -> src.main:app

The backend should listen on:

http://0.0.0.0:8005

2. Check backend health

In another terminal:

curl http://127.0.0.1:8005/healthcheck

Expected response:

{"status":"ok"}

3. Run the matching frontend

The memory-copy frontend normally runs in Docker and forwards /ircfa requests to this backend:

cd /home/nvidia/dev-rcfa-memory/fe-digitaltwin-v2

FRONTEND_PORT=8083 \
BACKEND_URL_IRCFA=http://host.docker.internal:8005 \
docker compose up --build

Open the real chat route:

http://127.0.0.1:8083/i-rcfa-app/chat

The request path is:

Browser -> frontend nginx /ircfa proxy -> backend :8005

Alternative local commands

Run with Make

From be-ircfa/:

make install
make run

make install runs poetry install. make run runs python run.py.

The Makefile also defines:

make test

which runs the unit tests under tests/unit/ through Poetry. The environment must have the test dependencies installed.

Run with Poetry directly

cd be-ircfa
poetry install
poetry run python run.py

Run Uvicorn directly

cd be-ircfa
poetry run uvicorn src.main:app --host 0.0.0.0 --port 8005 --reload

Do not run multiple backend processes against the same port.

Docker Compose note

docker-compose.yaml contains an older all-in-one deployment definition. It includes:

• the i-RCFA backend;
• Kong;
• vLLM services;
• llama.cpp services;
• optional local Gemma services.

That file is useful as a deployment/reference artifact, but it is not the recommended command for the current memory-copy learning setup. In particular, do not start the local model services just to run i-RCFA. Keep the backend pointed at the established remote Gemma configuration.

If you intentionally need to inspect the legacy Compose setup, review the file first and verify every environment variable and service before running:

cd be-ircfa
docker compose config
docker compose up --build

Important environment variables

The names below are read by src/config.py:

Variable	Purpose
HOST	Backend bind address, usually 0.0.0.0 when serving containers
PORT	Backend port, 8005 for the current memory-copy setup
DATABASE_HOSTNAME	Main PostgreSQL host
DATABASE_PORT	Main PostgreSQL port
DATABASE_CREDENTIAL_USER	Main PostgreSQL user
DATABASE_CREDENTIAL_PASSWORD	Main PostgreSQL password; keep secret
DATABASE_NAME	Main application database
COLLECTOR_HOSTNAME	Collector database host
COLLECTOR_PORT	Collector database port
COLLECTOR_CREDENTIAL_USER	Collector database user
COLLECTOR_CREDENTIAL_PASSWORD	Collector database password; keep secret
COLLECTOR_NAME	Collector database name
PFI_API_URL	PFI HTTP API base URL
RP_API_URL	Reliability HTTP API base URL
MODEL_SERVER_URL	OpenAI-compatible model server base URL
MODEL_SERVER_API_KEY	Model-service credential
PLANT_STATUS_URL	Plant-status endpoint
ENABLE_BACKGROUND_ANALYSIS	Enables background RCFA analysis behavior
ENABLE_DEBUG_EXPORT	Enables local debug snapshots; keep off in production

The active/default model is stored in the rcfa_model_config table. Do not assume a model is selected from Python defaults alone; verify the database configuration used by the running backend.

Main chat request flow

1. Frontend sends POST /chat/stream.
2. JWT authentication identifies the user.
3. router.py validates and converts the request into ChatRequest.
4. stream2.py loads history, RCFA state, memories and documents.
5. stream2.py creates the available technical tools.
6. The configured LLM adapter calls remote Gemma.
7. Gemma may request a tool.
8. The backend executes the tool and returns its result to Gemma.
9. The final answer is streamed back as events.
10. Chat history and approved explicit memories are saved.

The main files are:

• src/ircfa_interaction/router.py — HTTP chat routes;
• src/ircfa_interaction/stream2.py — orchestration and streaming;
• src/ircfa_interaction/llm/gemma_llamacpp.py — remote model adapter;
• src/ircfa_interaction/tools/ — controlled technical tools;
• src/ircfa_interaction/memory_policy.py — explicit-only memory policy;
• src/ircfa_interaction/memory_lifecycle.py — memory expiry, archive, pinning and limits.

Debugging and logs

The backend logs incoming requests, completion status, model activity, tool calls and failures. For the running manual backend, keep the terminal open and watch the logs while reproducing a problem.

To enable detailed debug snapshots in a development environment:

export ENABLE_DEBUG_EXPORT=true

Debug snapshots are written under the backend's debug output directory and may include:

• conversation metadata;
• LLM configuration;
• input/output messages;
• function-call results;
• retrieved context documents.

Treat these snapshots as sensitive because they may contain user questions, equipment information, documents, or model context. Keep ENABLE_DEBUG_EXPORT=false in production.

For a 503 Service Unavailable, inspect the request in this order:

1. Browser: did the request leave the chat screen?
2. Nginx: did /ircfa forward to the backend?
3. Backend: did /healthcheck and /chat/stream respond?
4. Tool/API: did PFI or reliability calls respond?
5. Remote Gemma: did /v1/chat/completions respond?
6. Database: did chat/memory saving commit?

A remote Gemma 503 usually means the remote model service was overloaded, restarting, timing out, or temporarily unavailable. It does not automatically mean the PFI database or memory system failed.

Testing and validation

Useful checks:

cd be-ircfa

# Syntax check selected backend files
poetry run python -m py_compile \
  src/main.py \
  src/api.py \
  src/ircfa_interaction/router.py \
  src/ircfa_interaction/stream2.py

# Unit tests, when pytest is installed in the Poetry environment
make test

The repository also contains integration and manual tests under tests/, plus exploratory scripts under scratch/.

Safety notes for this learning setup

• Keep using the established remote Gemma service for i-RCFA.
• Do not start local Ollama, llama.cpp or vLLM for the normal chat path.
• Do not modify /home/nvidia/dev-rcfa; it is the baseline application.
• Do not use /home/nvidia/hermes-webui/pltu_agent_zero for i-RCFA changes.
• The memory-copy backend currently may reuse the baseline .env and database. Create an isolated database before serious memory testing.
• The RAG path expects the embedding model dengcao/Qwen3-Embedding-8B:Q8_0; normal remote-LLM chat can work even when document retrieval is unavailable.
• Never commit .env files, API keys, passwords, tokens, or generated debug exports.