rag

Knowledge base — chunking, embedding storage (numpy), and retrieval.

KnowledgeBase

Vector-based knowledge base with TF-IDF embeddings and chunked retrieval.

Manages the lifecycle of the RAG knowledge base: building from source files, persisting to disk as a pickle file, loading into memory (with optional mmap support for low-memory environments), and querying via cosine similarity search.

The knowledge base is built from markdown files in processed/, stratcom/, searches/, and party_docs/mera25/ directories, chunked by markdown section and paragraph boundaries, and vectorized using the Embedder class.

__init__(embedder, mmap=False)

Initialize the knowledge base.

Parameters:

Name	Type	Description	Default
embedder	Embedder	Embedder instance used for vectorization. Must be fitted (vocab + idf populated) before querying.	required
mmap	bool	If True, loads vectors via memory-mapped file for reduced RAM usage. Recommended for Render free tier (512 MB RAM). Defaults to False.	False

build()

Build the knowledge base from source markdown files.

Collects all .md files from the KB source directories, chunks them, fits the embedder on the corpus, vectorizes all chunks, and persists everything to vectors.pkl. Also computes and returns a diff against the previous manifest for observability.

Returns:

Type	Description
int	Tuple of (total_chunk_count, diff_dict) where diff_dict
dict	is the result of diff_manifest() comparing old vs new manifests.

load()

Load the knowledge base from the vectors.pkl file.

In normal mode, loads vectors into RAM. In mmap mode, saves vectors to a temporary .npy file and maps them with np.load in read-only mmap mode, then immediately unlinks the temp file (vectors remain accessible via the mmap). Also restores the embedder's vocab and idf from the saved state.

needs_build()

Check whether the knowledge base needs to be built.

Returns:

Type	Description
bool	True if the vectors.pkl file does not exist, False otherwise.

query(query, top_k=TOP_K)

Search the knowledge base for chunks relevant to a query.

Automatically loads the knowledge base if vectors are not yet in memory. Delegates to _query_impl() for the actual cosine similarity computation.

Parameters:

Name	Type	Description	Default
query	str	Search query string.	required
top_k	int	Number of results to return. Defaults to TOP_K config.	TOP_K

Returns:

Type	Description
list[ChunkResult]	List of ChunkResult objects sorted by descending similarity score.

warmup()

Preload and page in the vectors matrix before serving traffic.

In mmap mode the matrix is read from disk page by page on first access; a dummy retrieval touches every block so the first real user query hits a warm page cache instead of slow cold disk. No-op in RAM mode (load() already read everything).

Safe to call at startup; read-only and idempotent.

compute_file_manifest()

Compute a manifest of all source files and their chunk counts.

Reads every markdown file in the KB source directories, chunks them, and records the number of chunks per file.

Returns:

Type	Description
dict[str, int]	Dictionary mapping relative file paths to chunk counts.

diff_manifest(old, new)

Compare two file manifests and produce a diff summary.

Identifies added files, removed files, and files whose chunk count changed between two builds.

Parameters:

Name	Type	Description	Default
old	dict[str, int]	Previous manifest dict mapping file paths to chunk counts.	required
new	dict[str, int]	New manifest dict mapping file paths to chunk counts.	required

Returns:

Type	Description
dict	Dictionary with keys: added, removed, changed (each a dict of
dict	file→count), and summary (with counts of added/removed/changed
dict	files and total chunks added/removed).