rag

rag

Knowledge base — chunking, embedding storage (numpy), and retrieval.

KnowledgeBase

Vector-based knowledge base with TF-IDF embeddings and chunked retrieval.

Manages the lifecycle of the RAG knowledge base: building from source files, persisting to disk as a pickle file, loading into memory (with optional mmap support for low-memory environments), and querying via cosine similarity search.

The knowledge base is built from markdown files in processed/ and stratcom/ directories, chunked by markdown section and paragraph boundaries, and vectorized using the Embedder class.

__init__

__init__(embedder: Embedder, mmap: bool = False)

Initialize the knowledge base.

Parameters:

Name	Type	Description	Default
embedder	Embedder	Embedder instance used for vectorization. Must be fitted (vocab + idf populated) before querying.	required
mmap	bool	If True, loads vectors via memory-mapped file for reduced RAM usage. Recommended for Render free tier (512 MB RAM). Defaults to False.	False

needs_build

needs_build() -> bool

Check whether the knowledge base needs to be built.

Returns:

Type	Description
bool	True if the vectors.pkl file does not exist, False otherwise.

build

build() -> tuple[int, dict]

Build the knowledge base from source markdown files.

Collects all .md files from processed/ and stratcom/, chunks them, fits the embedder on the corpus, vectorizes all chunks, and persists everything to vectors.pkl. Also computes and returns a diff against the previous manifest for observability.

Returns:

Type	Description
int	Tuple of (total_chunk_count, diff_dict) where diff_dict
dict	is the result of diff_manifest() comparing old vs new manifests.

load

load()

Load the knowledge base from the vectors.pkl file.

In normal mode, loads vectors into RAM. In mmap mode, saves vectors to a temporary .npy file and maps them with np.load in read-only mmap mode, then immediately unlinks the temp file (vectors remain accessible via the mmap). Also restores the embedder's vocab and idf from the saved state.

query

query(query: str, top_k: int = TOP_K) -> list[ChunkResult]

Search the knowledge base for chunks relevant to a query.

Automatically loads the knowledge base if vectors are not yet in memory. Delegates to _query_impl() for the actual cosine similarity computation.

Parameters:

Name	Type	Description	Default
query	str	Search query string.	required
top_k	int	Number of results to return. Defaults to TOP_K config.	TOP_K

Returns:

Type	Description
list[ChunkResult]	List of ChunkResult objects sorted by descending similarity score.

compute_file_manifest

compute_file_manifest() -> dict[str, int]

Compute a manifest of all source files and their chunk counts.

Reads every markdown file in processed/ and stratcom/, chunks them, and records the number of chunks per file.

Returns:

Type	Description
dict[str, int]	Dictionary mapping relative file paths to chunk counts.

diff_manifest

diff_manifest(
    old: dict[str, int], new: dict[str, int]
) -> dict

Compare two file manifests and produce a diff summary.

Identifies added files, removed files, and files whose chunk count changed between two builds.

Parameters:

Name	Type	Description	Default
old	dict[str, int]	Previous manifest dict mapping file paths to chunk counts.	required
new	dict[str, int]	New manifest dict mapping file paths to chunk counts.	required

Returns:

Type	Description
dict	Dictionary with keys: added, removed, changed (each a dict of
dict	file→count), and summary (with counts of added/removed/changed
dict	files and total chunks added/removed).