You need: a memory system that can ingest ~200 markdown files and answer a question, Python 3.9+, and a GitHub account. No model download is needed for anything except your own system.
git clone https://github.com/tonydzi/sqlite-graph-memory && cd sqlite-graph-memory
cp bench/adapters/template.py bench/adapters/my_system.pyFill in two methods:
setup(vault_dir, work_dir): load every.mdfile inbench/vault/into your system.ask(question): return{"retrieved": [...], "answer": ..., "citations": [...]}.retrievedis your ranked list of note ids. A note id is the filename without.md.answeris a string, orNonewhen the vault does not contain the answer. Returning the string"I don't know"counts as an answer, and it will be scored as a wrong one.citationsare{"note": id, "quote": text}, and the quote has to be copied word for word from that note.
Rename the key in SYSTEMS to the name you want on the board (lowercase, a-z0-9._-).
Do not name the file after your package (adapters/fidelis.py hides the real fidelis
from its own imports); run.py refuses such a file and says so. fidelis_memory.py is fine.
Put everything you set by hand into info()["config"]: model, prompt, top-k, thresholds.
Working adapters by people other than the system's authors, to copy from:
fidelis_memory.py (a library over mem0, local Ollama),
mem0_local.py (mem0 without an API key),
second_brain.py (a project that is not on PyPI, run from a
pinned clone) and sqlite_fts5.py (stdlib only).
Sketches for systems people have asked about. These are untested pointers, not working adapters. Check them against each project's current docs.
| system | in setup |
in ask |
|---|---|---|
| Letta | create an agent, insert each note into archival memory with its id in the text | send the question, read the archival-search tool calls for ids |
| Khoj | point a Khoj instance at bench/vault/ and wait for indexing |
/api/search?q=..., map file paths to ids |
| plain RAG | chunk + embed with your stack | top-k chunks -> ids; an LLM writes the answer and quotes a chunk |
python bench/run.py --system my-system --limit 5 # writes nothing
python bench/run.py --system my-system --submitted-by <your-github-handle> \
--notes "one honest sentence: what you tuned, what is weak"
python bench/leaderboard.py
python bench/validate.pyvalidate.py is exactly what CI runs on your pull request. If it prints VALID locally, CI
agrees with it.
Commit three things: bench/adapters/my_system.py, bench/results/my-system.json and the
regenerated bench/LEADERBOARD.md.
What happens next:
- CI recomputes every metric from your per-question outputs and checks every quote against the vault. Typos in the numbers are impossible because you do not type the numbers.
- Your row is labelled self-reported until it can be rerun in CI. If your adapter needs no API key and no paid service, say so in the PR. We add it to the reproduce job, and the label changes to yes.
- A maintainer answers within 48 hours, including "no, and here is why".
The vault and the questions come from bench/make_vault.py, never from hand edits, and CI
rejects hand edits. To add a question class, extend the generator, rerun it, and rerun the
systems already on the board (python bench/run.py --system <name> for each in-repo
adapter). A new vault hash makes every older result stale on purpose, because numbers from
two different test sets cannot share a table.
Good classes to add, if you want one: entity-name collisions (two people share a first
name; the question gives only that), contradictions with no superseded_by marker, and
questions whose evidence sits in a meeting note instead of the entity's own card.
- Do not tune on the questions. They are the test set. Tune on a vault you generate
yourself with another seed:
build(seed=...)inmake_vault.py. - Report what you ran, including the weak columns. The README of this bench reports ours.
- You keep the copyright to your code. No CLA. MIT, like the rest of the repo.