Split AGENTS.md into area guides so Codex reads it in full again (#1046)

This commit is contained in:
Thorsten Sommer authored and GitHub committed 2026-10-11 11:22:40 +02:00
1 parent 8badadb35f
commit 3a1f586f76
17 files changed
+435 -357

No files matched your search

@@ -0,0 +1,47 @@
# AGENTS.md
These instructions cover the indexing pipeline in `app/MindWork AI Studio/Tools/Services/Indexing/` and
`DataSourceEmbeddingService`. They add to the `AGENTS.md` in the repository root, which applies here as
well.
## Indexed data sources
The parts:
- **`IIndexedDataSource`** (`Settings/`) - what indexing needs to know about a data source: confidence level,
embedding provider and chunk settings. `IDataSourceBase` is what every data source has. Implement
`IDataSource` on top only when classic RAG, Semantic Search and the agents should see the data source. A data
source kept in a list of its own implements `IIndexedDataSource` alone, and the compiler keeps it out of
`DataSources`.
- **`IIndexedSourceIndexer`** - one per kind of data source. `Supports` claims the data sources, `ProcessAsync`
finds and reads the documents of one run, and `TrackChanges` / `StopTracking` notice changes on their own.
`FileSourceIndexer` is the reference: a file system watcher per data source, fingerprints over path, size and
write time.
- **`IndexedRunContext`** - one prepared run: both stores, the embedding provider, the manifest and the
collection. `IndexDocumentAsync` embeds and stores one document; the cleanup methods remove what a failed
attempt left behind.
- **`EmbeddingDocument`** - one document: its key, its index row, its display name, and how to read its chunks.
- **`DocumentRunProgress`** - counts the documents, records indexed and failed ones in the stores, publishes the
status and completes the run.
- **`TextChunker`** - cuts text into chunks the embedding provider accepts. Pick one of its strategies; do not
write a chunker of your own.
To add a kind of data source:
1. Write its indexer in `Tools/Services/Indexing/` and create it in `DataSourceEmbeddingService.CreateIndexers`,
which hands every indexer the same `TextChunker`.
2. Gate it in `IsSupportedIndexedSource`, behind a preview feature of its own while it is new.
3. When the data source is not kept in `DataSources`, add its list to `GetConfiguredIndexedSources`. Every lookup
by id and every pass over all data sources goes through it: the startup hash check,
`QueueAllInternalDataSourcesAsync` and `RefreshWatchers`.
4. Keep whatever the kind has to remember beyond its documents in tables of its own in the index store, added
by an EF Core migration (see `app/MindWork AI Studio/Tools/Databases/AGENTS.md`).
5. Report every status through `DocumentRunProgress`, so all rows of the embedding page behave alike.
Rules which are easy to break:
- **A document key is not a path.** Only files use their full path as the key. Never pass a key through the
`Path` APIs: on Windows, `Path.GetFullPath` reads a key like `mail:…` as a file with an alternate data stream.
- **Ids and signature are pinned.** The formats in `IndexedDocumentIds` and the embedding signature
(`DataSourceEmbeddingService.BuildEmbeddingSignature`) are fixed by tests, because every stored chunk and every
index depends on them. When the metadata stored next to a chunk changes, raise `CHUNK_METADATA_VERSION`
deliberately: that rebuilds every index.
- **The service decides when, the indexer decides how.** Whether changes are tracked at all depends on the
automatic refresh setting and the startup hash check, and only the embedding service decides that.
@@ -0,0 +1 @@
@AGENTS.md