mirror of
https://github.com/MindWorkAI/AI-Studio.git
synced 2026-10-13 09:13:48 +00:00
Split AGENTS.md into area guides so Codex reads it in full again (#1046)
This commit is contained in:
1 parent
8badadb35f
commit
3a1f586f76
17 files changed
+435
-357
No files matched your search
@@ -0,0 +1,47 @@
|
||||
# AGENTS.md
|
||||
|
||||
These instructions cover the indexing pipeline in `app/MindWork AI Studio/Tools/Services/Indexing/` and
|
||||
`DataSourceEmbeddingService`. They add to the `AGENTS.md` in the repository root, which applies here as
|
||||
well.
|
||||
|
||||
## Indexed data sources
|
||||
|
||||
The parts:
|
||||
- **`IIndexedDataSource`** (`Settings/`) - what indexing needs to know about a data source: confidence level,
|
||||
embedding provider and chunk settings. `IDataSourceBase` is what every data source has. Implement
|
||||
`IDataSource` on top only when classic RAG, Semantic Search and the agents should see the data source. A data
|
||||
source kept in a list of its own implements `IIndexedDataSource` alone, and the compiler keeps it out of
|
||||
`DataSources`.
|
||||
- **`IIndexedSourceIndexer`** - one per kind of data source. `Supports` claims the data sources, `ProcessAsync`
|
||||
finds and reads the documents of one run, and `TrackChanges` / `StopTracking` notice changes on their own.
|
||||
`FileSourceIndexer` is the reference: a file system watcher per data source, fingerprints over path, size and
|
||||
write time.
|
||||
- **`IndexedRunContext`** - one prepared run: both stores, the embedding provider, the manifest and the
|
||||
collection. `IndexDocumentAsync` embeds and stores one document; the cleanup methods remove what a failed
|
||||
attempt left behind.
|
||||
- **`EmbeddingDocument`** - one document: its key, its index row, its display name, and how to read its chunks.
|
||||
- **`DocumentRunProgress`** - counts the documents, records indexed and failed ones in the stores, publishes the
|
||||
status and completes the run.
|
||||
- **`TextChunker`** - cuts text into chunks the embedding provider accepts. Pick one of its strategies; do not
|
||||
write a chunker of your own.
|
||||
|
||||
To add a kind of data source:
|
||||
1. Write its indexer in `Tools/Services/Indexing/` and create it in `DataSourceEmbeddingService.CreateIndexers`,
|
||||
which hands every indexer the same `TextChunker`.
|
||||
2. Gate it in `IsSupportedIndexedSource`, behind a preview feature of its own while it is new.
|
||||
3. When the data source is not kept in `DataSources`, add its list to `GetConfiguredIndexedSources`. Every lookup
|
||||
by id and every pass over all data sources goes through it: the startup hash check,
|
||||
`QueueAllInternalDataSourcesAsync` and `RefreshWatchers`.
|
||||
4. Keep whatever the kind has to remember beyond its documents in tables of its own in the index store, added
|
||||
by an EF Core migration (see `app/MindWork AI Studio/Tools/Databases/AGENTS.md`).
|
||||
5. Report every status through `DocumentRunProgress`, so all rows of the embedding page behave alike.
|
||||
|
||||
Rules which are easy to break:
|
||||
- **A document key is not a path.** Only files use their full path as the key. Never pass a key through the
|
||||
`Path` APIs: on Windows, `Path.GetFullPath` reads a key like `mail:…` as a file with an alternate data stream.
|
||||
- **Ids and signature are pinned.** The formats in `IndexedDocumentIds` and the embedding signature
|
||||
(`DataSourceEmbeddingService.BuildEmbeddingSignature`) are fixed by tests, because every stored chunk and every
|
||||
index depends on them. When the metadata stored next to a chunk changes, raise `CHUNK_METADATA_VERSION`
|
||||
deliberately: that rebuilds every index.
|
||||
- **The service decides when, the indexer decides how.** Whether changes are tracked at all depends on the
|
||||
automatic refresh setting and the startup hash check, and only the embedding service decides that.
|
||||
@@ -0,0 +1 @@
|
||||
@AGENTS.md
|
||||
Reference in new issue
Block a user