AI-Studio/app/MindWork AI Studio/Tools/Services/Indexing/AGENTS.md

AGENTS.md

These instructions cover the indexing pipeline in app/MindWork AI Studio/Tools/Services/Indexing/ and DataSourceEmbeddingService. They add to the AGENTS.md in the repository root, which applies here as well.

Indexed data sources

The parts:

• IIndexedDataSource (Settings/) - what indexing needs to know about a data source: confidence level, embedding provider and chunk settings. IDataSourceBase is what every data source has. Implement IDataSource on top only when classic RAG, Semantic Search and the agents should see the data source. A data source kept in a list of its own implements IIndexedDataSource alone, and the compiler keeps it out of DataSources.
• IIndexedSourceIndexer - one per kind of data source. Supports claims the data sources, ProcessAsync finds and reads the documents of one run, and TrackChanges / StopTracking notice changes on their own. FileSourceIndexer is the reference: a file system watcher per data source, fingerprints over path, size and write time.
• IndexedRunContext - one prepared run: both stores, the embedding provider, the manifest and the collection. IndexDocumentAsync embeds and stores one document; the cleanup methods remove what a failed attempt left behind.
• EmbeddingDocument - one document: its key, its index row, its display name, and how to read its chunks.
• DocumentRunProgress - counts the documents, records indexed and failed ones in the stores, publishes the status and completes the run.
• TextChunker - cuts text into chunks the embedding provider accepts. Pick one of its strategies; do not write a chunker of your own.

To add a kind of data source:

1. Write its indexer in Tools/Services/Indexing/ and create it in DataSourceEmbeddingService.CreateIndexers, which hands every indexer the same TextChunker.
2. Gate it in IsSupportedIndexedSource, behind a preview feature of its own while it is new.
3. When the data source is not kept in DataSources, add its list to GetConfiguredIndexedSources. Every lookup by id and every pass over all data sources goes through it: the startup hash check, QueueAllInternalDataSourcesAsync and RefreshWatchers.
4. Keep whatever the kind has to remember beyond its documents in tables of its own in the index store, added by an EF Core migration (see app/MindWork AI Studio/Tools/Databases/AGENTS.md).
5. Report every status through DocumentRunProgress, so all rows of the embedding page behave alike.

Rules which are easy to break:

• A document key is not a path. Only files use their full path as the key. Never pass a key through the Path APIs: on Windows, Path.GetFullPath reads a key like mail:… as a file with an alternate data stream.
• Ids and signature are pinned. The formats in IndexedDocumentIds and the embedding signature (DataSourceEmbeddingService.BuildEmbeddingSignature) are fixed by tests, because every stored chunk and every index depends on them. When the metadata stored next to a chunk changes, raise CHUNK_METADATA_VERSION deliberately: that rebuilds every index.
• The service decides when, the indexer decides how. Whether changes are tracked at all depends on the automatic refresh setting and the startup hash check, and only the embedding service decides that.