AI-Studio/app/MindWork AI Studio/Tools/Services/Indexing/AGENTS.md

# AGENTS.md

These instructions cover the indexing pipeline in `app/MindWork AI Studio/Tools/Services/Indexing/` and
`DataSourceEmbeddingService`. They add to the `AGENTS.md` in the repository root, which applies here as
well.

## Indexed data sources

The parts:
- **`IIndexedDataSource`** (`Settings/`) - what indexing needs to know about a data source: confidence level,
  embedding provider and chunk settings. `IDataSourceBase` is what every data source has. Implement
  `IDataSource` on top only when classic RAG, Semantic Search and the agents should see the data source. A data
  source kept in a list of its own implements `IIndexedDataSource` alone, and the compiler keeps it out of
  `DataSources`.
- **`IIndexedSourceIndexer`** - one per kind of data source. `Supports` claims the data sources, `ProcessAsync`
  finds and reads the documents of one run, and `TrackChanges` / `StopTracking` notice changes on their own.
  `FileSourceIndexer` is the reference: a file system watcher per data source, fingerprints over path, size and
  write time.
- **`IndexedRunContext`** - one prepared run: both stores, the embedding provider, the manifest, the
  collection, and the scope for reporting prompt injections. `IndexDocumentAsync` embeds and stores one
  document; the cleanup methods remove what a failed attempt left behind.
- **`EmbeddingDocument`** - one document: its key, its index row, its display name, and how to read its chunks.
- **`DocumentRunProgress`** - counts the documents, records indexed and failed ones in the stores, publishes the
  status, and completes or pauses the run.
- **`RunShare`** - how much one run may do before the other data sources get their turn: five minutes, and for
  a mailbox at most 500 mails. A large data source is indexed in steps, one run each. A run which returns
  `MORE_TO_DO` gets its next step in the back lane of `DataSourceRunQueue`, behind every data source which begins
  a pass, so a mailbox waits for one step at most.
- **`IndexingPass`** - the steps of one data source, from its first step until one of them is done. It remembers
  which documents failed during an earlier step, so the next steps do not read them again, and which failures
  were reported to the user already.
- **`TextChunker`** - cuts text into chunks the embedding provider accepts. Pick one of its strategies; do not
  write a chunker of your own.

To add a kind of data source:
1. Write its indexer in `Tools/Services/Indexing/` and create it in `DataSourceEmbeddingService.CreateIndexers`,
   which hands every indexer the same `TextChunker`.
2. Gate it in `IsSupportedIndexedSource`, behind a preview feature of its own while it is new.
3. When the data source is not kept in `DataSources`, add its list to `GetConfiguredIndexedSources`. Every lookup
   by id and every pass over all data sources goes through it: the startup hash check,
   `QueueAllInternalDataSourcesAsync` and `RefreshWatchers`.
4. Keep whatever the kind has to remember beyond its documents in tables of its own in the index store, added
   by an EF Core migration (see `app/MindWork AI Studio/Tools/Databases/AGENTS.md`).
5. Report every status through `DocumentRunProgress`, so all rows of the embedding page behave alike.
6. Work in steps. Give the indexer its `Share`, and ask `DocumentRunProgress.HasDoneItsShare` before every
   document. Leave out the documents which failed during an earlier step of the pass: either ask
   `SkipIfFailedEarlierInPass` per document, or sort them out up front like `FileSourceIndexer.PartitionFiles`
   and count them with `RecordFailedEarlierInPass`. Once the share is done, return what `PauseRunAsync`
   returns: it decides whether the data source carries on in another run, or stops because its embedding
   provider failed for everything. A run which takes more than twice its share without asking once ends up as
   a warning in the log, while every other data source waits.
7. Filter everything the indexer reads: files through `ExtractedFileText.ReadAsync`, any other content through
   `PromptInjectionGuardService.SanitizeAsync`, both with `IndexedRunContext.PromptInjectionScope`. The service
   opens that scope per run, so the user hears about a run once, when it is over. Begin no action of your own
   in the indexer, or the user hears about one run several times.

Rules which are easy to break:
- **A document key is not a path.** Only files use their full path as the key. Never pass a key through the
  `Path` APIs: on Windows, `Path.GetFullPath` reads a key like `mail:…` as a file with an alternate data stream.
- **Ids and signature are pinned.** The formats in `IndexedDocumentIds` and the embedding signature
  (`DataSourceEmbeddingService.BuildEmbeddingSignature`) are fixed by tests, because every stored chunk and every
  index depends on them. When the metadata stored next to a chunk changes, raise `CHUNK_METADATA_VERSION`
  deliberately: that rebuilds every index.
- **The service decides when, the indexer decides how.** Whether changes are tracked at all depends on the
  automatic refresh setting and the startup hash check, and only the embedding service decides that.
- **A pause stores the source hash only when a partial index may be searched.** A stored source hash tells the
  data source selection that a run got through the whole data source (`IsIndexAwaitingRebuild`). Files pass
  `null` to `PauseRunAsync`, so a folder stays out of the selection until its first indexing or its rebuild is
  complete. The mailbox passes its hash, since every mail it indexed can be searched right away.
- **A pass lives only while the last step left more to do.** The service drops the pass of a data source after
  every other outcome and every exception, and when the stored index is reset. A retry the user asks for, and a
  step with other embedding settings, begin a new pass (`IndexingPass.ContinueOrBegin`). A pass kept beyond that
  leaves out documents which would work by now, for example once the embedding provider is reachable again.