AI-Studio/app/MindWork AI Studio/Tools/Services/Indexing/AGENTS.md

# AGENTS.md

These instructions cover the indexing pipeline in `app/MindWork AI Studio/Tools/Services/Indexing/` and
`DataSourceEmbeddingService`. They add to the `AGENTS.md` in the repository root, which applies here as
well.

## Indexed data sources

The parts:
- **`IIndexedDataSource`** (`Settings/`) - what indexing needs to know about a data source: confidence level,
  embedding provider and chunk settings. `IDataSourceBase` is what every data source has. Implement
  `IDataSource` on top only when classic RAG, Semantic Search and the agents should see the data source. A data
  source kept in a list of its own implements `IIndexedDataSource` alone, and the compiler keeps it out of
  `DataSources`.
- **`IIndexedSourceIndexer`** - one per kind of data source. `Supports` claims the data sources, `ProcessAsync`
  finds and reads the documents of one run, and `TrackChanges` / `StopTracking` notice changes on their own.
  `FileSourceIndexer` is the reference: a file system watcher per data source, fingerprints over path, size and
  write time, and once a fingerprint changed, the hash of the content (`FileContentHash`), so a file saved again
  without edits keeps its chunks without being read.
- **`IndexedRunContext`** - one prepared run: both stores, the embedding provider, the manifest, the
  collection, and the scope for reporting prompt injections. `IndexDocumentAsync` embeds one document into a
  new generation, next to the version it had stored; `DocumentRunProgress.RecordDocumentIndexedAsync` completes
  it (`CompleteDocumentAsync`), and `CleanupFailedDocumentAsync` sorts out a failed attempt. A document indexed
  before keeps its stored version whatever the failure, so never remove its manifest entry on a failure. Once
  the embedding provider fails, the attempt asks it nothing more and reads the rest of the document for the
  chunks with a stored vector; a document with nothing stored stops at the failure, since none of its chunks
  could take over a vector. Either way, the attempt stays next to the stored version, so the next one reuses
  its vectors; one which stopped for any other reason goes. Between attempts, a document keeps two generations at
  most: the one its row names, and an attempt which stays for its vectors. A run of the file indexer removes the
  others of every file which did not change since (`IndexedRunContext.RemoveLeftoverGenerationsAsync`), since no
  attempt would ever come for it. A chunk whose text the document had stored before keeps its vector (`ReusableVectors`),
  found by the hash of exactly the text that goes to the embedding provider (`ChunkTextHash`), so hash whatever
  is sent, and send whatever is stored. The new chunks belong to the next generation, which is part of their
  ids (`IndexedDocumentIds.CreateChunkIdSource`), so they never overwrite an old chunk, not even for a mail,
  whose fingerprint never changes.
- **`EmbeddingDocument`** - one document: its key, its index row, its display name, and how to read its chunks.
- **`DocumentRunProgress`** - counts the documents, records indexed and failed ones in the stores, publishes the
  status, and completes or pauses the run.
- **`RunShare`** - how much one run may do before the other data sources get their turn: five minutes, and for
  a mailbox at most 500 mails. A large data source is indexed in steps, one run each. A run which returns
  `MORE_TO_DO` gets its next step in the back lane of `DataSourceRunQueue`, behind every data source which begins
  a pass, so a mailbox waits for one step at most.
- **`PauseBetweenSteps`** - a gap after every step which worked on documents, so that many people can share one
  embedding server. The next run, of whichever data source, waits for a random pause between the bounds of the
  settings (`PauseRange`: 0 to 300 seconds, none by default), less the time which passed since the step ended.
  Not to be confused with the pause of the indexing as a whole (`IndexingPauses`), which a backup takes.
- **`IndexingPass`** - the steps of one data source, from its first step until one of them is done. It remembers
  which documents failed during an earlier step, so the next steps do not read them again, and which failures
  were reported to the user already.
- **`TextChunker`** - cuts text into chunks the embedding provider accepts. Pick one of its strategies; do not
  write a chunker of your own. It also says where each chunk starts and ends (`SourceLocation`: page, sheet and
  row, or line), from the segments the runtime delivered; never read a location back out of the chunk text.
  Only the pages go into the vector store, everything else about a location into the index store alone.

To add a kind of data source:
1. Write its indexer in `Tools/Services/Indexing/` and create it in `DataSourceEmbeddingService.CreateIndexers`,
   which hands every indexer the same `TextChunker`.
2. Gate it in `IsSupportedIndexedSource`, behind a preview feature of its own while it is new.
3. When the data source is not kept in `DataSources`, add its list to `GetConfiguredIndexedSources`. Every lookup
   by id and every pass over all data sources goes through it: the startup hash check,
   `QueueAllInternalDataSourcesAsync` and `RefreshWatchers`.
4. Keep whatever the kind has to remember beyond its documents in tables of its own in the index store, added
   by an EF Core migration (see `app/MindWork AI Studio/Tools/Databases/AGENTS.md`).
5. Report every status through `DocumentRunProgress`, so all rows of the embedding page behave alike.
6. Work in steps. Give the indexer its `Share`, and ask `DocumentRunProgress.HasDoneItsShare` before every
   document. Leave out the documents which failed during an earlier step of the pass: either ask
   `SkipIfFailedEarlierInPass` per document, or sort them out up front like `FileSourceIndexer.PartitionFiles`
   and count them with `RecordFailedEarlierInPass`. Once the share is done, return what `PauseRunAsync`
   returns: it decides whether the data source carries on in another run, or stops because its embedding
   provider failed for everything. A run which takes more than twice its share without asking once ends up as
   a warning in the log, while every other data source waits.
7. Filter everything the indexer reads: files through `ExtractedFileText.ReadAsync`, any other content through
   `PromptInjectionGuardService.SanitizeAsync`, both with `IndexedRunContext.PromptInjectionScope`. The service
   opens that scope per run, so the user hears about a run once, when it is over. Begin no action of your own
   in the indexer, or the user hears about one run several times.

Rules which are easy to break:
- **A document key is not a path.** Only files use their full path as the key. Never pass a key through the
  `Path` APIs: on Windows, `Path.GetFullPath` reads a key like `mail:…` as a file with an alternate data stream.
- **Ids and signature are pinned.** The formats in `IndexedDocumentIds` and the embedding signature
  (`DataSourceEmbeddingService.BuildEmbeddingSignature`) are fixed by tests, because every stored chunk and every
  index depends on them. `CHUNK_METADATA_VERSION` and `MAIL_TEXT_VERSION` are frozen: raising one rebuilds every
  index it is part of, so raise it only when every stored vector is wrong. A change to how documents are read, or
  to what is stored next to a chunk, raises neither. A new value which old chunks may lack without being wrong,
  such as the end page of a chunk, reads as unknown, and a chunk gains it once its document is indexed again.
- **A search finds the generation the row of a document names, and nothing else.** A row whose chunk count is
  zero names none yet. The keyword search joins on it (`SqliteIndexStoreClientImplementation.MatchChunks`), and
  the vector search leaves out `IndexStoreClient.GetHiddenChunksAsync`, which the runtime matches against the
  `generation` of every point. A new way to search the index, or to list chunks for a search, follows the same
  rule, or it finds failed attempts and documents which are not complete. Write a chunk into the index store
  before its point into the vector store, and remove the point before the chunk: a point the index store does
  not know cannot be hidden.
- **The service decides when, the indexer decides how.** Whether changes are tracked at all depends on the
  automatic refresh setting and the startup hash check, and only the embedding service decides that.
- **A pause stores the source hash only when a partial index may be searched.** A stored source hash tells the
  data source selection that a run got through the whole data source (`IsIndexAwaitingRebuild`). Files pass
  `null` to `PauseRunAsync`, so a folder stays out of the selection until its first indexing or its rebuild is
  complete. The mailbox passes its hash, since every mail it indexed can be searched right away.
- **Every run passes the pause.** A backup pauses the indexing with `PauseIndexingAsync` while it copies the
  stores. Runs wait for the end of a pause in `ProcessDataSourceRunAsync`, in front of the run gate, so start
  every run there. A run which starts any other way writes to the stores while they are copied, and the backup
  holds two stores which do not fit together. Discarding the stored index of a data source, when it is deleted
  or repaired, waits for the end of a pause as well (`WaitWhilePausedAsync`); the user interface asks
  `IsIndexingPaused` first and tells the user instead of waiting.
- **The pause between steps lies behind the run gate.** `ProcessDataSourceRunAsync` waits for it right after
  `runGate.WaitAsync`, before the run resolves anything: only one run waits, its token ends the wait, and its
  data source still shows as queued. Never wait for it in an indexer, where the run would hold its signature and
  its stores, and show as running. Afterward, the run looks up its data source anew, since the user may have
  changed it meanwhile. A step opens a gap only once it asked `HasDoneItsShare` (`IndexedRunContext.ShareStartedUtc`),
  so a run which found nothing to read leaves none, and a startup check over unchanged data sources does not
  slow down.
- **A pass lives only while the last step left more to do.** The service drops the pass of a data source after
  every other outcome and every exception, and when the stored index is reset. A retry the user asks for, and a
  step with other embedding settings or another selection of documents, begin a new pass
  (`IndexingPass.ContinueOrBegin`). A pass kept beyond that
  leaves out documents which would work by now, for example once the embedding provider is reachable again.