using AIStudio.Settings; namespace AIStudio.Tools.Services.Indexing; /// /// Indexes one kind of data source: knows how to find its documents and how to read them. /// /// /// Everything else is the same for every kind and stays out of here: the embedding service queues /// the runs, prepares each one in an IndexedRunContext and reports the progress, and the context /// embeds and stores the documents. A new kind of data source needs nothing but an indexer of its /// own, and tables of its own for whatever it has to remember beyond its documents. /// /// What an indexer does bring is how it notices that a data source changed. Files are watched by /// the file system; another kind may have to look again at an interval. Whether anything is tracked /// at all is not the indexer's to decide: the embedding service starts and stops the tracking. /// internal interface IIndexedSourceIndexer : IDisposable { /// /// Whether this indexer reads the documents of a data source. /// /// The data source. /// True when this indexer is the one for it. public bool Supports(IDataSourceBase dataSource); /// /// Works through the documents of the data source of a run, from finding them to completing the run. /// /// /// A run may stop after its share of the work and leave the rest to another one, so a data source /// which takes hours does not hold up all others for that long. Whether and when that run comes /// is the embedding service's to decide. /// /// The prepared run, whose data source this indexer supports. /// Why the run was started. /// The cancellation token. /// Whether another run has to carry on. public Task ProcessAsync(IndexedRunContext context, DataSourceEmbeddingRefreshMode refreshMode, CancellationToken token); /// /// Keeps track of changes to the given data sources, and of nothing else. /// /// /// Called whenever the configured data sources may have changed, always with all of those this /// indexer supports. A data source which was tracked before and is missing now is no longer /// tracked. /// /// The data sources to track. /// Asks the embedding service for a run of the data source with the given id. public void TrackChanges(IReadOnlyCollection dataSources, Func requestRun); /// /// Stops tracking changes to one data source, and drops what was about to be reported for it. /// /// The id of the data source. public void StopTracking(string dataSourceId); /// /// Stops tracking changes to any data source. /// public void StopTrackingAll(); }