Fixed Semantic Search leaving out passages that did not fit into a search (#1167)

documentation/Tools.md

@@ -102,6 +102,8 @@ A tool whose results come in pages takes a `page` argument starting at 1 and rep
 
 Paging stays stateless: every call brings its query and its page again. It has to, because tool results do not travel into later turns. `ToolInvocationTrace.Result` is not saved, and the tool conversation of a request is gone once the answer stands. Cap how deep a model may page, since every page fetches its whole window again, and refuse a page beyond the cap with the last page there is in the message.
 
+A page with a budget of characters ends right before the first result which no longer fits, and the next page starts with that result, see `CharacterPaging`. Never skip a long result to make room for shorter ones after it: the gap would sit in the middle of the ranking, and since every page is cut anew, no later page would ever bring the result back. A page is therefore worked out from the first one on, every time, and only a result too long for any page is shortened, then filtered again. `semantic_search` cuts its pages this way, see "Searching Data Sources" below.
+
 ## Tool Collections
 
 Some tools only make sense together. Searching mails without being able to read the ones found gets in the way, and reading them with less trust than the search asks for would protect nothing. Such tools form a collection: people select it as one entry, it needs one minimum provider confidence, and the user or an organization switches it off as one. The model still sees each of its tools on its own and calls each by its name.
@@ -221,15 +223,15 @@ A call has to pass both the restriction and the free address choice, so with the
 
 `semantic_search` lets the model search the data sources of a chat itself, with a query it works out from the conversation, whenever a question calls for it. The classic RAG process, `AISrcSelWithRetCtxVal`, searches them with every message instead, using the message as the query. One place decides which of the two runs, `ToolRegistry.GetEffectiveRetrievalModeAsync`. Semantic search is the default, and the user can choose the other way per chat through `DataSourceOptions.RetrievalMode`. Whenever the tool cannot be offered — a model without tool calling, tools or this tool switched off, a provider below a confidence the organization set for it — the classic process searches instead. That process steps back only when the answer is semantic search, so a chat never ends up searching nothing.
 
-The tool offers the data sources of the chat which the provider may use. With the automatic selection switched on, those are all data sources the provider may use: the AI which selects is the chat model itself. No agent takes part, so `DataSourceService` counts only the chat provider when it decides what may be used, see `DataSourceRetrievalMode`. The description of the tool lists the offered data sources by ID, name, kind, page size, and last page, with their own description shortened to 500 characters. The description of an ERI data source comes from its server, so it goes through the prompt-injection filter before it gets there. The listed IDs are the only values `data_source_ids` accepts.
+The tool offers the data sources of the chat which the provider may use. With the automatic selection switched on, those are all data sources the provider may use: the AI which selects is the chat model itself. No agent takes part, so `DataSourceService` counts only the chat provider when it decides what may be used, see `DataSourceRetrievalMode`. The description of the tool lists the offered data sources by ID, name, and kind, with their own description shortened to 500 characters. It names no page size: how many passages a page holds depends on how long they are, so each result tells whether a further page holds more. The description of an ERI data source comes from its server, so it goes through the prompt-injection filter before it gets there. The listed IDs are the only values `data_source_ids` accepts.
 
-An ERI data source has a single page, see `IDataSource.GetLastPage`, and asks its server for exactly as many matches as the classic process does. ERI v1 knows no offset, so a further page would have to ask for all pages before it as well, and a server which allows only the configured number of matches would refuse it: the model would hear that the data source could not be searched rather than that there is nothing more.
+An ERI data source has a single page of matches, see `IDataSource.GetLastPage`, and asks its server for exactly as many as the classic process does. ERI v1 knows no offset, so a further page would have to ask for all pages before it as well, and a server which allows only the configured number of matches would refuse it: the model would hear that the data source could not be searched rather than that there is nothing more. When these matches do not fit into one search, the model still turns pages within them, and each page asks the server again.
 
 An ERI data source whose server does not answer the check of `DataSourceService` (sign-in or security requirements failing, or slower than six seconds) cannot be offered, since its server's rules are unknown. It is not left out silently either: `AllowedSelectedDataSources.UnreachableDataSources` lists it when its configured security policy allows it for the provider, and the description of the tool names it, without an ID, as not reachable or not signed in to right now. A search of all data sources reports those as not searched, again without an ID. Otherwise, the model would answer from its own knowledge as though the chat had no data sources. When no data source of the chat can be searched at all, the tool stays out of the request: the model would call it all the same, and the call would cost a round of the answer without searching anything. The classic process, which steps back for Semantic Search, then writes the note into the system prompt instead; it asks `SemanticSearchTool.GetOfferedDataSourcesAsync`, so that the note appears exactly when the tool stays out. When it searches itself, the classic process names the same data sources with the same words after the passages it hands the model, also when it found nothing: `AllowedSelectedDataSources.GetUnreachableOfTheChat` decides which ones count for both, and `UnreachableDataSources` writes the note.
 
 Every check of an ERI data source signs in to its server, and one message checks several times: when the tool is offered, before every search, and with every message of the classic process. `DataSourceService` therefore keeps a server's answer for one minute in an `ERISecurityCheckCache` and joins a check which is still running, so that a server which limits sign-ins does not start refusing. A server which did not answer is asked again after 15 seconds, and a changed configuration of the data source asks anew right away. Expired answers are dropped whenever a check starts. The minute holds for an answer which lets the data reach any provider as well, on purpose: when an operator opened a server by mistake and corrects it, every AI Studio follows within that minute, and until then, a chat with a cloud provider may still search the server. The cache lives only as long as the app runs, so no answer sticks for longer.
 
-The data sources are checked again before each search, since rounds may have passed since they were offered, and then searched in parallel. A data source which fails is reported as not searched rather than left out, so that the model does not take its silence for finding nothing. Every passage goes through the same filter and into the same shape as with the classic RAG process, `IRetrievalContextExtensions.PrepareForModelAsync`: all passages of a search in one filter request, within one `PromptInjectionGuardService.BeginAction` per search, so that the user hears about what was filtered once per search, as soon as it is done. The result holds whole passages up to 100,000 characters, and the data sources take turns: first the best passage of each, then the second best of each. Otherwise, the data source listed first would take the whole budget. What does not fit is counted in the result, with a narrower query as the way out. An image an ERI server returns is not passed on, in neither process: written as base64 text, it is nothing a model can look at, and it would take more room than all other passages together. The passage tells the model that an image matched and where it is, so that it can point the user to it. What an ERI server says its content is, its category, is passed on as "Source category". The passages become sources through `IRetrievalContext.ToSources()`, as with the classic process, and only the data sources whose passages reached the model raise the requirements of the chat. The passages which reached the model are also kept with the answer, through `ToolExecutionResult.Passages` and `ContentText.AddSources`: the result of the call is not saved, but the user can still read what the answer rests on after the chat was reloaded. The title of a source names the pages of all its passages, see `SourcePassageExtensions.NamePassagesInTitles`.
+The data sources are checked again before each search, since rounds may have passed since they were offered, and then searched in parallel. A data source which fails is reported as not searched rather than left out, so that the model does not take its silence for finding nothing. Every passage goes through the same filter and into the same shape as with the classic RAG process, `IRetrievalContextExtensions.PrepareForModelAsync`: all passages of a search in one filter request, within one `PromptInjectionGuardService.BeginAction` per search, so that the user hears about what was filtered once per search, as soon as it is done. A page of the search holds passages up to 100,000 characters, measured after the filter, which can make a passage longer. A data source returns its matches in pages of its own, as many as it is set to return, see `RetrievalPaging`; each of them is cut into one or more pages of the search, numbered on across the pages of the data source, see `SemanticSearchTool.SearchAsync`. Where everything fits, both are the same. `has_more` is exact: when a page ends with a page of the data source which might have more, the search looks whether a further one holds any passage. A passage longer than a whole page is shortened, its surrounding content first, and filtered again, see `IRetrievalContextExtensions.ShortenToFitAsync`; its label tells the model, and the user is shown the shorter passage. A search of several data sources shares the first page among them. The data sources take turns: first the best passage of each, then the second best of each, and a data source stops at its first passage which no longer fits. Otherwise, the data source listed first would take the whole budget, or a shorter passage would take the place of a longer one above it. What does not fit is counted in the result, with a search of that data source alone, from page 1, as the way out; that search shows the passages read already once more. An image an ERI server returns is not passed on, in neither process: written as base64 text, it is nothing a model can look at, and it would take more room than all other passages together. The passage tells the model that an image matched and where it is, so that it can point the user to it. What an ERI server says its content is, its category, is passed on as "Source category". The passages become sources through `IRetrievalContext.ToSources()`, as with the classic process, and only the data sources whose passages reached the model raise the requirements of the chat. The passages which reached the model are also kept with the answer, through `ToolExecutionResult.Passages` and `ContentText.AddSources`: the result of the call is not saved, but the user can still read what the answer rests on after the chat was reloaded. The title of a source names the pages of all its passages, see `SourcePassageExtensions.NamePassagesInTitles`.
 
 **Where a chat may send data afterward.** A data source holds the documents of the user or of their organization, so each one decides, as a mailbox does, where a chat may still send data once a search found something in it, `IDataSource.OutboundDataRestriction`, see "Outbound Data Restrictions" above. New data sources start with `ONLY_LINKS_FROM_CHAT`, the level of internal pages, and so do the ones saved before the setting existed. What counts is the effective level, `DataSourceOutboundDataExtensions.GetEffectiveOutboundDataRestriction`: the level of the data source, tightened by the minimum of the organization, `DataSourceSecuritySettings.MinimumOutboundDataRestriction`, and to `ONLY_CONFIGURED_SERVICES` for an ERI data source whose security policy is `SELF_HOSTED`. Its data must not leave the infrastructure of the organization through a tool either, while its links into the intranet stay readable: that level still allows the wiki and the allowed private hosts. The rule holds at runtime, so neither a data source saved earlier nor one a configuration plugin provides has to be edited for it; the dialogs only stop offering the less strict levels.