Outbound data restrictions after searching data sources (#1106)

documentation/Tools.md

@@ -175,13 +175,14 @@ The instructions depend on the setting, so `read_web_page` words them per reques
 
 ## Outbound Data Restrictions
 
-Once internal content is in a chat, every tool which runs afterwards could carry it out: a web search in its query, `read_web_page` in the address it reads. The chat therefore records where it may still send data, `ChatThread.RequiredOutboundDataRestriction`, an `OutboundDataRequirement` with a level and the source which demands it. It only ever gets stricter, it is saved with the chat, and a later change of a setting does not loosen it, because the content was read under the old one. Three sources set it, each through `ToolExecutionResult.RequiredOutboundDataRestriction`, and only on a result which brought content in:
+Once internal content is in a chat, every tool which runs afterwards could carry it out: a web search in its query, `read_web_page` in the address it reads. The chat therefore records where it may still send data, `ChatThread.RequiredOutboundDataRestriction`, an `OutboundDataRequirement` with a level and the source which demands it. It only ever gets stricter, it is saved with the chat, and a later change of a setting does not loosen it, because the content was read under the old one. Four sources set it, and only when they brought content in. The tools do so through `ToolExecutionResult.RequiredOutboundDataRestriction`:
 
 - A mailbox, at its own level, see "Searching Mailboxes" below.
+- A data source, at its own level, once a search found something in it, by `semantic_search` or by the classic RAG process, which sets it on the chat itself, see "Searching Data Sources" below.
 - A search of the Confluence wiki with `search_confluence`.
 - An internal page read with `read_web_page`: one read from a non-public address, or one of the configured wiki, whatever its address. Only the page as it was read in the end counts, after every redirect. A public address which redirects into the intranet restricts the chat; an intranet address which redirects to a public page, such as a sign-in page, does not.
 
-The wiki and the internal pages share one setting, `read_web_page.internalPageOutboundDataRestriction`, which defaults to `ONLY_LINKS_FROM_CHAT`. The search loads a wiki page through the same reader, and the model opens its hits with `read_web_page` right after, so one setting gives the chat one level, and an organization one setting to lock. The tool blocked in the tool selection, and a call which was turned down, name the source: the mailbox, the wiki, or the host of the page, see `OutboundDataRequirementExtensions.GetToolBlockedMessage`.
+The wiki and the internal pages share one setting, `read_web_page.internalPageOutboundDataRestriction`, which defaults to `ONLY_LINKS_FROM_CHAT`. The search loads a wiki page through the same reader, and the model opens its hits with `read_web_page` right after, so one setting gives the chat one level, and an organization one setting to lock. The tool blocked in the tool selection, and a call which was turned down, name the source: the mailbox, the data source, the wiki, or the host of the page, see `OutboundDataRequirementExtensions.GetToolBlockedMessage`.
 
 What each level allows, see `ToolSelectionRules.IsOutboundDataAllowed`:
 
@@ -189,7 +190,7 @@ What each level allows, see `ToolSelectionRules.IsOutboundDataAllowed`:
 - `ONLY_LINKS_FROM_CHAT` keeps every tool from running which sends queries to a third party, such as `web_search`. `read_web_page` still reads the addresses given to the model, by the same rule as the free address choice, and the services configured in AI Studio.
 - `ONLY_CONFIGURED_SERVICES` allows the services configured in AI Studio only, not even an address the user wrote.
 
-The services configured in AI Studio are the mailboxes, the Confluence wiki, and the hosts matching `read_web_page.allowedPrivateHosts`. Without a wiki and without allowed hosts, `read_web_page` offers nothing on the strictest level.
+The services configured in AI Studio are the mailboxes, the data sources, the Confluence wiki, and the hosts matching `read_web_page.allowedPrivateHosts`. Without a wiki and without allowed hosts, `read_web_page` offers nothing on the strictest level.
 
 **Why an address counts as sending data.** A model which writes an address can put anything it read into it, in the path or the query, and the server at that address receives it. This is why `read_web_page` declares `ToolOutboundData.MODEL_CHOSEN_ADDRESSES` and keeps to the restriction itself, see `ReadWebPageTool.IsAllowedByOutboundDataRestriction`. An address given in the chat, by the user or in the result of a tool, is read word for word, so it carries nothing the model chose. Where a redirect leads was chosen by the server, so an address given in the chat may be redirected anywhere.
 
@@ -197,11 +198,11 @@ The services configured in AI Studio are the mailboxes, the Confluence wiki, and
 
 A call has to pass both the restriction and the free address choice, so with the choice switched off, a page of the wiki or of an allowed host counts only when its address was given to the model, e.g., as a hit of a wiki search. The instructions of `read_web_page` name what is left on the level of the chat, and a refusal never repeats the address.
 
-**The order of the tools.** A chat which still may search the web loses that as soon as internal content enters it. Only the tool which brings such content in knows when it does, so `search_mails`, `search_confluence`, and `read_web_page` each tell the model so in their own instructions, and tell it to search the web first when a task needs both. `WebSearchFirstRule` words that rule once for all of them; `search_mails` speaks for the whole mailbox collection, which is always offered together. The tools word their instructions per request and leave the rule out when the request does not offer `web_search`, because a rule about a tool the model does not get would only confuse it. They learn this from `ToolResolutionContext.CandidateToolIds`, the tools which passed every check of the request, which `ToolRegistry.GetRunnableToolsAsync` fills before it asks the first tool what it offers. `search_confluence` also leaves the rule out when the level of internal pages is `UNRESTRICTED`, and `read_web_page` also without a wiki and without allowed private hosts, because it cannot reach an internal page then. A mailbox may have any level, so `search_mails` says that reading it may end the web search. Once the search is gone, no instruction mentions it any more: a call of `web_search` in the same request is refused with the text of `OutboundDataRequirementExtensions.GetToolBlockedMessage`, which names the source and says that a new chat can search again, and the tool selection shows the user the same text.
+**The order of the tools.** A chat which still may search the web loses that as soon as internal content enters it. Only the tool which brings such content in knows when it does, so `search_mails`, `semantic_search`, `search_confluence`, and `read_web_page` each tell the model so in their own instructions, and tell it to search the web first when a task needs both. `WebSearchFirstRule` words that rule once for all of them; `search_mails` speaks for the whole mailbox collection, which is always offered together. The tools word their instructions per request and leave the rule out when the request does not offer `web_search`, because a rule about a tool the model does not get would only confuse it. They learn this from `ToolResolutionContext.CandidateToolIds`, the tools which passed every check of the request, which `ToolRegistry.GetRunnableToolsAsync` fills before it asks the first tool what it offers. `search_confluence` also leaves the rule out when the level of internal pages is `UNRESTRICTED`, and `read_web_page` also without a wiki and without allowed private hosts, because it cannot reach an internal page then. A mailbox may have any level, so `search_mails` says that reading it may end the web search. `semantic_search` leaves the rule out when every data source the chat may search is `UNRESTRICTED`: the ones it preselected, or all of them with the automatic selection. It decides from the settings alone, not from the data sources the checks of the request let through, because the instructions belong to the cached beginning of the request and must not change whenever an ERI server answers or not. The classic RAG process warns nobody: it searches before the model answers, so the web search ends with the first message which finds something in a restricted data source, and the tool selection then tells the user why. Once the search is gone, no instruction mentions it any more: a call of `web_search` in the same request is refused with the text of `OutboundDataRequirementExtensions.GetToolBlockedMessage`, which names the source and says that a new chat can search again, and the tool selection shows the user the same text.
 
 **Residual risk.** On `ONLY_LINKS_FROM_CHAT`, the model cannot write addresses, but it still chooses which of the given links to follow. A crafted page with many links could leak a few bits that way, limited by `ToolSelectionRules.MAX_TOOL_CALLS`. Organizations which do not accept this lock the level to `ONLY_CONFIGURED_SERVICES`.
 
-**Out of scope.** Files the user attaches or pastes are content the user brings in on purpose and restrict nothing. Neither do `semantic_search` and the classic RAG process yet.
+**Out of scope.** Files the user attaches or pastes are content the user brings in on purpose and restrict nothing.
 
 ## Searching Data Sources
 
@@ -217,6 +218,10 @@ Every check of an ERI data source signs in to its server, and one message checks
 
 The data sources are checked again before each search, since rounds may have passed since they were offered, and then searched in parallel. A data source which fails is reported as not searched rather than left out, so that the model does not take its silence for finding nothing. Every passage goes through the same filter and into the same shape as with the classic RAG process, `IRetrievalContext.AsMarkdown`, which gets the scope of one `PromptInjectionGuardService.BeginAction` per search, so that the user hears about what was filtered once per search, as soon as it is done. The result holds whole passages up to 100,000 characters, and the data sources take turns: first the best passage of each, then the second best of each. Otherwise, the data source listed first would take the whole budget. What does not fit is counted in the result, with a narrower query as the way out. An image an ERI server returns is not passed on, in neither process: written as base64 text, it is nothing a model can look at, and it would take more room than all other passages together. The passage tells the model that an image matched and where it is, so that it can point the user to it. What an ERI server says its content is, its category, is passed on as "Source category". The passages become sources through `IRetrievalContext.ToSources()`, as with the classic process, and only the data sources whose passages reached the model raise the requirements of the chat.
 
+**Where a chat may send data afterward.** A data source holds the documents of the user or of their organization, so each one decides, as a mailbox does, where a chat may still send data once a search found something in it, `IDataSource.OutboundDataRestriction`, see "Outbound Data Restrictions" above. New data sources start with `ONLY_LINKS_FROM_CHAT`, the level of internal pages, and so do the ones saved before the setting existed. What counts is the effective level, `DataSourceOutboundDataExtensions.GetEffectiveOutboundDataRestriction`: the level of the data source, tightened by the minimum of the organization, `DataSourceSecuritySettings.MinimumOutboundDataRestriction`, and to `ONLY_CONFIGURED_SERVICES` for an ERI data source whose security policy is `SELF_HOSTED`. Its data must not leave the infrastructure of the organization through a tool either, while its links into the intranet stay readable: that level still allows the wiki and the allowed private hosts. The rule holds at runtime, so neither a data source saved earlier nor one a configuration plugin provides has to be edited for it; the dialogs only stop offering the less strict levels.
+
+Both ways of searching ask one place what their passages demand, `DataSourceRequirements.Of`, with the data sources and the number of passages each one returned. Only the data sources with passages count, and the strictest effective level wins. On a tie, the data source which comes first by number, then by ID, is the one the chat names, so both ways name the same one, whatever order they searched in. The classic process asks right after retrieval, before the validation agent sees the passages, and counts a passage which that agent drops later as well. This keeps to the safe side, and a validation which fails or is canceled cannot skip the restriction. It sets the restriction on the chat before the request puts its tools together, so the restriction holds for the very answer the passages were found for. Provider confidence and data security stay as they were in the classic process, per selected data source and before retrieval, because its agents see the data.
+
 ## Searching Mailboxes
 
 `search_mails`, `read_mail`, and `count_mails` form the collection `mailboxes`, behind the preview `PRE_MAILBOXES_2026` on top of `PRE_RAG_2024`. None of them asks a mail server anything. They read the local index which `MailboxIndexer` keeps, through `MailboxRetrievalService`. Mailboxes are kept in a list of their own, `Data.Mailboxes`, so `semantic_search` and the classic RAG process never see one.