Web search restrictions after internal content (#1074)

Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>

documentation/Tools.md

@@ -153,9 +153,9 @@ The service reads HTML pages and text documents. An HTML page has its main conte
 
 What differs between callers is which targets are acceptable, and that follows from who chose the URL. `web_search` uses the public-only policy and never reads private, loopback, or link-local targets. `read_web_page` may reach an explicitly allowed private host, and only for a High-confidence provider. The `ReadWebContent` component sets `TargetChosenByUser`, which lifts the target restrictions entirely: the user typed the address, so their own network and a local server are legitimate. Never set that flag for a URL that reached AI Studio through a model.
 
-`read_web_page` remains the independent single-URL tool and may use its configured private-host allowlist and operating-system sign-in behavior for allowed HTTPS targets. An allowed private host can only be read by a High-confidence provider.
+`read_web_page` remains the independent single-URL tool and may use its configured private-host allowlist and operating-system sign-in behavior for allowed HTTPS targets. An allowed private host can only be read by a High-confidence provider. A page read from a private address, and every page of the configured wiki, is internal and restricts where the chat may send data afterwards, see "Outbound Data Restrictions" below.
 
-`search_confluence` builds a CQL query for the configured HTTPS Confluence Data Center site's `dosearchsite.action` page and loads it through `WebPageRetrievalService`, the same reader used by `read_web_page`. The model supplies a search phrase and optionally a space key, never a URL or CQL expression. The query uses the `siteSearch` field, which Confluence's own search box sends: it finds pages holding any of the words and ranks them by relevance, while the documented `text` field requires all of them and misses pages the search box finds. The tool returns the extracted search page as Markdown with links, after truncation and prompt-injection filtering, and lists the search page as its source. Every request, redirects included, must stay within the configured base URL; `WebPageRetrievalOptions.IsTargetAllowed` refuses a redirect before it is followed, so the query never reaches another host. The operating-system sign-in goes to the configured host only when all its addresses are private, the same rule `read_web_page` follows, and a redirect to Confluence's login page is reported as a missing sign-in instead of an empty search. The tool is offered to High-confidence providers only and checks that again before each search, because a lowered tool setting must not let internal wiki content reach a less trusted provider; the result raises the chat's continuing confidence requirement to High. Selecting `search_confluence` also selects `read_web_page` so the model can load a result's full content; the latter tool's private-host allowlist and other availability rules still apply, so a wiki with a private address has to be in that allowlist before any result opens.
+`search_confluence` builds a CQL query for the configured HTTPS Confluence Data Center site's `dosearchsite.action` page and loads it through `WebPageRetrievalService`, the same reader used by `read_web_page`. The model supplies a search phrase and optionally a space key, never a URL or CQL expression. The query uses the `siteSearch` field, which Confluence's own search box sends: it finds pages holding any of the words and ranks them by relevance, while the documented `text` field requires all of them and misses pages the search box finds. The tool returns the extracted search page as Markdown with links, after truncation and prompt-injection filtering, and lists the search page as its source. Every request, redirects included, must stay within the configured base URL; `WebPageRetrievalOptions.IsTargetAllowed` refuses a redirect before it is followed, so the query never reaches another host. The operating-system sign-in goes to the configured host only when all its addresses are private, the same rule `read_web_page` follows, and a redirect to Confluence's login page is reported as a missing sign-in instead of an empty search. The tool is offered to High-confidence providers only and checks that again before each search, because a lowered tool setting must not let internal wiki content reach a less trusted provider; the result raises the chat's continuing confidence requirement to High. Selecting `search_confluence` also selects `read_web_page` so the model can load a result's full content; the latter tool's private-host allowlist and other availability rules still apply, so a wiki with a private address has to be in that allowlist before any result opens. A successful search restricts where the chat may send data afterwards, at the level `read_web_page` applies to internal pages, see "Outbound Data Restrictions" below.
 
 Confluence Cloud is not supported yet. It offers neither `dosearchsite.action` as a server-rendered page nor the operating-system sign-in; its search needs Confluence's REST API with an API token instead, which is also the way to stop depending on the HTML of the Data Center search page.
 
@@ -173,14 +173,35 @@ Links in a tool result count as given with both values, a link on a page read be
 
 The instructions depend on the setting, so `read_web_page` words them per request through `ResolveSystemPromptInstructionsAsync`. Its registered instructions are those of `OFF`, and the token count below the message field counts with them. With `ON`, a request carries a shorter instruction, and the count comes out a few tokens high.
 
-### After Reading Mails
+## Outbound Data Restrictions
 
-`read_web_page` contacts addresses the model chooses, so a chat which read from a mailbox would keep it from running. It keeps to the outbound data restriction itself instead, see `ReadWebPageTool.IsAllowedByOutboundDataRestriction`:
+Once internal content is in a chat, every tool which runs afterwards could carry it out: a web search in its query, `read_web_page` in the address it reads. The chat therefore records where it may still send data, `ChatThread.RequiredOutboundDataRestriction`, an `OutboundDataRequirement` with a level and the source which demands it. It only ever gets stricter, it is saved with the chat, and a later change of a setting does not loosen it, because the content was read under the old one. Three sources set it, each through `ToolExecutionResult.RequiredOutboundDataRestriction`, and only on a result which brought content in:
 
-- `ONLY_LINKS_FROM_CHAT` allows the addresses given to the model, by the same rule as the free address choice, and the pages of the Confluence wiki configured for `search_confluence`.
-- `ONLY_CONFIGURED_SERVICES` allows the pages of that wiki only. Without a configured wiki, the tool offers nothing on this level.
+- A mailbox, at its own level, see "Searching Mailboxes" below.
+- A search of the Confluence wiki with `search_confluence`.
+- An internal page read with `read_web_page`: one read from a non-public address, or one of the configured wiki, whatever its address. Only the page as it was read in the end counts, after every redirect. A public address which redirects into the intranet restricts the chat; an intranet address which redirects to a public page, such as a sign-in page, does not.
 
-A wiki page whose address the model chose has to stay in the wiki, redirects included: the address may carry mail content, and a redirect elsewhere could carry it on. An address given to the model may be redirected anywhere, since whatever the redirect carries came from the server. A call has to pass both the restriction and the free address choice, so with the choice switched off, a wiki page counts only when its address was given to the model, e.g., as a hit of a wiki search. The instructions of the tool name what is left on the level of the chat, and the refusal never repeats the address.
+The wiki and the internal pages share one setting, `read_web_page.internalPageOutboundDataRestriction`, which defaults to `ONLY_LINKS_FROM_CHAT`. The search loads a wiki page through the same reader, and the model opens its hits with `read_web_page` right after, so one setting gives the chat one level, and an organization one setting to lock. The tool blocked in the tool selection, and a call which was turned down, name the source: the mailbox, the wiki, or the host of the page, see `OutboundDataRequirementExtensions.GetToolBlockedMessage`.
+
+What each level allows, see `ToolSelectionRules.IsOutboundDataAllowed`:
+
+- `UNRESTRICTED` allows every tool.
+- `ONLY_LINKS_FROM_CHAT` keeps every tool from running which sends queries to a third party, such as `web_search`. `read_web_page` still reads the addresses given to the model, by the same rule as the free address choice, and the services configured in AI Studio.
+- `ONLY_CONFIGURED_SERVICES` allows the services configured in AI Studio only, not even an address the user wrote.
+
+The services configured in AI Studio are the mailboxes, the Confluence wiki, and the hosts matching `read_web_page.allowedPrivateHosts`. Without a wiki and without allowed hosts, `read_web_page` offers nothing on the strictest level.
+
+**Why an address counts as sending data.** A model which writes an address can put anything it read into it, in the path or the query, and the server at that address receives it. This is why `read_web_page` declares `ToolOutboundData.MODEL_CHOSEN_ADDRESSES` and keeps to the restriction itself, see `ReadWebPageTool.IsAllowedByOutboundDataRestriction`. An address given in the chat, by the user or in the result of a tool, is read word for word, so it carries nothing the model chose. Where a redirect leads was chosen by the server, so an address given in the chat may be redirected anywhere.
+
+**What an address the model chose may reach.** A wiki page has to stay in the wiki, redirects included, through `WebPageRetrievalOptions.IsTargetAllowed`. A page on an allowed private host has to stay on non-public addresses, on every request, through `WebPageRetrievalOptions.NonPublicTargetsOnly`: a pattern such as `*.example.org` matches public hosts as well, so a matching name is not enough. The wiki is checked first, so a wiki on an allowed host stays confined to the wiki. This holds for chats restricted by a mailbox as well, which may therefore send mail content to an allowed host in an address the model chose, as they already could to the wiki.
+
+A call has to pass both the restriction and the free address choice, so with the choice switched off, a page of the wiki or of an allowed host counts only when its address was given to the model, e.g., as a hit of a wiki search. The instructions of `read_web_page` name what is left on the level of the chat, and a refusal never repeats the address.
+
+**The order of the tools.** A chat which still may search the web loses that as soon as internal content enters it. Only the tool which brings such content in knows when it does, so `search_mails`, `search_confluence`, and `read_web_page` each tell the model so in their own instructions, and tell it to search the web first when a task needs both. `WebSearchFirstRule` words that rule once for all of them; `search_mails` speaks for the whole mailbox collection, which is always offered together. The tools word their instructions per request and leave the rule out when the request does not offer `web_search`, because a rule about a tool the model does not get would only confuse it. They learn this from `ToolResolutionContext.CandidateToolIds`, the tools which passed every check of the request, which `ToolRegistry.GetRunnableToolsAsync` fills before it asks the first tool what it offers. `search_confluence` also leaves the rule out when the level of internal pages is `UNRESTRICTED`, and `read_web_page` also without a wiki and without allowed private hosts, because it cannot reach an internal page then. A mailbox may have any level, so `search_mails` says that reading it may end the web search. Once the search is gone, no instruction mentions it any more: a call of `web_search` in the same request is refused with the text of `OutboundDataRequirementExtensions.GetToolBlockedMessage`, which names the source and says that a new chat can search again, and the tool selection shows the user the same text.
+
+**Residual risk.** On `ONLY_LINKS_FROM_CHAT`, the model cannot write addresses, but it still chooses which of the given links to follow. A crafted page with many links could leak a few bits that way, limited by `ToolSelectionRules.MAX_TOOL_CALLS`. Organizations which do not accept this lock the level to `ONLY_CONFIGURED_SERVICES`.
+
+**Out of scope.** Files the user attaches or pastes are content the user brings in on purpose and restrict nothing. Neither do `semantic_search` and the classic RAG process yet.
 
 ## Searching Data Sources