Added mailboxes as local data sources (#1022)

documentation/Tools.md

@@ -8,11 +8,13 @@ A tool is a single `IToolImplementation` class in `app/MindWork AI Studio/Tools/
 
 The provider only sees local tools that are
 
+- present in this installation, which a tool of a preview feature only is while the preview is switched on, and
 - available for the current component and
 - selected by the user or defaults, or offering themselves from the context of the chat, and
 - supported by the model and
 - configured correctly and
 - allowed by the provider confidence rules and
+- allowed by the outbound data restriction of the chat and
 - able to offer something in this request.
 
 ## Provider API Shapes
@@ -53,6 +55,8 @@ User-visible names, descriptions, and icons come from the implementation's own m
 
 Use stable lower-case IDs with underscores, and keep `Id`, `ImplementationKey`, and `Function.Name` identical unless there is a clear compatibility reason not to. Give every argument and setting name a constant that the schema and the reading code share: the two then cannot drift apart.
 
+A tool which belongs to a preview feature returns false from `IToolImplementation.IsAvailable` while the preview is switched off. The registry then leaves it out of every list, the settings and the selections included, out of every request, where `CheckToolAsync` reports `NOT_AVAILABLE_HERE`, and out of the token count below the message field. A selection which names it keeps it, so the tool comes back with the preview. The property is asked whenever tools are listed, so keep it cheap; the mail tools ask `MailboxRetrievalService.AreMailboxesEnabled`.
+
 `VisibleIn.AllowedComponents` and `VisibleIn.DeniedComponents` are optional lists of `Components` values; a value outside the enum makes the definition invalid. When both lists are empty, the `Chat` and `Assistants` flags apply. As soon as either list has an entry, the lists replace those flags: an empty allow list starts by allowing every component, a non-empty allow list allows only its entries, and the deny list is applied last and always wins.
 
 Keep `Function.DescriptionForLLM` focused on what the tool does. This value is mapped to the provider's function `description` field and is only shown to the LLM. Put sequencing rules, answer-format guidance, or other behavior instructions in `SystemPromptInstructions`. When runnable tools are selected, their non-empty policy text is combined centrally and appended to the effective system prompt.
@@ -65,11 +69,22 @@ When a tool returns data that future messages must only send to providers at or
 
 `ToolExecutionResult.RequiredDataSecurity` is the other axis: a result from a data source which may only be used with self-hosted providers sets it to `SELF_HOSTED`, and the chat refuses every other provider from then on. Both only ever tighten, see `ChatThread.RequireProviderConfidence` and `ChatThread.RequireDataSecurity`, so raise them for what actually reached the model, not for everything the tool looked at. A search which found nothing brought nothing into the chat.
 
+The third requirement is where the chat may still send data. Mails come from strangers and may carry instructions meant for the AI, so each mailbox decides it, see `OutboundDataRestriction`. A tool which brings content of a mailbox into the chat sets `ToolExecutionResult.RequiredOutboundDataRestriction`, and the chat keeps the strictest level reached, together with the mailbox which demanded it, see `ChatThread.RequireOutboundDataRestriction`. Every tool therefore declares where its arguments go in `IToolImplementation.OutboundData`:
+
+| Value | Where the data goes |
+|---|---|
+| `NONE` | Nowhere beyond AI Studio and the provider of the model. |
+| `CONFIGURED_SERVICE` | To a service configured in AI Studio, such as the wiki of the organization, an ERI server, or the embedding provider of a data source. |
+| `THIRD_PARTY_QUERIES` | Queries the model writes go to a service somebody else runs, such as a web search engine. |
+| `MODEL_CHOSEN_ADDRESSES` | The tool contacts addresses the model chooses; the address alone can carry data out. A tool which says nothing counts as this, the most open kind. |
+
+Below `UNRESTRICTED`, only the first two may run, see `ToolSelectionRules.IsOutboundDataAllowed`. The registry does not offer the others (`ToolOfferBlockReason.OUTBOUND_DATA_RESTRICTED`), and `ToolExecutor` checks again before each call, because a mail tool can tighten the chat in the middle of a request. A tool which can tell allowed destinations from others on its own sets `EnforcesOutboundDataRestriction`. It is then offered on every level and has to refuse what goes too far on every call, as `read_web_page` does, see below.
+
 A result in `JsonContent` reaches the model the way `ToolExecutionResult.ToModelContent` writes it, which escapes only what JSON requires. Umlauts and other characters outside ASCII stay as they are rather than costing six characters each.
 
 ## Tools Which Offer Themselves
 
-Most tools are selected: by the user, by the defaults of a component, by a chat template, or by the rules of an assistant. A tool whose use follows from the chat instead sets `Activation = ToolActivation.CONTEXT`. Nobody can select such a tool, so it appears in no selection. `ToolRegistry.GetCatalogAsync` leaves it out of every list built for a component, and `ToolSelectionRules.NormalizeSelection` drops it from a selection which names it anyway, such as the one of a chat template. The tool list of the app settings still shows it, so that an organization can switch it off or raise the confidence it requires. `semantic_search` is the only such tool so far.
+Most tools are selected: by the user, by the defaults of a component, by a chat template, or by the rules of an assistant. A tool whose use follows from the chat instead sets `Activation = ToolActivation.CONTEXT`. Nobody can select such a tool, so it appears in no selection. `ToolRegistry.GetCatalogAsync` leaves it out of every list built for a component, and `ToolRegistry.NormalizeSelection` drops it from a selection which names it anyway, such as the one of a chat template. The tool list of the app settings still shows it, so that an organization can switch it off or raise the confidence it requires. `semantic_search` is the only such tool so far.
 
 The registry takes every context tool of the component as a candidate and runs it through the same checks as a selected one. A tool which passes them is then asked what it offers in this request, through `IToolImplementation.ResolveFunctionAsync`. Most tools leave that method alone and offer the function they registered. A tool which has to know the chat first returns a function tailored to it, or null when it has nothing to offer, and is then left out of the request. Only the description and the parameters of the answer count; the name and the strict mode stay as registered, because the calls of the model find their tool by its name. A resolution which throws costs that one tool and no other.
 
@@ -86,6 +101,22 @@ A tool whose results come in pages takes a `page` argument starting at 1 and rep
 
 Paging stays stateless: every call brings its query and its page again. It has to, because tool results do not travel into later turns. `ToolInvocationTrace.Result` is not saved, and the tool conversation of a request is gone once the answer stands. Cap how deep a model may page, since every page fetches its whole window again, and refuse a page beyond the cap with the last page there is in the message.
 
+## Tool Collections
+
+Some tools only make sense together. Searching mails without being able to read the ones found gets in the way, and reading them with less trust than the search asks for would protect nothing. Such tools form a collection: people select it as one entry, it needs one minimum provider confidence, and an organization switches it off as one. The model still sees each of its tools on its own and calls each by its name.
+
+A collection is an `IToolCollection` class next to its tools, registered in `Program.cs`. It states its `ToolCollectionDefinition`: its ID, the IDs of its tools in the order they are listed, its minimum provider confidence, and a description for a model which picks the tools of an assistant. Its name, description, and icon come from its own members, so they can be translated. Whether it exists right now follows from its tools: it disappears with the last of them, e.g., while their preview is switched off. `ToolRegistry` registers collections after the tools and leaves out what it cannot accept: a collection taking the ID of a tool, a tool which is not registered or which offers itself from the context of a chat, and a tool another collection claimed first.
+
+Every tool belongs to exactly one collection. A tool which belongs to no declared collection forms one of its own under its own ID. That is why the settings which used to name tools need no migration: `DisabledToolIds`, `MinimumProviderConfidenceByToolId`, the defaults of the components, chat templates, document analysis policies, and assistant plugins all name collections now, and the ID of such a tool is the ID of its collection. The ID of a tool in a declared collection stands for its whole collection wherever it appears. A selection stored before the tool joined the collection selects the collection, and an organization which names one tool of a collection switches the whole collection off or raises its confidence. Of several levels set for a collection and its tools, the highest applies, and the minimum a tool declares itself does not count once it belongs to a collection.
+
+Three methods of the registry translate between both views:
+
+- `ToolRegistry.NormalizeSelection` turns a selection into the collections which run. Every place which shows or stores a selection calls it.
+- `ToolRegistry.ExpandSelection` turns a selection into the tools which run. Preparing a request, counting its tokens, the security card of an assistant plugin, and its audit use it, because they are about what the model reads.
+- `ToolRegistry.GetCollectionId` names the collection of a tool.
+
+The settings of a tool stay with the tool, also inside a collection. An organization addresses them by `"<toolId>.<fieldName>"`, and the settings dialog of a collection shows one section per tool. `mailboxes` is the only declared collection so far, with `search_mails`, `read_mail`, and `count_mails`.
+
 ## Security
 
 Treat model-provided tool arguments as untrusted input. Refuse a wrong one rather than guessing what it meant: a placeholder such as `0` is not a page, and reading it as "no page" does something the model did not ask for. The model reads the refusal and tries again, so the message has to name the argument and the value that arrived, say what would be valid, and, for an optional argument, that leaving it out is always possible. `ToolArgumentReader` reads strings, positive integers, and values out of a fixed choice, alone or as a list, and words the refusals so; `WebSearchTool` shows how a tool uses it.
@@ -134,12 +165,23 @@ Every successfully retrieved page with readable content is also returned as a st
 
 `read_web_page.freeAddressChoice` decides whether the model may read addresses it chose itself. `OFF`, the default, tells the model to read only URLs which appear word for word in the conversation: in the system prompt, in a user message with the documents and data source content it carries, or in a tool result. When none fits and no other tool can find one, the model asks the user. `ON` lets it choose addresses as well. The values are the members of `FreeAddressChoice`, offered through `ToolSettingsOptionSources.FREE_ADDRESS_CHOICE`, and the setting follows the usual precedence of tool settings: a locked organization value, then the user's saved value, then an organization default.
 
-Both values are instructions to the model, not a technical check of where a URL came from. Such a check would have to know every way an address reaches the model: attachments are read from disk only when a message is sent, pages link relatively, and servers redirect, so a URL the model reads correctly may still match no spelling in the conversation. A technical check is left for a change of its own. What the application enforces is the same either way: the network target restrictions of `WebPageRetrievalService` and the prompt-injection filter.
+`OFF` is enforced as well: `read_web_page` refuses an address which was not given to the model, see `ChatThread.IsWebAddressGivenToTheModel`. Given means that the address stands in the system prompt the last request was sent with, in a user message or a document attached to it, or in the result of a tool. What the model wrote itself never counts, its earlier answers included. Each source is collected where its text exists in full: the system prompt in `PrepareSystemPrompt`, the attached documents in `ContentText.PrepareTextContentForAI`, since they are read from disk only when a message is sent, and the tool results in `ToolExecutor`. Relative links need no care of their own, because the page extraction makes every link absolute before the model sees it. Two addresses count as the same when they ask the server for the same, see `WebAddresses.CreateRequestKey`: scheme and host regardless of case, path and query exactly, the fragment not at all. A redirect may go anywhere, since the server rather than the model chose it.
+
+An address in a tool result counts only when it is no echo of the call: one which stands in the arguments, even as a part of one, is left out, because Semantic Search returns its query, and the model could otherwise turn any address it makes up into one a tool returned. The tool results are kept for the session only, like the results themselves, so after a restart a chat knows fewer addresses, never more. A refusal never repeats the address, for the same reason. On top of this, the network target restrictions of `WebPageRetrievalService` and the prompt-injection filter apply with both values.
 
 Links in a tool result count as given with both values, a link on a page read before included. Searching and then reading what was found is what these tools are for, and `search_confluence` opens its hits that way. Before this setting existed, the instructions forbade following a link which only retrieved content mentioned; that rule was dropped on purpose. Following a link word for word cannot carry anything out of the conversation. Putting parts of the conversation into an address could, so the instructions forbid that with both values.
 
 The instructions depend on the setting, so `read_web_page` words them per request through `ResolveSystemPromptInstructionsAsync`. Its registered instructions are those of `OFF`, and the token count below the message field counts with them. With `ON`, a request carries a shorter instruction, and the count comes out a few tokens high.
 
+### After Reading Mails
+
+`read_web_page` contacts addresses the model chooses, so a chat which read from a mailbox would keep it from running. It keeps to the outbound data restriction itself instead, see `ReadWebPageTool.IsAllowedByOutboundDataRestriction`:
+
+- `ONLY_LINKS_FROM_CHAT` allows the addresses given to the model, by the same rule as the free address choice, and the pages of the Confluence wiki configured for `search_confluence`.
+- `ONLY_CONFIGURED_SERVICES` allows the pages of that wiki only. Without a configured wiki, the tool offers nothing on this level.
+
+A wiki page whose address the model chose has to stay in the wiki, redirects included: the address may carry mail content, and a redirect elsewhere could carry it on. An address given to the model may be redirected anywhere, since whatever the redirect carries came from the server. A call has to pass both the restriction and the free address choice, so with the choice switched off, a wiki page counts only when its address was given to the model, e.g., as a hit of a wiki search. The instructions of the tool name what is left on the level of the chat, and the refusal never repeats the address.
+
 ## Searching Data Sources
 
 `semantic_search` lets the model search the data sources of a chat itself, with a query it works out from the conversation, whenever a question calls for it. The classic RAG process, `AISrcSelWithRetCtxVal`, searches them with every message instead, using the message as the query. One place decides which of the two runs, `ToolRegistry.GetEffectiveRetrievalModeAsync`. Semantic search is the default, and the user can choose the other way per chat through `DataSourceOptions.RetrievalMode`. Whenever the tool cannot be offered — a model without tool calling, tools or this tool switched off, a provider below a confidence the organization set for it — the classic process searches instead. That process steps back only when the answer is semantic search, so a chat never ends up searching nothing.
@@ -148,17 +190,34 @@ The tool offers the data sources of the chat which the provider may use. With th
 
 The data sources are checked again before each search, since rounds may have passed since they were offered, and then searched in parallel. A data source which fails is reported as not searched rather than left out, so that the model does not take its silence for finding nothing. Every passage goes through the same filter and into the same shape as with the classic RAG process, `IRetrievalContext.AsMarkdown`, within one `PromptInjectionGuardService.BeginAction()` scope, so that the user hears about what was filtered once per search. The result holds whole passages up to 100,000 characters, and the data sources take turns: first the best passage of each, then the second best of each. Otherwise, the data source listed first would take the whole budget. What does not fit is counted in the result, with a narrower query as the way out. The passages become sources through `IRetrievalContext.ToSources()`, as with the classic process, and only the data sources whose passages reached the model raise the requirements of the chat.
 
+## Searching Mailboxes
+
+`search_mails`, `read_mail`, and `count_mails` form the collection `mailboxes`, behind the preview `PRE_MAILBOXES_2026` on top of `PRE_RAG_2024`. None of them asks a mail server anything. They read the local index which `MailboxIndexer` keeps, through `MailboxRetrievalService`. Mailboxes are kept in a list of their own, `Data.Mailboxes`, so `semantic_search` and the classic RAG process never see one.
+
+- `search_mails` searches with a `query` by meaning and by words, and returns each mail once with the passage which matched best. Without a query, it lists the mails meeting the conditions, the most recently received first. Results come in pages per mailbox, so a page after the first needs exactly one mailbox. All mailboxes share a budget of 100,000 characters and take turns in it.
+- `read_mail` reads one mail in pages of 30,000 characters, an attachment by its number, and the header block on request. It names the mail this one replies to, when that one is in the index.
+- `count_mails` counts with the same conditions, by folder or by sender on request, and adds how many mails the folders hold on the server.
+
+The tools offer the mailboxes which `MailboxRetrievalService.GetReadableMailboxes` returns. The chat provider and the embedding provider both have to meet the level of a mailbox, because the embedding provider receives the query, which the model may have written from a mail. A mailbox requires a level from `VERY_LOW` to `HIGH`; `NONE`, `UNTRUSTED`, and `UNKNOWN` would let almost every provider through, so they close the mailbox instead. A mailbox on a server the organization does not allow is left out as well, see `MailServerPolicy`. Each call checks again, since rounds may have passed since the tools were offered, and `read_mail` finds a mail only in the mailboxes the provider may read.
+
+Mails are written by others. Everything of a mail which reaches the model, from the subject and the addresses to the passages and the names of attachments, goes through one `PromptInjectionGuardService` batch per call, as `PromptInjectionSource.MailContent`. Only the mailboxes whose content reached the model raise the requirements of the chat; of several, the strictest restriction wins, and the first mailbox demanding it is named. An organization's `DataMailboxes.MinimumOutboundDataRestriction` tightens a mailbox whose own level is less strict. Found and read mails become sources under `mailbox://<mailbox ID>/<mail ID>`, which the sources list shows as text without a link, see `SourceExtensions.IsMailSource`. AI Studio cannot read an encrypted mail, only its header, and the instructions tell the model to say so rather than guess.
+
+Logs name a mailbox and its ID, never a subject, a sender or recipient, a folder, or an attachment. `ToolExecutor` logs the message of every exception, so a mail tool must not throw with such a value either. A `folder` the mailboxes do not know is therefore answered in the result, together with the folders to choose from, rather than refused by an exception.
+
 ## Checklist
 
 - Add the `IToolImplementation` class, including its `GetDefinition()`.
 - Register the implementation in `Program.cs`.
+- When the tool belongs to a preview feature, return false from `IsAvailable` while the preview is switched off.
 - Put every argument and setting name in a constant that the schema and the reading code share.
 - Set `MinimumProviderConfidence` to what the tool actually exposes.
+- When the tool only makes sense together with others, put them into a tool collection, and set the minimum provider confidence there.
 - Mark a setting the tool cannot work without as `Required`, rather than saying so in its description.
 - Validate settings and model arguments, and refuse a wrong argument with a message the model can correct itself from.
 - Filter content fetched from outside AI Studio for prompt injections, and declare `ReturnsUntrustedExternalContent`.
 - Protect secrets and sensitive trace arguments.
-- Add provider-confidence checks when tool output may contain sensitive data, and raise `RequiredProviderConfidence` and `RequiredDataSecurity` for what actually reached the model.
+- Add provider-confidence checks when tool output may contain sensitive data, and raise `RequiredProviderConfidence`, `RequiredDataSecurity`, and `RequiredOutboundDataRestriction` for what actually reached the model.
+- Declare where the tool sends data in `OutboundData`. Set `EnforcesOutboundDataRestriction` only when the tool refuses what goes too far itself, on every call.
 - For a tool which offers itself from the context of the chat, set `Activation = ToolActivation.CONTEXT` and return null from `ResolveFunctionAsync` when there is nothing to offer. Keep a tailored function stable for the same chat, and cache what it fetches.
 - When the system prompt instructions follow a setting, register those of the default and word the current ones in `ResolveSystemPromptInstructionsAsync`.
 - Page with `page` and `has_more`, not with a total, and cap how deep the model may go.