Added a way to read what the AI received from your sources (#1100)

Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>

documentation/Enterprise IT.md

@@ -823,6 +823,14 @@ The workspace goes by its name, because a workspace has an ID of its own on ever
 
 On the chat page, the chat moves into the workspace with its first message, and only then is the workspace created when it does not exist yet. Opening the chat page or trying out a template creates no workspace, so one that users deleted stays gone until they actually start a chat with the template. An assistant chat launcher creates the workspace right away, because it opens its chat at once.
 
+## Attached files which change later
+
+A chat reads a document attached to a message once, when the message is sent for the first time, and keeps its text. Every later message of the chat sends the AI that same text, also when the file changed or is gone by now, and users can look up what the AI received. Users who want the AI to work with the current state of their files switch on reading changed files again in the chat options. The AI then receives the new content with the next message, and viewing the attachment shows what was sent last.
+
+| Setting | Effect |
+|---|---|
+| `DataChat.ReadChangedAttachmentsAgain` | `true` reads an attached file again when it changed since it was sent. A file which is gone keeps the text it was sent with. Starts as `false`. |
+
 ## Where a chat may send data after searching data sources
 
 The AI searches the data sources of a chat, local files and folders as well as external data sources (ERI servers), either itself through Semantic Search or with every message. Each data source decides where a chat may still send data once the AI found something in it, with the same levels as a mailbox, see [Where a chat may send data](#where-a-chat-may-send-data). New data sources start with `ONLY_LINKS_FROM_CHAT`, and so do the ones users added before the setting existed. An external data source whose security policy is `SELF_HOSTED` always gets `ONLY_CONFIGURED_SERVICES`, since its data must not leave your infrastructure through a tool either. Your Confluence wiki and the hosts in `read_web_page.allowedPrivateHosts` stay readable on that level, so the links of such a data source into your intranet still work, while a link to a public page does not.

documentation/Tools.md

@@ -216,7 +216,7 @@ An ERI data source whose server does not answer the check of `DataSourceService`
 
 Every check of an ERI data source signs in to its server, and one message checks several times: when the tool is offered, before every search, and with every message of the classic process. `DataSourceService` therefore keeps a server's answer for one minute in an `ERISecurityCheckCache` and joins a check which is still running, so that a server which limits sign-ins does not start refusing. A server which did not answer is asked again after 15 seconds, and a changed configuration of the data source asks anew right away. Expired answers are dropped whenever a check starts. The minute holds for an answer which lets the data reach any provider as well, on purpose: when an operator opened a server by mistake and corrects it, every AI Studio follows within that minute, and until then, a chat with a cloud provider may still search the server. The cache lives only as long as the app runs, so no answer sticks for longer.
 
-The data sources are checked again before each search, since rounds may have passed since they were offered, and then searched in parallel. A data source which fails is reported as not searched rather than left out, so that the model does not take its silence for finding nothing. Every passage goes through the same filter and into the same shape as with the classic RAG process, `IRetrievalContext.AsMarkdown`, which gets the scope of one `PromptInjectionGuardService.BeginAction` per search, so that the user hears about what was filtered once per search, as soon as it is done. The result holds whole passages up to 100,000 characters, and the data sources take turns: first the best passage of each, then the second best of each. Otherwise, the data source listed first would take the whole budget. What does not fit is counted in the result, with a narrower query as the way out. An image an ERI server returns is not passed on, in neither process: written as base64 text, it is nothing a model can look at, and it would take more room than all other passages together. The passage tells the model that an image matched and where it is, so that it can point the user to it. What an ERI server says its content is, its category, is passed on as "Source category". The passages become sources through `IRetrievalContext.ToSources()`, as with the classic process, and only the data sources whose passages reached the model raise the requirements of the chat.
+The data sources are checked again before each search, since rounds may have passed since they were offered, and then searched in parallel. A data source which fails is reported as not searched rather than left out, so that the model does not take its silence for finding nothing. Every passage goes through the same filter and into the same shape as with the classic RAG process, `IRetrievalContextExtensions.PrepareForModelAsync`: all passages of a search in one filter request, within one `PromptInjectionGuardService.BeginAction` per search, so that the user hears about what was filtered once per search, as soon as it is done. The result holds whole passages up to 100,000 characters, and the data sources take turns: first the best passage of each, then the second best of each. Otherwise, the data source listed first would take the whole budget. What does not fit is counted in the result, with a narrower query as the way out. An image an ERI server returns is not passed on, in neither process: written as base64 text, it is nothing a model can look at, and it would take more room than all other passages together. The passage tells the model that an image matched and where it is, so that it can point the user to it. What an ERI server says its content is, its category, is passed on as "Source category". The passages become sources through `IRetrievalContext.ToSources()`, as with the classic process, and only the data sources whose passages reached the model raise the requirements of the chat. The passages which reached the model are also kept with the answer, through `ToolExecutionResult.Passages` and `ContentText.AddSources`: the result of the call is not saved, but the user can still read what the answer rests on after the chat was reloaded. The title of a source names the pages of all its passages, see `SourcePassageExtensions.NamePassagesInTitles`.
 
 **Where a chat may send data afterward.** A data source holds the documents of the user or of their organization, so each one decides, as a mailbox does, where a chat may still send data once a search found something in it, `IDataSource.OutboundDataRestriction`, see "Outbound Data Restrictions" above. New data sources start with `ONLY_LINKS_FROM_CHAT`, the level of internal pages, and so do the ones saved before the setting existed. What counts is the effective level, `DataSourceOutboundDataExtensions.GetEffectiveOutboundDataRestriction`: the level of the data source, tightened by the minimum of the organization, `DataSourceSecuritySettings.MinimumOutboundDataRestriction`, and to `ONLY_CONFIGURED_SERVICES` for an ERI data source whose security policy is `SELF_HOSTED`. Its data must not leave the infrastructure of the organization through a tool either, while its links into the intranet stay readable: that level still allows the wiki and the allowed private hosts. The rule holds at runtime, so neither a data source saved earlier nor one a configuration plugin provides has to be edited for it; the dialogs only stop offering the less strict levels.