Fixed Semantic Search with external data sources (ERI servers) (#1090)

Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>

app/MindWork AI Studio/Provider/BaseProvider.cs

@@ -1445,9 +1445,22 @@ public abstract class BaseProvider : IProvider, ISecretId
     }
 
     /// <inheritdoc />
+    /// <remarks>
+    /// The ID and whether the provider is self-hosted come along, because the data source checks
+    /// trust a provider by them: Semantic Search decides with these settings which data sources
+    /// it offers, while the classic RAG process asks this provider itself. Without them, a
+    /// self-hosted provider, or one the organization trusts, could search a data source which asks
+    /// for such a provider in one process but not in the other.<br/><br/>
+    /// With the ID, the settings also know what the provider's own model list reported, e.g., the
+    /// context window of a self-hosted model, see ProviderExtensions.GetAutomaticModelProfile. So
+    /// the tools of a request and the way images are passed on see the model as the configured
+    /// provider does.
+    /// </remarks>
     public AIStudio.Settings.Provider CreateSettingsProvider(Model chatModel) => new()
     {
+        Id = this.ConfiguredProviderId,
         UsedLLMProvider = this.Provider,
+        IsSelfHosted = this.Provider is LLMProviders.SELF_HOSTED,
         Model = chatModel,
         InstanceName = this.InstanceName,
         CapabilityOverrides = this.CapabilityOverrides,

app/MindWork AI Studio/Settings/DataModel/DataSourceERI_V1.cs

@@ -93,26 +93,37 @@ public readonly record struct DataSourceERI_V1 : IERIDataSource
         return await this.RetrieveDataAsync(latestUserPrompt, lastUserPrompt.ToERIContentType, thread, this.MaxMatches, token) ?? [];
     }
 
+    /// <inheritdoc />
+    /// <remarks>
+    /// An ERI data source has a single page. ERI v1 knows no offset, so a further page would have
+    /// to ask the server for all pages before it as well. A server may refuse more matches than its
+    /// retrieval process allows, which often is the very number configured here: it would refuse
+    /// every further page, and the model would hear that the data source could not be searched
+    /// rather than that there is nothing more.
+    /// </remarks>
+    public int GetLastPage() => 1;
+
     /// <inheritdoc />
     public async Task<RetrievalPage> RetrieveDataAsync(string query, int page, ChatThread thread, CancellationToken token = default)
     {
-        var window = RetrievalPaging.GetWindowSize(page, this.MaxMatches);
+        var lastPage = this.GetLastPage();
+        if (page < 1 || page > lastPage)
+            throw new ArgumentOutOfRangeException(nameof(page), page, $"An ERI data source has pages from 1 to {lastPage}.");
+
         if (this.MaxMatches == 0)
             return RetrievalPage.EMPTY;
 
         //
         // ERI v1 knows no query apart from the latest user prompt, so the query takes its place; the
-        // thread still tells the server what the conversation is about. Nor does it know an offset:
-        // the server returns the whole window, and the page is cut from it here. Hence, the pages
-        // are only as stable as the order in which the server returns its matches. A server which
-        // returns fewer matches than asked for ends the paging early, which errs on the safe side.
+        // thread still tells the server what the conversation is about. Nor does it know an offset,
+        // so the data source has a single page, asking for exactly what the classic RAG process asks
+        // for, see GetLastPage:
         //
-        var contexts = await this.RetrieveDataAsync(query, ContentType.TEXT, thread, window, token);
+        var contexts = await this.RetrieveDataAsync(query, ContentType.TEXT, thread, this.MaxMatches, token);
         if (contexts is null)
             return RetrievalPage.EMPTY with { Gaps = [RetrievalGap.NOT_SEARCHED] };
 
-        var (pageContexts, hasMore) = RetrievalPaging.Cut(contexts, page, this.MaxMatches);
-        return new RetrievalPage(pageContexts, hasMore);
+        return new RetrievalPage(contexts.Take(this.MaxMatches).ToList(), false);
     }
 
     /// <returns>What the ERI server found, or null when it could not be searched.</returns>
@@ -155,6 +166,7 @@ public readonly record struct DataSourceERI_V1 : IERIDataSource
                                 Type = eriContext.ToRetrievalContentType(),
                                 Links = eriContext.Links,
                                 Category = eriContext.Type.ToRetrievalContentCategory(),
+                                SourceCategory = eriContext.Category ?? string.Empty,
                                 MatchedText = eriContext.MatchedContent,
                                 DataSourceName = eriContext.Name,
                                 SurroundingContent = eriContext.SurroundingContent,
@@ -169,6 +181,7 @@ public readonly record struct DataSourceERI_V1 : IERIDataSource
                                 Links = eriContext.Links,
                                 Source = eriContext.MatchedContent,
                                 Category = eriContext.Type.ToRetrievalContentCategory(),
+                                SourceCategory = eriContext.Category ?? string.Empty,
                                 SourceType = ContentImageSource.BASE64,
                                 DataSourceName = eriContext.Name,
                             });
@@ -183,10 +196,17 @@ public readonly record struct DataSourceERI_V1 : IERIDataSource
                 return genericRetrievalContexts;
             }
 
+            //
+            // The ERI client answers a cancellation like any other failure, since other callers
+            // pass their own timeout and expect an answer. Whoever cancelled the search, though,
+            // waits for it to stop; reporting the data source as not searched would carry on:
+            //
+            token.ThrowIfCancellationRequested();
             logger.LogWarning($"Was not able to retrieve data from the ERI data source '{this.Name}'. Message: {retrievalResponse.Message}");
             return null;
         }
 
+        token.ThrowIfCancellationRequested();
         logger.LogWarning($"Was not able to authenticate with the ERI data source '{this.Name}'. Message: {authResponse.Message}");
         return null;
     }

app/MindWork AI Studio/Settings/IDataSource.cs

@@ -20,6 +20,17 @@ public interface IDataSource : IDataSourceBase
     /// </summary>
     public ushort MaxMatches { get; init; }
 
+    /// <summary>
+    /// The last page which can be retrieved from this data source.
+    /// </summary>
+    /// <remarks>
+    /// By default, as many pages as RetrievalPaging.GetLastPage allows for the page size. A data
+    /// source which cannot page that far says so itself. It is a method rather than a property, so
+    /// that it never ends up in the settings.
+    /// </remarks>
+    /// <returns>The number of the last page, which is at least 1.</returns>
+    public int GetLastPage() => RetrievalPaging.GetLastPage(this.MaxMatches);
+
     /// <summary>
     /// Perform the data retrieval process.
     /// </summary>
@@ -43,7 +54,7 @@ public interface IDataSource : IDataSourceBase
     /// from covering the whole data source.
     /// </remarks>
     /// <param name="query">What to search for.</param>
-    /// <param name="page">The page to retrieve, from 1 up to RetrievalPaging.GetLastPage for MaxMatches.</param>
+    /// <param name="page">The page to retrieve, from 1 up to GetLastPage.</param>
     /// <param name="thread">The chat thread.</param>
     /// <param name="token">The cancellation token.</param>
     /// <returns>The retrieved data contexts of this page, whether the next page is worth asking for, and what the search could not cover.</returns>

app/MindWork AI Studio/Tools/AllowedSelectedDataSources.cs

@@ -1,4 +1,5 @@
 using AIStudio.Settings;
+using AIStudio.Settings.DataModel;
 
 namespace AIStudio.Tools;
 
@@ -17,9 +18,33 @@ namespace AIStudio.Tools;
 /// The same holds for the ones waiting for a repair, and they are a list of their own because the
 /// two reasons call for different words: one passes by itself, the other one waits for the user.
 /// A data source is in at most one of the two lists.
+///
+/// The ERI data sources whose server could not be reached are a list of their own, too: they are
+/// neither allowed nor refused, because their server's requirements are unknown. They must never be
+/// searched, but Semantic Search and the classic RAG process both name them to the model, so that it
+/// can tell the user what is missing instead of answering as though there were nothing.
 /// </remarks>
 /// <param name="AllowedDataSources">The allowed data sources.</param>
 /// <param name="SelectedDataSources">The selected data sources, which are a subset of the allowed data sources.</param>
 /// <param name="DataSourcesAwaitingReindex">The data sources which passed every check but cannot be searched until their index has been rebuilt.</param>
 /// <param name="DataSourcesNeedingRepair">The data sources which passed every check but whose index cannot be read anymore, so that only the user can get them back.</param>
-public readonly record struct AllowedSelectedDataSources(IReadOnlyList<IDataSource> AllowedDataSources, IReadOnlyList<IDataSource> SelectedDataSources, IReadOnlyList<IDataSource> DataSourcesAwaitingReindex, IReadOnlyList<IDataSource> DataSourcesNeedingRepair);
+/// <param name="UnreachableDataSources">The ERI data sources whose server could not be reached or signed in to right now, and whose configured security policy allows them for this provider. Selected or not.</param>
+public readonly record struct AllowedSelectedDataSources(IReadOnlyList<IDataSource> AllowedDataSources, IReadOnlyList<IDataSource> SelectedDataSources, IReadOnlyList<IDataSource> DataSourcesAwaitingReindex, IReadOnlyList<IDataSource> DataSourcesNeedingRepair, IReadOnlyList<IDataSource> UnreachableDataSources)
+{
+    /// <summary>
+    /// The data sources whose server could not be reached, which the model learns about in this chat.
+    /// </summary>
+    /// <remarks>
+    /// Only those the chat would have searched count: with the automatic selection all of them, since
+    /// any data source may be picked, otherwise only those the user selected. A data source configured
+    /// to return no matches is left out, because it would add nothing even when its server answers.
+    /// Semantic Search and the classic RAG process both ask here, so that the model hears the same in
+    /// either way.
+    /// </remarks>
+    /// <param name="options">The data source options of the chat.</param>
+    /// <returns>The unreachable data sources of the chat, in the order they were checked.</returns>
+    public IReadOnlyList<IDataSource> GetUnreachableOfTheChat(DataSourceOptions options) => this.UnreachableDataSources
+        .Where(dataSource => dataSource.MaxMatches > 0)
+        .Where(dataSource => options.AutomaticDataSourceSelection || options.PreselectedDataSourceIds.Contains(dataSource.Id))
+        .ToList();
+}

app/MindWork AI Studio/Tools/RAG/IRetrievalContext.cs

@@ -20,6 +20,15 @@ public interface IRetrievalContext
     /// </summary>
     public RetrievalContentCategory Category { get; init; }
 
+    /// <summary>
+    /// What the source itself says its content is, e.g., a dictionary, a book chapter, or a paper.
+    /// </summary>
+    /// <remarks>
+    /// The server of an ERI data source names it in its own words, so it is free text; an empty
+    /// string when the source says nothing.
+    /// </remarks>
+    public string SourceCategory { get; init; }
+
     /// <summary>
     /// What type of content is being retrieved? Like e.g., a project proposal, spreadsheet, art, etc.
     /// </summary>

app/MindWork AI Studio/Tools/RAG/IRetrievalContextExtensions.cs

@@ -24,6 +24,9 @@ public static class IRetrievalContextExtensions
     {
         contextBuilder.AppendLine($"Data source name: {retrievalContext.DataSourceName}");
         contextBuilder.AppendLine($"Content category: {retrievalContext.Category}");
+        if(!string.IsNullOrWhiteSpace(retrievalContext.SourceCategory))
+            contextBuilder.AppendLine($"Source category: {retrievalContext.SourceCategory.ReplaceLineEndings(" ").Trim()}");
+
         contextBuilder.AppendLine($"Content type: {retrievalContext.Type}");
         contextBuilder.AppendLine($"Content path: {retrievalContext.Path}");
 
@@ -38,6 +41,22 @@ public static class IRetrievalContextExtensions
             contextBuilder.AppendLine($"- {link}");
     }
 
+    /// <summary>
+    /// Writes what the AI is told about an image which matched, in place of the image itself.
+    /// </summary>
+    /// <remarks>
+    /// The image is not passed on. Written as base64 text, it is nothing a model can look at, and
+    /// a single image would take more room than all other passages together: Semantic Search
+    /// left it out for its size, and the classic RAG process sent it along with every message.
+    /// The description above still tells where the image is, so the model can point the user to it.
+    /// </remarks>
+    /// <param name="contextBuilder">The builder to write into, after the description of the context.</param>
+    internal static void AppendImageMatch(StringBuilder contextBuilder)
+    {
+        contextBuilder.AppendLine();
+        contextBuilder.AppendLine("Matched image content: an image matched here, but it is not shown to you. Do not guess what it shows. When it matters for the answer, tell the user that an image matched, and where to find it, using the path and the links above.");
+    }
+
     /// <summary>
     /// Writes what the AI is told about several retrieval contexts, filtered for prompt injections.
     /// </summary>
@@ -126,20 +145,9 @@ public static class IRetrievalContextExtensions
                 await FilterWhatWeHaveSoFar();
                 break;
 
-            case RetrievalImageContext imageContext:
-                //
-                // Filtering happens before the image is appended, and only covers the text
-                // around it. Base64 image data is not prose, and running it through the filter
-                // would have it treated as one enormous encoded carrier.
-                //
+            case RetrievalImageContext:
+                AppendImageMatch(contextBuilder);
                 await FilterWhatWeHaveSoFar();
-                contextBuilder.AppendLine();
-                contextBuilder.AppendLine("Matched image content as base64-encoded data:");
-                contextBuilder.AppendLine("````");
-                contextBuilder.AppendLine(await imageContext.TryAsBase64(token) is (success: true, { } base64Image)
-                        ? base64Image
-                        : string.Empty);
-                contextBuilder.AppendLine("````");
                 break;
 
             default:

app/MindWork AI Studio/Tools/RAG/RAGProcesses/AISrcSelWithRetCtxVal.cs

@@ -1,3 +1,5 @@
+using System.Text;
+
 using AIStudio.Chat;
 using AIStudio.Provider;
 using AIStudio.Settings;
@@ -137,6 +139,10 @@ public sealed class AISrcSelWithRetCtxVal : IRagProcess
                 // belongs into the selection instead, which names the preselected sources it
                 // cannot use.
                 //
+                // An ERI data source whose server is out of reach is another matter: it was not held
+                // back by a filter, it just could not answer this time. The model learns about it
+                // below, after the augmentation.
+                //
                 LOGGER.LogWarning("No data sources are selected. The RAG process is skipped.");
                 proceedWithRAG = false;
             }
@@ -189,6 +195,11 @@ public sealed class AISrcSelWithRetCtxVal : IRagProcess
                     {
                         dataContexts.AddRange(await retrievalTask);
                     }
+                    catch (OperationCanceledException) when (token.IsCancellationRequested)
+                    {
+                        // The user stopped the answer, which is no error of the data source:
+                        LOGGER.LogInformation("The retrieval process was cancelled.");
+                    }
                     catch (Exception e)
                     {
                         LOGGER.LogError(e, "An error occurred during the retrieval process.");
@@ -204,7 +215,15 @@ public sealed class AISrcSelWithRetCtxVal : IRagProcess
                 var augmentationProcess = new AugmentationOne();
                 chatThread = await augmentationProcess.ProcessAsync(provider, lastUserPrompt, chatThread, dataContexts, token);
             }
-            
+
+            //
+            // The data sources of the chat whose server could not be reached were never searched.
+            // The model learns about them, so that it does not take their silence for finding
+            // nothing. This holds even when nothing was retrieved, e.g., because every selected data
+            // source is out of reach:
+            //
+            chatThread.AugmentedData = WithUnreachableDataSources(chatThread.AugmentedData, dataSources.GetUnreachableOfTheChat(chatThread.DataSourceOptions));
+
             //
             // Add sources from the selected data
             //
@@ -231,6 +250,25 @@ public sealed class AISrcSelWithRetCtxVal : IRagProcess
 
     #endregion
 
+    /// <summary>
+    /// Adds the note on the data sources whose server could not be reached to what the model reads along with the passages.
+    /// </summary>
+    /// <param name="augmentedData">What the augmentation wrote, possibly nothing.</param>
+    /// <param name="unreachableDataSources">The data sources of the chat whose server could not be reached.</param>
+    /// <returns>The augmented data followed by the note, or the augmented data alone when every server answered.</returns>
+    internal static string WithUnreachableDataSources(string augmentedData, IReadOnlyList<IDataSource> unreachableDataSources)
+    {
+        if (unreachableDataSources.Count == 0)
+            return augmentedData;
+
+        var builder = new StringBuilder(augmentedData);
+        if (builder.Length > 0)
+            builder.AppendLine();
+
+        UnreachableDataSources.AppendTo(builder, unreachableDataSources);
+        return builder.ToString();
+    }
+
     /// <summary>
     /// Whether the model searches the data sources of this chat itself, through the tool semantic_search.
     /// </summary>

app/MindWork AI Studio/Tools/RAG/RetrievalImageContext.cs

@@ -12,6 +12,9 @@ public sealed class RetrievalImageContext : IRetrievalContext, IImageSource
     /// <inheritdoc />
     public required RetrievalContentCategory Category { get; init; }
 
+    /// <inheritdoc />
+    public string SourceCategory { get; init; } = string.Empty;
+
     /// <inheritdoc />
     public required RetrievalContentType Type { get; init; }
 

app/MindWork AI Studio/Tools/RAG/RetrievalPaging.cs

@@ -8,7 +8,8 @@ namespace AIStudio.Tools.RAG;
 /// Neither the vector store nor the keyword index knows an offset, and neither needs one: page p
 /// of size k is cut from the first p·k + 1 matches of every channel. The one match beyond the
 /// page tells whether a next page is worth asking for. The first page is therefore exactly what a
-/// search for k matches always returned.
+/// search for k matches always returned. A data source which cannot page that far, such as an ERI
+/// data source, says so through IDataSource.GetLastPage.
 /// </para>
 /// <para>
 /// Staying without state is not a shortcut but a requirement: tool results do not travel into
@@ -21,9 +22,9 @@ public static class RetrievalPaging
     /// How many matches a page beyond the first may fetch at most, per channel.
     /// </summary>
     /// <remarks>
-    /// Every page fetches its whole window again, from the vector store and the keyword index, or
-    /// from the ERI server. The first page is exempt: its size is what the user or the organization
-    /// configured, and fetching it is what the retrieval always did.
+    /// Every page fetches its whole window again, from the vector store and the keyword index. The
+    /// first page is exempt: its size is what the user or the organization configured, and
+    /// fetching it is what the retrieval always did.
     /// </remarks>
     public const int MAX_RESULT_WINDOW = 100;
 

app/MindWork AI Studio/Tools/RAG/RetrievalTextContext.cs

@@ -12,6 +12,9 @@ public sealed class RetrievalTextContext : IRetrievalContext
     
     /// <inheritdoc />
     public required RetrievalContentCategory Category { get; init; }
+
+    /// <inheritdoc />
+    public string SourceCategory { get; init; } = string.Empty;
     
     /// <inheritdoc />
     public required RetrievalContentType Type { get; init; }

app/MindWork AI Studio/Tools/RAG/UnreachableDataSources.cs

@@ -0,0 +1,32 @@
+using System.Text;
+
+using AIStudio.Settings;
+
+namespace AIStudio.Tools.RAG;
+
+/// <summary>
+/// Tells the model which data sources of the chat could not be searched because their server could not be reached.
+/// </summary>
+/// <remarks>
+/// Semantic Search writes this into the description of its tool, the classic RAG process next to
+/// the passages it found. Both use the same words, so that the model hears the same in either way.
+/// Only external data sources have a server which may be out of reach, hence their kind is not
+/// listed.<br/><br/>
+/// The note holds nothing but the names the data sources have in AI Studio, configured by the user
+/// or the organization, and no word of their servers. It therefore needs no prompt-injection filter,
+/// like the rest of the description of the tool.
+/// </remarks>
+public static class UnreachableDataSources
+{
+    /// <summary>
+    /// Writes the note on the given data sources.
+    /// </summary>
+    /// <param name="builder">The builder to write into.</param>
+    /// <param name="dataSources">The data sources whose server could not be reached, in the order to list them.</param>
+    public static void AppendTo(StringBuilder builder, IReadOnlyList<IDataSource> dataSources)
+    {
+        builder.AppendLine("These external data sources of this chat could not be reached or signed in to right now, so they cannot be searched. Their content is missing rather than empty; tell the user so when a question concerns them:");
+        foreach (var dataSource in dataSources)
+            builder.AppendLine($"- name='{dataSource.Name}'");
+    }
+}

app/MindWork AI Studio/Tools/Services/DataSourceService.cs

@@ -18,6 +18,7 @@ public sealed class DataSourceService
     // ReSharper disable once NotAccessedPositionalProperty.Local
     private readonly record struct ParticipatingProvider(string Role, bool IsTrusted, ConfidenceLevel ConfidenceLevel);
 
+    private readonly ERISecurityCheckCache securityChecks;
     private readonly DataSourceEmbeddingService embeddingService;
     private readonly RustService rustService;
     private readonly SettingsManager settingsManager;
@@ -29,6 +30,7 @@ public sealed class DataSourceService
         this.rustService = rustService;
         this.settingsManager = settingsManager;
         this.embeddingService = embeddingService;
+        this.securityChecks = new(this.FetchSecurityRequirementsAsync, TimeProvider.System);
 
         this.logger.LogInformation("The data source service has been initialized.");
     }
@@ -52,7 +54,7 @@ public sealed class DataSourceService
         if (selectedLLMProvider == Settings.Provider.NONE)
         {
             this.logger.LogWarning("The selected LLM provider is not set. We cannot filter the data sources by any means.");
-            return new([], [], [], []);
+            return new([], [], [], [], []);
         }
         
         var usingTrustedProvider = selectedLLMProvider.IsTrustedForDataSourceSecurityChecks(this.settingsManager);
@@ -82,7 +84,7 @@ public sealed class DataSourceService
         var usingTrustedProvider = selectedLLMProvider.IsTrustedForDataSourceSecurityChecks(this.settingsManager);
         var participatingProviders = this.GetParticipatingProviders(selectedLLMProvider.Id, dataSourceOptions, retrievalMode,
             new("chat provider", usingTrustedProvider, selectedLLMProvider.GetConfidenceLevel(this.settingsManager)));
-        var allowedDataSources = await this.GetAllowedDataSources(usingTrustedProvider, participatingProviders, requestedDataSources);
+        var allowedDataSources = (await this.GetAllowedDataSources(usingTrustedProvider, participatingProviders, requestedDataSources)).Allowed;
 
         //
         // Whoever asks this way has no list to show, so a data source which cannot be searched is
@@ -113,7 +115,7 @@ public sealed class DataSourceService
         if (selectedLLMProvider is NoProvider)
         {
             this.logger.LogWarning("The selected LLM provider is the default provider. We cannot filter the data sources by any means.");
-            return new([], [], [], []);
+            return new([], [], [], [], []);
         }
         
         var usingTrustedProvider = selectedLLMProvider.IsTrustedForDataSourceSecurityChecks(this.settingsManager);
@@ -180,7 +182,8 @@ public sealed class DataSourceService
     {
         var allDataSources = this.settingsManager.ConfigurationData.DataSources.ToList();
         var previousSelectedDataSourceIds = previousSelectedDataSources?.Select(source => source.Id).ToHashSet(StringComparer.Ordinal) ?? [];
-        var filteredDataSources = await this.GetAllowedDataSources(usingTrustedProvider, participatingProviders, allDataSources);
+        var checkedDataSources = await this.GetAllowedDataSources(usingTrustedProvider, participatingProviders, allDataSources);
+        var filteredDataSources = checkedDataSources.Allowed;
 
         //
         // Which of the sources that passed every check cannot answer a search right now. They are
@@ -202,7 +205,7 @@ public sealed class DataSourceService
         var usableDataSources = filteredDataSources.Where(source => !blockedIds.Contains(source.Id)).ToList();
         var filteredSelectedDataSources = usableDataSources.Where(source => previousSelectedDataSourceIds.Contains(source.Id)).ToList();
 
-        return new(usableDataSources, filteredSelectedDataSources, awaitingReindex, needingRepair);
+        return new(usableDataSources, filteredSelectedDataSources, awaitingReindex, needingRepair, checkedDataSources.Unreachable);
     }
 
     /// <summary>
@@ -257,10 +260,16 @@ public sealed class DataSourceService
         return needingRepair;
     }
 
-    private async Task<IReadOnlyList<IDataSource>> GetAllowedDataSources(bool usingTrustedProvider, IReadOnlyList<ParticipatingProvider> participatingProviders, IReadOnlyCollection<IDataSource> requestedDataSources)
+    /// <summary>
+    /// What the checks of several data sources found: those which may be used, and the ERI data sources whose server could not be reached.
+    /// </summary>
+    private readonly record struct CheckedDataSources(IReadOnlyList<IDataSource> Allowed, IReadOnlyList<IDataSource> Unreachable);
+
+    private async Task<CheckedDataSources> GetAllowedDataSources(bool usingTrustedProvider, IReadOnlyList<ParticipatingProvider> participatingProviders, IReadOnlyCollection<IDataSource> requestedDataSources)
     {
         var filteredDataSources = new List<IDataSource>(requestedDataSources.Count);
-        var tasks = new List<Task<IDataSource?>>(requestedDataSources.Count);
+        var unreachableDataSources = new List<IDataSource>();
+        var tasks = new List<Task<DataSourceCheck>>(requestedDataSources.Count);
 
         // Start all checks in parallel:
         foreach (var source in requestedDataSources)
@@ -270,15 +279,126 @@ public sealed class DataSourceService
         // Wait for all checks and collect the results:
         foreach (var task in tasks)
         {
-            var source = await task;
-            if (source is not null)
-                filteredDataSources.Add(source);
+            var check = await task;
+            if (check.AllowedSource is not null)
+                filteredDataSources.Add(check.AllowedSource);
+            else if (check.UnreachableSource is not null)
+                unreachableDataSources.Add(check.UnreachableSource);
         }
 
-        return filteredDataSources;
+        return new(filteredDataSources, unreachableDataSources);
     }
-    
-    private async Task<IDataSource?> CheckOneDataSource(IDataSource source, bool usingTrustedProvider, IReadOnlyList<ParticipatingProvider> participatingProviders)
+
+    /// <summary>
+    /// What the check of one data source found: it may be used, its ERI server could not be reached, or neither.
+    /// </summary>
+    private readonly record struct DataSourceCheck(IDataSource? AllowedSource, IDataSource? UnreachableSource);
+
+    private async Task<DataSourceCheck> CheckOneDataSource(IDataSource source, bool usingTrustedProvider, IReadOnlyList<ParticipatingProvider> participatingProviders)
+    {
+        //
+        // Unfortunately, we have to live-check any ERI source for its security requirements.
+        // Because the ERI server operator might change the security requirements at any time.
+        // The answer is kept for a minute, see ERISecurityCheckCache for why that is acceptable.
+        //
+        SecurityRequirements? eriSourceRequirements = null;
+        if (source is DataSourceERI_V1 eriSource)
+        {
+            var securityCheck = await this.securityChecks.GetAsync(eriSource);
+            if (!securityCheck.IsReachable)
+            {
+                //
+                // The source cannot be used, but whoever asks may still tell the model or the user
+                // about it, so that its silence is not taken for finding nothing. Only a source
+                // which its own configured policy allows for this provider counts: whatever the
+                // server would demand on top of that is unknown, and it is never searched anyway.
+                //
+                return new(null, MayBeNamedWhileUnreachable(eriSource, usingTrustedProvider) ? source : null);
+            }
+
+            eriSourceRequirements = securityCheck.Requirements;
+        }
+
+        return new(this.CheckPolicies(source, usingTrustedProvider, participatingProviders, eriSourceRequirements), null);
+    }
+
+    /// <summary>
+    /// Whether an ERI data source whose server could not be reached may still be named to a provider.
+    /// </summary>
+    /// <param name="source">The ERI data source.</param>
+    /// <param name="usingTrustedProvider">Whether the chat provider is trusted for data source security checks.</param>
+    /// <returns>True when the configured security policy of the data source allows it for this provider.</returns>
+    internal static bool MayBeNamedWhileUnreachable(IExternalDataSource source, bool usingTrustedProvider) => usingTrustedProvider.AllowsDataSourceSecurity(source.SecurityPolicy);
+
+    /// <summary>
+    /// Signs in to the server of an ERI data source and asks for its security requirements.
+    /// </summary>
+    /// <returns>Whether the server answered and the sign-in succeeded, and its requirements when it did.</returns>
+    private async Task<ERISecurityCheckResult> FetchSecurityRequirementsAsync(DataSourceERI_V1 source)
+    {
+        //
+        // Leave the caller first: this starts while the cache is locked, and signing in should
+        // not happen while holding the lock:
+        //
+        await Task.Yield();
+
+        using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(6));
+        using var client = ERIClientFactory.Get(ERIVersion.V1, source);
+        if(client is null)
+        {
+            this.logger.LogError($"Could not create ERI client for source '{source.Name}' (id={source.Id}). We skip this source.");
+            return ERISecurityCheckResult.UNREACHABLE;
+        }
+
+        this.logger.LogInformation($"Authenticating with ERI source '{source.Name}' (id={source.Id})...");
+        var loginResult = await client.AuthenticateAsync(this.rustService, cancellationToken: cancellationTokenSource.Token);
+        if (!loginResult.Successful)
+        {
+            this.logger.LogWarning($"Authentication with ERI source '{source.Name}' (id={source.Id}) failed. We skip this source. Reason: {loginResult.Message}");
+            return ERISecurityCheckResult.UNREACHABLE;
+        }
+
+        this.logger.LogInformation($"Checking security requirements for ERI source '{source.Name}' (id={source.Id})...");
+        var securityRequest = await client.GetSecurityRequirementsAsync(cancellationTokenSource.Token);
+        if (!securityRequest.Successful)
+        {
+            this.logger.LogWarning($"Could not retrieve security requirements for ERI source '{source.Name}' (id={source.Id}). We skip this source. Reason: {securityRequest.Message}");
+            return ERISecurityCheckResult.UNREACHABLE;
+        }
+
+        this.logger.LogInformation($"Security requirements for ERI source '{source.Name}' (id={source.Id}) retrieved successfully.");
+        return new(true, securityRequest.Data);
+    }
+
+    /// <summary>
+    /// Whether the server of an ERI data source allows the provider to see its data.
+    /// </summary>
+    /// <remarks>
+    /// Its operator owns the data, so the rule of the server applies to every provider, a trusted
+    /// one included. A server which allows no provider at all closes its data source this way, as
+    /// the info dialog of the data source tells the user, e.g., to stop all access while the
+    /// operator corrects a mistake. A rule this version of AI Studio does not know closes it, too:
+    /// better safe than sorry.
+    /// </remarks>
+    /// <param name="requirements">What the server requires.</param>
+    /// <param name="usingTrustedProvider">Whether the chat provider is trusted for data source security checks.</param>
+    /// <returns>True when the server allows the provider.</returns>
+    internal static bool ServerAllows(SecurityRequirements requirements, bool usingTrustedProvider) => requirements.AllowedProviderType switch
+    {
+        ProviderType.ANY => true,
+        ProviderType.SELF_HOSTED => usingTrustedProvider,
+        _ => false,
+    };
+
+    /// <summary>
+    /// Checks a data source against the trust it asks for.
+    /// </summary>
+    /// <param name="source">The data source.</param>
+    /// <param name="usingTrustedProvider">Whether the chat provider is trusted for data source security checks.</param>
+    /// <param name="participatingProviders">Every provider which gets to see the data.</param>
+    /// <param name="eriSourceRequirements">What the server of an ERI data source requires, or null for any other data source.</param>
+    /// <returns>The data source when it may be used, otherwise null.</returns>
+    private IDataSource? CheckPolicies(IDataSource source, bool usingTrustedProvider, IReadOnlyList<ParticipatingProvider> participatingProviders, SecurityRequirements? eriSourceRequirements)
     {
         if (source is IInternalDataSource internalSource)
         {
@@ -307,85 +427,28 @@ public sealed class DataSourceService
             return source;
         }
 
-        //
-        // Unfortunately, we have to live-check any ERI source for its security requirements.
-        // Because the ERI server operator might change the security requirements at any time.
-        //
-        SecurityRequirements? eriSourceRequirements = null;
-        if (source is DataSourceERI_V1 eriSource)
-        {
-            using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(6));
-            using var client = ERIClientFactory.Get(ERIVersion.V1, eriSource);
-            if(client is null)
-            {
-                this.logger.LogError($"Could not create ERI client for source '{source.Name}' (id={source.Id}). We skip this source.");
-                return null;
-            }
-
-            this.logger.LogInformation($"Authenticating with ERI source '{source.Name}' (id={source.Id})...");
-            var loginResult = await client.AuthenticateAsync(this.rustService, cancellationToken: cancellationTokenSource.Token);
-            if (!loginResult.Successful)
-            {
-                this.logger.LogWarning($"Authentication with ERI source '{source.Name}' (id={source.Id}) failed. We skip this source. Reason: {loginResult.Message}");
-                return null;
-            }
-
-            this.logger.LogInformation($"Checking security requirements for ERI source '{source.Name}' (id={source.Id})...");
-            var securityRequest = await client.GetSecurityRequirementsAsync(cancellationTokenSource.Token);
-            if (!securityRequest.Successful)
-            {
-                this.logger.LogWarning($"Could not retrieve security requirements for ERI source '{source.Name}' (id={source.Id}). We skip this source. Reason: {loginResult.Message}");
-                return null;
-            }
-
-            eriSourceRequirements = securityRequest.Data;
-            this.logger.LogInformation($"Security requirements for ERI source '{source.Name}' (id={source.Id}) retrieved successfully.");
-        }
-
         if (source is not IExternalDataSource externalSource)
             return source;
 
+        //
+        // The server of an ERI data source has the last word, on top of the policy configured
+        // here: its operator owns the data and may tighten the rules at any time. This holds for a
+        // trusted provider as well, see ServerAllows:
+        //
+        if (eriSourceRequirements is { } requirements && !ServerAllows(requirements, usingTrustedProvider))
+        {
+            this.logger.LogWarning($"The ERI source '{source.Name}' (id={source.Id}) does not allow this provider by the rules of its server ('{requirements.AllowedProviderType}'). We skip this source.");
+            return null;
+        }
+
         switch (externalSource.SecurityPolicy)
         {
+            //
+            // Case: The data source allows any provider type, and the server of an ERI data source
+            //       agreed above. Accept it.
+            //
             case DataSourceSecurity.ALLOW_ANY:
-                
-                //
-                // Case: The data source allows any provider type. We want to use a trusted provider.
-                //       There is no issue with this source. Accept it.
-                //
-                if(usingTrustedProvider)
-                    return source;
-
-                //
-                // Case: This is a local data source. When the source allows any provider type, we can use it.
-                //       Accept it.
-                //
-                if(eriSourceRequirements is null)
-                    return source;
-
-                //
-                // Case: The ERI source requires a self-hosted or organization-trusted provider. This misconfiguration happens
-                //       when the ERI server operator changes the security requirements. The ERI server
-                //       operator owns the data -- we have to respect their rules. We skip this source.
-                //
-                if (eriSourceRequirements is { AllowedProviderType: ProviderType.SELF_HOSTED })
-                {
-                    this.logger.LogWarning($"The ERI source '{source.Name}' (id={source.Id}) requires a self-hosted or organization-trusted provider. We skip this source.");
-                    return null;
-                }    
-                
-                //
-                // Case: The ERI source allows any provider type. The data source configuration is correct.
-                //       Accept it.
-                //
-                if(eriSourceRequirements is { AllowedProviderType: ProviderType.ANY })
-                    return source;
-
-                //
-                // Case: Missing rules. We skip this source. Better safe than sorry.
-                //
-                this.logger.LogWarning($"The ERI source '{source.Name}' (id={source.Id}) was filtered out due to missing rules.");
-                return null;
+                return source;
 
             //
             // Case: The data source requires a trusted provider. We want to use a trusted provider.

app/MindWork AI Studio/Tools/Services/ERISecurityCheckCache.cs

@@ -0,0 +1,107 @@
+using AIStudio.Settings.DataModel;
+
+namespace AIStudio.Tools.Services;
+
+/// <summary>
+/// Keeps what the servers of ERI data sources answered to the check of their security requirements.
+/// </summary>
+/// <remarks>
+/// One message asks this several times: Semantic Search checks the data sources when it is
+/// offered and again before every search, and the classic RAG process checks them with every
+/// message. Each check signs in, and a server which limits sign-ins or sessions would start
+/// refusing. A check which is still running is joined rather than started a second time.<br/><br/>
+/// The server's operator may change the requirements at any time, so an answer is kept for one
+/// minute only. That covers the rounds of one answer, while a change still takes effect soon. A
+/// server which did not answer is asked again sooner: it may well be back, and until then, its
+/// data source is not searched anyway. A changed configuration, e.g., another server or another
+/// account, asks anew right away.<br/><br/>
+/// The minute holds for an answer which lets the data reach any provider as well, and that is a
+/// deliberate trade-off. When an operator opened a server by mistake and corrects it, every AI
+/// Studio follows within that minute rather than with its next check: until then, a chat with a
+/// cloud provider may still search the server. Nothing outlasts the running app, though, so no
+/// answer, however open, sticks for longer.<br/><br/>
+/// Whenever a check starts, the expired ones are dropped, so that data sources which were deleted
+/// or changed do not pile up.
+/// </remarks>
+/// <param name="fetch">Signs in to the server of a data source and asks for its requirements.</param>
+/// <param name="timeProvider">Tells the time the checks are started at.</param>
+internal sealed class ERISecurityCheckCache(Func<DataSourceERI_V1, Task<ERISecurityCheckResult>> fetch, TimeProvider timeProvider)
+{
+    /// <summary>
+    /// How long the answer of an ERI server to the check of its security requirements is used.
+    /// </summary>
+    private static readonly TimeSpan ANSWER_LIFETIME = TimeSpan.FromMinutes(1);
+
+    /// <summary>
+    /// How long an ERI server which did not answer the check is not asked again, counted from the start of the check.
+    /// </summary>
+    private static readonly TimeSpan NO_ANSWER_LIFETIME = TimeSpan.FromSeconds(15);
+
+    /// <summary>
+    /// A check of an ERI server, which may still be running, together with the configuration it was started with.
+    /// </summary>
+    private readonly record struct SecurityCheck(DataSourceERI_V1 DataSource, Task<ERISecurityCheckResult> Result, DateTimeOffset StartedAt);
+
+    private readonly Lock checksLock = new();
+    private readonly Dictionary<string, SecurityCheck> checks = new(StringComparer.Ordinal);
+
+    /// <summary>
+    /// What the server of an ERI data source requires, asked anew only once the last answer expired.
+    /// </summary>
+    /// <param name="source">The ERI data source.</param>
+    /// <returns>Whether the server answered, and its requirements when it did.</returns>
+    public Task<ERISecurityCheckResult> GetAsync(DataSourceERI_V1 source)
+    {
+        lock (this.checksLock)
+        {
+            var now = timeProvider.GetUtcNow();
+            if (this.checks.TryGetValue(source.Id, out var known) && known.DataSource.Equals(source) && IsValid(known.Result, now - known.StartedAt))
+                return known.Result;
+
+            this.RemoveExpired(now);
+            var check = new SecurityCheck(source, fetch(source), now);
+            this.checks[source.Id] = check;
+            return check.Result;
+        }
+    }
+
+    /// <summary>
+    /// How many checks are kept right now, expired ones included.
+    /// </summary>
+    internal int Count
+    {
+        get
+        {
+            lock (this.checksLock)
+                return this.checks.Count;
+        }
+    }
+
+    /// <summary>
+    /// Whether a check of an ERI server may still be used instead of asking the server again.
+    /// </summary>
+    /// <param name="result">The check, which may still be running.</param>
+    /// <param name="age">How long ago the check was started.</param>
+    /// <returns>True when the check is still running or its answer has not expired yet.</returns>
+    internal static bool IsValid(Task<ERISecurityCheckResult> result, TimeSpan age)
+    {
+        if (!result.IsCompleted)
+            return true;
+
+        if (!result.IsCompletedSuccessfully)
+            return false;
+
+        return age < (result.Result.IsReachable ? ANSWER_LIFETIME : NO_ANSWER_LIFETIME);
+    }
+
+    private void RemoveExpired(DateTimeOffset now)
+    {
+        var expiredIds = this.checks
+            .Where(pair => !IsValid(pair.Value.Result, now - pair.Value.StartedAt))
+            .Select(pair => pair.Key)
+            .ToList();
+
+        foreach (var id in expiredIds)
+            this.checks.Remove(id);
+    }
+}

app/MindWork AI Studio/Tools/Services/ERISecurityCheckResult.cs

@@ -0,0 +1,16 @@
+using AIStudio.Tools.ERIClient.DataModel;
+
+namespace AIStudio.Tools.Services;
+
+/// <summary>
+/// What the server of an ERI data source answered to the check of its security requirements.
+/// </summary>
+/// <param name="IsReachable">Whether the server answered and the sign-in succeeded.</param>
+/// <param name="Requirements">What the server requires, or null when it could not be reached.</param>
+internal readonly record struct ERISecurityCheckResult(bool IsReachable, SecurityRequirements? Requirements)
+{
+    /// <summary>
+    /// The server could not be reached, or it refused the sign-in.
+    /// </summary>
+    public static readonly ERISecurityCheckResult UNREACHABLE = new(false, null);
+}

app/MindWork AI Studio/Tools/ToolCallingSystem/ToolCallingImplementations/SemanticSearch/OfferedDataSources.cs

@@ -0,0 +1,18 @@
+using AIStudio.Settings;
+
+namespace AIStudio.Tools.ToolCallingSystem.ToolCallingImplementations.SemanticSearch;
+
+/// <summary>
+/// The data sources Semantic Search offers the model in a request.
+/// </summary>
+/// <param name="Searchable">The data sources the model may search, in the order they are offered.</param>
+/// <param name="Unreachable">The ERI data sources of the chat whose server could not be reached or signed in to. The model learns their names, but can never search them.</param>
+internal readonly record struct OfferedDataSources(IReadOnlyList<IDataSource> Searchable, IReadOnlyList<IDataSource> Unreachable)
+{
+    public static readonly OfferedDataSources NONE = new([], []);
+
+    /// <summary>
+    /// Whether there is nothing at all to tell the model about.
+    /// </summary>
+    public bool IsEmpty => this.Searchable.Count == 0 && this.Unreachable.Count == 0;
+}

app/MindWork AI Studio/Tools/ToolCallingSystem/ToolCallingImplementations/SemanticSearch/SemanticSearchTool.cs

@@ -122,19 +122,22 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
     /// </summary>
     /// <remarks>
     /// Without a data source to offer, the tool stays out of the request: the model should not
-    /// learn about a search which can only come back empty.<br/><br/>
+    /// learn about a search which can only come back empty. A data source whose ERI server could
+    /// not be reached is the exception. The model learns about it even when nothing else can be
+    /// searched, so that it tells the user what is missing instead of answering as though the chat
+    /// had no data sources at all.<br/><br/>
     /// The descriptions of ERI data sources are asked from their servers and kept for a few
     /// minutes, so that asking for every request costs no more than the check of the data sources
     /// the classic RAG process makes with every message, too.
     /// </remarks>
     public async ValueTask<ToolFunctionDefinition?> ResolveFunctionAsync(ToolDefinition definition, ToolResolutionContext context, CancellationToken token = default)
     {
-        var dataSources = await this.GetOfferedDataSourcesAsync(context.Provider, context.ChatThread);
-        if (dataSources.Count == 0)
+        var offered = await this.GetOfferedDataSourcesAsync(context.Provider, context.ChatThread);
+        if (offered.IsEmpty)
             return null;
 
-        var descriptions = await Task.WhenAll(dataSources.Select(dataSource => descriptionService.GetDescriptionAsync(dataSource, token)));
-        return DescribeDataSources(definition.Function, dataSources.Zip(descriptions).ToList());
+        var descriptions = await Task.WhenAll(offered.Searchable.Select(dataSource => descriptionService.GetDescriptionAsync(dataSource, token)));
+        return DescribeDataSources(definition.Function, offered.Searchable.Zip(descriptions).ToList(), offered.Unreachable);
     }
 
     /// <summary>
@@ -148,36 +151,48 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
     /// Preparing a request knows the provider by its settings, running a call by the provider
     /// itself. Both ask the same question, so both come here.
     /// </remarks>
-    private Task<IReadOnlyList<IDataSource>> GetOfferedDataSourcesAsync(AIStudio.Settings.Provider provider, ChatThread thread) =>
+    private Task<OfferedDataSources> GetOfferedDataSourcesAsync(AIStudio.Settings.Provider provider, ChatThread thread) =>
         this.GetOfferedDataSourcesAsync(thread, (options, preselectedDataSources) => dataSourceService.GetDataSources(provider, options, DataSourceRetrievalMode.SEMANTIC_SEARCH, preselectedDataSources));
 
-    private Task<IReadOnlyList<IDataSource>> GetOfferedDataSourcesAsync(IProvider provider, ChatThread thread) =>
+    private Task<OfferedDataSources> GetOfferedDataSourcesAsync(IProvider provider, ChatThread thread) =>
         this.GetOfferedDataSourcesAsync(thread, (options, preselectedDataSources) => dataSourceService.GetDataSources(provider, options, DataSourceRetrievalMode.SEMANTIC_SEARCH, preselectedDataSources));
 
-    private async Task<IReadOnlyList<IDataSource>> GetOfferedDataSourcesAsync(ChatThread thread, Func<DataSourceOptions, IReadOnlyCollection<IDataSource>, Task<AllowedSelectedDataSources>> checkDataSources)
+    private async Task<OfferedDataSources> GetOfferedDataSourcesAsync(ChatThread thread, Func<DataSourceOptions, IReadOnlyCollection<IDataSource>, Task<AllowedSelectedDataSources>> checkDataSources)
     {
         // When the user wants AI Studio to search with every message, the model does not search itself:
         var options = thread.DataSourceOptions;
         if (options.RetrievalMode is not DataSourceRetrievalMode.SEMANTIC_SEARCH)
-            return [];
+            return OfferedDataSources.NONE;
 
         //
         // Data sources are a preview feature, and a chat keeps its data source options while the
         // feature is switched off, cf. AISrcSelWithRetCtxVal:
         //
         if (!PreviewFeatures.PRE_RAG_2024.IsEnabled(settingsManager) || !options.IsEnabled())
-            return [];
+            return OfferedDataSources.NONE;
 
         var preselectedDataSources = options.PreselectedDataSourceIds
             .Select(id => settingsManager.ConfigurationData.DataSources.FirstOrDefault(dataSource => dataSource.Id == id))
             .OfType<IDataSource>()
             .ToList();
 
-        var dataSources = await checkDataSources(options, preselectedDataSources);
+        return SelectOffered(options, await checkDataSources(options, preselectedDataSources));
+    }
+
+    /// <summary>
+    /// Picks the data sources to offer out of those which passed the checks.
+    /// </summary>
+    /// <param name="options">The data source options of the chat.</param>
+    /// <param name="dataSources">What the checks of the data source service found.</param>
+    /// <returns>The data sources the model may search, and those of the chat whose server could not be reached, each in the order they are offered.</returns>
+    internal static OfferedDataSources SelectOffered(DataSourceOptions options, AllowedSelectedDataSources dataSources)
+    {
         var offeredDataSources = options.AutomaticDataSourceSelection ? dataSources.AllowedDataSources : dataSources.SelectedDataSources;
 
         // A data source configured to return no matches would only ever come back empty:
-        return InOfferOrder(offeredDataSources.Where(dataSource => dataSource.MaxMatches > 0));
+        return new(
+            InOfferOrder(offeredDataSources.Where(dataSource => dataSource.MaxMatches > 0)),
+            InOfferOrder(dataSources.GetUnreachableOfTheChat(options)));
     }
 
     /// <summary>
@@ -206,22 +221,33 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
     /// </remarks>
     /// <param name="function">The function as registered.</param>
     /// <param name="dataSources">The data sources to offer, in the order to list them, each with its description.</param>
+    /// <param name="unreachableDataSources">The data sources of the chat whose server could not be reached, in the order to list them. They get no ID, as they cannot be searched.</param>
     /// <returns>The function to offer in this request.</returns>
-    internal static ToolFunctionDefinition DescribeDataSources(ToolFunctionDefinition function, IReadOnlyList<(IDataSource DataSource, string Description)> dataSources)
+    internal static ToolFunctionDefinition DescribeDataSources(ToolFunctionDefinition function, IReadOnlyList<(IDataSource DataSource, string Description)> dataSources, IReadOnlyList<IDataSource>? unreachableDataSources = null)
     {
         var description = new StringBuilder(DESCRIPTION);
         description.AppendLine();
         description.AppendLine();
-        description.AppendLine($"The data sources you may search, by the ID to pass in {DATA_SOURCE_IDS_ARGUMENT}:");
+        if (dataSources.Count == 0)
+            description.AppendLine("None of the data sources of this chat can be searched right now.");
+        else
+            description.AppendLine($"The data sources you may search, by the ID to pass in {DATA_SOURCE_IDS_ARGUMENT}:");
+
         foreach (var (dataSource, dataSourceDescription) in dataSources)
         {
-            description.Append($"- id={dataSource.Id}, name='{dataSource.Name}', type={GetKind(dataSource)}, results per page={dataSource.MaxMatches}, last page={RetrievalPaging.GetLastPage(dataSource.MaxMatches)}");
+            description.Append($"- id={dataSource.Id}, name='{dataSource.Name}', type={GetKind(dataSource)}, results per page={dataSource.MaxMatches}, last page={dataSource.GetLastPage()}");
             if (!string.IsNullOrWhiteSpace(dataSourceDescription))
                 description.Append($", description='{dataSourceDescription.Trim().Shorten(MAX_DESCRIPTION_CHARACTERS)}'");
 
             description.AppendLine();
         }
 
+        if (unreachableDataSources is { Count: > 0 })
+        {
+            description.AppendLine();
+            UnreachableDataSources.AppendTo(description, unreachableDataSources);
+        }
+
         return function with
         {
             DescriptionForLLM = description.ToString().TrimEnd(),
@@ -272,11 +298,14 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
         // changed the data sources of the chat, or the server of an ERI data source its rules, so
         // they are checked again, the same way as when they were offered:
         //
-        var offeredDataSources = await this.GetOfferedDataSourcesAsync(context.Provider, context.ChatThread);
-        if (offeredDataSources.Count == 0)
+        var offered = await this.GetOfferedDataSourcesAsync(context.Provider, context.ChatThread);
+        if (offered.IsEmpty)
             throw new ToolExecutionBlockedException(TB("None of the data sources of this chat can be searched right now."));
 
-        var request = ReadRequest(arguments, offeredDataSources);
+        var request = ReadRequest(arguments, offered.Searchable);
+
+        // They were never searched, so they raise none of the chat's requirements below:
+        var unreachableDataSources = GetUnreachableToReport(request, offered);
 
         //
         // The chat ends in the answer being written, which has no content yet. An ERI server reads
@@ -335,8 +364,11 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
         for (var index = 0; index < pages.Length; index++)
             dataSourceResults.Add(DescribeResult(request.DataSources[index], pages[index], resultCounts[index], leftOutCounts[index]));
 
+        foreach (var dataSource in unreachableDataSources)
+            dataSourceResults.Add(DescribeUnreachableResult(dataSource));
+
         var leftOutCount = leftOutCounts.Sum();
-        logger.LogInformation("Semantic search finished. ToolCallId={ToolCallId}, DataSourceCount={DataSourceCount}, Page={Page}, PassageCount={PassageCount}, LeftOutCount={LeftOutCount}", context.ToolCallId, request.DataSources.Count, request.Page, passageCount, leftOutCount);
+        logger.LogInformation("Semantic search finished. ToolCallId={ToolCallId}, DataSourceCount={DataSourceCount}, UnreachableCount={UnreachableCount}, Page={Page}, PassageCount={PassageCount}, LeftOutCount={LeftOutCount}", context.ToolCallId, request.DataSources.Count, unreachableDataSources.Count, request.Page, passageCount, leftOutCount);
 
         // Finding nothing is no error; the model reads it from the result counts:
         var requirements = GetRequirements(request.DataSources, resultCounts);
@@ -355,6 +387,37 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
         };
     }
 
+    /// <summary>
+    /// Which data sources whose server could not be reached a search reports as not searched.
+    /// </summary>
+    /// <remarks>
+    /// A search of all data sources reports them, so that the model does not take their silence
+    /// for finding nothing. A search of some data sources, or of a further page, was not meant for
+    /// them, and repeating them there would only cost tokens.
+    /// </remarks>
+    /// <param name="request">The search to run.</param>
+    /// <param name="offered">The data sources offered for this search.</param>
+    /// <returns>The unreachable data sources to report, in the order they are offered.</returns>
+    internal static IReadOnlyList<IDataSource> GetUnreachableToReport(SemanticSearchRequest request, OfferedDataSources offered)
+    {
+        var searchesAll = request.Page == 1 && request.DataSources.Count == offered.Searchable.Count;
+        return searchesAll ? offered.Unreachable : [];
+    }
+
+    /// <summary>
+    /// What the model learns about a data source whose server could not be reached, in the result of a search.
+    /// </summary>
+    /// <remarks>
+    /// Like the description of the tool, the result names such a data source without its ID: the
+    /// model cannot search it, so it has no use for an ID to pass.
+    /// </remarks>
+    internal static JsonObject DescribeUnreachableResult(IDataSource dataSource)
+    {
+        var result = DescribeResult(dataSource, RetrievalPage.EMPTY with { Gaps = [RetrievalGap.NOT_SEARCHED] }, 0, 0);
+        result.Remove("id");
+        return result;
+    }
+
     /// <summary>
     /// What the chat has to require from now on, because of the passages which reached the model.
     /// </summary>
@@ -414,7 +477,7 @@ public sealed class SemanticSearchTool(SettingsManager settingsManager, DataSour
         if (dataSources.Count != 1)
             throw new ArgumentException($"Argument '{PAGE_ARGUMENT}' may be above 1 only for exactly one data source in '{DATA_SOURCE_IDS_ARGUMENT}', but was {page} for {dataSources.Count}. Name the one data source to page through, or leave '{PAGE_ARGUMENT}' out to get the first page of each.");
 
-        var lastPage = RetrievalPaging.GetLastPage(dataSources[0].MaxMatches);
+        var lastPage = dataSources[0].GetLastPage();
         if (page > lastPage)
             throw new ArgumentException($"Argument '{PAGE_ARGUMENT}' must be at most {lastPage} for the data source '{dataSources[0].Id}', but was {page}. Rephrase the query to find other passages.");
 

app/MindWork AI Studio/wwwroot/changelog/v26.10.1.md

@@ -45,6 +45,8 @@
 - Improved attaching files to a chat. You can now attach any file that contains readable text, for example, program code or configuration files of a kind AI Studio did not know before.
 - Improved adding a local data source when you have not configured an embedding provider yet. Until now, you only found out when saving the dialog, after filling in everything else. The data source settings now tell you right away and let you add an embedding provider right there, without losing what you entered.
 - Improved the dialogs for managing your data sources, profiles, and chat templates, which you open, for example, from a chat. The buttons for adding and importing now stay at the bottom of the dialog, so you no longer need to scroll past a long list to reach them. Thanks, Paul Koudelka (`PaulKoudelka`), for this contribution.
+- Improved how often AI Studio signs in to the server of an external data source (ERI server). AI Studio now keeps the server's answer to its security check for a minute, so a server which limits sign-ins no longer refuses the searches of the AI.
+- Improved the search results from external data sources (ERI servers), which now tell the AI what kind of content a result comes from, for example, a dictionary or a book chapter, when the server names it.
 - Changed the banner on the start page to a fresh design that shows what AI Studio can do today.
 - Changed when a chat you add to a workspace in the workspace list appears there. Like any other new chat, it now appears once there is something to keep, for example, your first message, instead of right away.
 - Fixed a warning about the confidence level that stayed in the dialog for adding a local data source, even after you chose an embedding provider that meets it. The dialog now checks again whenever you change either of them, so a warning also appears when you switch to a provider that falls short.
@@ -52,6 +54,11 @@
 - Fixed two confidence levels sharing the same name in the German user interface. Moderate is called "Mäßig" again, and Medium stays "Mittel".
 - Fixed renaming an external data source (ERI server) leaving its secret behind under the old name in the keychain of your operating system.
 - Fixed the ERI server assistant deleting files in a folder you no longer used. When it updates a server, it deletes the files it created before, and after you chose another folder for the server, it also deleted them in the old one. It now only deletes files within the folder you chose.
+- Fixed the AI not searching an external data source (ERI server) which only allows self-hosted providers, although you chatted with a self-hosted provider or one your organization trusts. This happened when the AI searched your data sources itself; it did not even learn about the data source.
+- Fixed the AI being unable to search an external data source (ERI server) that limits the number of results. When the AI searched your data sources itself, AI Studio asked the server for one result more than you configured, and such a server refused the search. AI Studio now asks for exactly the number you configured, and the AI no longer pages through the results of an external data source, which such a server would refuse as well.
+- Fixed the AI answering from its own knowledge without telling you when the server of an external data source (ERI server) could not be reached or refused to sign you in. The AI now learns which of your external data sources are unavailable, so it can tell you that their content is missing.
+- Fixed images from external data sources (ERI servers) crowding out the other search results. AI Studio passed each image to the AI as a long block of text which the AI cannot look at. The AI now learns that an image matched and where to find it, so it can point you to it.
+- Fixed a security issue with external data sources (ERI servers) whose server does not allow any AI provider. AI Studio now follows this rule for self-hosted providers and for providers your organization trusts as well.
 - Fixed the Search Confluence tool missing pages that the search of your Confluence wiki finds. The tool now searches the same way as the search box of your wiki.
 - Fixed a security issue in the Search Confluence and Read Web Page tools. Once the AI has searched your Confluence wiki or read a page in your intranet or behind a VPN, the chat no longer searches the web, and the tool selection tells you why. In the settings of the Read Web Page tool, you decide which web pages such a chat may still read.
 - Fixed AI Studio starting only partly when the configuration of your organization took a while to apply. The plugins late in line never started, so the app stayed in English instead of your language, and the configuration of your organization was missing, together with its models and its welcome text. Now every plugin gets its own time to start, and a slow one no longer holds up the others.