Added mailboxes as local data sources (#1022)
Build and Release / Determine run mode (push) Waiting to run
Build and Release / Read metadata (push) Blocked by required conditions
Build and Release / Sync Flatpak repo (push) Blocked by required conditions
Build and Release / Collect Flatpak artifacts (push) Blocked by required conditions
Build and Release / Verify (push) Waiting to run
Build and Release / Build app (${{ matrix.dotnet_runtime }}) (-aarch64-pc-windows-msvc.exe, win-arm64, windows-latest, aarch64-pc-windows-msvc, nsis,updater, nsis) (push) Blocked by required conditions
Build and Release / Build app (${{ matrix.dotnet_runtime }}) (-aarch64-unknown-linux-gnu, linux-arm64, ubuntu-22.04-arm, aarch64-unknown-linux-gnu, appimage,updater, appimage) (push) Blocked by required conditions
Build and Release / Build app (${{ matrix.dotnet_runtime }}) (-x86_64-apple-darwin, osx-x64, macos-latest, x86_64-apple-darwin, dmg,app,updater, dmg) (push) Blocked by required conditions
Build and Release / Build app (${{ matrix.dotnet_runtime }}) (-x86_64-pc-windows-msvc.exe, win-x64, windows-latest, x86_64-pc-windows-msvc, nsis,updater, nsis) (push) Blocked by required conditions
Build and Release / Prepare & create release (push) Blocked by required conditions
Build and Release / Publish release (push) Blocked by required conditions
Build and Release / Build app (${{ matrix.dotnet_runtime }}) (-aarch64-apple-darwin, osx-arm64, macos-latest, aarch64-apple-darwin, dmg,app,updater, dmg) (push) Blocked by required conditions
Build and Release / Build app (${{ matrix.dotnet_runtime }}) (-x86_64-unknown-linux-gnu, linux-x64, ubuntu-22.04, x86_64-unknown-linux-gnu, appimage,updater, appimage) (push) Blocked by required conditions

This commit is contained in:
Thorsten Sommer authored and GitHub committed 2026-10-04 12:26:26 +02:00
1 parent 93be539e50
commit c4400c0ff6
343 files changed
+26051 -3122

No files matched your search

+54 -1
View File
@@ -186,12 +186,15 @@ When adding configuration plugin capabilities:
When adding, changing, or removing model-driven tools, keep these parts in sync:
- `app/MindWork AI Studio/Tools/ToolCallingSystem/ToolCallingImplementations/` for the `IToolImplementation` class, which states its own `ToolDefinition` through `GetDefinition()`, written with `ToolSettingsSchemaBuilder` for its settings and `ToolParameterSchemaBuilder` for the arguments the model passes. There are no tool definition files; a tool arriving from elsewhere brings an `IToolDefinitionSource` instead.
- `app/MindWork AI Studio/Program.cs` for DI registration of the implementation. Registering it as an `IToolImplementation` is enough, because `CodeToolDefinitionSource` collects the definitions of all of them.
- `app/MindWork AI Studio/Tools/ToolCallingSystem/ToolSelectionRules.cs` when the shared tool-call limits change. A tool's own minimum provider confidence belongs in its definition, not here.
- An `IToolCollection` next to the tools, registered in `Program.cs`, when tools only make sense together. Selections, `DataTools.DisabledToolIds`, and the minimum provider confidence name tool collections; a tool outside a declared collection forms one under its own ID, and the ID of a tool inside one stands for its whole collection. Normalize a selection with `ToolRegistry.NormalizeSelection`, and expand it into tools with `ToolRegistry.ExpandSelection` only where the view of the model counts.
- `app/MindWork AI Studio/Tools/ToolCallingSystem/ToolSelectionRules.cs` when the shared tool-call limits change. A tool's own minimum provider confidence belongs in its definition, or in the definition of its collection, not here.
- `app/MindWork AI Studio/Tools/ToolCallingSystem/ToolSettingsOptionSources.cs` when a tool setting offers a fixed choice the app maintains, such as languages. Prefer this over spelling the values out in the settings schema; it keeps the list in one place and gives the user translated names.
- `app/MindWork AI Studio/Plugins/configuration/plugin.lua` to document each setting's field name, meaning, and data type. Tool settings need no code to be centrally manageable: an organization addresses them by `"<toolId>.<fieldName>"` in `DataTools.LockedToolSettings` or `DataTools.DefaultToolSettings`.
Tool implementations must treat model-provided arguments as untrusted input. Validate settings and arguments, protect secrets with `SensitiveTraceArgumentNames`, use `ToolExecutionBlockedException` for intentional policy blocks, and check provider confidence before returning sensitive data to the model.
A tool which belongs to a preview feature returns false from `IToolImplementation.IsAvailable` while the preview is switched off; the registry then leaves it out of every list, every request, and the token count, so no component has to check that preview for the tool. Every tool declares in `IToolImplementation.OutboundData` where its arguments go: a chat which read from a mailbox keeps the tools whose data goes further than the mailbox allows from being offered and from running, see `ToolSelectionRules.IsOutboundDataAllowed`. A tool which brings content of a mailbox into the chat raises `ToolExecutionResult.RequiredOutboundDataRestriction`, next to `RequiredProviderConfidence` and `RequiredDataSecurity`. "Searching Mailboxes" in `documentation/Tools.md` explains the mail tools.
A tool which offers itself from the context of a chat instead of being selected, such as `semantic_search`, sets `Activation = ToolActivation.CONTEXT` and tailors its function to each request in `ResolveFunctionAsync`. Code which decides something on behalf of a request — whether the classic RAG process steps back, say — asks `ToolRegistry.GetOfferBlockReasonAsync` or `ToolRegistry.GetEffectiveRetrievalModeAsync` with the provider settings of the request (`IProvider.CreateSettingsProvider`), never a check of its own: two answers which drift apart leave a chat searching nothing or twice.
## Model Capabilities
@@ -221,6 +224,56 @@ RAG is available as a beta preview feature. Architecture:
- **Index database** - SQLite, holding the file fingerprints and the chunk texts for full-text search; see "Databases" below
- **File processing** - Extracts text from PDF, DOCX, XLSX via Rust runtime
### Indexed data sources
Everything AI Studio embeds itself runs through one pipeline in `app/MindWork AI Studio/Tools/Services/Indexing/`,
driven by `DataSourceEmbeddingService`, which queues the runs, prepares each one and owns the statuses. A new
kind of data source plugs into this pipeline instead of building its own.
The parts:
- **`IIndexedDataSource`** (`Settings/`) - what indexing needs to know about a data source: confidence level,
embedding provider and chunk settings. `IDataSourceBase` is what every data source has. Implement
`IDataSource` on top only when classic RAG, Semantic Search and the agents should see the data source. A data
source kept in a list of its own implements `IIndexedDataSource` alone, and the compiler keeps it out of
`DataSources`.
- **`IIndexedSourceIndexer`** - one per kind of data source. `Supports` claims the data sources, `ProcessAsync`
finds and reads the documents of one run, and `TrackChanges` / `StopTracking` notice changes on their own.
`FileSourceIndexer` is the reference: a file system watcher per data source, fingerprints over path, size and
write time.
- **`IndexedRunContext`** - one prepared run: both stores, the embedding provider, the manifest and the
collection. `IndexDocumentAsync` embeds and stores one document; the cleanup methods remove what a failed
attempt left behind.
- **`EmbeddingDocument`** - one document: its key, its index row, its display name, and how to read its chunks.
- **`DocumentRunProgress`** - counts the documents, records indexed and failed ones in the stores, publishes the
status and completes the run.
- **`TextChunker`** - cuts text into chunks the embedding provider accepts. Pick one of its strategies; do not
write a chunker of your own.
To add a kind of data source:
1. Write its indexer in `Tools/Services/Indexing/` and create it in `DataSourceEmbeddingService.CreateIndexers`,
which hands every indexer the same `TextChunker`.
2. Gate it in `IsSupportedIndexedSource`, behind a preview feature of its own while it is new.
3. When the data source is not kept in `DataSources`, add its list to `GetConfiguredIndexedSources`. Every lookup
by id and every pass over all data sources goes through it: the startup hash check,
`QueueAllInternalDataSourcesAsync` and `RefreshWatchers`.
4. Keep whatever the kind has to remember beyond its documents in tables of its own in the index store, added
by an EF Core migration (see "Databases").
5. Report every status through `DocumentRunProgress`, so all rows of the embedding page behave alike.
Rules which are easy to break:
- **A document key is not a path.** Only files use their full path as the key. Never pass a key through the
`Path` APIs: on Windows, `Path.GetFullPath` reads a key like `mail:…` as a file with an alternate data stream.
- **Ids and signature are pinned.** The formats in `IndexedDocumentIds` and the embedding signature
(`DataSourceEmbeddingService.BuildEmbeddingSignature`) are fixed by tests, because every stored chunk and every
index depends on them. When the metadata stored next to a chunk changes, raise `CHUNK_METADATA_VERSION`
deliberately: that rebuilds every index.
- **Every content path goes through the prompt injection filter.** Files pass the sanitizer of the runtime while
their text is extracted; a new kind of data source needs its own pass through `PromptInjectionGuardService`.
- **The service decides when, the indexer decides how.** Whether changes are tracked at all depends on the
automatic refresh setting and the startup hash check, and only the embedding service decides that.
- **Moving a `TB()` text into another class gives it a new I18N key**, so its translation is made anew during
the next localization run.
## Databases
Local RAG runs on two databases, addressed through `DatabaseRole`:
@@ -343,12 +343,13 @@ public sealed class AssistantAuditAgent(ILogger<AssistantAuditAgent> logger, ILo
/// </summary>
/// <remarks>
/// The description is the one the tool gives a model, which is exactly what the assistant's
/// model would read. A tool this installation does not know is listed by its ID alone: the
/// plugin still asks for it, and a name nobody can resolve is itself worth seeing.
/// model would read. A collection is listed by its tools for the same reason. A tool this
/// installation does not know is listed by its ID alone: the plugin still asks for it, and a
/// name nobody can resolve is itself worth seeing.
/// </remarks>
private string FormatRequestedTools(PluginAssistants plugin)
{
var toolIds = ToolSelectionRules.NormalizeSelection(plugin.AssistantToolIds ?? plugin.ChatLaunchConfiguration?.ToolIds ?? []);
var toolIds = toolRegistry.ExpandSelection(plugin.AssistantToolIds ?? plugin.ChatLaunchConfiguration?.ToolIds ?? []);
if (toolIds.Count == 0)
return "None. This plugin does not request any tools.";
@@ -1,4 +1,5 @@
@using AIStudio.Chat
@using AIStudio.Settings.DataModel
@inherits AssistantLowerBase
@typeparam TSettings
@@ -180,7 +181,7 @@
@* No selection where the assistant's own rules already name the tools: *@
@if (this.SettingsManager.AreToolsEnabled() && this.AssistantManagedToolIds is null && this.SettingsManager.IsToolSelectionVisible(this.Component))
{
<ToolSelection Component="@this.Component" LLMProvider="@this.ProviderSettings" SelectedToolIds="@this.SelectedToolIds" SelectedToolIdsChanged="@this.SelectedToolIdsChanged" Disabled="@this.IsProcessing" />
<ToolSelection Component="@this.Component" LLMProvider="@this.ProviderSettings" RequiredOutboundDataRestriction="@(this.ChatThread?.RequiredOutboundDataRestriction ?? OutboundDataRequirement.NONE)" SelectedToolIds="@this.SelectedToolIds" SelectedToolIdsChanged="@this.SelectedToolIdsChanged" Disabled="@this.IsProcessing" />
}
<MudSpacer />
@@ -1,6 +1,7 @@
using AIStudio.Chat;
using AIStudio.Provider;
using AIStudio.Settings;
using AIStudio.Settings.DataModel;
using AIStudio.Dialogs.Settings;
using AIStudio.Tools.AIJobs;
using AIStudio.Tools.AssistantSessions;
@@ -197,7 +198,7 @@ public abstract partial class AssistantBase<TSettings> : AssistantLowerBase wher
this.ProviderSettings = this.SettingsManager.GetPreselectedProvider(this.Component);
this.CurrentProfile = this.SettingsManager.GetPreselectedProfile(this.Component);
this.CurrentChatTemplate = this.SettingsManager.GetPreselectedChatTemplate(this.Component);
this.SelectedToolIds = this.SettingsManager.GetDefaultToolIds(this.Component);
this.SelectedToolIds = this.ToolRegistry.GetDefaultToolIds(this.Component);
await this.OnDefaultsAppliedAsync();
this.assistantSessionKey = new(this.Component, this.AssistantSessionInstanceId);
await this.AttachAssistantSessionIfAvailable();
@@ -408,7 +409,7 @@ public abstract partial class AssistantBase<TSettings> : AssistantLowerBase wher
this.ProviderSettings = this.SettingsManager.GetPreselectedProvider(this.Component);
this.CurrentProfile = this.SettingsManager.GetPreselectedProfile(this.Component);
this.CurrentChatTemplate = this.SettingsManager.GetPreselectedChatTemplate(this.Component);
this.SelectedToolIds = this.SettingsManager.GetDefaultToolIds(this.Component);
this.SelectedToolIds = this.ToolRegistry.GetDefaultToolIds(this.Component);
}
/// <summary>
@@ -431,18 +432,20 @@ public abstract partial class AssistantBase<TSettings> : AssistantLowerBase wher
/// always has the last word: a tool asking for more confidence than the selected provider has
/// never reaches the model, no matter who put it on the list. That filter belongs here rather
/// than into the stored selection, because a provider with too little confidence must not cost
/// the user a tool for good.
/// the user a tool for good. The same goes for a tool which a mailbox the thread read from
/// keeps back.
/// </remarks>
protected HashSet<string> GetRunnableToolIds()
/// <param name="outboundDataRestriction">Where the thread the tools run in may still send data, see ChatThread.RequiredOutboundDataRestriction.</param>
protected HashSet<string> GetRunnableToolIds(OutboundDataRestriction outboundDataRestriction)
{
if (this.AssistantManagedToolIds is not null)
return this.ToolRegistry.FilterToolIdsForProvider(this.ProviderSettings, this.AssistantManagedToolIds);
return this.ToolRegistry.FilterToolIdsForProvider(this.ProviderSettings, this.AssistantManagedToolIds, outboundDataRestriction);
// What the user cannot see, the assistant does not use:
if (!this.SettingsManager.IsToolSelectionVisible(this.Component))
return [];
return this.ToolRegistry.FilterToolIdsForProvider(this.ProviderSettings, this.SelectedToolIds);
return this.ToolRegistry.FilterToolIdsForProvider(this.ProviderSettings, this.SelectedToolIds, outboundDataRestriction);
}
/// <summary>
@@ -454,7 +457,7 @@ public abstract partial class AssistantBase<TSettings> : AssistantLowerBase wher
/// </remarks>
protected Task SelectedToolIdsChanged(HashSet<string> updatedToolIds)
{
this.SelectedToolIds = ToolSelectionRules.NormalizeSelection(updatedToolIds);
this.SelectedToolIds = this.ToolRegistry.NormalizeSelection(updatedToolIds);
return Task.CompletedTask;
}
@@ -518,7 +521,7 @@ public abstract partial class AssistantBase<TSettings> : AssistantLowerBase wher
this.ChatThread.SelectedProvider = this.ProviderSettings.Id;
this.ChatThread.RuntimeComponent = this.Component;
this.ChatThread.SelectedToolIds = [..this.SelectedToolIds];
this.ChatThread.RuntimeSelectedToolIds = this.GetRunnableToolIds();
this.ChatThread.RuntimeSelectedToolIds = this.GetRunnableToolIds(this.ChatThread.RequiredOutboundDataRestriction.Restriction);
this.ChatThread.RuntimeToolsAreAssistantManaged = this.AssistantManagedToolIds is not null;
}
@@ -1014,7 +1017,7 @@ public abstract partial class AssistantBase<TSettings> : AssistantLowerBase wher
reader.Restore(RESULTING_CONTENT_BLOCK_STATE_KEY, value => this.ResultingContentBlock = value);
reader.Restore(INPUT_ISSUES_STATE_KEY, value => this.InputIssues = value);
reader.Restore(IS_PROCESSING_STATE_KEY, value => this.IsProcessing = value);
reader.Restore(SELECTED_TOOL_IDS_STATE_KEY, value => this.SelectedToolIds = ToolSelectionRules.NormalizeSelection(value));
reader.Restore(SELECTED_TOOL_IDS_STATE_KEY, value => this.SelectedToolIds = this.ToolRegistry.NormalizeSelection(value));
this.RestoreCustomAssistantSessionState(reader);
}
@@ -1,6 +1,7 @@
using AIStudio.Chat;
using AIStudio.Provider;
using AIStudio.Settings;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.ToolCallingSystem;
namespace AIStudio.Assistants.BatchProcessing;
@@ -110,7 +111,9 @@ public partial class AssistantBatchProcessing
Name = this.Title,
Blocks = [],
RuntimeComponent = this.Component,
RuntimeSelectedToolIds = this.GetRunnableToolIds(),
// A thread of its own, which has read no mailbox yet:
RuntimeSelectedToolIds = this.GetRunnableToolIds(OutboundDataRestriction.UNRESTRICTED),
// Always true here, unlike in the assistant base: a batch run takes its tools from the
// selected policy or from its own field, never from the tool selection in the footer.
@@ -6,7 +6,6 @@ using AIStudio.Tools.PluginSystem;
using AIStudio.Tools.PluginSystem.Assistants;
using AIStudio.Tools.PluginSystem.Assistants.DataModel;
using AIStudio.Tools.Services;
using AIStudio.Tools.ToolCallingSystem;
using Microsoft.AspNetCore.Components;
using DialogOptions = AIStudio.Dialogs.DialogOptions;
@@ -326,7 +325,7 @@ public partial class AssistantBuilder : AssistantBaseCore<NoSettingsPanel>
state.Restore(LAUNCHER_PROFILE_ID_STATE_KEY, value => this.launcherProfileId = value);
state.Restore(LAUNCHER_CHAT_TEMPLATE_ID_STATE_KEY, value => this.launcherChatTemplateId = value);
state.Restore(LAUNCHER_DATA_SOURCE_IDS_STATE_KEY, value => this.launcherDataSourceIds = value);
state.Restore(LAUNCHER_TOOL_IDS_STATE_KEY, value => this.launcherToolIds = ToolSelectionRules.NormalizeSelection(value));
state.Restore(LAUNCHER_TOOL_IDS_STATE_KEY, value => this.launcherToolIds = this.ToolRegistry.NormalizeSelection(value));
state.Restore(SELECTED_ASSISTANT_COMPONENTS_STATE_KEY, value => this.selectedAssistantComponents = value);
state.Restore(SELECTED_OUTPUT_LANGUAGE_STATE_KEY, value => this.selectedOutputLanguage = value);
state.Restore(CUSTOM_OUTPUT_LANGUAGE_STATE_KEY, value => this.customOutputLanguage = value);
@@ -542,7 +541,7 @@ public partial class AssistantBuilder : AssistantBaseCore<NoSettingsPanel>
return null;
var dataSourceIds = this.launcherDataSourceIds.Distinct(StringComparer.OrdinalIgnoreCase).ToArray();
var toolIds = ToolSelectionRules.NormalizeSelection(this.launcherToolIds).ToArray();
var toolIds = this.ToolRegistry.NormalizeSelection(this.launcherToolIds).ToArray();
return new(
this.launcherWorkspaceName.Trim(),
NullIfEmpty(this.launcherProviderId),
@@ -9,7 +9,6 @@ using AIStudio.Tools.PluginSystem;
using AIStudio.Tools.PluginSystem.Assistants;
using AIStudio.Tools.PluginSystem.Assistants.DataModel;
using AIStudio.Tools.Services;
using AIStudio.Tools.ToolCallingSystem;
using Lua;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.WebUtilities;
@@ -171,7 +170,7 @@ public partial class AssistantDynamic : AssistantBaseCore<NoSettingsPanel>
this.systemPrompt = pluginAssistant.SystemPrompt;
this.submitText = pluginAssistant.SubmitText;
this.allowProfiles = pluginAssistant.AllowProfiles;
this.assistantToolIds = ReadPluginToolIds(pluginAssistant);
this.assistantToolIds = this.ReadPluginToolIds(pluginAssistant);
this.showFooterProfileSelection = !pluginAssistant.HasEmbeddedProfileSelection;
this.pluginPath = pluginAssistant.PluginPath;
var pluginHash = pluginAssistant.ComputeAuditHash();
@@ -366,7 +365,7 @@ public partial class AssistantDynamic : AssistantBaseCore<NoSettingsPanel>
this.systemPrompt = updatedPlugin.SystemPrompt;
this.submitText = updatedPlugin.SubmitText;
this.allowProfiles = updatedPlugin.AllowProfiles;
this.assistantToolIds = ReadPluginToolIds(updatedPlugin);
this.assistantToolIds = this.ReadPluginToolIds(updatedPlugin);
this.showFooterProfileSelection = !updatedPlugin.HasEmbeddedProfileSelection;
this.pluginPath = updatedPlugin.PluginPath;
var pluginHash = updatedPlugin.ComputeAuditHash();
@@ -391,7 +390,7 @@ public partial class AssistantDynamic : AssistantBaseCore<NoSettingsPanel>
/// a plugin installed later, and dropping it here would silently turn a plugin that names tools
/// into one that lets the user choose.
/// </remarks>
private static HashSet<string>? ReadPluginToolIds(PluginAssistants plugin) => plugin.AssistantToolIds is { } toolIds ? ToolSelectionRules.NormalizeSelection(toolIds) : null;
private HashSet<string>? ReadPluginToolIds(PluginAssistants plugin) => plugin.AssistantToolIds is { } toolIds ? this.ToolRegistry.NormalizeSelection(toolIds) : null;
private string ResolveImageSource(AssistantImage image)
{
File diff suppressed because it is too large. Load diff
+88 -2
View File
@@ -7,6 +7,7 @@ using AIStudio.Settings;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.ToolCallingSystem;
using AIStudio.Tools.ERIClient.DataModel;
using AIStudio.Tools.Web;
namespace AIStudio.Chat;
@@ -126,6 +127,26 @@ public sealed record ChatThread
_ => this.DataSecurity,
};
/// <summary>
/// Where this chat may still send data, because of the mailboxes it read from.
/// </summary>
/// <remarks>
/// The third thing the thread ratchets, next to the data security and the required provider
/// confidence: once mail content is in the thread, every tool which runs afterwards could carry
/// it out, in a search query or in the address of a web page. It only ever gets stricter, and a
/// later change of the mailbox settings does not loosen it either, because the content was
/// read under the old one. A chat which read no mailbox demands nothing.
/// </remarks>
[JsonInclude]
public OutboundDataRequirement RequiredOutboundDataRestriction { get; private set; } = OutboundDataRequirement.NONE;
/// <summary>
/// Tightens where this chat may still send data to what the data brought in demands, and
/// never loosens it.
/// </summary>
/// <param name="requirement">What the data brought in demands.</param>
public void RequireOutboundDataRestriction(OutboundDataRequirement requirement) => this.RequiredOutboundDataRestriction = this.RequiredOutboundDataRestriction.StricterOf(requirement);
/// <summary>
/// The name of the chat thread. Usually generated by an AI model or manually edited by the user.
/// </summary>
@@ -147,6 +168,64 @@ public sealed record ChatThread
[JsonIgnore]
public HashSet<string> RuntimeSelectedToolIds { get; set; } = [];
/// <summary>
/// The web addresses tools returned in this chat since it was opened, as request keys, see
/// WebAddresses.CreateRequestKey.
/// </summary>
/// <remarks>
/// Addresses the model wrote into the call itself are left out, see ToolCallOutcome.ReturnedWebAddresses.
/// Not stored, like the results of the tools themselves: after a restart, a chat knows fewer
/// addresses, never more.
/// </remarks>
[JsonIgnore]
public HashSet<string> RuntimeWebAddressesFromTools { get; } = new(StringComparer.Ordinal);
/// <summary>
/// The web addresses in the system prompt the last request was sent with, as request keys.
/// </summary>
/// <remarks>
/// Collected by PrepareSystemPrompt, because the system prompt is put together only then: from
/// the chat template, the content of the data sources the RAG process found, the profile and
/// the tool policy. Not stored, since every request puts it together anew.
/// </remarks>
[JsonIgnore]
public IReadOnlySet<string> RuntimeSystemPromptWebAddresses { get; internal set; } = new HashSet<string>(StringComparer.Ordinal);
/// <summary>
/// Whether a web address was given to the model rather than chosen by it.
/// </summary>
/// <remarks>
/// Given means: it stands in the system prompt, see RuntimeSystemPromptWebAddresses, in a
/// message of the user or a document attached to one, see ContentText.RuntimeAttachmentWebAddresses,
/// or a tool returned it, see RuntimeWebAddressesFromTools. What the model wrote itself never
/// counts, its answers included, because the model could have put anything of the chat into it.
/// The comparison is the one of WebAddresses.CreateRequestKey, so an address counts when it
/// asks the server for the same.<br/><br/>
/// Shared by both rules of Read Web Page which depend on it: a free address choice switched
/// off, and a mailbox which allows the addresses from the chat only.
/// </remarks>
/// <param name="url">The address the model wants to read.</param>
/// <returns>True when the address was given to the model.</returns>
public bool IsWebAddressGivenToTheModel(Uri url)
{
var requestKey = WebAddresses.CreateRequestKey(url);
if (this.RuntimeWebAddressesFromTools.Contains(requestKey) || this.RuntimeSystemPromptWebAddresses.Contains(requestKey))
return true;
var messagesOfTheUser = this.Blocks
.Where(block => block.Role is ChatRole.USER)
.Select(block => block.Content)
.OfType<ContentText>()
.ToList();
if (messagesOfTheUser.Any(content => content.RuntimeAttachmentWebAddresses.Contains(requestKey)))
return true;
return messagesOfTheUser
.SelectMany(content => WebAddresses.Find(content.Text))
.Any(address => WebAddresses.TryCreateRequestKey(address, out var key) && string.Equals(key, requestKey, StringComparison.Ordinal));
}
/// <summary>
/// Whether the tools of this run were named by the assistant's own rules instead of chosen by
/// the user.
@@ -187,6 +266,13 @@ public sealed record ChatThread
this.SystemPrompt = prepared.BasePrompt;
LOGGER.LogInformation(prepared.Explanation);
var systemPromptWebAddresses = new HashSet<string>(StringComparer.Ordinal);
foreach (var address in WebAddresses.Find(prepared.Text))
if (WebAddresses.TryCreateRequestKey(address, out var requestKey))
systemPromptWebAddresses.Add(requestKey);
this.RuntimeSystemPromptWebAddresses = systemPromptWebAddresses;
return prepared.Text;
}
@@ -388,8 +474,8 @@ public sealed record ChatThread
/// because they are the user's choice rather than the result of a message.<br/><br/>
///
/// What stays as well is everything the thread ratchets for security reasons, namely the data
/// security and the required provider confidence. Both only ever tighten, because the data which
/// raised them was seen by this thread. Removing the message that brought it in does not unsee
/// security, the required provider confidence, and where the chat may still send data. They only
/// ever tighten, because the data which raised them was seen by this thread. Removing the message that brought it in does not unsee
/// it, so the chat keeps demanding the same of every provider which continues it.
/// </remarks>
/// <param name="content">The content to keep as the last block.</param>
+21 -1
View File
@@ -8,6 +8,7 @@ using AIStudio.Tools.RAG.RAGProcesses;
using AIStudio.Tools.Rust;
using AIStudio.Tools.Security;
using AIStudio.Tools.ToolCallingSystem;
using AIStudio.Tools.Web;
namespace AIStudio.Chat;
@@ -386,9 +387,23 @@ public sealed class ContentText : IContent
#endregion
/// <summary>
/// The web addresses in the documents attached to this message, as request keys, see
/// WebAddresses.CreateRequestKey.
/// </summary>
/// <remarks>
/// The documents are read from disk only when a message is sent, so their addresses are
/// collected right then, see PrepareTextContentForAI, and count as given to the model, see
/// ChatThread.IsWebAddressGivenToTheModel. Not stored: every request reads the documents anew,
/// and a document changed or gone since no longer gives an address.
/// </remarks>
[JsonIgnore]
public IReadOnlySet<string> RuntimeAttachmentWebAddresses { get; internal set; } = new HashSet<string>(StringComparer.Ordinal);
public async Task<string> PrepareTextContentForAI()
{
var sb = new StringBuilder();
var attachmentWebAddresses = new HashSet<string>(StringComparer.Ordinal);
sb.AppendLine(this.Text);
if(this.FileAttachments.Count > 0)
@@ -492,6 +507,10 @@ public sealed class ContentText : IContent
documentBlocks.AppendLine("````");
documentBlocks.AppendLine(extraction.Content);
documentBlocks.AppendLine("````");
foreach (var address in WebAddresses.Find(extraction.Content))
if (WebAddresses.TryCreateRequestKey(address, out var requestKey))
attachmentWebAddresses.Add(requestKey);
}
if (documentBlocks.Length > 0)
@@ -510,7 +529,8 @@ public sealed class ContentText : IContent
}
}
}
this.RuntimeAttachmentWebAddresses = attachmentWebAddresses;
return sb.ToString();
}
@@ -15,4 +15,24 @@ public static class StringExtensions
return input[(endIndex + CLOSE_TAG.Length)..];
}
/// <summary>
/// Shortens a text to the given number of characters, and marks the cut with three dots.
/// </summary>
/// <remarks>
/// Never cuts between the two halves of a surrogate pair, which no JSON writer takes.<br/><br/>
/// Takes and returns a string rather than a span on purpose: most texts are short enough, and
/// those come back as the very same instance. Only a text which is cut costs one new string.
/// </remarks>
/// <param name="text">The text.</param>
/// <param name="maxCharacters">How many characters of the text to keep at most, the dots not counted.</param>
/// <returns>The text as it was when it is short enough, otherwise its beginning followed by three dots.</returns>
public static string Shorten(this string text, int maxCharacters)
{
if (text.Length <= maxCharacters)
return text;
var end = char.IsHighSurrogate(text[maxCharacters - 1]) ? maxCharacters - 1 : maxCharacters;
return string.Concat(text.AsSpan(0, end).TrimEnd(), "...");
}
}
@@ -18,6 +18,9 @@ public partial class AssistantPluginSecurityCard : MSGComponentBase
[Inject]
private IDialogService DialogService { get; init; } = null!;
[Inject]
private ToolRegistry ToolRegistry { get; init; } = null!;
private PluginAssistantSecurityState SecurityState => this.Plugin is null
? new PluginAssistantSecurityState()
: PluginAssistantSecurityResolver.Resolve(this.SettingsManager, this.Plugin);
@@ -29,9 +32,10 @@ public partial class AssistantPluginSecurityCard : MSGComponentBase
/// Tools are a capability, not a detail: an assistant allowed to search the web or read a page
/// can carry what a user typed out of the app. Whoever decides whether to enable this plugin
/// should see that beforehand, which is why the count sits in the header next to the audit
/// level and the tools themselves are named in the details.
/// level and the tools themselves are named in the details. A collection is named by its
/// tools, since each of them is a capability of its own, see ToolRegistry.ExpandSelection.
/// </remarks>
private IReadOnlyList<string> PluginToolIds => ToolSelectionRules.NormalizeSelection(this.Plugin?.AssistantToolIds ?? this.Plugin?.ChatLaunchConfiguration?.ToolIds ?? [])
private IReadOnlyList<string> PluginToolIds => this.ToolRegistry.ExpandSelection(this.Plugin?.AssistantToolIds ?? this.Plugin?.ChatLaunchConfiguration?.ToolIds ?? [])
.OrderBy(x => x, StringComparer.Ordinal)
.ToList();
@@ -136,7 +136,7 @@
@if (this.SettingsManager.AreToolsEnabled())
{
<ToolSelection Component="Components.CHAT" LLMProvider="@this.Provider" SelectedToolIds="@this.selectedToolIds" SelectedToolIdsChanged="@this.SelectedToolIdsChanged" Disabled="@this.IsCurrentChatStreaming" />
<ToolSelection Component="Components.CHAT" LLMProvider="@this.Provider" RequiredOutboundDataRestriction="@(this.ChatThread?.RequiredOutboundDataRestriction ?? OutboundDataRequirement.NONE)" SelectedToolIds="@this.selectedToolIds" SelectedToolIdsChanged="@this.SelectedToolIdsChanged" Disabled="@this.IsCurrentChatStreaming" />
}
@if (PreviewFeatures.PRE_RAG_2024.IsEnabled(this.SettingsManager))
@@ -729,7 +729,7 @@ public partial class ChatComponent : MSGComponentBase
{
if (this.currentChatTemplate.ToolIds is not { } templateToolIds)
{
this.selectedToolIds = ToolSelectionRules.NormalizeSelection(this.SettingsManager.GetDefaultToolIds(Tools.Components.CHAT));
this.selectedToolIds = this.ToolRegistry.GetDefaultToolIds(Tools.Components.CHAT);
return;
}
@@ -1149,7 +1149,7 @@ public partial class ChatComponent : MSGComponentBase
this.StateHasChanged();
this.ChatThread!.RuntimeComponent = Tools.Components.CHAT;
this.ChatThread.SelectedToolIds = [..this.selectedToolIds];
this.ChatThread.RuntimeSelectedToolIds = this.ToolRegistry.FilterToolIdsForProvider(this.Provider, this.selectedToolIds);
this.ChatThread.RuntimeSelectedToolIds = this.ToolRegistry.FilterToolIdsForProvider(this.Provider, this.selectedToolIds, this.ChatThread.RequiredOutboundDataRestriction.Restriction);
await this.AIJobService.TryStartChatGenerationAsync(new ChatGenerationRequest
{
ChatThread = this.ChatThread,
@@ -1178,11 +1178,13 @@ public partial class ChatComponent : MSGComponentBase
/// the footer would keep showing the tools of the chat before it.
/// </remarks>
private void ApplyToolSelectionOfLoadedChat() =>
this.selectedToolIds = ToolSelectionRules.NormalizeSelection(this.ChatThread?.SelectedToolIds ?? this.SettingsManager.GetDefaultToolIds(Tools.Components.CHAT));
this.selectedToolIds = this.ChatThread?.SelectedToolIds is { } storedToolIds
? this.ToolRegistry.NormalizeSelection(storedToolIds)
: this.ToolRegistry.GetDefaultToolIds(Tools.Components.CHAT);
private void SelectedToolIdsChanged(HashSet<string> updatedToolIds)
{
this.selectedToolIds = ToolSelectionRules.NormalizeSelection(updatedToolIds);
this.selectedToolIds = this.ToolRegistry.NormalizeSelection(updatedToolIds);
//
// The thread keeps the selection so that reopening the chat tomorrow brings the same tools
@@ -1667,8 +1669,9 @@ public partial class ChatComponent : MSGComponentBase
/// The tools the next request would offer the model.
/// </summary>
/// <remarks>
/// Filtered for the provider the same way they are before sending, so that a tool the provider
/// is not trusted enough to receive does not count either.
/// Filtered for the provider and the chat the same way they are before sending, so that a tool
/// the provider is not trusted enough to receive does not count, and neither does one which a
/// mailbox the chat read from keeps back. A new chat read no mailbox, so nothing is kept back.
///
/// Asked for once and used twice: their policy goes into the system prompt, and their schemas
/// travel next to it in the request body. Both cost tokens, and both change the moment somebody
@@ -1679,14 +1682,18 @@ public partial class ChatComponent : MSGComponentBase
/// the request asks for.
///
/// Read Web Page likewise counts with its registered instructions, those of its default free
/// address choice. With the choice switched on, a request carries a shorter instruction, so the
/// count comes out a few tokens high.
/// address choice in a chat which read no mailbox. With the choice switched on, a request
/// carries a shorter instruction, and in a chat which read a mailbox a longer one, so the count
/// comes out a few tokens off. One case it gets wrong as a whole: a mailbox which allows the
/// configured services only, with no wiki configured, leaves Read Web Page nothing to offer, and
/// only the request finds that out, see ReadWebPageTool.ResolveFunctionAsync.
/// </remarks>
/// <param name="offersSemanticSearch">Whether the next request offers Semantic Search, see OffersSemanticSearchAsync.</param>
/// <returns>The definitions of the selected tools, and of Semantic Search when it is offered.</returns>
private IReadOnlyList<ToolDefinition> GetRunnableToolDefinitions(bool offersSemanticSearch)
{
var definitions = this.ToolRegistry.FilterToolIdsForProvider(this.Provider, this.selectedToolIds)
var outboundDataRestriction = (this.ChatThread?.RequiredOutboundDataRestriction ?? OutboundDataRequirement.NONE).Restriction;
var definitions = this.ToolRegistry.FilterToolIdsForProvider(this.Provider, this.selectedToolIds, outboundDataRestriction)
.Select(this.ToolRegistry.GetDefinition)
.Where(definition => definition is not null)
.Select(definition => definition!)
@@ -39,6 +39,7 @@ public partial class DataSourceCloudEmbeddingWarning : MSGComponentBase
{
DataSourceType.LOCAL_DIRECTORY => T("All files in this folder and its subfolders"),
DataSourceType.LOCAL_FILE => T("The selected file"),
DataSourceType.MAILBOX => T("Every mail of this mailbox in the selected folder and period, together with the text of its attachments,"),
_ => T("The selected data")
};
@@ -46,6 +47,7 @@ public partial class DataSourceCloudEmbeddingWarning : MSGComponentBase
{
DataSourceType.LOCAL_DIRECTORY => string.Format(T("All files in the folder '{0}' and its subfolders"), this.SourcePath),
DataSourceType.LOCAL_FILE => string.Format(T("The file '{0}'"), this.SourcePath),
DataSourceType.MAILBOX => string.Format(T("Every mail of the mailbox '{0}' in the selected folder and period, together with the text of its attachments,"), this.SourcePath),
_ => string.Format(T("The data source '{0}'"), this.SourcePath)
};
}
@@ -3,18 +3,25 @@
@inherits MSGComponentBase
<MudJustifiedText Typo="Typo.body1" Class="mb-3">
@T("You might configure different data sources. A data source can include one file, all files in a directory, or data from your company. Later, you can incorporate these data sources as needed when the AI requires this data to complete a certain task.")
@if (this.AreMailboxesEnabled)
{
@T("You might configure different data sources. A data source can include one file, all files in a directory, a mailbox, or data from your company. Later, you can incorporate these data sources as needed when the AI requires this data to complete a certain task.")
}
else
{
@T("You might configure different data sources. A data source can include one file, all files in a directory, or data from your company. Later, you can incorporate these data sources as needed when the AI requires this data to complete a certain task.")
}
</MudJustifiedText>
<MudStack Row="true" AlignItems="AlignItems.Center" Justify="Justify.SpaceBetween" Class="mb-3" Wrap="Wrap.Wrap">
<MudTextSwitch Label="@T("Automatic local data source refresh")" Value="@this.SettingsManager.ConfigurationData.App.DataSourceIndexing.AutomaticRefresh" LabelOn="@T("Local data sources refresh when files change.")" LabelOff="@T("Local data sources refresh only when triggered manually.")" ValueChanged="@this.AutomaticRefreshChanged"/>
<MudTextSwitch Label="@T("Automatic local data source refresh")" Value="@this.SettingsManager.ConfigurationData.App.DataSourceIndexing.AutomaticRefresh" LabelOn="@(this.AreMailboxesEnabled ? T("Local data sources refresh when files change, mailboxes every 16 minutes.") : T("Local data sources refresh when files change."))" LabelOff="@T("Local data sources refresh only when triggered manually.")" ValueChanged="@this.AutomaticRefreshChanged"/>
<MudTooltip Text="@T("Refresh all")">
<MudIconButton Color="Color.Primary" Icon="@Icons.Material.Filled.Sync" Disabled="@(!this.HasRefreshableDataSources())" OnClick="@this.RefreshAllDataSources"/>
</MudTooltip>
</MudStack>
@{ var embeddingStatuses = this.DataSourceEmbeddingService.GetStatuses().ToDictionary(status => status.DataSourceId, StringComparer.OrdinalIgnoreCase); }
<MudTable Items="@this.SettingsManager.ConfigurationData.DataSources" Hover="@true" Class="border-dashed border rounded-lg">
<MudTable T="IDataSourceBase" Items="@this.ConfiguredDataSources" Hover="@true" Class="border-dashed border rounded-lg">
<ColGroup>
<col style="width: 3em;"/>
<col/>
@@ -28,7 +35,7 @@
<MudTh>@T("Name")</MudTh>
<MudTh>@T("Type")</MudTh>
<MudTh>@T("Embedding")</MudTh>
<MudTh>@T("Indexed files")</MudTh>
<MudTh>@T("Indexed")</MudTh>
<MudTh>@T("Actions")</MudTh>
</HeaderContent>
<RowTemplate>
@@ -38,12 +45,12 @@
<MudTd Style="white-space: nowrap;">@context.Type.GetDisplayName()</MudTd>
<MudTd Style="white-space: nowrap;">@this.GetEmbeddingName(context)</MudTd>
<MudTd>
@if (context is IInternalDataSource)
@if (context is IIndexedDataSource)
{
<MudTooltip Text="@this.GetIndexingStatusTooltip(embeddingStatus)">
<MudStack Row="true" AlignItems="AlignItems.Center" Spacing="1">
<MudIcon Icon="@Icons.Material.Filled.Circle" Color="@GetIndexingStatusColor(embeddingStatus)" Size="Size.Small" Style="width: 0.75rem; height: 0.75rem; flex-shrink: 0;"/>
<MudText Typo="Typo.body2">@(embeddingStatus is null ? T("Not available") : string.Format(T("{0} of {1}"), embeddingStatus.IndexedFiles, embeddingStatus.TotalFiles))</MudText>
<MudText Typo="Typo.body2">@(embeddingStatus is null ? T("Not available") : string.Format(T("{0} of {1}"), embeddingStatus.IndexedDocuments.CompactCount(), embeddingStatus.TotalDocuments.CompactCount()))</MudText>
</MudStack>
</MudTooltip>
}
@@ -56,9 +63,12 @@
<MudTd>
<MudStack Row="true" Class="mb-2 mt-2" Spacing="1" Wrap="Wrap.NoWrap">
<MudTooltip Text="@T("Information")">
<MudIconButton Color="Color.Info" Icon="@Icons.Material.Outlined.Info" OnClick="@(() => this.ShowInformation(context))"/>
</MudTooltip>
@if (context is IDataSource dataSource)
{
<MudTooltip Text="@T("Information")">
<MudIconButton Color="Color.Info" Icon="@Icons.Material.Outlined.Info" OnClick="@(() => this.ShowInformation(dataSource))"/>
</MudTooltip>
}
@*
Outside the two branches below on purpose: an index which cannot be read is a
matter of this machine, not of the configuration. Hiding the repair for a data
@@ -80,7 +90,7 @@
else
{
<MudTooltip Text="@T("Edit")">
<MudIconButton Color="Color.Info" Icon="@Icons.Material.Filled.Edit" OnClick="@(() => this.EditDataSource(context))"/>
<MudIconButton Color="Color.Info" Icon="@Icons.Material.Filled.Edit" OnClick="@(() => this.EditEntry(context))"/>
</MudTooltip>
<MudTooltip Text="@T("Refresh")">
<MudIconButton Color="Color.Primary" Icon="@Icons.Material.Filled.Sync" Disabled="@(!this.CanRefreshDataSource(context))" OnClick="@(() => this.RefreshDataSource(context))"/>
@@ -90,7 +100,7 @@
<AdminExportButton OnClick="@(() => this.ExportDataSource(context))" />
}
<MudTooltip Text="@T("Delete")">
<MudIconButton Color="Color.Error" Icon="@Icons.Material.Filled.Delete" OnClick="@(() => this.DeleteDataSource(context))"/>
<MudIconButton Color="Color.Error" Icon="@Icons.Material.Filled.Delete" OnClick="@(() => this.DeleteEntry(context))"/>
</MudTooltip>
}
</MudStack>
@@ -98,7 +108,7 @@
</RowTemplate>
</MudTable>
@if (this.SettingsManager.ConfigurationData.DataSources.Count == 0)
@if (!this.ConfiguredDataSources.Any())
{
<MudText Typo="Typo.h6" Class="mt-3">
@T("No data sources configured yet.")
@@ -115,4 +125,11 @@
<MudMenuItem OnClick="@(() => this.AddDataSource(DataSourceType.LOCAL_FILE))">
@T("Local File")
</MudMenuItem>
@if (PreviewFeatures.PRE_MAILBOXES_2026.IsEnabled(this.SettingsManager))
{
@* Shown with a lock rather than left out, like the locked buttons for adding providers: *@
<MudMenuItem OnClick="@this.AddMailbox" Disabled="@(!this.MayAddMailbox)" Icon="@(this.MayAddMailbox ? null : Icons.Material.Filled.Lock)">
@(this.MayAddMailbox ? T("Mailbox (IMAP)") : T("Mailbox (IMAP), not allowed by your organization"))
</MudMenuItem>
}
</MudMenu>
@@ -1,6 +1,7 @@
using AIStudio.Dialogs;
using AIStudio.Settings;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.Databases;
using AIStudio.Tools.ERIClient.DataModel;
using AIStudio.Tools.PluginSystem;
using AIStudio.Tools.Services;
@@ -26,8 +27,21 @@ public partial class DataSourceManagement : MSGComponentBase
[Inject]
private RustService RustService { get; init; } = null!;
[Inject]
private DatabaseClientProvider DatabaseClientProvider { get; init; } = null!;
private readonly List<ConfigurationSelectData<string>> availableEmbeddingProviders = new();
/// <summary>
/// The data sources and then the mailboxes, which the table shows together.
/// </summary>
private IEnumerable<IDataSourceBase> ConfiguredDataSources => this.SettingsManager.ConfigurationData.DataSources
.Concat(this.SettingsManager.ConfigurationData.Mailboxes.Cast<IDataSourceBase>());
private bool AreMailboxesEnabled => PreviewFeatures.PRE_MAILBOXES_2026.IsEnabled(this.SettingsManager);
private bool MayAddMailbox => this.SettingsManager.ConfigurationData.App.AllowUserToAddMailbox;
#region Overrides of ComponentBase
protected override async Task OnInitializedAsync()
@@ -77,7 +91,7 @@ public partial class DataSourceManagement : MSGComponentBase
if (status is null || status.State is DataSourceEmbeddingState.IDLE or DataSourceEmbeddingState.QUEUED or DataSourceEmbeddingState.RUNNING)
return Color.Warning;
return status.State is DataSourceEmbeddingState.FAILED || status.FailedFiles > 0
return status.State is DataSourceEmbeddingState.FAILED || status.FailedDocuments > 0
? Color.Error
: Color.Success;
}
@@ -95,17 +109,18 @@ public partial class DataSourceManagement : MSGComponentBase
if (status is null)
return T("Waiting for indexing status");
if (status.PermanentlySkippedFiles == 0)
// Only files are skipped for want of readable text. A mail is read from the server, never from a file:
if (status.PermanentlySkippedDocuments == 0 || status.DataSourceType is DataSourceType.MAILBOX)
return status.StateLabel;
return $"{status.StateLabel} — {string.Format(T("{0} files were skipped because they contain no readable text. AI Studio reads them again once they change."), status.PermanentlySkippedFiles)}";
return $"{status.StateLabel} — {string.Format(T("{0} files were skipped because they contain no readable text. AI Studio reads them again once they change."), status.PermanentlySkippedDocuments)}";
}
private string GetEmbeddingName(IDataSource dataSource)
private string GetEmbeddingName(IDataSourceBase dataSource)
{
if(dataSource is IInternalDataSource internalDataSource)
if(dataSource is IIndexedDataSource indexedDataSource)
{
var matchedEmbedding = this.SettingsManager.ConfigurationData.EmbeddingProviders.FirstOrDefault(x => x.Id == internalDataSource.EmbeddingId);
var matchedEmbedding = this.SettingsManager.ConfigurationData.EmbeddingProviders.FirstOrDefault(x => x.Id == indexedDataSource.EmbeddingId);
if(matchedEmbedding == default)
return T("No valid embedding");
@@ -118,14 +133,14 @@ public partial class DataSourceManagement : MSGComponentBase
return T("Unknown");
}
private bool CanRefreshDataSource(IDataSource dataSource)
private bool CanRefreshDataSource(IDataSourceBase dataSource)
{
return this.DataSourceEmbeddingService.CanRefreshDataSource(dataSource);
}
private bool HasRefreshableDataSources()
{
return this.SettingsManager.ConfigurationData.DataSources.Any(this.CanRefreshDataSource);
return this.ConfiguredDataSources.Any(this.CanRefreshDataSource);
}
/// <remarks>
@@ -134,12 +149,12 @@ public partial class DataSourceManagement : MSGComponentBase
/// every internal data source regardless of state, and singling this one out would say more
/// about the state than that button ever has.
/// </remarks>
private bool CanRepairDataSource(IDataSource dataSource)
private bool CanRepairDataSource(IDataSourceBase dataSource)
{
return this.DataSourceEmbeddingService.NeedsIndexRepair(dataSource);
}
private async Task RepairDataSource(IDataSource dataSource)
private async Task RepairDataSource(IDataSourceBase dataSource)
{
if (!this.CanRepairDataSource(dataSource))
return;
@@ -161,7 +176,7 @@ public partial class DataSourceManagement : MSGComponentBase
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
}
private async Task RefreshDataSource(IDataSource dataSource)
private async Task RefreshDataSource(IDataSourceBase dataSource)
{
if (!this.CanRefreshDataSource(dataSource))
return;
@@ -235,7 +250,7 @@ public partial class DataSourceManagement : MSGComponentBase
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
}
private async Task ExportDataSource(IDataSource dataSource)
private async Task ExportDataSource(IDataSourceBase dataSource)
{
if (!this.SettingsManager.ConfigurationData.App.ShowAdminSettings)
return;
@@ -442,6 +457,112 @@ public partial class DataSourceManagement : MSGComponentBase
}
}
/// <summary>
/// Edits a row of the table, which is either a data source or a mailbox.
/// </summary>
private Task EditEntry(IDataSourceBase entry) => entry switch
{
DataSourceMailbox mailbox => this.EditMailbox(mailbox),
IDataSource dataSource => this.EditDataSource(dataSource),
_ => Task.CompletedTask,
};
/// <summary>
/// Deletes a row of the table, which is either a data source or a mailbox.
/// </summary>
private Task DeleteEntry(IDataSourceBase entry) => entry switch
{
DataSourceMailbox mailbox => this.DeleteMailbox(mailbox),
IDataSource dataSource => this.DeleteDataSource(dataSource),
_ => Task.CompletedTask,
};
private async Task AddMailbox()
{
if (!this.MayAddMailbox)
return;
var dialogParameters = new DialogParameters<DataSourceMailboxDialog>
{
{ x => x.IsEditing, false },
{ x => x.AvailableEmbeddings, this.availableEmbeddingProviders }
};
var dialogReference = await this.DialogService.ShowAsync<DataSourceMailboxDialog>(T("Add Mailbox"), dialogParameters, DialogOptions.FULLSCREEN);
var dialogResult = await dialogReference.Result;
if (dialogResult is null || dialogResult.Canceled)
return;
var mailbox = (DataSourceMailbox)dialogResult.Data!;
//
// The dialog refuses to add a mailbox once the organization rules them out, but the rule
// may change while it stores the password. Then the password goes again, as far as possible:
//
if (!this.MayAddMailbox)
{
await this.RustService.DeleteSecret(mailbox, SecretStoreType.DATA_SOURCE);
return;
}
mailbox = mailbox with { Num = this.SettingsManager.ConfigurationData.NextDataSourceNum++ };
this.SettingsManager.ConfigurationData.Mailboxes.Add(mailbox);
await this.SettingsManager.StoreSettings();
await this.DataSourceEmbeddingService.QueueDataSourceAsync(mailbox);
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
}
private async Task EditMailbox(DataSourceMailbox mailbox)
{
if (await MailboxEditing.EditAsync(this.DialogService, this.SettingsManager, this.DataSourceEmbeddingService, mailbox.Id))
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
}
/// <summary>
/// Deletes a mailbox from AI Studio: its settings, its password, its index, and its recorded sign-in failure.
/// </summary>
/// <remarks>
/// The sign-in failure is deleted on its own, since it outlives the index on purpose: repairing
/// an index must not give the server a new attempt with a password it already refused.
/// </remarks>
private async Task DeleteMailbox(DataSourceMailbox mailbox)
{
if (mailbox.IsEnterpriseConfiguration)
return;
var dialogParameters = new DialogParameters<ConfirmDialog>
{
{ x => x.Message, string.Format(T("Are you sure you want to delete the mailbox '{0}'? Your mails stay on the server as they are. AI Studio only deletes its index of them and the stored password."), mailbox.Name) },
};
var dialogReference = await this.DialogService.ShowAsync<ConfirmDialog>(T("Delete Mailbox"), dialogParameters, DialogOptions.FULLSCREEN);
var dialogResult = await dialogReference.Result;
if (dialogResult is null || dialogResult.Canceled)
return;
var deleteSecretResponse = await this.RustService.DeleteSecret(mailbox, SecretStoreType.DATA_SOURCE);
if (!deleteSecretResponse.Success)
{
await this.DialogService.ShowMessageBox(
T("Delete Mailbox"),
string.Format(T("The password of this mailbox could not be deleted from the operating system, so the mailbox was kept. The issue was: {0}"), deleteSecretResponse.Issue),
T("Close"));
return;
}
this.SettingsManager.ConfigurationData.Mailboxes.Remove(mailbox);
await this.SettingsManager.StoreSettings();
await this.DataSourceEmbeddingService.RemoveDataSourceAsync(mailbox);
var indexStore = await this.DatabaseClientProvider.GetIndexStoreAsync();
await indexStore.ClearMailboxAuthFailureAsync(mailbox.Id, CancellationToken.None);
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
}
private async Task ShowInformation(IDataSource dataSource)
{
switch (dataSource)
@@ -0,0 +1,28 @@
@using AIStudio.Tools.Mail
@inherits MSGComponentBase
<MudTreeView T="string" Items="@this.treeItems" SelectionMode="SelectionMode.SingleSelection" SelectedValue="@this.SelectedFolder" SelectedValueChanged="@this.SelectFolder" Hover="@true" Dense="@true" Disabled="@this.Disabled" Class="mb-3">
<ItemTemplate Context="item">
<MudTreeViewItem T="string" Value="@item.Value" Text="@item.Text" Icon="@item.Icon" @bind-Expanded="@item.Expanded" CanExpand="@item.Expandable" Items="@item.Children" Disabled="@(!this.CanBeSelected(item.Value))"/>
</ItemTemplate>
</MudTreeView>
@if (this.CreateFolder is not null && !this.Disabled)
{
<MudStack Row="@true" AlignItems="AlignItems.Center" Class="mb-3">
<MudTextField
T="string"
@bind-Text="@this.newFolderName"
Label="@T("New folder")"
Immediate="@true"
MaxLength="@ImapMailboxConnector.MAX_FOLDER_NAME_LENGTH"
HelperText="@this.NewFolderHelperText"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.CreateNewFolder"
AdornmentColor="Color.Info"
Variant="Variant.Outlined"/>
<MudButton Variant="Variant.Filled" Color="Color.Primary" StartIcon="@Icons.Material.Filled.CreateNewFolder" Disabled="@(!this.CanCreateFolder)" OnClick="@this.CreateNewFolder">
@T("Create")
</MudButton>
</MudStack>
}
@@ -0,0 +1,234 @@
using AIStudio.Tools.Mail;
using Microsoft.AspNetCore.Components;
namespace AIStudio.Components;
/// <summary>
/// Shows the folders of a mailbox as the server lists them, picks one of them, and creates new ones.
/// </summary>
/// <remarks>
/// The whole mailbox is the root of the tree and stands for no folder at all, i.e., an empty full
/// name. The trash and the junk folder cannot be picked: the synchronization leaves them out, so a
/// mailbox restricted to one of them would never hold a mail.
/// </remarks>
public partial class MailFolderPicker : MSGComponentBase
{
/// <summary>
/// The folders of the mailbox, as the server lists them.
/// </summary>
/// <remarks>
/// The tree is built anew whenever another list arrives, so pass a new list instead of changing this one.
/// </remarks>
[Parameter]
public IReadOnlyList<MailServerFolder> Folders { get; set; } = [];
/// <summary>
/// The full name of the picked folder, or empty for the whole mailbox.
/// </summary>
[Parameter]
public string SelectedFolder { get; set; } = string.Empty;
[Parameter]
public EventCallback<string> SelectedFolderChanged { get; set; }
[Parameter]
public bool Disabled { get; set; }
/// <summary>
/// Creates a folder on the server, from the full name of the folder to create it in (empty at
/// the top level) and the name of the new folder.
/// </summary>
/// <remarks>
/// Returns the new folder, or null when the server did not create it, in which case the caller
/// tells the user why. Without it, there is no way to create a folder here.
/// </remarks>
[Parameter]
public Func<string, string, Task<MailServerFolder?>>? CreateFolder { get; set; }
private IReadOnlyCollection<TreeItemData<string>> treeItems = [];
private IReadOnlyList<MailServerFolder>? treeFolders;
private Dictionary<string, MailServerFolder> foldersByName = new(StringComparer.Ordinal);
private string newFolderName = string.Empty;
private bool isCreatingFolder;
#region Overrides of ComponentBase
protected override void OnParametersSet()
{
if (!ReferenceEquals(this.treeFolders, this.Folders))
{
this.treeFolders = this.Folders;
this.foldersByName = IndexByName(this.Folders);
this.treeItems = BuildTree(this.Folders, this.SelectedFolder, T("Whole mailbox"));
}
base.OnParametersSet();
}
#endregion
private char NewFolderSeparator => this.foldersByName.TryGetValue(this.SelectedFolder, out var parent)
? parent.DirectorySeparator
: this.Folders.FirstOrDefault()?.DirectorySeparator ?? '\0';
private bool IsNewFolderNameValid => ImapMailboxConnector.IsValidFolderName(this.newFolderName.Trim(), this.NewFolderSeparator);
private bool CanCreateFolder => !this.isCreatingFolder && this.IsNewFolderNameValid;
private string NewFolderHelperText
{
get
{
if (!string.IsNullOrWhiteSpace(this.newFolderName) && !this.IsNewFolderNameValid)
return T("This name is too long, or it contains a character the server reserves for folder paths.");
return string.IsNullOrEmpty(this.SelectedFolder)
? T("Creates the folder on the server right away, at the top level of the mailbox.")
: string.Format(T("Creates the folder on the server right away, inside '{0}'."), this.SelectedFolder);
}
}
private bool CanBeSelected(string? fullName)
{
if (fullName is null)
return false;
if (fullName.Length == 0)
return true;
return this.foldersByName.TryGetValue(fullName, out var folder) && CanBeSelected(folder);
}
private async Task SelectFolder(string? fullName)
{
if (!this.CanBeSelected(fullName) || fullName == this.SelectedFolder)
return;
this.SelectedFolder = fullName!;
await this.SelectedFolderChanged.InvokeAsync(this.SelectedFolder);
}
private async Task CreateNewFolder()
{
if (this.CreateFolder is null || !this.CanCreateFolder)
return;
this.isCreatingFolder = true;
try
{
var createdFolder = await this.CreateFolder(this.SelectedFolder, this.newFolderName.Trim());
if (createdFolder is null)
return;
this.newFolderName = string.Empty;
this.SelectedFolder = createdFolder.FullName;
await this.SelectedFolderChanged.InvokeAsync(this.SelectedFolder);
}
finally
{
this.isCreatingFolder = false;
}
}
/// <summary>
/// Whether a folder can be picked, cf. the remarks of this component.
/// </summary>
internal static bool CanBeSelected(MailServerFolder folder) => folder.SpecialUse is not (MailFolderSpecialUse.TRASH or MailFolderSpecialUse.JUNK);
/// <summary>
/// Builds the tree of folders below the whole mailbox, with the way to the picked folder expanded.
/// </summary>
/// <remarks>
/// A folder whose parent the server did not list goes to the top level, so that nothing listed
/// gets lost. Each level starts with the inbox, and the other folders follow by name.
/// </remarks>
/// <param name="folders">The folders as the server lists them.</param>
/// <param name="selectedFolder">The full name of the picked folder, or empty for the whole mailbox.</param>
/// <param name="wholeMailboxText">The text of the root, which stands for the whole mailbox.</param>
/// <returns>The root of the tree, as the only item.</returns>
internal static List<TreeItemData<string>> BuildTree(IReadOnlyList<MailServerFolder> folders, string selectedFolder, string wholeMailboxText)
{
var foldersByName = IndexByName(folders);
var childrenByParent = foldersByName.Values
.ToLookup(folder => foldersByName.ContainsKey(folder.ParentFullName) && folder.ParentFullName != folder.FullName ? folder.ParentFullName : string.Empty, StringComparer.Ordinal);
//
// Every folder on the way to the picked one is expanded. Should a server list two folders as
// each other's parent, the way ends where it meets itself:
//
var expandedFolders = new HashSet<string>(StringComparer.Ordinal);
var current = selectedFolder;
while (foldersByName.TryGetValue(current, out var folder) && expandedFolders.Add(folder.ParentFullName))
current = folder.ParentFullName;
var topLevelFolders = BuildChildren(string.Empty);
return
[
new TreeItemData<string>
{
Value = string.Empty,
Text = wholeMailboxText,
Icon = Icons.Material.Filled.Mail,
Expanded = true,
Expandable = topLevelFolders.Count > 0,
Children = topLevelFolders,
},
];
List<TreeItemData<string>> BuildChildren(string parentFullName) => childrenByParent[parentFullName]
.OrderByDescending(folder => folder.IsInbox)
.ThenBy(folder => folder.Name, StringComparer.OrdinalIgnoreCase)
.ThenBy(folder => folder.FullName, StringComparer.Ordinal)
.Select(folder =>
{
var children = BuildChildren(folder.FullName);
return new TreeItemData<string>
{
Value = folder.FullName,
Text = folder.Name,
Icon = GetIcon(folder),
Expanded = expandedFolders.Contains(folder.FullName),
Expandable = children.Count > 0,
Children = children,
};
})
.ToList();
}
/// <remarks>
/// The empty full name belongs to the whole mailbox, so a folder listed under it would become
/// its own child.
/// </remarks>
private static Dictionary<string, MailServerFolder> IndexByName(IEnumerable<MailServerFolder> folders)
{
var foldersByName = new Dictionary<string, MailServerFolder>(StringComparer.Ordinal);
foreach (var folder in folders)
{
if (folder.FullName.Length > 0)
foldersByName.TryAdd(folder.FullName, folder);
}
return foldersByName;
}
private static string GetIcon(MailServerFolder folder)
{
if (folder.IsInbox)
return Icons.Material.Filled.Inbox;
return folder.SpecialUse switch
{
MailFolderSpecialUse.ALL => Icons.Material.Filled.AllInbox,
MailFolderSpecialUse.ARCHIVE => Icons.Material.Filled.Archive,
MailFolderSpecialUse.DRAFTS => Icons.Material.Filled.Drafts,
MailFolderSpecialUse.FLAGGED => Icons.Material.Filled.Flag,
MailFolderSpecialUse.IMPORTANT => Icons.Material.Filled.LabelImportant,
MailFolderSpecialUse.JUNK => Icons.Material.Filled.Report,
MailFolderSpecialUse.SENT => Icons.Material.Filled.Send,
MailFolderSpecialUse.TRASH => Icons.Material.Filled.Delete,
_ => Icons.Material.Filled.Folder,
};
}
}
@@ -57,9 +57,10 @@ public partial class ManagedToolsWarning : MSGComponentBase
if (this.ToolIds.Count is 0 || !this.SettingsManager.AreToolsEnabled())
return [];
var collectionIds = this.ToolRegistry.NormalizeSelection(this.ToolIds);
return this.availableTools
.Where(x => this.ToolIds.Contains(x.Definition.Id) && x.IsActive && !x.ConfigurationState.IsConfigured)
.Select(x => x.Implementation.GetDisplayName())
.Where(x => collectionIds.Contains(x.Id) && x.IsActive && !x.ConfigurationState.IsConfigured)
.Select(x => x.DisplayName)
.ToList();
}
}
@@ -82,10 +83,11 @@ public partial class ManagedToolsWarning : MSGComponentBase
? ConfidenceLevel.NONE
: this.ProviderSettings.UsedLLMProvider.GetConfidence(this.SettingsManager).Level;
var collectionIds = this.ToolRegistry.NormalizeSelection(this.ToolIds);
return this.availableTools
.Where(x => this.ToolIds.Contains(x.Definition.Id) && x.IsActive)
.Where(x => collectionIds.Contains(x.Id) && x.IsActive)
.Where(x => !ToolSelectionRules.IsProviderConfidenceAllowed(providerConfidence, x.MinimumProviderConfidence))
.Select(x => x.Implementation.GetDisplayName())
.Select(x => x.DisplayName)
.ToList();
}
}
@@ -16,13 +16,13 @@
</HeaderContent>
<RowTemplate>
<MudTd>
<MudIcon Icon="@context.Implementation.Icon" Color="Color.Info" />
<MudIcon Icon="@context.Icon" Color="Color.Info" />
</MudTd>
<MudTd>
<MudText Typo="Typo.body1">@context.Implementation.GetDisplayName()</MudText>
<MudText Typo="Typo.body1">@context.DisplayName</MudText>
</MudTd>
<MudTd>
<MudText Typo="Typo.body2">@context.Implementation.GetDescription()</MudText>
<MudText Typo="Typo.body2">@context.Description</MudText>
</MudTd>
<MudTd>
<MudMenu StartIcon="@Icons.Material.Filled.Security" EndIcon="@Icons.Material.Filled.KeyboardArrowDown" Label="@this.GetCurrentConfidenceLevelName(context)" Variant="Variant.Filled" Style="@this.SetCurrentConfidenceLevelColorStyle(context)" Disabled="@this.IsToolConfidenceManaged()">
@@ -55,9 +55,9 @@
<MudTd>
<MudStack Row="true" Wrap="Wrap.NoWrap" AlignItems="AlignItems.Center" Spacing="0">
<MudTooltip Text="@T("Settings")">
<MudIconButton Icon="@Icons.Material.Filled.Settings" OnClick="@(async () => await this.OpenSettings(context.Definition.Id))" aria-label="@T("Settings")" />
<MudIconButton Icon="@Icons.Material.Filled.Settings" OnClick="@(async () => await this.OpenSettings(context))" aria-label="@T("Settings")" />
</MudTooltip>
<AdminExportButton OnClick="@(() => this.OpenExport(context.Definition.Id))" />
<AdminExportButton OnClick="@(() => this.OpenExport(context))" />
</MudStack>
</MudTd>
</RowTemplate>
@@ -21,11 +21,11 @@ public partial class SettingsPanelTools : SettingsPanelBase
await base.OnInitializedAsync();
}
private async Task OpenSettings(string toolId)
private async Task OpenSettings(ToolCatalogItem item)
{
var parameters = new DialogParameters<ToolSettingsDialog>
{
{ x => x.ToolId, toolId },
{ x => x.CollectionId, item.Id },
};
var dialog = await this.DialogService.ShowAsync<ToolSettingsDialog>(null, parameters, Dialogs.DialogOptions.FULLSCREEN);
@@ -34,14 +34,14 @@ public partial class SettingsPanelTools : SettingsPanelBase
this.StateHasChanged();
}
private async Task OpenExport(string toolId)
private async Task OpenExport(ToolCatalogItem item)
{
if (!this.SettingsManager.ConfigurationData.App.ShowAdminSettings)
return;
var parameters = new DialogParameters<ToolSettingsExportDialog>
{
{ x => x.ToolId, toolId },
{ x => x.CollectionId, item.Id },
};
await this.DialogService.ShowAsync<ToolSettingsExportDialog>(null, parameters, Dialogs.DialogOptions.FULLSCREEN);
@@ -54,13 +54,18 @@ public partial class SettingsPanelTools : SettingsPanelBase
_ => string.Format(this.T("Missing required settings: {0}"), string.Join(", ", item.ConfigurationState.MissingRequiredFields.Select(fieldName => this.GetFieldDisplayName(item, fieldName))))
};
/// <remarks>
/// The missing fields are those of the first tool whose settings are incomplete, see
/// ToolCatalogItem.ConfigurationState, so that tool names them.
/// </remarks>
private string GetFieldDisplayName(ToolCatalogItem item, string fieldName)
{
var fieldDefinition = item.Definition.SettingsSchema.Properties.GetValueOrDefault(fieldName);
if (fieldDefinition is null)
var tool = item.Tools.FirstOrDefault(tool => !tool.ConfigurationState.IsConfigured);
var fieldDefinition = tool?.Definition.SettingsSchema.Properties.GetValueOrDefault(fieldName);
if (tool is null || fieldDefinition is null)
return fieldName;
return item.Implementation.GetSettingsFieldLabel(fieldName, fieldDefinition);
return tool.Implementation.GetSettingsFieldLabel(fieldName, fieldDefinition);
}
private IEnumerable<ConfidenceLevel> GetSelectableConfidenceLevels() =>
@@ -85,7 +90,7 @@ public partial class SettingsPanelTools : SettingsPanelBase
private async Task ChangeMinimumProviderConfidence(ToolCatalogItem item, ConfidenceLevel confidenceLevel)
{
this.SettingsManager.SetMinimumProviderConfidenceForTool(item.Definition.Id, confidenceLevel, item.Definition.MinimumProviderConfidence);
this.ToolRegistry.SetMinimumProviderConfidence(item.Id, confidenceLevel);
await this.SettingsManager.StoreSettings();
this.items = await this.ToolRegistry.GetCatalogAsync(this.ToolRegistry.GetAllDefinitions());
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
@@ -26,6 +26,13 @@
<MudIconButton Icon="@Icons.Material.Filled.FolderOpen" Size="Size.Small" OnClick="@(() => this.ShowInFileManager(document))"/>
</MudTooltip>
}
else if (entry.IsMail)
{
@* Nothing opens a mail by its address yet, so a link would lead nowhere. *@
<MudText Typo="Typo.body1" Inline="true">
@entry.Title
</MudText>
}
else
{
<MudLink Href="@entry.Link" Target="_blank" Typo="Typo.body1">
@@ -79,7 +79,7 @@ public partial class SourcesList : MSGComponentBase
foreach (var numberedSource in group.Sources)
{
var document = numberedSource.Source.TryGetDocumentLocation(out var location) ? location : (SourceDocumentLocation?)null;
entries.Add(new(numberedSource.Number, numberedSource.Source.Title, numberedSource.Source.URL, document));
entries.Add(new(numberedSource.Number, numberedSource.Source.Title, numberedSource.Source.URL, document, numberedSource.Source.IsMailSource()));
}
this.groups.Add(new(group.Heading, entries));
@@ -160,5 +160,6 @@ public partial class SourcesList : MSGComponentBase
/// <param name="Title">The title of the source.</param>
/// <param name="Link">The address of the source, which a web source is opened by.</param>
/// <param name="Document">The document the source names, or null when it names none.</param>
private readonly record struct SourceEntry(int Number, string Title, string Link, SourceDocumentLocation? Document);
/// <param name="IsMail">Whether the source names a mail, which is shown as text.</param>
private readonly record struct SourceEntry(int Number, string Title, string Link, SourceDocumentLocation? Document, bool IsMail);
}
@@ -38,17 +38,17 @@ public partial class ToolDefaultsConfiguration : MSGComponentBase
this.Component is not AIStudio.Tools.Components.CHAT &&
!this.SettingsManager.IsToolSelectionVisible(this.Component);
private bool IsToolDisabled(string toolId) => !this.SettingsManager.IsToolActive(toolId);
private bool IsToolDisabled(string toolId) => !this.ToolRegistry.IsToolActive(toolId);
protected override async Task OnInitializedAsync()
{
this.availableTools = (await this.ToolRegistry.GetCatalogAsync(this.Component))
.Select(x => new ConfigurationSelectData<string>(x.Implementation.GetDisplayName(), x.Definition.Id))
.Select(x => new ConfigurationSelectData<string>(x.DisplayName, x.Id))
.ToList();
await base.OnInitializedAsync();
}
private HashSet<string> GetSelectedValues() => this.SettingsManager.GetDefaultToolIds(this.Component);
private HashSet<string> GetSelectedValues() => this.ToolRegistry.GetDefaultToolIds(this.Component);
private void UpdateSelection(HashSet<string> values) => this.SettingsManager.ConfigurationData.Tools.DefaultToolIdsByComponent[this.Component.ToString()] = [..ToolSelectionRules.NormalizeSelection(values)];
private void UpdateSelection(HashSet<string> values) => this.SettingsManager.ConfigurationData.Tools.DefaultToolIdsByComponent[this.Component.ToString()] = [..this.ToolRegistry.NormalizeSelection(values)];
}
@@ -43,7 +43,7 @@
<div class="tool-selection-rows">
@foreach (var item in this.catalog)
{
var isSelected = this.SelectedToolIds.Contains(item.Definition.Id);
var isSelected = this.SelectedToolIds.Contains(item.Id);
var warningText = this.GetWarningText(item);
<div class="tool-selection-row">
<MudStack Row="true" AlignItems="AlignItems.Center" Justify="Justify.SpaceBetween" Spacing="1">
@@ -66,19 +66,19 @@
box and would otherwise swallow the clicks landing in that strip.
*@
<MudCheckBox T="bool" Size="Size.Small" Dense="@true" Color="Color.Primary" Value="@isSelected" ReadOnly="@true" Disabled="@this.IsRowDisabled(item)" Style="pointer-events: none;" />
<MudIcon Icon="@item.Implementation.Icon" Color="Color.Info" />
<MudIcon Icon="@item.Icon" Color="Color.Info" />
@if (!item.IsActive)
{
<MudTooltip Text="@T("This tool has been disabled by your organization.")">
<MudIcon Icon="@Icons.Material.Filled.Lock" Color="Color.Error" Size="Size.Small" />
</MudTooltip>
}
<MudTooltip Text="@item.Implementation.GetDescription()">
<MudText Typo="Typo.body1">@item.Implementation.GetDisplayName()</MudText>
<MudTooltip Text="@item.Description">
<MudText Typo="Typo.body1">@item.DisplayName</MudText>
</MudTooltip>
</MudStack>
</MudButton>
<MudIconButton Icon="@Icons.Material.Filled.Settings" Size="Size.Small" OnClick="@(async () => await this.OpenSettings(item.Definition.Id))" />
<MudIconButton Icon="@Icons.Material.Filled.Settings" Size="Size.Small" OnClick="@(async () => await this.OpenSettings(item))" />
</MudStack>
@if (!string.IsNullOrWhiteSpace(warningText))
{
@@ -1,5 +1,6 @@
using AIStudio.Dialogs.Settings;
using AIStudio.Provider;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.ToolCallingSystem;
using Microsoft.AspNetCore.Components;
@@ -14,6 +15,17 @@ public partial class ToolSelection : MSGComponentBase
[Parameter]
public required AIStudio.Settings.Provider LLMProvider { get; set; }
/// <summary>
/// Where the chat of this selection may still send data, because of the mailboxes it read from.
/// </summary>
/// <remarks>
/// The selection shows a tool this keeps back as unavailable, with the reason, the same way it
/// does for a provider with too little confidence. Keeping the tool out of the request is the
/// business of the tool registry and the tool executor; this only explains it.
/// </remarks>
[Parameter]
public required OutboundDataRequirement RequiredOutboundDataRestriction { get; set; }
[Parameter]
public HashSet<string> SelectedToolIds { get; set; } = [];
@@ -37,7 +49,7 @@ public partial class ToolSelection : MSGComponentBase
protected override void OnParametersSet()
{
this.SelectedToolIds = ToolSelectionRules.NormalizeSelection(this.SelectedToolIds);
this.SelectedToolIds = this.ToolRegistry.NormalizeSelection(this.SelectedToolIds);
base.OnParametersSet();
}
@@ -76,10 +88,11 @@ public partial class ToolSelection : MSGComponentBase
/// <remarks>
/// The switch and the row click share this, so both agree on when a tool is out of reach: the
/// organization disabled it, it is not configured, the provider lacks the confidence it needs,
/// a response is running, or the model cannot call tools in the first place.
/// the chat read a mailbox which keeps the tool back, a response is running, or the model
/// cannot call tools in the first place.
/// </remarks>
private bool IsRowDisabled(ToolCatalogItem item) => !item.IsActive || !item.ConfigurationState.IsConfigured || this.IsBlockedByProviderConfidence(item) ||
this.Disabled || !this.SupportsTools;
this.IsBlockedByOutboundDataRestriction(item) || this.Disabled || !this.SupportsTools;
/// <summary>
/// Switches a tool when the user clicks anywhere in its row.
@@ -94,21 +107,21 @@ public partial class ToolSelection : MSGComponentBase
if (this.IsRowDisabled(item))
return;
await this.ChangeSelection(item.Definition.Id, !this.SelectedToolIds.Contains(item.Definition.Id));
await this.ChangeSelection(item.Id, !this.SelectedToolIds.Contains(item.Id));
}
private async Task ChangeSelection(string toolId, bool isSelected)
private async Task ChangeSelection(string collectionId, bool isSelected)
{
if (isSelected && !this.SettingsManager.IsToolActive(toolId))
if (isSelected && !this.ToolRegistry.IsToolActive(collectionId))
return;
var updated = new HashSet<string>(this.SelectedToolIds, StringComparer.Ordinal);
if (isSelected)
updated.Add(toolId);
updated.Add(collectionId);
else
updated.Remove(toolId);
updated.Remove(collectionId);
updated = ToolSelectionRules.NormalizeSelection(updated);
updated = this.ToolRegistry.NormalizeSelection(updated);
this.SelectedToolIds = updated;
await this.SelectedToolIdsChanged.InvokeAsync(updated);
}
@@ -118,6 +131,15 @@ public partial class ToolSelection : MSGComponentBase
private bool IsBlockedByProviderConfidence(ToolCatalogItem item) => !ToolSelectionRules.IsProviderConfidenceAllowed(this.ProviderConfidence, GetMinimumProviderConfidence(item));
/// <summary>
/// Whether the mailboxes the chat read from keep every tool of this entry back.
/// </summary>
/// <remarks>
/// Only then is there nothing left to select. Should a mailbox keep back some tools of a
/// collection, the others still run, and the request leaves out the rest on its own.
/// </remarks>
private bool IsBlockedByOutboundDataRestriction(ToolCatalogItem item) => item.Tools.All(tool => !ToolSelectionRules.IsOutboundDataAllowed(this.RequiredOutboundDataRestriction.Restriction, tool.Implementation));
private string? GetProviderConfidenceHint(ToolCatalogItem item)
{
if (!this.IsBlockedByProviderConfidence(item))
@@ -138,7 +160,7 @@ public partial class ToolSelection : MSGComponentBase
/// </remarks>
private string GetWarningText(ToolCatalogItem item)
{
var warnings = new List<string>(3);
var warnings = new List<string>(4);
if (!item.ConfigurationState.IsConfigured)
warnings.Add(string.IsNullOrWhiteSpace(item.ConfigurationState.Message) ? T("Required settings are missing. Configure this tool before enabling it.") : item.ConfigurationState.Message);
@@ -149,14 +171,17 @@ public partial class ToolSelection : MSGComponentBase
if (!string.IsNullOrWhiteSpace(providerConfidenceHint))
warnings.Add(providerConfidenceHint);
if (this.IsBlockedByOutboundDataRestriction(item))
warnings.Add(this.RequiredOutboundDataRestriction.GetToolBlockedMessage(this.SettingsManager.ConfigurationData.Mailboxes));
return string.Join(' ', warnings);
}
private async Task OpenSettings(string toolId)
private async Task OpenSettings(ToolCatalogItem item)
{
var parameters = new DialogParameters<ToolSettingsDialog>
{
{ x => x.ToolId, toolId },
{ x => x.CollectionId, item.Id },
};
var dialog = await this.DialogService.ShowAsync<ToolSettingsDialog>(null, parameters, Dialogs.DialogOptions.FULLSCREEN);
@@ -51,27 +51,27 @@ public partial class ToolSelectionField : MSGComponentBase
protected override void OnParametersSet()
{
// Like ToolSelection, the field shows the tools which will actually run, also when the
// selection is read-only. See ToolSelectionRules.NormalizeSelection:
this.SelectedToolIds = ToolSelectionRules.NormalizeSelection(this.SelectedToolIds);
// Like ToolSelection, the field shows the collections which will actually run, also when the
// selection is read-only. See ToolRegistry.NormalizeSelection:
this.SelectedToolIds = this.ToolRegistry.NormalizeSelection(this.SelectedToolIds);
base.OnParametersSet();
}
protected override async Task OnInitializedAsync()
{
this.availableTools = (await this.ToolRegistry.GetCatalogAsync(this.Component))
.Select(x => new ConfigurationSelectData<string>(x.Implementation.GetDisplayName(), x.Definition.Id))
.Select(x => new ConfigurationSelectData<string>(x.DisplayName, x.Id))
.ToList();
this.ApplyFilters([], [ Event.CONFIGURATION_CHANGED ]);
await base.OnInitializedAsync();
}
private bool IsToolLocked(string toolId) => !this.SettingsManager.IsToolActive(toolId);
private bool IsToolLocked(string toolId) => !this.ToolRegistry.IsToolActive(toolId);
private async Task OptionChangedAsync(HashSet<string> updatedToolIds)
{
this.SelectedToolIds = ToolSelectionRules.NormalizeSelection(updatedToolIds);
this.SelectedToolIds = this.ToolRegistry.NormalizeSelection(updatedToolIds);
await this.SelectedToolIdsChanged.InvokeAsync(this.SelectedToolIds);
}
@@ -81,7 +81,7 @@ public partial class ToolSelectionField : MSGComponentBase
{
case Event.CONFIGURATION_CHANGED:
this.availableTools = (await this.ToolRegistry.GetCatalogAsync(this.Component))
.Select(x => new ConfigurationSelectData<string>(x.Implementation.GetDisplayName(), x.Definition.Id))
.Select(x => new ConfigurationSelectData<string>(x.DisplayName, x.Id))
.ToList();
await this.InvokeAsync(this.StateHasChanged);
@@ -45,6 +45,7 @@ public partial class ConfigurationPluginDeleteDialog : MSGComponentBase
Add(summary.DocumentAnalysisPolicies, this.T("{0} document analysis policy"), this.T("{0} document analysis policies"));
Add(summary.MandatoryInfos, this.T("{0} mandatory information"), this.T("{0} mandatory informations"));
Add(summary.Introductions, this.T("{0} introduction on the welcome page"), this.T("{0} introductions on the welcome page"));
Add(summary.MailboxProviders, this.T("{0} mail server offered for new mailboxes"), this.T("{0} mail servers offered for new mailboxes"));
// Data sources are called out separately: removing them also deletes their credentials from
// the operating system's keychain, which the user cannot undo by reinstalling the plugin.
@@ -25,7 +25,7 @@ public partial class DataSourceERI_V1Dialog : MSGComponentBase, ISecretId
public DataSourceERI_V1 DataSource { get; set; }
[Inject]
private ILogger<ProviderDialog> Logger { get; init; } = null!;
private ILogger<DataSourceERI_V1Dialog> Logger { get; init; } = null!;
[Inject]
private RustService RustService { get; init; } = null!;
@@ -87,8 +87,10 @@ public partial class DataSourceERI_V1Dialog : MSGComponentBase, ISecretId
// Configure the spellchecking for the instance name input:
this.SettingsManager.InjectSpellchecking(SPELLCHECK_ATTRIBUTES);
// Load the used instance names:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant()).ToList();
// Load the used instance names, those of the mailboxes included:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant())
.Concat(this.SettingsManager.ConfigurationData.Mailboxes.Select(x => x.Name.ToLowerInvariant()))
.ToList();
// When editing, we need to load the data:
if(this.IsEditing)
@@ -331,6 +333,19 @@ public partial class DataSourceERI_V1Dialog : MSGComponentBase, ISecretId
await this.form.Validate();
return;
}
//
// The OS keyring stores the secret under the name of the data source, so a renamed one
// got a new entry just now, and the old one goes once the new one is in place. A name
// which differs only in case is no rename here: some keyrings ignore case, so both
// names lead to the same entry, and deleting the old one would delete the secret.
//
if (this.IsEditing && !string.Equals(this.DataSource.Name, addedDataSource.Name, StringComparison.OrdinalIgnoreCase))
{
var deleteResponse = await this.RustService.DeleteSecret(this.DataSource, SecretStoreType.DATA_SOURCE);
if (!deleteResponse.Success)
this.Logger.LogWarning($"Failed to delete the secret of the data source '{this.dataId}' stored under its previous name: {deleteResponse.Issue}");
}
}
this.MudDialog.Close(DialogResult.Ok(addedDataSource));
@@ -7,7 +7,7 @@
<DialogContent>
@* A drop anywhere in this dialog belongs to the dialog, not to the page behind it: *@
<PathDropZone IsArea="@true">
<MudForm @ref="@this.form" @bind-IsValid="@this.dataIsValid" @bind-Errors="@this.dataIssues">
<MudForm @ref="@this.form" @bind-IsValid="@this.dataIsValid" @bind-Errors="@this.dataIssues" FieldChanged="@this.RevalidateAfterFieldChange">
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudTextField
T="string"
@@ -48,7 +48,7 @@
@if (!this.CanChangeSource)
{
<MudAlert Severity="Severity.Info" Variant="Variant.Outlined" Class="mb-3">
@T("The documents of this data source are already prepared, so its folder cannot be changed. Another folder holds other documents, which makes it another data source: please add one for it. The embedding method below can be changed.")
@T("The documents of this data source are already prepared, so its folder cannot be changed. Another folder holds other documents, which makes it another data source: please add one for it. The embedding provider below can be changed.")
</MudAlert>
}
@if (this.CanChangeSource)
@@ -72,9 +72,9 @@
}
<MudJustifiedText Typo="Typo.body1" Class="mb-3">
@T("In order for the AI to be able to determine the appropriate data at any time, you must choose an embedding method.")
@T("In order for the AI to be able to determine the appropriate data at any time, you must choose an embedding provider.")
</MudJustifiedText>
<MudSelect @bind-Value="@this.dataEmbeddingId" Label="@T("Embedding")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateEmbeddingProviderAccess">
<MudSelect @ref="@this.embeddingSelect" @bind-Value="@this.dataEmbeddingId" Label="@T("Embedding")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateEmbeddingProviderAccess">
@foreach (var embedding in this.AvailableEmbeddings)
{
<MudSelectItem Value="@embedding.Value">
@@ -106,7 +106,7 @@
<ManagePandocDependency IntroText="@T("For some data types, such as Office files, MindWork AI Studio requires the open-source application Pandoc.")"/>
<MudSelect @bind-Value="@this.dataConfidenceLevel" Text="@this.dataConfidenceLevel.GetName()" Label="@T("Required provider confidence level")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateDataSourceConfidenceLevel">
<MudSelect @ref="@this.confidenceLevelSelect" @bind-Value="@this.dataConfidenceLevel" Text="@this.dataConfidenceLevel.GetName()" Label="@T("Required provider confidence level")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateDataSourceConfidenceLevel">
@foreach (var level in this.ConfidenceLevels)
{
<MudSelectItem Value="@level.Value">
@@ -138,6 +138,7 @@
@T("Optional expert settings for how this data source is split before embedding.")
</MudJustifiedText>
<MudNumericField
@ref="@this.maxChunkTokenLengthField"
T="int"
@bind-Value="@this.dataMaxChunkTokenLength"
Label="@T("Token limit")"
@@ -151,6 +152,7 @@
AdornmentIcon="@Icons.Material.Filled.FormatListNumbered"
AdornmentColor="Color.Info"/>
<MudNumericField
@ref="@this.chunkOverlapTokenLengthField"
T="int"
@bind-Value="@this.dataChunkOverlapTokenLength"
Label="@T("Token overlap")"
@@ -7,6 +7,9 @@ using AIStudio.Tools.Validation;
using Microsoft.AspNetCore.Components;
using MudBlazor.Interfaces;
using MudBlazor.Utilities;
namespace AIStudio.Dialogs;
public partial class DataSourceLocalDirectoryDialog : MSGComponentBase
@@ -63,11 +66,18 @@ public partial class DataSourceLocalDirectoryDialog : MSGComponentBase
private int dataChunkOverlapTokenLength = DataSourceEmbeddingService.DEFAULT_CHUNK_OVERLAP_TOKEN_LENGTH;
private ushort dataMaxMatches = 10;
private bool showExpertSettings;
private bool revalidateAfterRender;
private ConfidenceLevel dataConfidenceLevel = ConfidenceLevel.UNKNOWN;
// We get the form reference from Blazor code to validate it manually:
private MudForm form = null!;
// The fields whose rules read other fields, see RevalidateDependentFields:
private MudSelect<string> embeddingSelect = null!;
private MudSelect<ConfidenceLevel> confidenceLevelSelect = null!;
private MudNumericField<int> maxChunkTokenLengthField = null!;
private MudNumericField<int> chunkOverlapTokenLengthField = null!;
public DataSourceLocalDirectoryDialog()
{
this.dataSourceValidation = new()
@@ -88,8 +98,10 @@ public partial class DataSourceLocalDirectoryDialog : MSGComponentBase
// Configure the spellchecking for the instance name input:
this.SettingsManager.InjectSpellchecking(SPELLCHECK_ATTRIBUTES);
// Load the used instance names:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant()).ToList();
// Load the used instance names, those of the mailboxes included:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant())
.Concat(this.SettingsManager.ConfigurationData.Mailboxes.Select(x => x.Name.ToLowerInvariant()))
.ToList();
// When editing, we need to load the data:
if(this.IsEditing)
@@ -116,7 +128,14 @@ public partial class DataSourceLocalDirectoryDialog : MSGComponentBase
// We don't want to show validation errors when the user opens the dialog.
if(!this.IsEditing && firstRender)
this.form.ResetValidation();
// A check asked for in code waits until the fields hold their new values, cf. ToggleExpertSettings:
if (this.revalidateAfterRender)
{
this.revalidateAfterRender = false;
await this.RevalidateDependentFields(changedField: null);
}
await base.OnAfterRenderAsync(firstRender);
}
@@ -197,6 +216,14 @@ public partial class DataSourceLocalDirectoryDialog : MSGComponentBase
private void Cancel() => this.MudDialog.Cancel();
/// <summary>
/// Gives the fields which are checked against each other a fresh verdict: the embedding provider
/// and the required confidence level, and the token limits, which depend on the embedding provider.
/// </summary>
private Task RevalidateDependentFields(IFormComponent? changedField) => DependentFieldValidation.RevalidateAsync(changedField, this.embeddingSelect, this.confidenceLevelSelect, this.maxChunkTokenLengthField, this.chunkOverlapTokenLengthField);
private Task RevalidateAfterFieldChange(FormFieldChangedEventArgs change) => this.RevalidateDependentFields(change.Field);
private string? ValidateMaxChunkTokenLength(int maxChunkTokenLength)
{
if (!this.showExpertSettings)
@@ -234,6 +261,10 @@ public partial class DataSourceLocalDirectoryDialog : MSGComponentBase
this.showExpertSettings = !this.showExpertSettings;
if (this.showExpertSettings && this.dataMaxChunkTokenLength < 1)
this.dataMaxChunkTokenLength = this.ProviderMaxChunkTokenLength;
// The token limits are only checked while they are shown. The field learns the limit set
// above only with the next render, so it is checked after that:
this.revalidateAfterRender = true;
}
private string GetExpertStyles => this.showExpertSettings ? "border-2 border-dashed rounded pa-2" : string.Empty;
@@ -7,7 +7,7 @@
<DialogContent>
@* A drop anywhere in this dialog belongs to the dialog, not to the page behind it: *@
<PathDropZone IsArea="@true">
<MudForm @ref="@this.form" @bind-IsValid="@this.dataIsValid" @bind-Errors="@this.dataIssues">
<MudForm @ref="@this.form" @bind-IsValid="@this.dataIsValid" @bind-Errors="@this.dataIssues" FieldChanged="@this.RevalidateAfterFieldChange">
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudTextField
T="string"
@@ -48,7 +48,7 @@
@if (!this.CanChangeSource)
{
<MudAlert Severity="Severity.Info" Variant="Variant.Outlined" Class="mb-3">
@T("The documents of this data source are already prepared, so its file cannot be changed. Another file holds other content, which makes it another data source: please add one for it. The embedding method below can be changed.")
@T("The documents of this data source are already prepared, so its file cannot be changed. Another file holds other content, which makes it another data source: please add one for it. The embedding provider below can be changed.")
</MudAlert>
}
@if (this.CanChangeSource)
@@ -72,9 +72,9 @@
}
<MudJustifiedText Typo="Typo.body1" Class="mb-3">
@T("In order for the AI to be able to determine the appropriate data at any time, you must choose an embedding method.")
@T("In order for the AI to be able to determine the appropriate data at any time, you must choose an embedding provider.")
</MudJustifiedText>
<MudSelect @bind-Value="@this.dataEmbeddingId" Label="@T("Embedding")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateEmbeddingProviderAccess">
<MudSelect @ref="@this.embeddingSelect" @bind-Value="@this.dataEmbeddingId" Label="@T("Embedding")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateEmbeddingProviderAccess">
@foreach (var embedding in this.AvailableEmbeddings)
{
<MudSelectItem Value="@embedding.Value">
@@ -106,7 +106,7 @@
<ManagePandocDependency IntroText="@T("For some data types, such as Office files, MindWork AI Studio requires the open-source application Pandoc.")"/>
<MudSelect @bind-Value="@this.dataConfidenceLevel" Text="@this.dataConfidenceLevel.GetName()" Label="@T("Required provider confidence level")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateDataSourceConfidenceLevel">
<MudSelect @ref="@this.confidenceLevelSelect" @bind-Value="@this.dataConfidenceLevel" Text="@this.dataConfidenceLevel.GetName()" Label="@T("Required provider confidence level")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateDataSourceConfidenceLevel">
@foreach (var level in this.ConfidenceLevels)
{
<MudSelectItem Value="@level.Value">
@@ -138,6 +138,7 @@
@T("Optional expert settings for how this data source is split before embedding.")
</MudJustifiedText>
<MudNumericField
@ref="@this.maxChunkTokenLengthField"
T="int"
@bind-Value="@this.dataMaxChunkTokenLength"
Label="@T("Token limit")"
@@ -151,6 +152,7 @@
AdornmentIcon="@Icons.Material.Filled.FormatListNumbered"
AdornmentColor="Color.Info"/>
<MudNumericField
@ref="@this.chunkOverlapTokenLengthField"
T="int"
@bind-Value="@this.dataChunkOverlapTokenLength"
Label="@T("Token overlap")"
@@ -7,6 +7,9 @@ using AIStudio.Tools.Validation;
using Microsoft.AspNetCore.Components;
using MudBlazor.Interfaces;
using MudBlazor.Utilities;
namespace AIStudio.Dialogs;
public partial class DataSourceLocalFileDialog : MSGComponentBase
@@ -63,11 +66,18 @@ public partial class DataSourceLocalFileDialog : MSGComponentBase
private int dataChunkOverlapTokenLength = DataSourceEmbeddingService.DEFAULT_CHUNK_OVERLAP_TOKEN_LENGTH;
private ushort dataMaxMatches = 10;
private bool showExpertSettings;
private bool revalidateAfterRender;
private ConfidenceLevel dataConfidenceLevel = ConfidenceLevel.UNKNOWN;
// We get the form reference from Blazor code to validate it manually:
private MudForm form = null!;
// The fields whose rules read other fields, see RevalidateDependentFields:
private MudSelect<string> embeddingSelect = null!;
private MudSelect<ConfidenceLevel> confidenceLevelSelect = null!;
private MudNumericField<int> maxChunkTokenLengthField = null!;
private MudNumericField<int> chunkOverlapTokenLengthField = null!;
public DataSourceLocalFileDialog()
{
this.dataSourceValidation = new()
@@ -88,8 +98,10 @@ public partial class DataSourceLocalFileDialog : MSGComponentBase
// Configure the spellchecking for the instance name input:
this.SettingsManager.InjectSpellchecking(SPELLCHECK_ATTRIBUTES);
// Load the used instance names:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant()).ToList();
// Load the used instance names, those of the mailboxes included:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant())
.Concat(this.SettingsManager.ConfigurationData.Mailboxes.Select(x => x.Name.ToLowerInvariant()))
.ToList();
// When editing, we need to load the data:
if(this.IsEditing)
@@ -116,7 +128,14 @@ public partial class DataSourceLocalFileDialog : MSGComponentBase
// We don't want to show validation errors when the user opens the dialog.
if(!this.IsEditing && firstRender)
this.form.ResetValidation();
// A check asked for in code waits until the fields hold their new values, cf. ToggleExpertSettings:
if (this.revalidateAfterRender)
{
this.revalidateAfterRender = false;
await this.RevalidateDependentFields(changedField: null);
}
await base.OnAfterRenderAsync(firstRender);
}
@@ -197,6 +216,14 @@ public partial class DataSourceLocalFileDialog : MSGComponentBase
private void Cancel() => this.MudDialog.Cancel();
/// <summary>
/// Gives the fields which are checked against each other a fresh verdict: the embedding provider
/// and the required confidence level, and the token limits, which depend on the embedding provider.
/// </summary>
private Task RevalidateDependentFields(IFormComponent? changedField) => DependentFieldValidation.RevalidateAsync(changedField, this.embeddingSelect, this.confidenceLevelSelect, this.maxChunkTokenLengthField, this.chunkOverlapTokenLengthField);
private Task RevalidateAfterFieldChange(FormFieldChangedEventArgs change) => this.RevalidateDependentFields(change.Field);
private string? ValidateMaxChunkTokenLength(int maxChunkTokenLength)
{
if (!this.showExpertSettings)
@@ -234,6 +261,10 @@ public partial class DataSourceLocalFileDialog : MSGComponentBase
this.showExpertSettings = !this.showExpertSettings;
if (this.showExpertSettings && this.dataMaxChunkTokenLength < 1)
this.dataMaxChunkTokenLength = this.ProviderMaxChunkTokenLength;
// The token limits are only checked while they are shown. The field learns the limit set
// above only with the next render, so it is checked after that:
this.revalidateAfterRender = true;
}
private string GetExpertStyles => this.showExpertSettings ? "border-2 border-dashed rounded pa-2" : string.Empty;
@@ -0,0 +1,452 @@
@using AIStudio.Settings
@using AIStudio.Settings.DataModel
@using AIStudio.Tools.Mail
@using AIStudio.Tools.Validation
@using AIStudio.Provider
@inherits MSGComponentBase
<MudDialog>
<DialogContent>
<MudForm @ref="@this.form" @bind-IsValid="@this.dataIsValid" @bind-Errors="@this.dataIssues" FieldChanged="@this.RevalidateAfterFieldChange">
@if (this.IsAddingNotAllowed)
{
<MudAlert Severity="Severity.Error" Variant="Variant.Outlined" Class="mb-6">
<MudJustifiedText Typo="Typo.body1">
@T("Your organization no longer allows adding mailboxes, so this one cannot be added.")
</MudJustifiedText>
</MudAlert>
}
@if (this.IsOnServerNotAllowed)
{
<MudAlert Severity="Severity.Error" Variant="Variant.Outlined" Class="mb-6">
<MudJustifiedText Typo="Typo.body1">
@MailboxConnectionFailure.SERVER_NOT_ALLOWED.GetDescription()
</MudJustifiedText>
</MudAlert>
}
@if (this.authFailure is not null)
{
<MudAlert Severity="Severity.Error" Variant="Variant.Outlined" Class="mb-6">
<MudJustifiedText Typo="Typo.body1">
@this.AuthFailureText
</MudJustifiedText>
@if (!string.IsNullOrWhiteSpace(this.authFailure.FailureMessage))
{
<MudJustifiedText Typo="Typo.body2" Class="mt-2">
@string.Format(T("The server answered: {0}"), this.authFailure.FailureMessage)
</MudJustifiedText>
}
</MudAlert>
}
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudTextField
T="string"
@bind-Text="@this.dataName"
Label="@T("Data Source Name")"
Class="mb-6"
MaxLength="@DataSourceValidation.MAX_NAME_LENGTH"
Counter="@DataSourceValidation.MAX_NAME_LENGTH"
Immediate="@true"
Validation="@this.dataSourceValidation.ValidatingName"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.Lightbulb"
AdornmentColor="Color.Info"
UserAttributes="@SPELLCHECK_ATTRIBUTES"
Variant="Variant.Outlined"
/>
@if (this.CanChangeSource)
{
<MudJustifiedText Typo="Typo.body1" Class="mb-3">
@if (this.AllowsOnlyOrganizationMailServers)
{
@T("Your organization allows mailboxes only on its own mail servers. Choose one of them here, and AI Studio fills in the server settings for you.")
}
else if (this.HasOrganizationProviders)
{
@T("Choose your mail provider here, and AI Studio fills in the server settings for you. The mail servers of your organization come first, marked with a building. Alternatively, you can enter the technical details yourself.")
}
else
{
@T("Choose your mail provider here, and AI Studio fills in the server settings for you. Alternatively, e.g., for a mail server of your organization, you can enter the technical details yourself.")
}
</MudJustifiedText>
@if (this.AllowsOnlyOrganizationMailServers && !this.HasOrganizationProviders)
{
<MudAlert Severity="Severity.Warning" Variant="Variant.Outlined" Dense="@true" Class="mb-3">
@T("Your organization has not set up any of its mail servers here yet. Please ask your IT department.")
</MudAlert>
}
@* No adornment: the field shows the chosen entry together with its icon, see the entries below. They use the size and spacing of ProviderLabel, so a logo and a Material icon line up. *@
<MudSelect @ref="@this.providerSelect" T="string" Value="@this.SelectedProviderKey" ValueChanged="@this.SelectProvider" Text="@this.SelectedProviderText" Placeholder="@T("Please choose a mail server")" Label="@T("Provider")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" Validation="@this.ValidateOrganizationProvider">
@if (!this.AllowsOnlyOrganizationMailServers)
{
<MudSelectItem T="string" Value="@string.Empty">
<MudStack Row="@true" AlignItems="AlignItems.Center" Spacing="1">
<MudIcon Icon="@Icons.Material.Filled.Edit" Color="Color.Info"/>
<span>@T("Another provider")</span>
</MudStack>
</MudSelectItem>
}
@foreach (var organizationProvider in this.organizationProviders)
{
<MudSelectItem T="string" Value="@GetProviderKey(organizationProvider)">
<MudStack Row="@true" AlignItems="AlignItems.Center" Spacing="1">
<MudIcon Icon="@Icons.Material.Filled.Business" Color="Color.Info"/>
<span>@organizationProvider.Name</span>
</MudStack>
</MudSelectItem>
}
@if (!this.AllowsOnlyOrganizationMailServers)
{
foreach (var template in MailboxProviderTemplates.ALL)
{
<MudSelectItem T="string" Value="@GetProviderKey(template)">
<MudStack Row="@true" AlignItems="AlignItems.Center" Spacing="1">
@if (string.IsNullOrEmpty(template.IconUrl))
{
<MudIcon Icon="@Icons.Material.Filled.Email" Color="Color.Info"/>
}
else
{
<img src="@template.IconUrl" alt="" aria-hidden="true" draggable="false" class="provider-icon"/>
}
<span>@template.Name</span>
</MudStack>
</MudSelectItem>
}
}
</MudSelect>
}
@if (this.selectedOrganizationProvider is { } chosenOrganizationProvider)
{
<MudAlert Severity="Severity.Info" Variant="Variant.Outlined" Dense="@true" Class="mb-6">
<MudJustifiedText Typo="Typo.body2" Class="mb-1">
@T("Your organization offers this mail server. Sign in with your own username and password.")
</MudJustifiedText>
@if (!string.IsNullOrEmpty(chosenOrganizationProvider.UsernameHint))
{
<MudJustifiedText Typo="Typo.body2" Class="mb-1">
@string.Format(T("Username: {0}"), chosenOrganizationProvider.UsernameHint)
</MudJustifiedText>
}
@if (!string.IsNullOrEmpty(chosenOrganizationProvider.HelpUrl))
{
<MudLink Href="@chosenOrganizationProvider.HelpUrl" Target="_blank" Typo="Typo.body2">
@string.Format(T("Instructions of {0}"), chosenOrganizationProvider.Name)
</MudLink>
}
</MudAlert>
}
else if (this.selectedTemplate is not null)
{
<MudAlert Severity="Severity.Info" Variant="Variant.Outlined" Dense="@true" Class="mb-6">
@foreach (var requirement in this.selectedTemplate.Requirements.GetDescriptions())
{
<MudJustifiedText Typo="Typo.body2" Class="mb-1">
@requirement
</MudJustifiedText>
}
<MudJustifiedText Typo="Typo.body2" Class="mb-1">
@string.Format(T("Username: {0}"), this.selectedTemplate.UsernameFormat.GetDescription())
</MudJustifiedText>
<MudLink Href="@this.selectedTemplate.HelpUrl" Target="_blank" Typo="Typo.body2">
@string.Format(T("Instructions of {0}"), this.selectedTemplate.Name)
</MudLink>
</MudAlert>
}
@* Settings AI Studio fills in wait behind a button, so the sign-in follows the choice of the provider directly: *@
@if (this.AreServerSettingsFilledIn)
{
<MudButton OnClick="@this.ToggleServerDetails" Variant="Variant.Text" Color="Color.Primary" StartIcon="@Icons.Material.Filled.Dns" Class="@this.ServerDetailsButtonClass">
@(this.showServerDetails ? T("Hide Server Details") : T("Show Server Details"))
</MudButton>
}
else
{
<MudText Typo="Typo.h6" Class="mb-1">
@T("Server")
</MudText>
}
<MudCollapse Expanded="@this.AreServerDetailsShown">
<div class="border-dashed border rounded-lg pa-3 mb-6">
<MudStack Row="@true">
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudTextField
T="string"
@bind-Text="@this.dataHost"
Label="@T("IMAP server host")"
Immediate="@true"
ReadOnly="@(!this.CanChangeSource || this.AllowsOnlyOrganizationMailServers)"
Validation="@this.ValidateHost"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.Dns"
AdornmentColor="Color.Info"
Variant="Variant.Outlined"
UserAttributes="@SPELLCHECK_ATTRIBUTES"/>
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudNumericField
T="int"
@bind-Value="@this.dataPort"
Label="@T("Port")"
Immediate="@true"
ReadOnly="@this.AllowsOnlyOrganizationMailServers"
Min="1" Max="65535"
Validation="@this.dataSourceValidation.ValidatePort"
Variant="Variant.Outlined"/>
</MudStack>
<MudSelect T="MailboxTransportSecurity" Value="@this.dataTransportSecurity" ValueChanged="@this.SelectTransportSecurity" Text="@this.dataTransportSecurity.GetName()" Label="@T("Encryption")" Class="mt-3" ReadOnly="@this.AllowsOnlyOrganizationMailServers" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" AdornmentIcon="@Icons.Material.Filled.Lock" Validation="@DataSourceValidation.ValidateMailboxTransportSecurity">
@foreach (var transportSecurity in ConfigurationSelectDataFactory.GetMailboxTransportSecurityData())
{
<MudSelectItem T="MailboxTransportSecurity" Value="@transportSecurity.Value">
@transportSecurity.Name
</MudSelectItem>
}
</MudSelect>
</div>
</MudCollapse>
<MudText Typo="Typo.h6" Class="mb-1">
@T("Sign-in")
</MudText>
<div class="@this.SignInGroupClass">
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudTextField
T="string"
@bind-Text="@this.dataUsername"
Label="@T("Username")"
Class="mb-3"
Immediate="@true"
ReadOnly="@(!this.CanChangeSource)"
Validation="@DataSourceValidation.ValidateMailboxUsername"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.Person2"
AdornmentColor="Color.Info"
Variant="Variant.Outlined"
UserAttributes="@SPELLCHECK_ATTRIBUTES"/>
<SecretInputField @bind-Secret="@this.dataPassword" Label="@T("Password")" Validation="@this.dataSourceValidation.ValidateMailboxPassword" Class="mb-3"/>
<MudStack Row="@true" AlignItems="AlignItems.Center">
<MudButton Variant="Variant.Filled" Color="@this.TestResultColor" StartIcon="@this.TestResultIcon" Disabled="@(!this.CanTestConnection)" OnClick="@this.TestConnection">
@T("Test connection & load folders")
</MudButton>
@if (this.isTestingConnection)
{
<MudProgressCircular Size="Size.Small" Indeterminate="@true"/>
}
<MudText Typo="Typo.body1">
@this.TestResultText
</MudText>
</MudStack>
</div>
<MudText Typo="Typo.h6" Class="mb-1">
@T("Folder")
</MudText>
<MudJustifiedText Typo="Typo.body1" Class="mb-3">
@T("Select the folder AI Studio indexes, together with its subfolders. The AI only reads from this folder. Without a folder, AI Studio indexes the whole mailbox, apart from the trash and the junk folder.")
</MudJustifiedText>
@if (!this.CanChangeSource)
{
<MudAlert Severity="Severity.Info" Variant="Variant.Outlined" Class="mb-3">
@T("The mails of this mailbox are already prepared, so its server, its username, and its folder cannot be changed. Another account or folder holds other mails, which makes it another mailbox: please add one for it. The embedding provider below can be changed.")
</MudAlert>
}
@if (this.CanChangeSource && this.ConnectionTestSucceeded)
{
<MailFolderPicker Folders="@this.serverFolders" @bind-SelectedFolder="@this.dataRootFolder" CreateFolder="@this.CreateFolder"/>
}
else
{
<MudTextField
T="string"
Text="@this.RootFolderText"
Label="@T("Selected folder")"
Class="mb-3"
ReadOnly="@true"
HelperText="@(this.CanChangeSource ? T("Test the connection to select another folder.") : string.Empty)"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.Folder"
Variant="Variant.Outlined"/>
}
@if (!string.IsNullOrWhiteSpace(this.folderIssue))
{
<MudAlert Severity="Severity.Error" Variant="Variant.Outlined" Dense="@true" Class="mb-3">
@this.folderIssue
</MudAlert>
}
<MudSelect T="MailboxMaxAge" @bind-Value="@this.dataMaxAge" Text="@this.dataMaxAge.GetName()" Label="@T("Period")" Class="mb-1 mt-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" AdornmentIcon="@Icons.Material.Filled.DateRange">
@foreach (var maxAge in ConfigurationSelectDataFactory.GetMailboxMaxAgeData())
{
<MudSelectItem T="MailboxMaxAge" Value="@maxAge.Value">
@maxAge.Name
</MudSelectItem>
}
</MudSelect>
<MudJustifiedText Typo="Typo.body2" Class="mb-6">
@T("AI Studio indexes the newest mails first. Flagged mails are always indexed, however old they are.")
</MudJustifiedText>
<MudTextSwitch @bind-Value="@this.dataIndexAttachments" Color="Color.Primary" Label="@T("Index the text of attachments?")" LabelOn="@T("Yes, index the text of attached documents")" LabelOff="@T("No, only list the names of attachments")"/>
@if (this.dataIndexAttachments)
{
@* ReSharper disable once CSharpWarnings::CS8974 *@
<MudNumericField
T="int"
@bind-Value="@this.dataMaxAttachmentSizeMegabytes"
Label="@T("Largest attachment to index, in MB")"
Class="mb-6"
Min="@DataSourceValidation.MIN_ATTACHMENT_SIZE_MEGABYTES"
Max="@DataSourceValidation.MAX_ATTACHMENT_SIZE_MEGABYTES"
Immediate="@true"
Validation="@DataSourceValidation.ValidateMailboxMaxAttachmentSize"
HelperText="@T("Of a larger attachment, only the name is indexed.")"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.AttachFile"
AdornmentColor="Color.Info"
Variant="Variant.Outlined"/>
}
<MudJustifiedText Typo="Typo.body1" Class="mb-3 mt-3">
@T("In order for the AI to be able to determine the appropriate data at any time, you must choose an embedding provider.")
</MudJustifiedText>
<MudSelect @ref="@this.embeddingSelect" @bind-Value="@this.dataEmbeddingId" Label="@T("Embedding")" Class="mb-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateEmbeddingProviderAccess">
@foreach (var embedding in this.AvailableEmbeddings)
{
<MudSelectItem Value="@embedding.Value">
@if (this.GetEmbeddingProvider(embedding.Value) is { } provider)
{
<ProviderLabel ProviderType="@provider.UsedLLMProvider" CustomIconDataUrl="@provider.CustomIconDataUrl" Text="@embedding.Name" />
}
else
{
@embedding.Name
}
</MudSelectItem>
}
</MudSelect>
@if (!string.IsNullOrWhiteSpace(this.dataEmbeddingId))
{
if (this.SelectedCloudEmbedding)
{
<DataSourceCloudEmbeddingWarning DataSourceType="DataSourceType.MAILBOX" SourcePath="@this.dataUsername" @bind-UserAcknowledged="@this.dataUserAcknowledgedCloudEmbedding" Validation="@this.dataSourceValidation.ValidateUserAcknowledgedCloudEmbedding"/>
}
else
{
<MudJustifiedText Typo="Typo.body1" Color="Color.Tertiary" Class="mb-3">
@T("The embedding you selected runs locally or in your organization. Your data is not sent to the cloud.")
</MudJustifiedText>
}
}
<MudSelect @ref="@this.confidenceLevelSelect" T="ConfidenceLevel" @bind-Value="@this.dataConfidenceLevel" Text="@this.ConfidenceLevelText" Placeholder="@T("Please select a level")" Label="@T("Required provider confidence level")" Class="mb-1 mt-3" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" Validation="@this.dataSourceValidation.ValidateMailboxConfidenceLevel">
@foreach (var level in ConfigurationSelectDataFactory.GetMailboxConfidenceLevelsData())
{
<MudSelectItem T="ConfidenceLevel" Value="@level.Value">
@level.Name
</MudSelectItem>
}
</MudSelect>
<MudJustifiedText Typo="Typo.body2" Class="mb-6">
@T("Only providers with at least this confidence level see the content of your mails: the chat provider, the embedding provider, and every provider which checks a mail for you.")
</MudJustifiedText>
<MudSelect T="OutboundDataRestriction" @bind-Value="@this.dataOutboundDataRestriction" Text="@this.dataOutboundDataRestriction.GetName()" Label="@T("Where a chat may send data after reading mails")" Class="mb-1" OpenIcon="@Icons.Material.Filled.ExpandMore" AdornmentColor="Color.Info" Adornment="Adornment.Start" AdornmentIcon="@Icons.Material.Filled.Shield">
@foreach (var restriction in ConfigurationSelectDataFactory.GetOutboundDataRestrictionData(this.MinimumOutboundDataRestriction))
{
<MudSelectItem T="OutboundDataRestriction" Value="@restriction.Value">
@restriction.Name
</MudSelectItem>
}
</MudSelect>
<MudJustifiedText Typo="Typo.body2" Class="mb-1">
@this.dataOutboundDataRestriction.GetDescription()
</MudJustifiedText>
@if (this.IsOutboundDataRestrictionLimited)
{
<MudJustifiedText Typo="Typo.body2" Class="mb-1">
@T("Your organization does not allow less strict settings than the ones offered here.")
</MudJustifiedText>
}
<MudJustifiedText Typo="Typo.body2" Class="mb-6">
@T("Mails come from strangers and may contain instructions meant for the AI. This setting decides where a chat may still send data once it has read from this mailbox. Mind that opening a link can be enough to set something off, e.g., to confirm a subscription, an order, or a payment.")
</MudJustifiedText>
<MudStack Class="mb-3">
<MudButton OnClick="@this.ToggleExpertSettings" Variant="Variant.Text" Color="Color.Primary" StartIcon="@Icons.Material.Filled.Tune">
@(this.showExpertSettings ? T("Hide Expert Settings") : T("Show Expert Settings"))
</MudButton>
<MudDivider/>
<MudCollapse Expanded="@this.showExpertSettings" Class="@this.GetExpertStyles">
<MudJustifiedText Typo="Typo.body1" Class="mb-3">
@T("Optional expert settings for how this data source is split before embedding.")
</MudJustifiedText>
<MudNumericField
@ref="@this.maxChunkTokenLengthField"
T="int"
@bind-Value="@this.dataMaxChunkTokenLength"
Label="@T("Token limit")"
Class="mb-3"
Min="1"
Immediate="@true"
Validation="@this.ValidateMaxChunkTokenLength"
HelperText="@this.MaxChunkTokenLengthHelperText"
Variant="Variant.Outlined"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.FormatListNumbered"
AdornmentColor="Color.Info"/>
<MudNumericField
@ref="@this.chunkOverlapTokenLengthField"
T="int"
@bind-Value="@this.dataChunkOverlapTokenLength"
Label="@T("Token overlap")"
Class="mb-3"
Min="0"
Immediate="@true"
Validation="@this.ValidateChunkOverlapTokenLength"
HelperText="@this.ChunkOverlapTokenLengthHelperText"
Variant="Variant.Outlined"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.CompareArrows"
AdornmentColor="Color.Info"/>
<MudNumericField
T="ushort"
Min="10"
@bind-Value="@this.dataMaxMatches"
Label="@T("How many mails do you want at most per search?")"
Variant="Variant.Outlined"
Step="10"
Adornment="Adornment.Start"
AdornmentIcon="@Icons.Material.Filled.Search"
AdornmentColor="Color.Info"/>
</MudCollapse>
</MudStack>
</MudForm>
<Issues IssuesData="@this.dataIssues"/>
</DialogContent>
<DialogActions>
<MudButton OnClick="@this.Cancel" Variant="Variant.Filled">
@T("Cancel")
</MudButton>
<MudButton OnClick="@this.Store" Variant="Variant.Filled" Color="Color.Primary" Disabled="@this.IsAddingNotAllowed">
@if(this.IsEditing)
{
@T("Update")
}
else
{
@T("Add")
}
</MudButton>
</DialogActions>
</MudDialog>
@@ -0,0 +1,743 @@
using AIStudio.Components;
using AIStudio.Provider;
using AIStudio.Settings;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.Databases;
using AIStudio.Tools.Databases.IndexStore;
using AIStudio.Tools.Mail;
using AIStudio.Tools.PluginSystem;
using AIStudio.Tools.Services;
using AIStudio.Tools.Validation;
using Microsoft.AspNetCore.Components;
using MudBlazor.Interfaces;
using MudBlazor.Utilities;
namespace AIStudio.Dialogs;
/// <summary>
/// Adds or edits a mailbox.
/// </summary>
/// <remarks>
/// Every connection test is exactly one sign-in, started by the user. Its outcome only touches the
/// recorded sign-in failure of the mailbox when the test used the settings and the password stored
/// for it: a failure with a password the user is still typing says nothing about the stored one,
/// and neither does a success.
/// </remarks>
public partial class DataSourceMailboxDialog : MSGComponentBase
{
[CascadingParameter]
private IMudDialogInstance MudDialog { get; set; } = null!;
[Parameter]
public bool IsEditing { get; set; }
[Parameter]
public DataSourceMailbox DataSource { get; set; }
/// <summary>
/// Whether the server, the username, and the folder of this mailbox must stay as they are.
/// </summary>
/// <remarks>
/// Set once the index holds something for this mailbox. The embedding is not locked along with
/// them: it can be changed, and DataSourceReindexWarning asks what that costs.
/// </remarks>
[Parameter]
public bool LockSource { get; set; }
[Parameter]
public IReadOnlyList<ConfigurationSelectData<string>> AvailableEmbeddings { get; set; } = [];
[Inject]
private IDialogService DialogService { get; init; } = null!;
[Inject]
private DataSourceEmbeddingService DataSourceEmbeddingService { get; init; } = null!;
[Inject]
private DatabaseClientProvider DatabaseClientProvider { get; init; } = null!;
[Inject]
private RustService RustService { get; init; } = null!;
[Inject]
private ILogger<DataSourceMailboxDialog> Logger { get; init; } = null!;
/// <summary>
/// How long a connection test may take, signing in and listing the folders included.
/// </summary>
private static readonly TimeSpan CONNECTION_TEST_TIMEOUT = TimeSpan.FromMinutes(2);
private static readonly Dictionary<string, object?> SPELLCHECK_ATTRIBUTES = new();
private readonly DataSourceValidation dataSourceValidation;
/// <summary>
/// The names of all data sources and mailboxes. A mailbox needs a name of its own among both.
/// </summary>
private List<string> UsedDataSourcesNames { get; set; } = [];
private bool dataIsValid;
private string[] dataIssues = [];
private string dataSecretStorageIssue = string.Empty;
private string dataEditingPreviousInstanceName = string.Empty;
private uint dataNum;
private string dataId = Guid.NewGuid().ToString();
private string dataName = string.Empty;
private string dataHost = string.Empty;
private int dataPort = MailboxTransportSecurityExtensions.SSL_ON_CONNECT_PORT;
private MailboxTransportSecurity dataTransportSecurity = MailboxTransportSecurity.SSL_ON_CONNECT;
private string dataUsername = string.Empty;
private string dataPassword = string.Empty;
private string dataRootFolder = string.Empty;
private MailboxMaxAge dataMaxAge = MailboxMaxAge.LAST_12_MONTHS;
private bool dataIndexAttachments = true;
private int dataMaxAttachmentSizeMegabytes = 10;
private OutboundDataRestriction dataOutboundDataRestriction = OutboundDataRestriction.ONLY_CONFIGURED_SERVICES;
private bool dataUserAcknowledgedCloudEmbedding;
private string dataEmbeddingId = string.Empty;
private int dataMaxChunkTokenLength;
private int dataChunkOverlapTokenLength = DataSourceEmbeddingService.DEFAULT_CHUNK_OVERLAP_TOKEN_LENGTH;
private ushort dataMaxMatches = 10;
private ConfidenceLevel dataConfidenceLevel = ConfidenceLevel.NONE;
private bool showExpertSettings;
private bool showServerDetails;
private bool revalidateAfterRender;
private IReadOnlyList<DataMailboxProvider> organizationProviders = [];
private MailServerPolicy mailServerPolicy = MailServerPolicy.ANY_SERVER;
private DataMailboxProvider? selectedOrganizationProvider;
private MailboxProviderTemplate? selectedTemplate;
private MailboxAuthFailure? authFailure;
private string storedPassword = string.Empty;
private ConnectionSettings? testedSettings;
private MailboxConnectionFailure? testFailure;
private bool isTestingConnection;
private IReadOnlyList<MailServerFolder> serverFolders = [];
private string folderIssue = string.Empty;
// We get the form reference from Blazor code to validate it manually:
private MudForm form = null!;
// The fields whose rules read other fields, see RevalidateDependentFields. The provider is only
// rendered while the source can be changed:
private MudSelect<string>? providerSelect;
private MudSelect<string> embeddingSelect = null!;
private MudSelect<ConfidenceLevel> confidenceLevelSelect = null!;
private MudNumericField<int> maxChunkTokenLengthField = null!;
private MudNumericField<int> chunkOverlapTokenLengthField = null!;
public DataSourceMailboxDialog()
{
this.dataSourceValidation = new()
{
GetSelectedCloudEmbedding = () => this.SelectedCloudEmbedding,
GetSelectedEmbeddingProvider = () => this.SelectedEmbedding,
GetConfidenceLevel = () => this.dataConfidenceLevel,
GetSettingsManager = () => this.SettingsManager,
GetPreviousDataSourceName = () => this.dataEditingPreviousInstanceName,
GetUsedDataSourceNames = () => this.UsedDataSourcesNames,
GetSecretStorageIssue = () => this.dataSecretStorageIssue,
GetTestedConnection = () => this.testedSettings == this.CurrentSettings,
GetTestedConnectionResult = () => this.ConnectionTestSucceeded,
};
}
#region Overrides of ComponentBase
protected override async Task OnInitializedAsync()
{
// Configure the spellchecking for the instance name input:
this.SettingsManager.InjectSpellchecking(SPELLCHECK_ATTRIBUTES);
// A mailbox and a data source must not share a name:
this.UsedDataSourcesNames = this.SettingsManager.ConfigurationData.DataSources.Select(x => x.Name.ToLowerInvariant())
.Concat(this.SettingsManager.ConfigurationData.Mailboxes.Select(x => x.Name.ToLowerInvariant()))
.ToList();
// When editing, we need to load the data:
if (this.IsEditing)
{
this.dataEditingPreviousInstanceName = this.DataSource.Name.ToLowerInvariant();
this.dataNum = this.DataSource.Num;
this.dataId = this.DataSource.Id;
this.dataName = this.DataSource.Name;
this.dataHost = this.DataSource.Host;
this.dataPort = this.DataSource.Port;
this.dataTransportSecurity = this.DataSource.TransportSecurity;
this.dataUsername = this.DataSource.Username;
this.dataRootFolder = this.DataSource.RootFolder;
this.dataMaxAge = this.DataSource.MaxAge;
this.dataIndexAttachments = this.DataSource.IndexAttachments;
this.dataMaxAttachmentSizeMegabytes = this.DataSource.MaxAttachmentSizeMegabytes;
this.dataOutboundDataRestriction = this.DataSource.OutboundDataRestriction;
this.dataEmbeddingId = this.DataSource.EmbeddingId;
this.dataMaxChunkTokenLength = this.DataSource.MaxChunkTokenLength;
this.dataChunkOverlapTokenLength = this.DataSource.ChunkOverlapTokenLength;
this.dataMaxMatches = this.DataSource.MaxMatches;
this.dataConfidenceLevel = this.DataSource.ConfidenceLevel;
// A mail server of the organization goes first, see FitProviderSelection:
this.selectedTemplate = MailboxProviderTemplates.ALL.FirstOrDefault(template => template.Host.Length > 0 && template.Host.Equals(this.DataSource.Host.Trim(), StringComparison.OrdinalIgnoreCase));
var requestedSecret = await this.RustService.GetSecret(this.DataSource, SecretStoreType.DATA_SOURCE, isTrying: true);
if (requestedSecret.Success)
{
this.storedPassword = await requestedSecret.Secret.Decrypt(Program.ENCRYPTION);
this.dataPassword = this.storedPassword;
}
else
this.dataSecretStorageIssue = string.Format(T("Failed to load the password from the operating system. The message was: {0}. You might ignore this message and provide the password again."), requestedSecret.Issue);
var indexStore = await this.DatabaseClientProvider.GetIndexStoreAsync();
this.authFailure = await indexStore.GetMailboxAuthFailureAsync(this.dataId, CancellationToken.None);
}
this.ApplyOrganizationRules();
await base.OnInitializedAsync();
// A configuration plugin may change what the organization allows while the dialog is open:
this.ApplyFilters([], [ Event.CONFIGURATION_CHANGED ]);
}
protected override async Task OnAfterRenderAsync(bool firstRender)
{
// Reset the validation when not editing and on the first render.
// We don't want to show validation errors when the user opens the dialog.
if (!this.IsEditing && firstRender)
this.form.ResetValidation();
// A check asked for in code waits until the fields hold their new values, cf. ToggleExpertSettings:
if (this.revalidateAfterRender)
{
this.revalidateAfterRender = false;
await this.RevalidateDependentFields(changedField: null);
}
await base.OnAfterRenderAsync(firstRender);
}
#endregion
#region Overrides of MSGComponentBase
protected override Task ProcessIncomingMessage<T>(ComponentBase? sendingComponent, Event triggeredEvent, T? data) where T : default
{
switch (triggeredEvent)
{
case Event.CONFIGURATION_CHANGED:
case Event.PLUGINS_RELOADED:
this.ApplyOrganizationRules();
// A choice the organization does not offer anymore shows its error, once the fields hold the new values:
this.revalidateAfterRender = true;
this.StateHasChanged();
break;
}
return base.ProcessIncomingMessage(sendingComponent, triggeredEvent, data);
}
#endregion
/// <summary>
/// What a connection test depends on. A test only counts for the settings it was made with.
/// </summary>
private readonly record struct ConnectionSettings(string Host, int Port, MailboxTransportSecurity TransportSecurity, string Username, string Password);
private ConnectionSettings CurrentSettings => new(this.dataHost.Trim(), this.dataPort, this.dataTransportSecurity, this.dataUsername.Trim(), this.dataPassword);
private ConnectionSettings StoredSettings => new(this.DataSource.Host.Trim(), this.DataSource.Port, this.DataSource.TransportSecurity, this.DataSource.Username.Trim(), this.storedPassword);
/// <summary>
/// Whether the settings to connect with are the ones stored for this mailbox.
/// </summary>
private bool UsesStoredSettings(ConnectionSettings settings) => this.IsEditing && settings == this.StoredSettings;
/// <summary>
/// A new mailbox is only added once a sign-in worked, and an edited one once more when its connection changed.
/// </summary>
private bool RequiresConnectionTest => !this.UsesStoredSettings(this.CurrentSettings);
private bool ConnectionTestSucceeded => this.testedSettings == this.CurrentSettings && this.testFailure is null;
private bool CanChangeSource => !this.IsEditing || !this.LockSource;
private bool CanTestConnection => !this.isTestingConnection
&& DataSourceValidation.ValidateMailboxHost(this.dataHost) is null
&& this.dataSourceValidation.ValidatePort(this.dataPort) is null
&& DataSourceValidation.ValidateMailboxTransportSecurity(this.dataTransportSecurity) is null
&& DataSourceValidation.ValidateMailboxUsername(this.dataUsername) is null
&& !string.IsNullOrEmpty(this.dataPassword);
/// <summary>
/// The key of the chosen provider, see GetProviderKey, or empty for another provider.
/// </summary>
private string SelectedProviderKey => this.selectedOrganizationProvider is { } organizationProvider
? GetProviderKey(organizationProvider)
: this.selectedTemplate is { } template ? GetProviderKey(template) : string.Empty;
private string SelectedProviderText => this.selectedOrganizationProvider?.Name
?? (this.AllowsOnlyOrganizationMailServers ? string.Empty : this.selectedTemplate?.Name ?? T("Another provider"));
/// <summary>
/// Whether the organization offers any mail servers of its own.
/// </summary>
private bool HasOrganizationProviders => this.organizationProviders.Count > 0;
/// <summary>
/// Whether the organization allows mailboxes only on its own mail servers. The user then picks
/// one of them and enters nothing about the server.
/// </summary>
private bool AllowsOnlyOrganizationMailServers => this.mailServerPolicy.AllowsOnlyOrganizationMailServers;
/// <summary>
/// Whether the edited mailbox is on a server the organization does not allow.
/// </summary>
private bool IsOnServerNotAllowed => this.IsEditing && !this.mailServerPolicy.IsAllowed(this.DataSource.Host);
/// <summary>
/// Whether the organization stopped allowing new mailboxes while the dialog was open to add one.
/// </summary>
private bool IsAddingNotAllowed => !this.IsEditing && !this.SettingsManager.ConfigurationData.App.AllowUserToAddMailbox;
private string? ValidateOrganizationProvider(string providerKey) => this.AllowsOnlyOrganizationMailServers && this.organizationProviders.All(provider => GetProviderKey(provider) != providerKey)
? T("Please choose one of the mail servers of your organization.")
: null;
/// <summary>
/// Checks the host, unless it comes from the organization: then the choice of its mail server is
/// checked instead, and nobody is told about a host they cannot enter.
/// </summary>
private string? ValidateHost(string host) => this.AllowsOnlyOrganizationMailServers ? null : DataSourceValidation.ValidateMailboxHost(host);
/// <summary>
/// Whether AI Studio fills in the server settings, from a mail server of the organization or
/// from a public template which knows the host. They are hidden then, until the user asks for them.
/// </summary>
private bool AreServerSettingsFilledIn => this.AllowsOnlyOrganizationMailServers || this.selectedOrganizationProvider is not null || this.selectedTemplate is { Host.Length: > 0 };
private bool AreServerDetailsShown => !this.AreServerSettingsFilledIn || this.showServerDetails;
// While the details are hidden, the button stands in for the whole server section and keeps its distance:
private string ServerDetailsButtonClass => this.showServerDetails ? "mb-3" : "mb-6";
private void ToggleServerDetails() => this.showServerDetails = !this.showServerDetails;
private string SignInGroupClass => this.authFailure is null
? "border-dashed border rounded-lg pa-3 mb-6"
: "border-dashed border-2 mud-border-error rounded-lg pa-3 mb-6";
private string AuthFailureText => this.authFailure is null
? string.Empty
: string.Format(T("Signing in to this mailbox failed on {0}. Presumably your password changed. AI Studio does not try again on its own, so that your account is not locked. Enter your new password below and save it, or test the connection."), this.authFailure.FailedAtUtc.ToLocalTime().ToString("g", I18N.I.Culture));
private string RootFolderText => string.IsNullOrEmpty(this.dataRootFolder) ? T("Whole mailbox") : this.dataRootFolder;
private string ConfidenceLevelText => this.dataConfidenceLevel is ConfidenceLevel.NONE ? string.Empty : this.dataConfidenceLevel.GetName();
private OutboundDataRestriction MinimumOutboundDataRestriction => this.SettingsManager.ConfigurationData.MailboxSettings.MinimumOutboundDataRestriction;
private bool IsOutboundDataRestrictionLimited => this.MinimumOutboundDataRestriction is not OutboundDataRestriction.UNRESTRICTED;
private string TestResultText
{
get
{
if (this.isTestingConnection)
return T("Testing the connection ...");
if (this.testedSettings != this.CurrentSettings)
return this.RequiresConnectionTest ? T("Not tested yet.") : string.Empty;
return this.testFailure is { } failure ? failure.GetDescription() : T("Connection successful.");
}
}
private Color TestResultColor => this.testedSettings != this.CurrentSettings
? Color.Default
: this.testFailure is null ? Color.Success : Color.Error;
private string TestResultIcon => this.testedSettings != this.CurrentSettings
? Icons.Material.Outlined.HourglassEmpty
: this.testFailure is null ? Icons.Material.Outlined.CheckCircle : Icons.Material.Outlined.Error;
private EmbeddingProvider? GetEmbeddingProvider(string providerId)
{
var provider = this.SettingsManager.GetEmbeddingProviderById(providerId);
return provider == EmbeddingProvider.NONE ? null : provider;
}
private EmbeddingProvider? SelectedEmbedding => this.SettingsManager.ConfigurationData.EmbeddingProviders
.FirstOrDefault(x => x.Id == this.dataEmbeddingId);
private bool SelectedCloudEmbedding => this.SelectedEmbedding is { IsSelfHosted: false };
private int ProviderMaxChunkTokenLength => this.SelectedEmbedding?.EffectiveTokenLimit ?? EmbeddingProvider.DEFAULT_TOKEN_LIMIT;
private string MaxChunkTokenLengthHelperText => string.Format(
T("Maximum number of tokens per chunk for this data source. The embedding provider default is {0} tokens."),
this.ProviderMaxChunkTokenLength);
private string ChunkOverlapTokenLengthHelperText => string.Format(
T("Number of tokens repeated at the start of the next chunk. The default overlap is {0} tokens."),
DataSourceEmbeddingService.DEFAULT_CHUNK_OVERLAP_TOKEN_LENGTH);
private DataSourceMailbox CreateDataSource() => new()
{
Id = this.dataId,
Num = this.dataNum,
Name = this.dataName,
Type = DataSourceType.MAILBOX,
EmbeddingId = this.dataEmbeddingId,
MaxChunkTokenLength = this.dataMaxChunkTokenLength,
ChunkOverlapTokenLength = this.dataChunkOverlapTokenLength,
ConfidenceLevel = this.dataConfidenceLevel,
// Kept out of reach of the form while the source is locked, so a stale field cannot point an
// indexed mailbox somewhere else:
Host = this.CanChangeSource ? this.dataHost.Trim() : this.DataSource.Host,
Username = this.CanChangeSource ? this.dataUsername.Trim() : this.DataSource.Username,
RootFolder = this.CanChangeSource ? this.dataRootFolder : this.DataSource.RootFolder,
Port = this.dataPort,
TransportSecurity = this.dataTransportSecurity,
AuthMethod = MailboxAuthMethod.PASSWORD,
MaxAge = this.dataMaxAge,
IndexAttachments = this.dataIndexAttachments,
MaxAttachmentSizeMegabytes = this.dataMaxAttachmentSizeMegabytes,
OutboundDataRestriction = this.dataOutboundDataRestriction,
MaxMatches = this.dataMaxMatches,
};
//
// A key per choice rather than the name: an organization may well call its mail server like
// one of the well-known providers, e.g., "Microsoft Exchange Server".
//
private static string GetProviderKey(DataMailboxProvider organizationProvider) => $"organization:{organizationProvider.Id}";
private static string GetProviderKey(MailboxProviderTemplate template) => $"template:{template.Name}";
/// <summary>
/// Reads what the organization decides about mailboxes and fits the dialog to it.
/// </summary>
/// <remarks>
/// Runs when the dialog opens and again whenever a configuration plugin changes while it is
/// open. Either way, the dialog offers what it would offer when opened anew, and keeps what the
/// user entered as far as the organization allows it.
/// </remarks>
private void ApplyOrganizationRules()
{
// The mail servers of the organization come from the running configuration plugins:
this.organizationProviders = PluginFactory.GetMailboxProviders();
this.mailServerPolicy = MailServerPolicy.Read(this.SettingsManager);
this.FitProviderSelection();
// A level the organization ruled out is not offered anymore, so the dialog switches to the
// one which applies anyway, see MailToolResults.GetRequirements:
this.dataOutboundDataRestriction = this.dataOutboundDataRestriction.StricterOf(this.MinimumOutboundDataRestriction);
}
/// <summary>
/// Fits the chosen provider to the mail servers the organization offers right now.
/// </summary>
private void FitProviderSelection()
{
//
// While the source can be changed, a chosen mail server of the organization is followed by
// its id: when the organization changed how to reach it, its new settings apply. A locked
// source keeps its host, so only the host tells which server of the organization it is on:
//
if (this.selectedOrganizationProvider is { } chosenOrganizationProvider && this.CanChangeSource)
{
var currentOrganizationProvider = this.organizationProviders.FirstOrDefault(provider => provider.Id == chosenOrganizationProvider.Id);
if (currentOrganizationProvider is not null && (currentOrganizationProvider.Host != chosenOrganizationProvider.Host || currentOrganizationProvider.Port != chosenOrganizationProvider.Port || currentOrganizationProvider.TransportSecurity != chosenOrganizationProvider.TransportSecurity))
this.SelectProvider(GetProviderKey(currentOrganizationProvider));
else
this.selectedOrganizationProvider = currentOrganizationProvider;
}
else
this.selectedOrganizationProvider = null;
this.selectedOrganizationProvider ??= this.organizationProviders.FirstOrDefault(provider => MailServerHosts.AreSame(provider.Host, this.dataHost));
// The mail servers of the organization go before the public templates, which are not offered
// at all while the organization allows its own mail servers only:
if (this.selectedOrganizationProvider is not null || this.AllowsOnlyOrganizationMailServers)
this.selectedTemplate = null;
// A new mailbox with a single mail server to choose from has nothing to choose:
if (!this.IsEditing && this.AllowsOnlyOrganizationMailServers && this.selectedOrganizationProvider is null && this.organizationProviders.Count == 1)
this.SelectProvider(GetProviderKey(this.organizationProviders[0]));
}
private void SelectProvider(string providerKey)
{
this.selectedOrganizationProvider = this.organizationProviders.FirstOrDefault(candidate => GetProviderKey(candidate) == providerKey);
this.selectedTemplate = MailboxProviderTemplates.ALL.FirstOrDefault(candidate => GetProviderKey(candidate) == providerKey);
if (this.selectedOrganizationProvider is { } organizationProvider)
{
this.dataHost = organizationProvider.Host;
this.dataPort = organizationProvider.Port;
this.dataTransportSecurity = organizationProvider.TransportSecurity;
}
else if (this.selectedTemplate is { } template)
{
this.dataHost = template.Host;
this.dataPort = template.Port;
this.dataTransportSecurity = template.TransportSecurity;
}
}
private void SelectTransportSecurity(MailboxTransportSecurity transportSecurity)
{
//
// Who switches the encryption usually means the port that comes with it, unless they
// entered a port of their own:
//
if (this.dataPort == this.dataTransportSecurity.GetUsualPort() && transportSecurity.GetUsualPort() is { } usualPort)
this.dataPort = usualPort;
this.dataTransportSecurity = transportSecurity;
}
private async Task TestConnection()
{
if (!this.CanTestConnection)
return;
var settings = this.CurrentSettings;
var serverAnswer = string.Empty;
this.isTestingConnection = true;
this.folderIssue = string.Empty;
try
{
using var timeout = new CancellationTokenSource(CONNECTION_TEST_TIMEOUT);
await using var connector = new ImapMailboxConnector();
await connector.ConnectAsync(this.CreateDataSource(), settings.Password, MailServerPolicy.Read(this.SettingsManager), timeout.Token);
this.serverFolders = await connector.GetFoldersAsync(timeout.Token);
this.testFailure = null;
this.Logger.LogInformation($"Tested the connection of the mailbox '{this.dataId}' successfully.");
}
catch (MailboxConnectionException e)
{
this.serverFolders = [];
this.testFailure = e.Failure;
serverAnswer = e.InnerException?.Message ?? string.Empty;
this.Logger.LogWarning($"Testing the connection of the mailbox '{this.dataId}' failed: {e.Failure} ({e.InnerException?.GetType().Name ?? "no inner exception"}).");
}
catch (OperationCanceledException)
{
this.serverFolders = [];
this.testFailure = MailboxConnectionFailure.NETWORK_UNAVAILABLE;
this.Logger.LogWarning($"Testing the connection of the mailbox '{this.dataId}' took too long.");
}
finally
{
this.testedSettings = settings;
this.isTestingConnection = false;
}
await this.RecordSignInOutcomeAsync(settings, this.testFailure, serverAnswer);
await this.form.Validate();
}
private async Task<MailServerFolder?> CreateFolder(string parentFullName, string name)
{
var settings = this.CurrentSettings;
var serverAnswer = string.Empty;
this.folderIssue = string.Empty;
MailboxConnectionFailure? failure = null;
try
{
using var timeout = new CancellationTokenSource(CONNECTION_TEST_TIMEOUT);
await using var connector = new ImapMailboxConnector();
await connector.ConnectAsync(this.CreateDataSource(), settings.Password, MailServerPolicy.Read(this.SettingsManager), timeout.Token);
var createdFolder = await connector.CreateFolderAsync(parentFullName, name, timeout.Token);
this.serverFolders = [..this.serverFolders, createdFolder];
this.Logger.LogInformation($"Created a folder in the mailbox '{this.dataId}'.");
return createdFolder;
}
catch (MailboxConnectionException e)
{
failure = e.Failure;
serverAnswer = e.InnerException?.Message ?? string.Empty;
this.folderIssue = e.Failure is MailboxConnectionFailure.SERVER_ERROR
? T("The server did not create the folder. Perhaps a folder of this name exists already.")
: e.Failure.GetDescription();
this.Logger.LogWarning($"Creating a folder in the mailbox '{this.dataId}' failed: {e.Failure} ({e.InnerException?.GetType().Name ?? "no inner exception"}).");
return null;
}
catch (ArgumentException)
{
this.folderIssue = T("This name is too long, or it contains a character the server reserves for folder paths.");
return null;
}
catch (OperationCanceledException)
{
this.folderIssue = MailboxConnectionFailure.NETWORK_UNAVAILABLE.GetDescription();
return null;
}
finally
{
if (failure is MailboxConnectionFailure.AUTHENTICATION_FAILED)
await this.RecordSignInOutcomeAsync(settings, failure, serverAnswer);
}
}
/// <summary>
/// Records what a sign-in with the stored settings showed, cf. the remarks of this dialog.
/// </summary>
/// <param name="settings">The settings the sign-in used.</param>
/// <param name="failure">Why the sign-in failed, or null when it worked.</param>
/// <param name="serverAnswer">What the server answered to a failed sign-in, which often says why, e.g., that it requires an app password.</param>
private async Task RecordSignInOutcomeAsync(ConnectionSettings settings, MailboxConnectionFailure? failure, string serverAnswer)
{
if (!this.UsesStoredSettings(settings))
return;
var indexStore = await this.DatabaseClientProvider.GetIndexStoreAsync();
switch (failure)
{
case null:
await indexStore.ClearMailboxAuthFailureAsync(this.dataId, CancellationToken.None);
this.authFailure = null;
break;
case MailboxConnectionFailure.AUTHENTICATION_FAILED:
this.authFailure = MailboxAuthFailure.FromServerAnswer(serverAnswer);
await indexStore.UpsertMailboxAuthFailureAsync(this.dataId, this.authFailure, CancellationToken.None);
break;
}
}
private async Task Store()
{
if (this.IsAddingNotAllowed)
return;
await this.form.Validate();
if (this.RequiresConnectionTest && this.dataSourceValidation.ValidateTestedConnection() is { } testIssue)
{
Array.Resize(ref this.dataIssues, this.dataIssues.Length + 1);
this.dataIssues[^1] = testIssue;
this.dataIsValid = false;
}
this.dataSecretStorageIssue = string.Empty;
// When the data is not valid, we don't store it:
if (!this.dataIsValid)
return;
var mailbox = this.CreateDataSource();
//
// Only when editing: while adding, DataSource is still default, and nothing has been
// prepared for a mailbox which does not exist yet.
//
if (this.IsEditing && !await DataSourceReindexWarning.ConfirmDataSourceChangeAsync(this.DialogService, this.SettingsManager, this.DataSourceEmbeddingService, this.DataSource, mailbox))
return;
//
// The OS keyring stores the password under the name of the mailbox, so a renamed mailbox
// gets a new entry, and the old one goes once the new one is in place:
//
var isRenamed = this.IsEditing && !string.Equals(this.DataSource.Name, mailbox.Name, StringComparison.Ordinal);
var isNewPassword = !this.IsEditing || !string.Equals(this.dataPassword, this.storedPassword, StringComparison.Ordinal);
if (isNewPassword || isRenamed)
{
var storeResponse = await this.RustService.SetSecret(mailbox, this.dataPassword, SecretStoreType.DATA_SOURCE);
if (!storeResponse.Success)
{
this.dataSecretStorageIssue = string.Format(T("Failed to store the password in the operating system. The message was: {0}. Please try again."), storeResponse.Issue);
await this.form.Validate();
return;
}
}
//
// A name which differs only in case keeps the old entry: some keyrings ignore case, so both
// names lead to the same entry, and deleting the old one would delete the password:
//
if (isRenamed && !string.Equals(this.DataSource.Name, mailbox.Name, StringComparison.OrdinalIgnoreCase))
{
var deleteResponse = await this.RustService.DeleteSecret(this.DataSource, SecretStoreType.DATA_SOURCE);
if (!deleteResponse.Success)
this.Logger.LogWarning($"Failed to delete the password of the mailbox '{this.dataId}' stored under its previous name: {deleteResponse.Issue}");
}
// A new password earns the next synchronization one attempt to sign in:
if (this.IsEditing && isNewPassword)
{
var indexStore = await this.DatabaseClientProvider.GetIndexStoreAsync();
await indexStore.ClearMailboxAuthFailureAsync(this.dataId, CancellationToken.None);
}
this.MudDialog.Close(DialogResult.Ok(mailbox));
}
private void Cancel() => this.MudDialog.Cancel();
/// <summary>
/// Gives the fields which are checked against each other a fresh verdict: the embedding provider
/// and the required confidence level, and the token limits, which depend on the embedding provider.
/// The provider joins them, because its rule reads what the organization allows.
/// </summary>
private Task RevalidateDependentFields(IFormComponent? changedField) => DependentFieldValidation.RevalidateAsync(changedField, this.providerSelect, this.embeddingSelect, this.confidenceLevelSelect, this.maxChunkTokenLengthField, this.chunkOverlapTokenLengthField);
private Task RevalidateAfterFieldChange(FormFieldChangedEventArgs change) => this.RevalidateDependentFields(change.Field);
private string? ValidateMaxChunkTokenLength(int maxChunkTokenLength)
{
if (!this.showExpertSettings)
return null;
if (maxChunkTokenLength < 1)
return T("Please enter a token limit of at least 1.");
var providerMaxChunkTokenLength = this.ProviderMaxChunkTokenLength;
if (maxChunkTokenLength > providerMaxChunkTokenLength)
return string.Format(T("The data source token limit must not be larger than the embedding provider token limit ({0})."), providerMaxChunkTokenLength);
return null;
}
private string? ValidateChunkOverlapTokenLength(int chunkOverlapTokenLength)
{
if (!this.showExpertSettings)
return null;
if (chunkOverlapTokenLength < 0)
return T("Please enter 0 or a positive overlap length.");
var effectiveMaxChunkTokenLength = this.dataMaxChunkTokenLength > 0
? this.dataMaxChunkTokenLength
: this.ProviderMaxChunkTokenLength;
if (chunkOverlapTokenLength >= effectiveMaxChunkTokenLength)
return T("The overlap must be smaller than the effective token limit.");
return null;
}
private void ToggleExpertSettings()
{
this.showExpertSettings = !this.showExpertSettings;
if (this.showExpertSettings && this.dataMaxChunkTokenLength < 1)
this.dataMaxChunkTokenLength = this.ProviderMaxChunkTokenLength;
// The token limits are only checked while they are shown. The field learns the limit set
// above only with the next render, so it is checked after that:
this.revalidateAfterRender = true;
}
private string GetExpertStyles => this.showExpertSettings ? "border-2 border-dashed rounded pa-2" : string.Empty;
}
@@ -72,6 +72,7 @@ public partial class PluginImportDialog : MSGComponentBase
Add(summary.DocumentAnalysisPolicies, this.T("{0} document analysis policy"), this.T("{0} document analysis policies"));
Add(summary.MandatoryInfos, this.T("{0} mandatory information you have to accept before using AI Studio"), this.T("{0} mandatory information you have to accept before using AI Studio"));
Add(summary.Introductions, this.T("{0} introduction on the welcome page"), this.T("{0} introductions on the welcome page"));
Add(summary.MailboxProviders, this.T("{0} mail server offered for new mailboxes"), this.T("{0} mail servers offered for new mailboxes"));
return contents;
@@ -3,22 +3,22 @@
<MudDialog>
<TitleContent>
<MudText Typo="Typo.h6" Class="d-flex align-center">
<MudIcon Icon="@this.implementation?.Icon" Class="mr-2" />
@(this.implementation?.GetDisplayName() ?? T("Tool Settings"))
<MudIcon Icon="@this.item?.Icon" Class="mr-2" />
@(this.item?.DisplayName ?? T("Tool Settings"))
</MudText>
</TitleContent>
<DialogContent>
@if (this.toolDefinition is null)
@if (this.item is null)
{
<MudText Typo="Typo.body1">@T("The selected tool could not be loaded.")</MudText>
}
else
{
<MudJustifiedText Typo="Typo.body1" Class="mb-4">
@this.implementation?.GetDescription()
@this.item.Description
</MudJustifiedText>
@if (!this.SettingsManager.IsToolActive(this.toolDefinition.Id))
@if (!this.ToolRegistry.IsToolActive(this.item.Id))
{
<MudAlert Severity="Severity.Warning" Class="mb-4">@T("This tool has been disabled by your organization.")</MudAlert>
}
@@ -33,47 +33,59 @@
<MudAlert Severity="Severity.Warning" Class="mb-4">@warning</MudAlert>
}
@foreach (var group in this.BuildVisibleFieldGroups())
@foreach (var toolSection in this.sections)
{
<MudPaper Class="pa-3 mb-4 border-dashed border rounded-lg">
@if (this.ShowsGroupHeader(group))
var visibleGroups = BuildVisibleFieldGroups(toolSection);
@if (this.ShowsToolHeaders && visibleGroups.Count > 0)
{
<MudStack Row="@true" AlignItems="AlignItems.Center" Justify="Justify.SpaceBetween" Wrap="Wrap.Wrap" Class="mb-3">
<MudText Typo="Typo.subtitle2">@this.GetGroupLabel(group.Key)</MudText>
<MudStack Row="@true" AlignItems="AlignItems.Center" Spacing="2" Wrap="Wrap.Wrap">
@foreach (var link in this.GetGroupLinks(group.Key))
{
<MudButton Variant="Variant.Filled" Size="Size.Small" StartIcon="@link.Icon" Href="@link.Url" Target="_blank">
@link.Label
</MudButton>
}
<MudText Typo="Typo.subtitle1" Class="d-flex align-center mb-2">
<MudIcon Icon="@toolSection.Implementation.Icon" Class="mr-2" />
@toolSection.Implementation.GetDisplayName()
</MudText>
}
@foreach (var group in visibleGroups)
{
<MudPaper Class="pa-3 mb-4 border-dashed border rounded-lg">
@if (ShowsGroupHeader(toolSection, group))
{
<MudStack Row="@true" AlignItems="AlignItems.Center" Justify="Justify.SpaceBetween" Wrap="Wrap.Wrap" Class="mb-3">
<MudText Typo="Typo.subtitle2">@this.GetGroupLabel(toolSection, group.Key)</MudText>
<MudStack Row="@true" AlignItems="AlignItems.Center" Spacing="2" Wrap="Wrap.Wrap">
@foreach (var link in GetGroupLinks(toolSection, group.Key))
{
<MudButton Variant="Variant.Filled" Size="Size.Small" StartIcon="@link.Icon" Href="@link.Url" Target="_blank">
@link.Label
</MudButton>
}
</MudStack>
</MudStack>
</MudStack>
}
@foreach (var property in group.Fields)
{
var fieldName = property.Key;
var field = property.Value;
var fieldOptions = field.GetOptions();
if (fieldOptions.Count > 0)
{
<MudSelect T="string" Label="@this.GetFieldLabel(fieldName, field)" Value="@this.GetValue(fieldName)" ValueChanged="@(value => this.UpdateValue(fieldName, value))" Variant="Variant.Outlined" Margin="Margin.Dense" HelperText="@this.GetFieldDescription(fieldName, field)" Placeholder="@this.GetFieldPlaceholder(fieldName, field)" Class="mb-3 configuration-help-justified" Disabled="@this.IsFieldDisabled(fieldName)">
@if (!this.toolDefinition.SettingsSchema.Required.Contains(fieldName))
{
<MudSelectItem T="string" Value="@string.Empty">@T("Not set")</MudSelectItem>
}
@foreach (var option in fieldOptions)
{
<MudSelectItem T="string" Value="@option.Value">@option.Label</MudSelectItem>
}
</MudSelect>
}
else
@foreach (var property in group.Fields)
{
<MudTextField T="string" Label="@this.GetFieldLabel(fieldName, field)" Value="@this.GetValue(fieldName)" ValueChanged="@(value => this.UpdateValue(fieldName, value))" Variant="Variant.Outlined" Margin="Margin.Dense" Class="mb-3 configuration-help-justified" HelperText="@this.GetFieldDescription(fieldName, field)" Placeholder="@this.GetFieldPlaceholder(fieldName, field)" InputType="@(field.Secret ? InputType.Password : InputType.Text)" Disabled="@this.IsFieldDisabled(fieldName)" />
var fieldName = property.Key;
var field = property.Value;
var fieldOptions = field.GetOptions();
if (fieldOptions.Count > 0)
{
<MudSelect T="string" Label="@GetFieldLabel(toolSection, fieldName, field)" Value="@GetValue(toolSection, fieldName)" ValueChanged="@(value => this.UpdateValue(toolSection, fieldName, value))" Variant="Variant.Outlined" Margin="Margin.Dense" HelperText="@this.GetFieldDescription(toolSection, fieldName, field)" Placeholder="@GetFieldPlaceholder(toolSection, fieldName, field)" Class="mb-3 configuration-help-justified" Disabled="@this.IsFieldDisabled(toolSection, fieldName)">
@if (!toolSection.Definition.SettingsSchema.Required.Contains(fieldName))
{
<MudSelectItem T="string" Value="@string.Empty">@T("Not set")</MudSelectItem>
}
@foreach (var option in fieldOptions)
{
<MudSelectItem T="string" Value="@option.Value">@option.Label</MudSelectItem>
}
</MudSelect>
}
else
{
<MudTextField T="string" Label="@GetFieldLabel(toolSection, fieldName, field)" Value="@GetValue(toolSection, fieldName)" ValueChanged="@(value => this.UpdateValue(toolSection, fieldName, value))" Variant="Variant.Outlined" Margin="Margin.Dense" Class="mb-3 configuration-help-justified" HelperText="@this.GetFieldDescription(toolSection, fieldName, field)" Placeholder="@GetFieldPlaceholder(toolSection, fieldName, field)" InputType="@(field.Secret ? InputType.Password : InputType.Text)" Disabled="@this.IsFieldDisabled(toolSection, fieldName)" />
}
}
</MudPaper>
}
</MudPaper>
}
}
</DialogContent>
@@ -81,8 +93,8 @@
<MudButton OnClick="@this.Close" Variant="Variant.Text">
@T("Cancel")
</MudButton>
<MudButton OnClick="@this.Save" Variant="Variant.Filled" Disabled="@(this.toolDefinition is null)">
<MudButton OnClick="@this.Save" Variant="Variant.Filled" Disabled="@(this.item is null)">
@T("Save")
</MudButton>
</DialogActions>
</MudDialog>
</MudDialog>
@@ -4,10 +4,22 @@ using Microsoft.AspNetCore.Components;
namespace AIStudio.Dialogs.Settings;
/// <summary>
/// Shows and saves the settings of one entry of the tool catalog.
/// </summary>
/// <remarks>
/// An entry is a tool collection, see ToolCollectionDefinition. Its tools keep their settings to
/// themselves, so the dialog shows one section per tool, and an organization still addresses each
/// field as "toolId.fieldName". A tool which belongs to no declared collection forms an entry of its
/// own, and the dialog looks as it always did for it: one section without a heading.
/// </remarks>
public partial class ToolSettingsDialog : SettingsDialogBase
{
/// <summary>
/// The ID of the collection whose settings to show. The ID of one of its tools stands for the whole collection.
/// </summary>
[Parameter]
public string ToolId { get; set; } = string.Empty;
public string CollectionId { get; set; } = string.Empty;
[Inject]
private ToolRegistry ToolRegistry { get; init; } = null!;
@@ -15,25 +27,34 @@ public partial class ToolSettingsDialog : SettingsDialogBase
[Inject]
private ToolSettingsService ToolSettingsService { get; init; } = null!;
private ToolDefinition? toolDefinition;
private IToolImplementation? implementation;
private Dictionary<string, string> values = new(StringComparer.Ordinal);
private IReadOnlyList<FieldGroup> fieldGroups = [];
private ToolCatalogItem? item;
private IReadOnlyList<ToolSection> sections = [];
private string validationMessage = string.Empty;
protected override async Task OnInitializedAsync()
{
await base.OnInitializedAsync();
this.toolDefinition = this.ToolRegistry.GetDefinition(this.ToolId);
if (this.toolDefinition is not null)
{
this.implementation = this.ToolRegistry.GetImplementation(this.toolDefinition.ImplementationKey);
this.values = await this.ToolSettingsService.GetSettingsAsync(this.toolDefinition);
this.fieldGroups = BuildFieldGroups(this.toolDefinition);
}
this.item = await this.ToolRegistry.GetCatalogItemAsync(this.CollectionId);
if (this.item is null)
return;
var loadedSections = new List<ToolSection>(this.item.Tools.Count);
foreach (var tool in this.item.Tools)
loadedSections.Add(new(tool.Definition, tool.Implementation, await this.ToolSettingsService.GetSettingsAsync(tool.Definition), BuildFieldGroups(tool.Definition)));
this.sections = loadedSections;
}
private string GetValue(string fieldName) => this.values.GetValueOrDefault(fieldName, string.Empty);
/// <summary>
/// Whether each section is headed by the name of its tool.
/// </summary>
/// <remarks>
/// Only when the entry holds more than one tool. Otherwise the title of the dialog names the
/// tool already.
/// </remarks>
private bool ShowsToolHeaders => this.sections.Count > 1;
private static string GetValue(ToolSection section, string fieldName) => section.Values.GetValueOrDefault(fieldName, string.Empty);
/// <summary>
/// Splits the tool's settings fields into the groups the tool declared for them.
@@ -66,7 +87,7 @@ public partial class ToolSettingsDialog : SettingsDialogBase
}
/// <summary>
/// The groups as they are rendered right now, without the fields the tool is hiding.
/// The groups of one tool as they are rendered right now, without the fields the tool is hiding.
/// </summary>
/// <remarks>
/// Which fields make sense can depend on what is filled in, so this is built on every
@@ -76,15 +97,12 @@ public partial class ToolSettingsDialog : SettingsDialogBase
/// Cheap enough to be called more than once per render: a tool has a handful of settings,
/// and asking the tool about one of them costs a dictionary lookup or two.
/// </remarks>
private IReadOnlyList<FieldGroup> BuildVisibleFieldGroups()
private static IReadOnlyList<FieldGroup> BuildVisibleFieldGroups(ToolSection section)
{
if (this.implementation is null)
return this.fieldGroups;
var visibleGroups = new List<FieldGroup>();
foreach (var group in this.fieldGroups)
foreach (var group in section.FieldGroups)
{
var visibleFields = group.Fields.Where(field => this.implementation.IsSettingsFieldVisible(field.Key, this.values)).ToList();
var visibleFields = group.Fields.Where(field => section.Implementation.IsSettingsFieldVisible(field.Key, section.Values)).ToList();
if (visibleFields.Count > 0)
visibleGroups.Add(new FieldGroup(group.Key, visibleFields));
}
@@ -100,84 +118,98 @@ public partial class ToolSettingsDialog : SettingsDialogBase
/// heading above the only box would say nothing the dialog's title does not say already.
/// As soon as there is a second box, each of them has to state which one it is — the box
/// holding the fields that belong to no group in particular included.<br/><br/>
/// It counts the boxes that are actually rendered, so a group the tool hides entirely does
/// not leave the remaining box with a heading it does not need.
/// It counts the boxes of the tool that are actually rendered, so a group the tool hides
/// entirely does not leave the remaining box with a heading it does not need.
/// </remarks>
private bool ShowsGroupHeader(FieldGroup group) => this.BuildVisibleFieldGroups().Count > 1 || !string.IsNullOrEmpty(group.Key);
private static bool ShowsGroupHeader(ToolSection section, FieldGroup group) => BuildVisibleFieldGroups(section).Count > 1 || !string.IsNullOrEmpty(group.Key);
/// <remarks>
/// The ungrouped fields have no name of their own, so the label hook hands back their
/// empty group name. A tool may still name them through that same hook; when it does not,
/// they are simply what is left over next to the named groups.
/// </remarks>
private string GetGroupLabel(string groupKey)
private string GetGroupLabel(ToolSection section, string groupKey)
{
var label = this.implementation?.GetSettingsGroupLabel(groupKey) ?? groupKey;
var label = section.Implementation.GetSettingsGroupLabel(groupKey);
return string.IsNullOrEmpty(label) ? T("General") : label;
}
private IReadOnlyList<ToolSettingsGroupLink> GetGroupLinks(string groupKey) => this.implementation?.GetSettingsGroupLinks(groupKey) ?? [];
private static IReadOnlyList<ToolSettingsGroupLink> GetGroupLinks(ToolSection section, string groupKey) => section.Implementation.GetSettingsGroupLinks(groupKey);
/// <summary>
/// What the tool wants to say about the settings as they stand right now.
/// What the tools want to say about their settings as they stand right now.
/// </summary>
/// <remarks>
/// Asked on every render, so a warning follows the value it is about instead of waiting for
/// the next save. These are not errors: they describe settings that are allowed and do
/// something other than what they look like, and the dialog saves them either way.
/// </remarks>
private IReadOnlyList<string> GetSettingsWarnings() => this.implementation?.GetSettingsWarnings(this.values) ?? [];
private IEnumerable<string> GetSettingsWarnings() => this.sections.SelectMany(section => section.Implementation.GetSettingsWarnings(section.Values));
private string GetFieldLabel(string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
this.implementation?.GetSettingsFieldLabel(fieldName, fieldDefinition) ?? fieldDefinition.Title;
private static string GetFieldLabel(ToolSection section, string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
section.Implementation.GetSettingsFieldLabel(fieldName, fieldDefinition);
private string GetFieldDescription(string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
this.GetFieldDescriptionWithDefault(fieldName, fieldDefinition);
private static string GetFieldDefaultValue(ToolSection section, string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
section.Implementation.GetSettingsFieldDefaultValue(fieldName, fieldDefinition) ?? string.Empty;
private string GetFieldDefaultValue(string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
this.implementation?.GetSettingsFieldDefaultValue(fieldName, fieldDefinition) ?? string.Empty;
private string GetFieldDescriptionWithDefault(string fieldName, ToolSettingsFieldDefinition fieldDefinition)
private string GetFieldDescription(ToolSection section, string fieldName, ToolSettingsFieldDefinition fieldDefinition)
{
var description = this.implementation?.GetSettingsFieldDescription(fieldName, fieldDefinition) ?? fieldDefinition.Description;
var defaultValue = this.GetFieldDefaultValue(fieldName, fieldDefinition);
var description = section.Implementation.GetSettingsFieldDescription(fieldName, fieldDefinition);
var defaultValue = GetFieldDefaultValue(section, fieldName, fieldDefinition);
if (string.IsNullOrWhiteSpace(defaultValue))
return description;
return string.Format(T("{0} Default: {1}"), description, defaultValue);
}
private bool IsFieldDisabled(string fieldName) =>
this.toolDefinition is not null && this.ToolSettingsService.IsFieldLocked(this.toolDefinition, fieldName);
private bool IsFieldDisabled(ToolSection section, string fieldName) => this.ToolSettingsService.IsFieldLocked(section.Definition, fieldName);
private string GetFieldPlaceholder(string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
string.IsNullOrWhiteSpace(this.GetValue(fieldName)) ? this.GetFieldDefaultValue(fieldName, fieldDefinition) : string.Empty;
private static string GetFieldPlaceholder(ToolSection section, string fieldName, ToolSettingsFieldDefinition fieldDefinition) =>
string.IsNullOrWhiteSpace(GetValue(section, fieldName)) ? GetFieldDefaultValue(section, fieldName, fieldDefinition) : string.Empty;
private void UpdateValue(string fieldName, string? value)
private void UpdateValue(ToolSection section, string fieldName, string? value)
{
this.values[fieldName] = value ?? string.Empty;
section.Values[fieldName] = value ?? string.Empty;
this.validationMessage = string.Empty;
}
/// <summary>
/// Checks the settings of every tool, and saves those of the tools which have any.
/// </summary>
/// <remarks>
/// Nothing is saved while one tool is not satisfied, so a collection never ends up with some
/// of its tools configured and the others not, which would leave it unusable all the same.
/// </remarks>
private async Task Save()
{
if (this.toolDefinition is null)
if (this.item is null)
return;
var validationState = await this.ToolSettingsService.ValidateSettingsAsync(this.toolDefinition, this.values, this.implementation);
if (!validationState.IsConfigured)
foreach (var section in this.sections)
{
var validationState = await this.ToolSettingsService.ValidateSettingsAsync(section.Definition, section.Values, section.Implementation);
if (validationState.IsConfigured)
continue;
this.validationMessage = !string.IsNullOrWhiteSpace(validationState.Message)
? validationState.Message
: string.Format(T("Please configure the required settings: {0}"), string.Join(", ", validationState.MissingRequiredFields));
return;
}
await this.ToolSettingsService.SaveSettingsAsync(this.toolDefinition, this.values);
foreach (var section in this.sections.Where(section => section.Definition.SettingsSchema.Properties.Count > 0))
await this.ToolSettingsService.SaveSettingsAsync(section.Definition, section.Values);
this.MudDialog.Close();
}
/// <param name="Definition">The tool.</param>
/// <param name="Implementation">The implementation of the tool, which labels and checks its fields.</param>
/// <param name="Values">The values as they are being edited.</param>
/// <param name="FieldGroups">The groups of its fields, in the order the tool declared them.</param>
private sealed record ToolSection(ToolDefinition Definition, IToolImplementation Implementation, Dictionary<string, string> Values, IReadOnlyList<FieldGroup> FieldGroups);
/// <param name="Key">The group's name from the schema, or empty for the ungrouped fields.</param>
/// <param name="Fields">The fields of this group, in the order the tool declared them.</param>
private sealed record FieldGroup(string Key, List<KeyValuePair<string, ToolSettingsFieldDefinition>> Fields);
}
}
@@ -22,7 +22,7 @@
<MudProgressCircular Indeterminate="@true" Size="Size.Small" />
<MudText>@T("Loading tool configuration...")</MudText>
}
else if (this.toolDefinition is null || this.implementation is null)
else if (this.item is null)
{
@if (string.IsNullOrWhiteSpace(this.message))
{
@@ -31,7 +31,7 @@
}
else
{
<MudText Typo="Typo.subtitle1" Class="mb-2">@this.implementation.GetDisplayName()</MudText>
<MudText Typo="Typo.subtitle1" Class="mb-2">@this.item.DisplayName</MudText>
<MudText Typo="Typo.body2" Class="mb-4">
@T("Export saved settings as Lua code for your configuration plugin. You can combine exports and adapt the code before deploying it.")
</MudText>
@@ -8,8 +8,11 @@ namespace AIStudio.Dialogs.Settings;
public partial class ToolSettingsExportDialog : SettingsDialogBase
{
/// <summary>
/// The ID of the collection whose settings to export. The ID of one of its tools stands for the whole collection.
/// </summary>
[Parameter]
public string ToolId { get; set; } = string.Empty;
public string CollectionId { get; set; } = string.Empty;
[Inject]
private ToolRegistry ToolRegistry { get; init; } = null!;
@@ -20,12 +23,11 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
[Inject]
private ILogger<ToolSettingsExportDialog> Logger { get; init; } = null!;
private ToolDefinition? toolDefinition;
private IToolImplementation? implementation;
private IReadOnlyList<ExportableSettings> areas = [];
private ToolCatalogItem? item;
private IReadOnlyList<ToolSettingsExportArea> areas = [];
private HashSet<string> selectedAreaIds = new(StringComparer.Ordinal);
private HashSet<string> configuredSecretFields = new(StringComparer.Ordinal);
private HashSet<string> emptyFieldNames = new(StringComparer.Ordinal);
private readonly HashSet<ToolField> configuredSecretFields = [];
private readonly HashSet<ToolField> emptyFields = [];
private ToolSettingsExportMode mode = ToolSettingsExportMode.LOCKED;
private bool includeSecrets;
private bool includeMinimumProviderConfidence = true;
@@ -39,7 +41,7 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
private bool AllAreasSelected => this.areas.Count > 0 && this.areas.All(area => this.selectedAreaIds.Contains(area.Id));
private bool HasSelectedSecrets => this.areas.Any(area => this.selectedAreaIds.Contains(area.Id) && area.FieldNames.Any(this.configuredSecretFields.Contains));
private bool HasSelectedSecrets => this.SelectedFields.Any(this.configuredSecretFields.Contains);
private bool CanIncludeSecrets => this.HasSelectedSecrets && PluginFactory.EnterpriseEncryption?.IsAvailable is true;
@@ -51,16 +53,20 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
/// the administrator never filled in still exports. Locked, those empty values are what the
/// recipient is left with and cannot change, which is worth saying before the export.
/// </remarks>
private int EmptySelectedFieldCount => this.areas
private int EmptySelectedFieldCount => this.SelectedFields.Count(this.emptyFields.Contains);
/// <summary>
/// The fields of all selected areas, each once.
/// </summary>
private IEnumerable<ToolField> SelectedFields => this.areas
.Where(area => this.selectedAreaIds.Contains(area.Id))
.SelectMany(area => area.FieldNames)
.Distinct(StringComparer.Ordinal)
.Count(this.emptyFieldNames.Contains);
.SelectMany(area => area.Area.FieldNames.Select(fieldName => new ToolField(area.Tool.Definition.Id, fieldName)))
.Distinct();
private bool WarnAboutEmptyLockedSettings => this.mode is ToolSettingsExportMode.LOCKED && this.EmptySelectedFieldCount > 0;
private bool CanExport => this.IsAdmin && !this.isLoading && !this.isExporting && !this.isDisposed &&
this.toolDefinition is not null && this.implementation is not null && (this.selectedAreaIds.Count > 0 || this.includeMinimumProviderConfidence);
this.item is not null && (this.selectedAreaIds.Count > 0 || this.includeMinimumProviderConfidence);
protected override async Task OnInitializedAsync()
{
@@ -73,40 +79,38 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
try
{
this.toolDefinition = this.ToolRegistry.GetDefinition(this.ToolId);
if (this.toolDefinition is null)
this.item = await this.ToolRegistry.GetCatalogItemAsync(this.CollectionId);
if (this.item is null)
return;
this.implementation = this.ToolRegistry.GetImplementation(this.toolDefinition.ImplementationKey);
if (this.implementation is null)
return;
this.areas = this.implementation.GetExportableSettings(this.toolDefinition);
this.areas = ToolSettingsService.GetExportAreas(this.item.Tools);
this.selectedAreaIds = this.areas.Select(area => area.Id).ToHashSet(StringComparer.Ordinal);
// Retain only field names, never the values themselves, so no plaintext secret lives
// in this component. ExportAsync reads effective settings again when the
// administrator exports.
var values = await this.ToolSettingsService.GetSettingsAsync(this.toolDefinition);
this.configuredSecretFields = this.toolDefinition.SettingsSchema.Properties
.Where(property => property.Value.Secret && values.TryGetValue(property.Key, out var value) && !string.IsNullOrWhiteSpace(value))
.Select(property => property.Key)
.ToHashSet(StringComparer.Ordinal);
foreach (var tool in this.item.Tools)
{
var values = await this.ToolSettingsService.GetSettingsAsync(tool.Definition);
var properties = tool.Definition.SettingsSchema.Properties;
this.configuredSecretFields.UnionWith(properties
.Where(property => property.Value.Secret && values.TryGetValue(property.Key, out var value) && !string.IsNullOrWhiteSpace(value))
.Select(property => new ToolField(tool.Definition.Id, property.Key)));
// A field the export writes as an empty value: it has to be present, because a
// missing one is skipped rather than exported, and it has to be a non-secret,
// because an empty secret is skipped as well.
this.emptyFieldNames = this.toolDefinition.SettingsSchema.Properties
.Where(property => !property.Value.Secret && values.TryGetValue(property.Key, out var value) && string.IsNullOrWhiteSpace(value))
.Select(property => property.Key)
.ToHashSet(StringComparer.Ordinal);
// A field the export writes as an empty value: it has to be present, because a
// missing one is skipped rather than exported, and it has to be a non-secret,
// because an empty secret is skipped as well.
this.emptyFields.UnionWith(properties
.Where(property => !property.Value.Secret && values.TryGetValue(property.Key, out var value) && string.IsNullOrWhiteSpace(value))
.Select(property => new ToolField(tool.Definition.Id, property.Key)));
}
}
catch (Exception e)
{
// A runtime error may contain secret data, so it goes to the log for diagnosis but
// never into the dialog:
this.Logger.LogError(e, "Failed to load the configuration of the tool '{ToolId}' for export.", this.ToolId);
this.toolDefinition = null;
this.Logger.LogError(e, "Failed to load the configuration of the tool collection '{CollectionId}' for export.", this.CollectionId);
this.item = null;
this.message = T("The tool configuration could not be loaded. Please close this dialog and try again.");
}
finally
@@ -142,13 +146,13 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
private string GetMinimumProviderConfidenceName()
{
var confidence = this.toolDefinition is null ? ConfidenceLevel.NONE : this.ToolRegistry.GetMinimumProviderConfidence(this.toolDefinition);
var confidence = this.item is null ? ConfidenceLevel.NONE : this.ToolRegistry.GetMinimumProviderConfidence(this.item.Id);
return confidence is ConfidenceLevel.NONE ? T("No minimum confidence level chosen") : confidence.GetName();
}
private async Task Export()
{
if (!this.CanExport || this.toolDefinition is null || this.implementation is null)
if (!this.CanExport || this.item is null)
return;
this.isExporting = true;
@@ -164,7 +168,7 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
IncludeMinimumProviderConfidence = this.includeMinimumProviderConfidence,
};
var result = await this.ToolSettingsService.ExportAsync(this.toolDefinition, this.implementation, options);
var result = await this.ToolSettingsService.ExportAsync(this.item.Tools, options, this.item.Id, this.ToolRegistry.GetMinimumProviderConfidence(this.item.Id));
if (this.isDisposed || !this.IsAdmin)
return;
@@ -187,7 +191,7 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
}
catch (Exception e)
{
this.Logger.LogError(e, "Failed to export the configuration of the tool '{ToolId}'.", this.ToolId);
this.Logger.LogError(e, "Failed to export the configuration of the tool collection '{CollectionId}'.", this.CollectionId);
this.message = T("The tool configuration could not be exported. Please try again.");
}
finally
@@ -201,4 +205,9 @@ public partial class ToolSettingsExportDialog : SettingsDialogBase
this.isDisposed = true;
base.DisposeResources();
}
/// <summary>
/// One settings field of one tool. A collection may hold two tools whose fields share a name.
/// </summary>
private readonly record struct ToolField(string ToolId, string FieldName);
}
@@ -487,15 +487,18 @@ public partial class MainLayout : LayoutComponentBase, IMessageBusReceiver, ILan
this.embeddingItem = new NavBarItem(T("Data sources"), embeddingIcon.icon, embeddingIcon.lightcolor, embeddingIcon.darkcolor, Routes.EMBEDDINGS, false);
}
/// <remarks>
/// Speaks of neither files nor mails: the overview sums up data sources of every kind.
/// </remarks>
private string EmbeddingNavigationTooltip => this.embeddingOverview.State switch
{
DataSourceEmbeddingState.QUEUED => T("Embeddings are waiting to be processed."),
DataSourceEmbeddingState.RUNNING => string.Format(
T("Embeddings are running: {0} of {1} files are indexed."),
this.embeddingOverview.IndexedFiles,
this.embeddingOverview.TotalFiles),
DataSourceEmbeddingState.FAILED => this.embeddingOverview.FailedFiles > 0
? string.Format(T("Some embeddings failed. {0} file(s) need attention."), this.embeddingOverview.FailedFiles)
T("Embeddings are running: {0} of {1} are indexed."),
this.embeddingOverview.IndexedDocuments.CompactCount(),
this.embeddingOverview.TotalDocuments.CompactCount()),
DataSourceEmbeddingState.FAILED => this.embeddingOverview.FailedDocuments > 0
? string.Format(T("Some embeddings failed: {0} could not be indexed."), this.embeddingOverview.FailedDocuments.CompactCount())
: T("Some embeddings failed and need attention."),
// The entry is always visible, so its resting state needs words as well. An empty tooltip
@@ -62,6 +62,7 @@
<ItemGroup>
<PackageReference Include="CodeBeam.MudBlazor.Extensions" Version="8.3.0" />
<PackageReference Include="HtmlAgilityPack" Version="1.12.4" />
<PackageReference Include="MailKit" Version="4.18.1" />
<PackageReference Include="Microsoft.Data.Sqlite.Core" Version="9.0.20" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="9.0.20" />
<PackageReference Include="Microsoft.Extensions.FileProviders.Embedded" Version="9.0.20" />
+65 -16
View File
@@ -1,4 +1,5 @@
@attribute [Route(Routes.EMBEDDINGS)]
@using AIStudio.Tools.Services
@inherits MSGComponentBase
<MudStack Spacing="3" Class="pr-2 pb-4" Style="height: 100%; min-height: 0; overflow-y: auto;">
@@ -13,11 +14,17 @@
<MudText Typo="Typo.body1" Class="mt-2">
@T("AI Studio indexes local RAG data sources in the background. Finished files stay recorded so unchanged files can be skipped after a restart, while added or deleted files are detected during the next run. The same applies to documents without readable text, such as scanned pages: AI Studio remembers them and reads them again only once they change.")
</MudText>
@if (this.AreMailboxesEnabled)
{
<MudText Typo="Typo.body1" Class="mt-2">
@T("Mailboxes are synced every 16 minutes while the automatic refresh is on. Each mail is embedded only once: when it moves to another folder, AI Studio merely notes where it lies now.")
</MudText>
}
<MudStack Row="true" Class="mt-3" Wrap="Wrap.Wrap" Spacing="2">
<MudChip T="string" Color="Color.Success" Variant="Variant.Filled">@string.Format(T("Indexed files: {0}"), this.TotalIndexedFiles)</MudChip>
<MudChip T="string" Color="Color.Info" Variant="Variant.Filled">@string.Format(T("Pending files: {0}"), this.TotalPendingFiles)</MudChip>
<MudChip T="string" Color="Color.Default" Variant="Variant.Filled">@string.Format(T("Skipped files: {0}"), this.TotalPermanentlySkippedFiles)</MudChip>
<MudChip T="string" Color="Color.Error" Variant="Variant.Filled">@string.Format(T("Failed files: {0}"), this.TotalFailedFiles)</MudChip>
<MudChip T="string" Color="Color.Success" Variant="Variant.Filled">@string.Format(T("Indexed: {0}"), this.FormatNumber(this.TotalIndexedDocuments))</MudChip>
<MudChip T="string" Color="Color.Info" Variant="Variant.Filled">@string.Format(T("Pending: {0}"), this.FormatNumber(this.TotalPendingDocuments))</MudChip>
<MudChip T="string" Color="Color.Default" Variant="Variant.Filled">@string.Format(T("Skipped: {0}"), this.FormatNumber(this.TotalPermanentlySkippedDocuments))</MudChip>
<MudChip T="string" Color="Color.Error" Variant="Variant.Filled">@string.Format(T("Failed: {0}"), this.FormatNumber(this.TotalFailedDocuments))</MudChip>
</MudStack>
@if (this.IsWorkingThroughDataSources)
{
@@ -74,27 +81,34 @@
<MudProgressLinear Value="@status.ProgressPercent" Rounded="@true" Color="@GetStatusColor(status)" />
<MudText Typo="Typo.body2">
@this.GetFileProgressText(status)
@this.GetProgressText(status)
</MudText>
@if (status.PermanentlySkippedFiles > 0)
@if (IsMailbox(status))
{
<MudText Typo="Typo.body2">
@string.Format(T("Skipped files: {0}. AI Studio reads them again once they change."), status.PermanentlySkippedFiles)
@this.GetLastSyncText(status)
</MudText>
}
@if (status.FailedFiles > 0)
@if (status.PermanentlySkippedDocuments > 0)
{
<MudText Typo="Typo.body2">
@string.Format(T("Failed files: {0}"), status.FailedFiles)
@this.GetSkippedText(status)
</MudText>
}
@if (!string.IsNullOrWhiteSpace(status.CurrentFile))
@if (status.FailedDocuments > 0)
{
<MudText Typo="Typo.body2">
@string.Format(T("Current file: {0}"), status.CurrentFile)
@this.GetFailedText(status)
</MudText>
}
@if (!string.IsNullOrWhiteSpace(status.CurrentDocument))
{
<MudText Typo="Typo.body2">
@this.GetCurrentDocumentText(status)
</MudText>
}
@@ -112,7 +126,7 @@
<MudStack Spacing="3">
<MudStack Row="@true" Wrap="Wrap.Wrap" Spacing="2" AlignItems="AlignItems.Center">
<MudChip T="string" Size="Size.Small" Color="@group.Cause.Color" Variant="Variant.Outlined">
@(group.Cause.IsPermanent ? T("Skipped until the file changes") : T("Tried again during the next run"))
@(group.Cause.IsPermanent ? (IsMailbox(status) ? T("Skipped") : T("Skipped until the file changes")) : T("Tried again during the next run"))
</MudChip>
@if (!string.IsNullOrWhiteSpace(group.EmbeddingProviderName))
{
@@ -132,15 +146,22 @@
<col style="width: 7em;"/>
</ColGroup>
<HeaderContent>
<MudTh>@T("File")</MudTh>
<MudTh>@(IsMailbox(status) ? T("Mail") : T("File"))</MudTh>
<MudTh>@T("Noticed")</MudTh>
<MudTh>@T("Actions")</MudTh>
</HeaderContent>
<RowTemplate>
<MudTd Style="white-space: normal; overflow-wrap: anywhere;">
@* Captions render as spans, so each of them needs to be told to take its own line. *@
<MudText Typo="Typo.body2">@GetFileName(context.FilePath)</MudText>
<MudText Typo="Typo.caption" Class="mud-text-secondary d-block">@context.FilePath</MudText>
@if (IsMailbox(status))
{
<MudText Typo="Typo.body2">@GetMailName(context)</MudText>
}
else
{
<MudText Typo="Typo.body2">@GetFileName(context.DocumentKey)</MudText>
<MudText Typo="Typo.caption" Class="mud-text-secondary d-block">@context.DocumentKey</MudText>
}
@if (group.Cause.ShowsMessagePerFile)
{
<MudText Typo="Typo.caption" Color="Color.Warning" Class="d-block">@context.Reason</MudText>
@@ -150,7 +171,7 @@
<MudText Typo="Typo.caption">@GetOccurrenceText(context)</MudText>
</MudTd>
<MudTd>
@if (CanShowInFileManager(context))
@if (CanShowInFileManager(status, context))
{
<MudTooltip Text="@T("Show this file in the file browser of your system")">
<MudIconButton Icon="@Icons.Material.Filled.FolderOpen" Color="Color.Info" Size="Size.Small" OnClick="@(async () => await this.ShowInFileManager(context))"/>
@@ -179,6 +200,34 @@
@status.LastError
</MudAlert>
}
@*
What the user has to decide comes with the buttons to decide it. The
sentence above says what is at stake; reading it and then having to find
the settings of the mailbox is where people give up.
*@
@if (status.Attention is DataSourceAttention.AUTH_FAILED)
{
<MudStack Row="@true" Wrap="Wrap.Wrap" Spacing="2">
<MudButton Variant="Variant.Filled" Color="Color.Primary" StartIcon="@Icons.Material.Filled.Key" OnClick="@(async () => await this.ChangeMailboxSettings(status))">
@T("Change the password")
</MudButton>
<MudButton Variant="Variant.Outlined" Color="Color.Primary" StartIcon="@Icons.Material.Filled.Sync" OnClick="@(async () => await this.RefreshDataSource(status))">
@T("Try to sign in again")
</MudButton>
</MudStack>
}
else if (status is { Attention: DataSourceAttention.MASS_REMOVAL_PENDING, PendingRemovalCount: not null })
{
<MudStack Row="@true" Wrap="Wrap.Wrap" Spacing="2">
<MudButton Variant="Variant.Filled" Color="Color.Warning" StartIcon="@Icons.Material.Filled.PlaylistRemove" OnClick="@(async () => await this.ApproveMailRemoval(status))">
@T("Remove them from the index")
</MudButton>
<MudButton Variant="Variant.Outlined" Color="Color.Primary" StartIcon="@Icons.Material.Filled.Settings" OnClick="@(async () => await this.ChangeMailboxSettings(status))">
@T("Change the settings of the mailbox")
</MudButton>
</MudStack>
}
</MudStack>
</ChildContent>
</MudExpansionPanel>
+103 -22
View File
@@ -44,18 +44,24 @@ public partial class Embeddings : MSGComponentBase
/// </remarks>
private CultureInfo currentCulture = CultureInfo.InvariantCulture;
private int TotalIndexedFiles => this.Statuses.Sum(status => status.IndexedFiles);
/// <remarks>
/// The sums above every data source count files and mails alike, which is why their chips name
/// neither.
/// </remarks>
private int TotalIndexedDocuments => this.Statuses.Sum(status => status.IndexedDocuments);
private int TotalPendingFiles => this.Statuses.Sum(status => Math.Max(0, status.TotalFiles - status.IndexedFiles - status.FailedFiles - status.PermanentlySkippedFiles));
private int TotalPendingDocuments => this.Statuses.Sum(status => Math.Max(0, status.TotalDocuments - status.IndexedDocuments - status.FailedDocuments - status.PermanentlySkippedDocuments));
private int TotalFailedFiles => this.Statuses.Sum(status => status.FailedFiles);
private int TotalFailedDocuments => this.Statuses.Sum(status => status.FailedDocuments);
private int TotalPermanentlySkippedFiles => this.Statuses.Sum(status => status.PermanentlySkippedFiles);
private int TotalPermanentlySkippedDocuments => this.Statuses.Sum(status => status.PermanentlySkippedDocuments);
private bool AreMailboxesEnabled => PreviewFeatures.PRE_MAILBOXES_2026.IsEnabled(this.SettingsManager);
/// <remarks>
/// The chips above count files, which says nothing about how far the list of data sources itself
/// has come. While several of them wait their turn, this is the one line saying so. With a single
/// data source there is nothing to say: its own row already tells the whole story.
/// The chips above count documents, which says nothing about how far the list of data sources
/// itself has come. While several of them wait their turn, this is the one line saying so. With a
/// single data source there is nothing to say: its own row already tells the whole story.
/// </remarks>
private bool IsWorkingThroughDataSources => this.Statuses.Count > 1 && this.Statuses.Any(status => status.State is DataSourceEmbeddingState.RUNNING or DataSourceEmbeddingState.QUEUED);
@@ -141,7 +147,7 @@ public partial class Embeddings : MSGComponentBase
/// data source at once.
/// </remarks>
private static bool IsWorthOpening(DataSourceEmbeddingStatus status) =>
status.State is DataSourceEmbeddingState.RUNNING or DataSourceEmbeddingState.QUEUED or DataSourceEmbeddingState.FAILED || status.FailedFiles > 0;
status.State is DataSourceEmbeddingState.RUNNING or DataSourceEmbeddingState.QUEUED or DataSourceEmbeddingState.FAILED || status.FailedDocuments > 0;
/// <remarks>
/// MudBlazor keeps track of which panel is open on its own, so this only records the decision.
@@ -173,7 +179,7 @@ public partial class Embeddings : MSGComponentBase
}
/// <summary>
/// What the panel of a data source says about its progress through the files.
/// What the panel of a data source says about its progress through its files or mails.
/// </summary>
/// <remarks>
/// While a file is being worked on, the sentence names that file and how far into it we are.
@@ -186,26 +192,63 @@ public partial class Embeddings : MSGComponentBase
/// while it is being read, and hanging the choice on the block number let the line jump back
/// and forth between two entirely different sentences at every file. Now the beginning of the
/// sentence stays put and the blocks are appended to it as soon as the first one arrives.
///
/// A mail has no pages: those of its attachments would not say which attachment they are in.
/// </remarks>
private string GetFileProgressText(DataSourceEmbeddingStatus status)
private string GetProgressText(DataSourceEmbeddingStatus status)
{
if (status.State is not DataSourceEmbeddingState.RUNNING || string.IsNullOrWhiteSpace(status.CurrentFile))
return string.Format(T("{0} of {1} files are indexed."), this.FormatNumber(status.IndexedFiles), this.FormatNumber(status.TotalFiles));
var isMailbox = IsMailbox(status);
if (status.State is not DataSourceEmbeddingState.RUNNING || string.IsNullOrWhiteSpace(status.CurrentDocument))
{
return isMailbox
? string.Format(T("{0} of {1} mails are indexed."), this.FormatNumber(status.IndexedDocuments), this.FormatNumber(status.TotalDocuments))
: string.Format(T("{0} of {1} files are indexed."), this.FormatNumber(status.IndexedDocuments), this.FormatNumber(status.TotalDocuments));
}
//
// Everything already dealt with, plus the one in hand. Skipped and failed files are part of
// that: they are behind us in the folder, and leaving them out would let the number fall
// behind the file whose name is shown right next to it.
//
var currentFileNumber = Math.Min(status.TotalFiles, status.IndexedFiles + status.PermanentlySkippedFiles + status.FailedFiles + 1);
var currentNumber = this.FormatNumber(Math.Min(status.TotalDocuments, status.IndexedDocuments + status.PermanentlySkippedDocuments + status.FailedDocuments + 1));
var total = this.FormatNumber(status.TotalDocuments);
if (isMailbox)
{
return status.CurrentDocumentBlock is { } mailBlock
? string.Format(T("Mail {0} of {1} is being indexed: block {2}."), currentNumber, total, this.FormatNumber(mailBlock))
: string.Format(T("Mail {0} of {1} is being indexed."), currentNumber, total);
}
return status switch
{
{ CurrentFileBlock: { } block, CurrentFilePage: { } page } => string.Format(T("File {0} of {1} is being indexed: block {2}, page {3}."), this.FormatNumber(currentFileNumber), this.FormatNumber(status.TotalFiles), this.FormatNumber(block), this.FormatNumber(page)),
{ CurrentFileBlock: { } block } => string.Format(T("File {0} of {1} is being indexed: block {2}."), this.FormatNumber(currentFileNumber), this.FormatNumber(status.TotalFiles), this.FormatNumber(block)),
_ => string.Format(T("File {0} of {1} is being indexed."), this.FormatNumber(currentFileNumber), this.FormatNumber(status.TotalFiles)),
{ CurrentDocumentBlock: { } block, CurrentDocumentPage: { } page } => string.Format(T("File {0} of {1} is being indexed: block {2}, page {3}."), currentNumber, total, this.FormatNumber(block), this.FormatNumber(page)),
{ CurrentDocumentBlock: { } block } => string.Format(T("File {0} of {1} is being indexed: block {2}."), currentNumber, total, this.FormatNumber(block)),
_ => string.Format(T("File {0} of {1} is being indexed."), currentNumber, total),
};
}
private string GetSkippedText(DataSourceEmbeddingStatus status) => IsMailbox(status)
? string.Format(T("Skipped mails: {0}."), this.FormatNumber(status.PermanentlySkippedDocuments))
: string.Format(T("Skipped files: {0}. AI Studio reads them again once they change."), this.FormatNumber(status.PermanentlySkippedDocuments));
private string GetFailedText(DataSourceEmbeddingStatus status) => IsMailbox(status)
? string.Format(T("Failed mails: {0}"), this.FormatNumber(status.FailedDocuments))
: string.Format(T("Failed files: {0}"), this.FormatNumber(status.FailedDocuments));
private string GetCurrentDocumentText(DataSourceEmbeddingStatus status) => IsMailbox(status)
? string.Format(T("Current mail: {0}"), status.CurrentDocument)
: string.Format(T("Current file: {0}"), status.CurrentDocument);
private static bool IsMailbox(DataSourceEmbeddingStatus status) => status.DataSourceType is DataSourceType.MAILBOX;
/// <remarks>
/// A mailbox can be searched long before it was worked through once: its first sync takes
/// hours with a large one, newest mails first. What the index lacks until then are older mails.
/// </remarks>
private string GetLastSyncText(DataSourceEmbeddingStatus status) => status.LastSyncUtc is { } lastSyncUtc
? string.Format(T("Last complete sync: {0}"), lastSyncUtc.ToLocalTime().ToString("g", this.currentCulture))
: T("Not synced completely yet. AI Studio works through the mailbox piece by piece, the newest mails first, and older mails are still missing.");
private string FormatNumber(int value) => value.ToString("N0", this.currentCulture);
private static Color GetStatusColor(DataSourceEmbeddingStatus status) => status.State switch
@@ -213,7 +256,7 @@ public partial class Embeddings : MSGComponentBase
DataSourceEmbeddingState.RUNNING => Color.Warning,
DataSourceEmbeddingState.QUEUED => Color.Info,
DataSourceEmbeddingState.FAILED => Color.Error,
DataSourceEmbeddingState.COMPLETED when status.FailedFiles > 0 => Color.Warning,
DataSourceEmbeddingState.COMPLETED when status.FailedDocuments > 0 => Color.Warning,
DataSourceEmbeddingState.COMPLETED => Color.Success,
_ => Color.Default,
};
@@ -238,7 +281,7 @@ public partial class Embeddings : MSGComponentBase
/// </remarks>
private IReadOnlyList<FailureGroup> GetFailureGroups(DataSourceEmbeddingStatus status) => status.Failures
.GroupBy(this.GetFailureCause)
.Select(group => new FailureGroup(group.Key, GetEmbeddingProviderName(group), group.OrderBy(failure => failure.FilePath, StringComparer.OrdinalIgnoreCase).ToList()))
.Select(group => new FailureGroup(group.Key, GetEmbeddingProviderName(group), group.OrderBy(failure => IsMailbox(status) ? GetMailName(failure) : failure.DocumentKey, StringComparer.OrdinalIgnoreCase).ToList()))
.OrderBy(group => group.Cause.Priority)
.ThenByDescending(group => group.Failures.Count)
.ThenBy(group => group.Cause.Title, StringComparer.OrdinalIgnoreCase)
@@ -285,9 +328,19 @@ public partial class Embeddings : MSGComponentBase
/// <remarks>
/// A failure which was not about one file, such as a folder which is gone, carries the name of
/// the data source instead of a path. There is nothing to show for those.
/// the data source instead of a path. There is nothing to show for those, nor for a mail, which
/// lies on a server.
/// </remarks>
private static bool CanShowInFileManager(DataSourceEmbeddingFailure failure) => !string.IsNullOrWhiteSpace(failure.FilePath) && Path.IsPathRooted(failure.FilePath);
private static bool CanShowInFileManager(DataSourceEmbeddingStatus status, DataSourceEmbeddingFailure failure) => !IsMailbox(status) && !string.IsNullOrWhiteSpace(failure.DocumentKey) && Path.IsPathRooted(failure.DocumentKey);
/// <summary>
/// How a failed mail is called in the list: by its subject, which is all the user knows it by.
/// </summary>
/// <remarks>
/// The key of a mail is a hash, which means nothing to anybody. A failure which was not about
/// one mail carries the name of the mailbox instead.
/// </remarks>
private static string GetMailName(DataSourceEmbeddingFailure failure) => string.IsNullOrWhiteSpace(failure.DisplayName) ? failure.DocumentKey : failure.DisplayName;
/// <summary>
/// Opens the file browser of the system and selects the file in it.
@@ -302,7 +355,7 @@ public partial class Embeddings : MSGComponentBase
OpenPathResponse response;
try
{
response = await this.RustService.TryOpenPathInRuntimeFileManager(failure.FilePath);
response = await this.RustService.TryOpenPathInRuntimeFileManager(failure.DocumentKey);
}
catch (Exception e)
{
@@ -326,7 +379,7 @@ public partial class Embeddings : MSGComponentBase
{
return this.DataSourceEmbeddingService.CanRefreshDataSource(status.DataSourceId) &&
status is { VectorStoreUnreadable: false, State: not DataSourceEmbeddingState.RUNNING and not DataSourceEmbeddingState.QUEUED } &&
(status.State is DataSourceEmbeddingState.FAILED || status.FailedFiles > 0);
(status.State is DataSourceEmbeddingState.FAILED || status.FailedDocuments > 0);
}
/// <remarks>
@@ -356,6 +409,34 @@ public partial class Embeddings : MSGComponentBase
await this.InvokeAsync(this.StateHasChanged);
}
/// <summary>
/// Opens the settings of a mailbox, e.g. to enter the current password after a refused sign-in.
/// </summary>
/// <remarks>
/// Saving them syncs the mailbox right away. With a new password, that sync is allowed to sign
/// in. The page reloads through CONFIGURATION_CHANGED, which it receives like everybody else.
/// </remarks>
private async Task ChangeMailboxSettings(DataSourceEmbeddingStatus status)
{
if (await MailboxEditing.EditAsync(this.DialogService, this.SettingsManager, this.DataSourceEmbeddingService, status.DataSourceId))
await this.MessageBus.SendMessage<bool>(this, Event.CONFIGURATION_CHANGED);
}
/// <summary>
/// Agrees to the removal a mailbox held back, for the very number the user read.
/// </summary>
private async Task ApproveMailRemoval(DataSourceEmbeddingStatus status)
{
if (status.PendingRemovalCount is not { } removalCount)
return;
if (!await this.DataSourceEmbeddingService.ApprovePendingMailRemovalAsync(status.DataSourceId, removalCount))
await this.MessageBus.SendWarning(new(Icons.Material.Filled.PlaylistRemove, T("Your agreement could not be recorded: either the number of mails to remove changed in the meantime, or the index cannot be reached. AI Studio asks you again after the next sync.")));
this.ReloadStatuses();
await this.InvokeAsync(this.StateHasChanged);
}
private async Task RepairDataSource(DataSourceEmbeddingStatus status)
{
if (!await DataSourceRepair.ConfirmAndRepairAsync(this.DialogService, this.DataSourceEmbeddingService, status.DataSourceId, status.DataSourceName))
@@ -83,7 +83,7 @@ Each assistant plugin lives in its own directory under the assistants plugin roo
- `ASSISTANT` is the root table. Every assistant requires `Title` and `Description`.
- Form assistants additionally require `SystemPrompt`, `SubmitText`, `AllowProfiles`, and a nested `UI` definition.
- Direct chat launchers instead require a `LaunchBehavior`: either `"OPEN_WORKSPACE_CHAT_BY_NAME"` together with a `WorkspaceName`, or `"OPEN_TEMPORARY_CHAT"` for a chat that belongs to no workspace. AI Studio stops reading the form-only fields as soon as a launch behavior is active, so older launchers that still carry them keep working.
- `ToolIds` is optional for both kinds and names the tools the assistant runs with, such as `{"web_search"}`. When present, it must list at least one unique, non-empty tool ID; omit the field instead of writing an empty list. For a form assistant, naming tools takes the choice away from users: the tool selection disappears, and the assistant always runs with exactly these tools. For a launcher, the tools are merely preselected and users may change them once the chat is open. Naming a tool is a wish, not a permission: a tool switched off in the settings stays off, one whose settings are incomplete cannot run, and every tool still has to meet the confidence requirements of the provider in use. A tool ID unknown to the installation is skipped.
- `ToolIds` is optional for both kinds and names the tools the assistant runs with, such as `{"web_search"}`. When present, it must list at least one unique, non-empty tool ID; omit the field instead of writing an empty list. For a form assistant, naming tools takes the choice away from users: the tool selection disappears, and the assistant always runs with exactly these tools. For a launcher, the tools are merely preselected and users may change them once the chat is open. Naming a tool is a wish, not a permission: a tool switched off in the settings stays off, one whose settings are incomplete cannot run, and every tool still has to meet the confidence requirements of the provider in use. A tool ID unknown to the installation is skipped. Tools which only make sense together form a tool collection, such as `mailboxes` for searching, reading, and counting mails; name such tools by the ID of their collection. The ID of one of its tools selects the whole collection as well.
- `DEPLOYED_USING_CONFIG_SERVER` identifies who manages the assistant plugin. Set it to `false` for locally managed plugins. A missing field is also treated as local for compatibility with existing plugins. Enterprise-distributed plugins must set it to `true` and cannot be revised with AI in AI Studio.
- `AI_STUDIO_ASSISTANT_BUILDER = {Generated = true, SchemaVersion = 1}` is reserved for plugins generated by the AI Studio Assistant Builder. It enables Builder-specific actions such as safe deletion and must not be added to manually authored or enterprise-distributed assistants. Newly generated Builder assistants always set `DEPLOYED_USING_CONFIG_SERVER = false` explicitly.
- `UI.Type` is always `"FORM"` and `UI.Children` is a list of component tables.
@@ -303,6 +303,38 @@ CONFIG["DATA_SOURCES"] = {}
-- ["MaxMatches"] = 10,
-- }
-- Mail servers of your organization, offered for new mailboxes:
-- The mailbox dialog lists them before the well-known public providers. Choosing one fills in
-- the host, the port, and the encryption; username and password stay with each user. Large
-- organizations run more than one mail system, so a configuration may define several mail
-- servers, and those of all configurations are offered together.
--
-- Fields:
-- Id Required GUID, which stays the same across versions of your configuration.
-- Name Required name shown to the user, e.g., the name of the mail system.
-- Host Required host name or IP address of the IMAP server, without a scheme
-- or a port.
-- TransportSecurity SSL_ON_CONNECT (the default, TLS from the first byte on) or STARTTLS.
-- There is no way without encryption.
-- Port The default is 993 with SSL_ON_CONNECT and 143 with STARTTLS.
-- UsernameHint Optional text telling users what to enter as their username.
-- HelpUrl Optional http or https address of your instructions, e.g., on how to
-- enable IMAP for a mailbox.
-- An entry with an invalid field is left out as a whole, and the log names the field.
-- To allow mailboxes on these mail servers only, see DataMailboxes.AllowOnlyOrganizationMailServers.
CONFIG["MAILBOX_PROVIDERS"] = {}
-- An example mail server:
-- CONFIG["MAILBOX_PROVIDERS"][#CONFIG["MAILBOX_PROVIDERS"]+1] = {
-- ["Id"] = "00000000-0000-0000-0000-000000000000",
-- ["Name"] = "Exchange (headquarters)",
-- ["Host"] = "imap.example.org",
-- ["TransportSecurity"] = "SSL_ON_CONNECT",
-- ["Port"] = 993,
-- ["UsernameHint"] = "Your account as DOMAIN\\username",
-- ["HelpUrl"] = "https://intranet.example.org/mail/imap",
-- }
CONFIG["SETTINGS"] = {}
-- ------
@@ -395,6 +427,12 @@ CONFIG["SETTINGS"] = {}
-- CONFIG["SETTINGS"]["DataApp.AllowUserToAddEmbeddingProvider"] = false
-- CONFIG["SETTINGS"]["DataApp.AllowUserToAddTranscriptionProvider"] = false
-- Configure the permission to add mailboxes. A mailbox is a data source, not a provider, so
-- DataApp.AllowUserToAddProvider does not apply to it. When set to false, the menu entry for
-- adding a mailbox stays visible but is disabled. Mailboxes the user added before stay; to keep
-- the AI from reading them, switch off the tool collection mailboxes in DataTools.DisabledToolIds.
-- CONFIG["SETTINGS"]["DataApp.AllowUserToAddMailbox"] = false
-- Configure the user permission to import plugin archives from disk.
-- When set to false, the import button on the plugins page stays visible but is disabled.
-- CONFIG["SETTINGS"]["DataApp.AllowUserToImportPlugins"] = false
@@ -749,7 +787,19 @@ CONFIG["SETTINGS"] = {}
-- but the global tool settings remain available to administrators.
-- CONFIG["SETTINGS"]["DataTools.EnableTools"] = false
-- Disable individual tools by their stable tool ID. The default is an empty set.
-- Tools and tool collections:
-- Some tools only make sense together, e.g., searching, reading, and counting mails. They form a
-- tool collection: users select it as one, it needs one minimum provider confidence, and you
-- switch it off as one. The model still calls each of its tools by name. Wherever the settings
-- below, chat templates, or document analysis policies ask for a tool ID, use the ID of the
-- collection for such tools. A tool which belongs to no collection keeps its own ID.
-- The ID of a tool in a collection stands for its whole collection: naming it switches off the
-- whole collection or sets its confidence, and of several levels set for a collection and its
-- tools, the highest applies.
-- Collection IDs include: mailboxes (search_mails, read_mail, and count_mails; while the mailbox
-- preview is enabled)
-- Disable individual tools or tool collections by their stable ID. The default is an empty set.
-- Unknown IDs are safely ignored and can be deployed before a future tool is installed.
-- semantic_search lets the model search the data sources of a chat itself. Nobody selects it:
-- it offers itself whenever a chat has data sources to search. Disabling it makes AI Studio
@@ -757,11 +807,13 @@ CONFIG["SETTINGS"] = {}
-- tool usage.
-- CONFIG["SETTINGS"]["DataTools.DisabledToolIds"] = { "web_search" }
-- Configure the minimum provider confidence level required for individual tools.
-- Configure the minimum provider confidence level required for individual tools or tool
-- collections.
-- Tool IDs include: web_search, read_web_page, search_confluence, semantic_search
-- Collection IDs include: mailboxes
-- Allowed values are: NONE, UNTRUSTED, VERY_LOW, LOW, MODERATE, MEDIUM, HIGH
-- Defaults: web_search = VERY_LOW, read_web_page = VERY_LOW, search_confluence = HIGH,
-- semantic_search = NONE
-- semantic_search = NONE, mailboxes = VERY_LOW
-- search_confluence always searches with a HIGH-confidence provider only, whatever value is
-- set here.
-- semantic_search offers a provider only the data sources whose own confidence level it meets,
@@ -771,12 +823,14 @@ CONFIG["SETTINGS"] = {}
-- ["web_search"] = "VERY_LOW",
-- ["read_web_page"] = "VERY_LOW",
-- ["search_confluence"] = "HIGH",
-- ["semantic_search"] = "NONE"
-- ["semantic_search"] = "NONE",
-- ["mailboxes"] = "VERY_LOW"
-- }
-- Configure the settings of individual tools. Keys are "<tool ID>.<field name>", values are
-- always strings. This works for every tool, including tools added by plugins, because nothing
-- here needs to be known to AI Studio in advance.
-- here needs to be known to AI Studio in advance. Settings belong to the tool, also when it is
-- part of a tool collection, so these keys always start with the tool ID.
--
-- Two tables decide how firmly a value applies:
-- LockedToolSettings - the user cannot change it, and it is reapplied on every update.
@@ -840,9 +894,9 @@ CONFIG["SETTINGS"] = {}
-- freeAddressChoice Whether the AI may read web addresses it chose itself. Allowed values are:
-- OFF -> the AI reads only addresses which appear in the chat, such as in
-- a message, an attached document, or a data source, or which a
-- tool returned, such as a search hit. This is the default.
-- tool returned, such as a search hit. AI Studio refuses every
-- other address. This is the default.
-- ON -> the AI may also choose addresses itself.
-- Both are instructions to the AI, not a technical block of any address.
-- allowedPrivateHosts Comma-separated private or VPN host patterns. Public pages need not be
-- listed. Wildcards match subdomains only, so add the root domain
-- separately. Allowed private hosts require a provider with HIGH
@@ -983,6 +1037,30 @@ CONFIG["SETTINGS"] = {}
-- "00000000-0000-0000-0000-000000000001",
-- }
-- Configure the least strict outbound data restriction a mailbox may have. Mails come from
-- strangers and may contain instructions meant for the AI, so each mailbox decides where a chat
-- may still send data once it has read mails from it:
-- ONLY_CONFIGURED_SERVICES Only services configured in AI Studio, such as the mailbox itself
-- or your Confluence. No web pages, no web search.
-- ONLY_LINKS_FROM_CHAT Also web pages whose addresses stand in the chat, written by the
-- user or returned by a tool. No web search, and no addresses the AI
-- chooses itself.
-- UNRESTRICTED Every tool the user selected.
-- A mailbox set to a less strict level gets this one whenever its mails reach a chat, and the
-- mailbox dialog no longer offers the less strict levels. A chat which read mails before keeps
-- the level it got then, until it reads mails again. The default is UNRESTRICTED, which leaves
-- the choice to the user. New mailboxes start with ONLY_CONFIGURED_SERVICES either way.
-- CONFIG["SETTINGS"]["DataMailboxes.MinimumOutboundDataRestriction"] = "ONLY_LINKS_FROM_CHAT"
-- Configure whether mailboxes may only be on the mail servers your configurations offer in
-- CONFIG["MAILBOX_PROVIDERS"]. The default is false. When set to true, AI Studio connects to no
-- other IMAP server: before every connection, it compares the host of the mailbox with the hosts
-- of those mail servers; port and encryption make no difference. A mailbox a user added on another
-- server before stops synchronizing at once, and the AI no longer reads it. Its local index stays,
-- so the mailbox comes back as it was, should you allow its server again. Without any mail server
-- in CONFIG["MAILBOX_PROVIDERS"], this blocks every mailbox.
-- CONFIG["SETTINGS"]["DataMailboxes.AllowOnlyOrganizationMailServers"] = true
-- Configure the data source selection agent.
-- This agent is used when chat data source options enable AI-based data source selection.
-- The provider must be one of the provider IDs defined in CONFIG["LLM_PROVIDERS"].
@@ -1098,11 +1176,12 @@ CONFIG["CHAT_TEMPLATES"] = {}
-- ["SystemPrompt"] = "You are <Company Name>'s research assistant. Answer from our own documents and say where each answer comes from.",
-- ["AllowProfileUsage"] = true,
--
-- -- Optional: the tools a chat with this template starts with, by tool ID.
-- -- Optional: the tools a chat with this template starts with, by tool or collection ID.
-- -- A tool ID unknown to the installation is ignored, and so is a tool your
-- -- organization switched off. A tool has to meet the confidence requirements of the
-- -- provider in use, so it may stay unavailable even though this template names it.
-- -- Tool IDs include: web_search, read_web_page, search_confluence
-- -- Collection IDs include: mailboxes
-- -- Selecting search_confluence also selects read_web_page. semantic_search cannot be
-- -- selected here: it offers itself whenever the chat has data sources to search.
-- ["ToolIds"] = {
@@ -1213,12 +1292,13 @@ CONFIG["DOCUMENT_ANALYSIS_POLICIES"] = {}
-- -- Allowed values are: NONE, VERY_LOW, LOW, MODERATE, MEDIUM, HIGH
-- ["MinimumProviderConfidence"] = "MEDIUM",
--
-- -- Optional: the tools an analysis with this policy may use, by tool ID.
-- -- Optional: the tools an analysis with this policy may use, by tool or collection ID.
-- -- This is a limit, not a preselection: a tool which is not listed here cannot be
-- -- used for this policy. Omitting the list, or leaving it empty, means no tools.
-- -- A listed tool must still meet the confidence requirements of the provider in
-- -- use, so a tool may stay unavailable even though this policy permits it.
-- -- Tool IDs include: web_search, read_web_page, search_confluence
-- -- Collection IDs include: mailboxes
-- -- Allowing search_confluence also allows read_web_page.
-- ["AllowedToolIds"] = { "web_search" },
--
+7
View File
@@ -14,6 +14,7 @@ using AIStudio.Tools.Security;
using AIStudio.Tools.Services;
using AIStudio.Tools.ToolCallingSystem.Harness;
using AIStudio.Tools.ToolCallingSystem.ToolCallingImplementations;
using AIStudio.Tools.ToolCallingSystem.ToolCallingImplementations.Mailboxes;
using AIStudio.Tools.ToolCallingSystem.ToolCallingImplementations.SemanticSearch;
using AIStudio.Tools.ToolCallingSystem.ToolCallingImplementations.WebSearch;
using AIStudio.Tools.ToolCallingSystem.ToolCallingImplementations.WebSearch.SearXNG;
@@ -182,6 +183,10 @@ internal sealed class Program
builder.Services.AddSingleton<IWebSearchBackend, TavilySearchBackend>();
builder.Services.AddSingleton<IToolImplementation, WebSearchTool>();
builder.Services.AddSingleton<IToolImplementation, SemanticSearchTool>();
builder.Services.AddSingleton<IToolImplementation, SearchMailsTool>();
builder.Services.AddSingleton<IToolImplementation, ReadMailTool>();
builder.Services.AddSingleton<IToolImplementation, CountMailsTool>();
builder.Services.AddSingleton<IToolCollection, MailboxToolCollection>();
builder.Services.AddSingleton<IToolDefinitionSource, CodeToolDefinitionSource>();
builder.Services.AddSingleton<ToolRegistry>();
builder.Services.AddSingleton<ToolExecutor>();
@@ -205,7 +210,9 @@ internal sealed class Program
builder.Services.AddSingleton<DataSourceService>();
builder.Services.AddSingleton<DataSourceDescriptionService>();
builder.Services.AddSingleton<DataSourceEmbeddingService>();
builder.Services.AddSingleton<LocalIndexSearchService>();
builder.Services.AddSingleton<DataSourceLocalRetrievalService>();
builder.Services.AddSingleton<MailboxRetrievalService>();
builder.Services.AddSingleton<DirectChatService>();
builder.Services.AddScoped<PandocAvailabilityService>();
@@ -92,7 +92,7 @@ public sealed class ProviderOpenAI() : BaseProvider(LLMProviders.OPEN_AI, new Ur
// asks for: to the user it is the same act, whoever performs the search.
//
var minimumWebSearchConfidence = toolRegistry?.GetMinimumProviderConfidence(ToolSelectionRules.WEB_SEARCH_TOOL_ID) ?? ConfidenceLevel.NONE;
var isWebSearchAllowed = settingsManager.IsToolActive(ToolSelectionRules.WEB_SEARCH_TOOL_ID) &&
var isWebSearchAllowed = toolRegistry?.IsToolActive(ToolSelectionRules.WEB_SEARCH_TOOL_ID) is true &&
ToolSelectionRules.IsProviderConfidenceAllowed(providerConfidence, minimumWebSearchConfidence);
IList<object> providerTools = modelProfile.Has(Capability.WEB_SEARCH) && isWebSearchAllowed
? [ ProviderTools.WEB_SEARCH ]
@@ -306,7 +306,46 @@ public static class ConfigurationSelectDataFactory
yield return new(level.GetName(), level);
}
}
public static IEnumerable<ConfigurationSelectData<ConfidenceLevel>> GetMailboxConfidenceLevelsData()
{
foreach (var level in Enum.GetValues<ConfidenceLevel>())
{
if (!level.IsAllowedMailboxConfidence())
continue;
yield return new(level.GetName(), level);
}
}
public static IEnumerable<ConfigurationSelectData<MailboxTransportSecurity>> GetMailboxTransportSecurityData()
{
foreach (var transportSecurity in Enum.GetValues<MailboxTransportSecurity>())
{
if (transportSecurity is MailboxTransportSecurity.UNKNOWN)
continue;
yield return new(transportSecurity.GetName(), transportSecurity);
}
}
public static IEnumerable<ConfigurationSelectData<MailboxMaxAge>> GetMailboxMaxAgeData()
{
foreach (var maxAge in Enum.GetValues<MailboxMaxAge>())
yield return new(maxAge.GetName(), maxAge);
}
/// <summary>
/// The restrictions a mailbox may have.
/// </summary>
/// <param name="minimumRestriction">The least strict restriction the organization allows, see DataMailboxes.MinimumOutboundDataRestriction.</param>
public static IEnumerable<ConfigurationSelectData<OutboundDataRestriction>> GetOutboundDataRestrictionData(OutboundDataRestriction minimumRestriction)
{
foreach (var restriction in Enum.GetValues<OutboundDataRestriction>())
if (restriction.StricterOf(minimumRestriction) == restriction)
yield return new(restriction.GetName(), restriction);
}
public static IEnumerable<ConfigurationSelectData<Themes>> GetThemesData()
{
foreach (var theme in Enum.GetValues<Themes>())
@@ -42,7 +42,25 @@ public sealed class Data
/// A collection of data sources configured.
/// </summary>
public List<IDataSource> DataSources { get; set; } = [];
/// <summary>
/// A collection of mailboxes configured.
/// </summary>
/// <remarks>
/// A list of its own rather than a part of DataSources: classic RAG, Semantic Search and the
/// agents read DataSources, and none of them is meant to see a mailbox. It also keeps the
/// settings readable for older versions of AI Studio, so the settings version stays the same.
/// An older version skips a property it does not know, but would fail to read the whole file
/// over a data source type it does not know. It drops the list, though, the next time it stores
/// the settings. The numbers come from NextDataSourceNum, which the data sources use as well.
/// </remarks>
public List<DataSourceMailbox> Mailboxes { get; set; } = [];
/// <summary>
/// Settings which apply to all mailboxes.
/// </summary>
public DataMailboxes MailboxSettings { get; init; } = new(x => x.MailboxSettings);
/// <summary>
/// List of configured profiles.
/// </summary>
@@ -181,6 +181,15 @@ public sealed class DataApp(Expression<Func<Data, DataApp>>? configSelection = n
/// </summary>
public bool AllowUserToAddTranscriptionProvider { get; set; } = ManagedConfiguration.Register(configSelection, n => n.AllowUserToAddTranscriptionProvider, true);
/// <summary>
/// Should the user be allowed to add mailboxes?
/// </summary>
/// <remarks>
/// Independent of AllowUserToAddProvider, because a mailbox is a data source, not a provider.
/// Mailboxes the user added before stay, and the user may still edit or delete them.
/// </remarks>
public bool AllowUserToAddMailbox { get; set; } = ManagedConfiguration.Register(configSelection, n => n.AllowUserToAddMailbox, true);
/// <summary>
/// Should the user be allowed to import plugin archives from disk?
/// </summary>
@@ -0,0 +1,148 @@
using System.Diagnostics.CodeAnalysis;
using AIStudio.Tools.Mail;
using AIStudio.Tools.PluginSystem;
using Lua;
namespace AIStudio.Settings.DataModel;
/// <summary>
/// A mail server of the organization, which a configuration plugin offers for new mailboxes.
/// </summary>
/// <remarks>
/// Live plugin content rather than a stored configuration object: it is a choice in the mailbox
/// dialog, not something the user owns. Large organizations run more than one mail system, so a
/// plugin may define several, and the ones of all plugins are offered together. Username and
/// password stay with each user.
/// </remarks>
public sealed record DataMailboxProvider : ILivePluginContent
{
/// <summary>
/// The stable ID of the mail server.
/// </summary>
public string Id { get; private init; } = string.Empty;
/// <summary>
/// The ID of the enterprise configuration plugin that provides this mail server.
/// </summary>
public Guid EnterpriseConfigurationPluginId { get; private init; } = Guid.Empty;
/// <summary>
/// The name shown to the user, e.g., the name of the mail system.
/// </summary>
public string Name { get; private init; } = string.Empty;
/// <summary>
/// The host of the IMAP server, as the organization wrote it.
/// </summary>
public string Host { get; private init; } = string.Empty;
/// <summary>
/// The port of the IMAP server.
/// </summary>
public int Port { get; private init; } = MailboxTransportSecurityExtensions.SSL_ON_CONNECT_PORT;
/// <summary>
/// How the connection to the IMAP server is encrypted.
/// </summary>
public MailboxTransportSecurity TransportSecurity { get; private init; } = MailboxTransportSecurity.SSL_ON_CONNECT;
/// <summary>
/// What the user enters as the username, in the words of the organization. Empty when the organization gave no hint.
/// </summary>
public string UsernameHint { get; private init; } = string.Empty;
/// <summary>
/// The page of the organization which explains how to set up the mailbox. Empty when there is none.
/// </summary>
public string HelpUrl { get; private init; } = string.Empty;
/// <summary>
/// Reads a mail server from an entry of the table MAILBOX_PROVIDERS.
/// </summary>
/// <remarks>
/// An entry with an invalid field is dropped as a whole, also when only an optional field is
/// invalid. Offering a server with a part of its settings left out would let the user connect
/// differently than the organization meant.
/// </remarks>
/// <param name="index">The position of the entry, for the log.</param>
/// <param name="table">The entry.</param>
/// <param name="configPluginId">The ID of the configuration plugin which defines the entry.</param>
/// <param name="logger">The logger to report an invalid entry to.</param>
/// <param name="provider">The mail server, when the entry is valid.</param>
/// <returns>True when the entry is valid.</returns>
public static bool TryParseConfiguration(int index, LuaTable table, Guid configPluginId, ILogger logger, [NotNullWhen(true)] out DataMailboxProvider? provider)
{
provider = null;
if (!table.TryGetValue("Id", out var idValue) || !idValue.TryRead<string>(out var idText) || !Guid.TryParse(idText, out var id))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid ID. The ID must be a valid GUID.", index);
return false;
}
if (!table.TryGetValue("Name", out var nameValue) || !nameValue.TryRead<string>(out var name) || string.IsNullOrWhiteSpace(name))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid Name field.", index);
return false;
}
if (!table.TryGetValue("Host", out var hostValue) || !hostValue.TryRead<string>(out var host) || !MailServerHosts.TryGetIdnHost(host, out _))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid Host field. The host must be a host name or an IP address, without a scheme or a port.", index);
return false;
}
var transportSecurity = MailboxTransportSecurity.SSL_ON_CONNECT;
if (table.TryGetValue("TransportSecurity", out var transportSecurityValue) && (!transportSecurityValue.TryRead<string>(out var transportSecurityText) || !EnumNames.TryParse(transportSecurityText, out transportSecurity) || transportSecurity is MailboxTransportSecurity.UNKNOWN))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid TransportSecurity field. Allowed values are SSL_ON_CONNECT and STARTTLS.", index);
return false;
}
var port = transportSecurity.GetUsualPort() ?? MailboxTransportSecurityExtensions.SSL_ON_CONNECT_PORT;
if (table.TryGetValue("Port", out var portValue) && (!portValue.TryRead(out port) || port is < 1 or > 65535))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid Port field. The port must be a number from 1 to 65535.", index);
return false;
}
var usernameHint = string.Empty;
if (table.TryGetValue("UsernameHint", out var usernameHintValue))
{
if (!usernameHintValue.TryRead<string>(out var usernameHintText))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid UsernameHint field. The hint must be a text.", index);
return false;
}
usernameHint = usernameHintText.Trim();
}
var helpUrl = string.Empty;
if (table.TryGetValue("HelpUrl", out var helpUrlValue))
{
if (!helpUrlValue.TryRead<string>(out var helpUrlText) || !Uri.TryCreate(helpUrlText.Trim(), UriKind.Absolute, out var helpUri) || (helpUri.Scheme != Uri.UriSchemeHttps && helpUri.Scheme != Uri.UriSchemeHttp))
{
logger.LogWarning("The configured mailbox provider {ProviderIndex} does not contain a valid HelpUrl field. The URL must be an absolute http or https address.", index);
return false;
}
helpUrl = helpUri.OriginalString;
}
provider = new DataMailboxProvider
{
Id = id.ToString(),
EnterpriseConfigurationPluginId = configPluginId,
Name = name.Trim(),
Host = host.Trim(),
Port = port,
TransportSecurity = transportSecurity,
UsernameHint = usernameHint,
HelpUrl = helpUrl,
};
return true;
}
}
@@ -0,0 +1,36 @@
using System.Linq.Expressions;
namespace AIStudio.Settings.DataModel;
/// <summary>
/// Settings which apply to all mailboxes, as opposed to the settings of each one.
/// </summary>
public sealed class DataMailboxes(Expression<Func<Data, DataMailboxes>>? configSelection = null)
{
/// <summary>
/// The default constructor for the JSON deserializer.
/// </summary>
public DataMailboxes() : this(null)
{
}
/// <summary>
/// The least strict outbound data restriction a mailbox may have.
/// </summary>
/// <remarks>
/// Only an organization sets it; the default leaves every level to the user. A mailbox set to a
/// less strict level gets this one when its mails reach a chat, see
/// MailToolResults.GetRequirements, and its dialog no longer offers the less strict levels.
/// </remarks>
public OutboundDataRestriction MinimumOutboundDataRestriction { get; set; } = ManagedConfiguration.Register(configSelection, n => n.MinimumOutboundDataRestriction, OutboundDataRestriction.UNRESTRICTED);
/// <summary>
/// Whether mailboxes may only be on the mail servers which configuration plugins offer.
/// </summary>
/// <remarks>
/// Only an organization sets it. AI Studio then connects to no other server, and the mail tools
/// leave a mailbox on another server out, see MailServerPolicy. Its index stays, so the mailbox
/// comes back as it was once the organization allows its server again.
/// </remarks>
public bool AllowOnlyOrganizationMailServers { get; set; } = ManagedConfiguration.Register(configSelection, n => n.AllowOnlyOrganizationMailServers, false);
}
@@ -0,0 +1,132 @@
using System.Text.Json.Serialization;
using AIStudio.Provider;
using AIStudio.Tools.Services;
namespace AIStudio.Settings.DataModel;
/// <summary>
/// An e-mail mailbox on an IMAP server, which AI Studio embeds and indexes itself.
/// </summary>
/// <remarks>
/// Mailboxes are kept in Data.Mailboxes rather than in DataSources, which is why this is no
/// IDataSource: classic RAG, Semantic Search and the agents never see a mailbox. Only the mail
/// tools read from one.
/// </remarks>
public readonly record struct DataSourceMailbox : IIndexedDataSource, ISecretId
{
public DataSourceMailbox()
{
}
/// <inheritdoc />
public uint Num { get; init; }
/// <inheritdoc />
public string Id { get; init; } = Guid.Empty.ToString();
/// <inheritdoc />
public string Name { get; init; } = string.Empty;
/// <inheritdoc />
public DataSourceType Type { get; init; } = DataSourceType.MAILBOX;
/// <inheritdoc />
public bool IsEnterpriseConfiguration { get; init; }
/// <inheritdoc />
public Guid EnterpriseConfigurationPluginId { get; init; } = Guid.Empty;
/// <inheritdoc />
public string EmbeddingId { get; init; } = Guid.Empty.ToString();
/// <inheritdoc />
public int MaxChunkTokenLength { get; init; }
/// <inheritdoc />
public int ChunkOverlapTokenLength { get; init; } = DataSourceEmbeddingService.DEFAULT_CHUNK_OVERLAP_TOKEN_LENGTH;
/// <inheritdoc />
/// <remarks>
/// There is no default, the user has to choose one of the levels which IsAllowedMailboxConfidence
/// accepts. Until then, the level is NONE, and no provider may read the mailbox.
/// </remarks>
public ConfidenceLevel ConfidenceLevel { get; init; } = ConfidenceLevel.NONE;
/// <summary>
/// The host name of the IMAP server, e.g., imap.example.org.
/// </summary>
public string Host { get; init; } = string.Empty;
/// <summary>
/// The port of the IMAP server. The default is the one for IMAP with TLS from the first byte on.
/// </summary>
public int Port { get; init; } = MailboxTransportSecurityExtensions.SSL_ON_CONNECT_PORT;
/// <summary>
/// How the connection to the IMAP server is encrypted.
/// </summary>
public MailboxTransportSecurity TransportSecurity { get; init; } = MailboxTransportSecurity.SSL_ON_CONNECT;
/// <summary>
/// How AI Studio signs in to the IMAP server.
/// </summary>
public MailboxAuthMethod AuthMethod { get; init; } = MailboxAuthMethod.PASSWORD;
/// <summary>
/// The username to sign in with, often the e-mail address.
/// </summary>
public string Username { get; init; } = string.Empty;
/// <summary>
/// The folder to which the synchronization and all mail tools are restricted, together with its subfolders.
/// </summary>
/// <remarks>
/// The full path as the server names it, including the server's own hierarchy delimiter. Empty
/// means the whole mailbox, apart from the trash and the junk folder.
/// </remarks>
public string RootFolder { get; init; } = string.Empty;
/// <summary>
/// How far back the index reaches. Flagged mails are indexed regardless of their age.
/// </summary>
public MailboxMaxAge MaxAge { get; init; } = MailboxMaxAge.LAST_12_MONTHS;
/// <summary>
/// Whether the text of attached documents is indexed as well.
/// </summary>
public bool IndexAttachments { get; init; } = true;
/// <summary>
/// The size in megabytes up to which the text of an attachment is indexed. Of a larger one, only the name is.
/// </summary>
public int MaxAttachmentSizeMegabytes { get; init; } = 10;
/// <summary>
/// Where a chat may still send data, once it has read from this mailbox.
/// </summary>
/// <remarks>
/// An organization may demand a stricter one, see DataMailboxes.MinimumOutboundDataRestriction.
/// </remarks>
public OutboundDataRestriction OutboundDataRestriction { get; init; } = OutboundDataRestriction.ONLY_CONFIGURED_SERVICES;
/// <summary>
/// The maximum number of mails one search returns. Searched page by page, it is the size of a page.
/// </summary>
public ushort MaxMatches { get; init; } = 10;
#region Implementation of ISecretId
/// <remarks>
/// The OS keyring stores the password under this ID together with the name of the mailbox, so
/// that the user recognizes the entry there. Renaming a mailbox therefore stores the password
/// anew, and deletes the old entry.
/// </remarks>
[JsonIgnore]
string ISecretId.SecretId => this.IsEnterpriseConfiguration ? $"{ISecretId.ENTERPRISE_KEY_PREFIX}::{this.Id}" : this.Id;
[JsonIgnore]
string ISecretId.SecretName => this.Name;
#endregion
}
@@ -24,4 +24,9 @@ public enum DataSourceType
/// External data source accessed via an ERI server, cf. https://github.com/MindWorkAI/ERI.
/// </summary>
ERI_V1,
/// <summary>
/// An e-mail mailbox on an IMAP server, kept in a list of its own, cf. DataSourceMailbox.
/// </summary>
MAILBOX,
}
@@ -19,6 +19,7 @@ public static class DataSourceTypeExtension
DataSourceType.LOCAL_FILE => TB("Local File"),
DataSourceType.LOCAL_DIRECTORY => TB("Local Directory"),
DataSourceType.ERI_V1 => TB("External ERI Server (v1)"),
DataSourceType.MAILBOX => TB("Mailbox"),
_ => TB("None"),
};
@@ -13,6 +13,9 @@ public sealed class DataTools(Expression<Func<Data, DataTools>>? configSelection
/// </summary>
public Dictionary<string, Dictionary<string, string>> Settings { get; set; } = [];
/// <summary>
/// The tool collections preselected per component, see DisabledToolIds for the IDs.
/// </summary>
public Dictionary<string, HashSet<string>> DefaultToolIdsByComponent { get; set; } = [];
public HashSet<string> VisibleToolSelectionComponents { get; set; } = [];
@@ -22,11 +25,27 @@ public sealed class DataTools(Expression<Func<Data, DataTools>>? configSelection
x => x.EnableTools,
true);
/// <summary>
/// The tool collections an organization switched off.
/// </summary>
/// <remarks>
/// The IDs are those of tool collections. A tool outside of a declared collection forms one of
/// its own under its own ID, which is why the names of this and the other settings still speak
/// of tools. The ID of a tool inside a declared collection stands for its whole collection, see
/// ToolRegistry.GetCollectionId.
/// </remarks>
public HashSet<string> DisabledToolIds { get; set; } = ManagedConfiguration.Register(
configSelection,
x => x.DisabledToolIds,
[]);
/// <summary>
/// The minimum provider confidence per tool collection, see DisabledToolIds for the IDs.
/// </summary>
/// <remarks>
/// When the ID of a collection and those of its tools carry different levels, the highest one
/// applies, see SettingsManager.GetMinimumProviderConfidenceResolution.
/// </remarks>
public Dictionary<string, string> MinimumProviderConfidenceByToolId { get; set; } = ManagedConfiguration.Register(
configSelection,
x => x.MinimumProviderConfidenceByToolId,
@@ -0,0 +1,23 @@
namespace AIStudio.Settings.DataModel;
/// <summary>
/// How AI Studio signs in to an IMAP server.
/// </summary>
public enum MailboxAuthMethod
{
/// <summary>
/// A method this version of AI Studio does not know, e.g., one written by a newer version. AI Studio never signs in then.
/// </summary>
/// <remarks>
/// It is the member with the underlying value 0 on purpose: when the settings file holds a value
/// TolerantEnumConverter cannot read, it falls back to that member. Falling back to a password
/// instead would sign in with a secret meant for something else, and every such attempt counts
/// as a failed sign-in, which can lock the account.
/// </remarks>
UNKNOWN = 0,
/// <summary>
/// A username and a password, or an app password where the provider asks for one.
/// </summary>
PASSWORD,
}
@@ -0,0 +1,19 @@
namespace AIStudio.Settings.DataModel;
/// <summary>
/// How far back the index of a mailbox reaches, by the date the server received a mail. Flagged mails are indexed regardless of their age.
/// </summary>
public enum MailboxMaxAge
{
// The shortest period is deliberately the member with the underlying value 0: when the settings
// file holds a value TolerantEnumConverter cannot read, it falls back to that member. A shorter
// period than the user chose removes mails from the index, which the protection against mass
// removal asks about first. A longer one would embed mails nobody asked for, which takes hours
// for a large mailbox and costs money with a cloud embedding.
LAST_3_MONTHS = 0,
LAST_6_MONTHS,
LAST_12_MONTHS,
LAST_24_MONTHS,
ALL,
}
@@ -0,0 +1,36 @@
using AIStudio.Tools.PluginSystem;
namespace AIStudio.Settings.DataModel;
public static class MailboxMaxAgeExtensions
{
private static string TB(string fallbackEN) => I18N.I.T(fallbackEN, typeof(MailboxMaxAgeExtensions).Namespace, nameof(MailboxMaxAgeExtensions));
public static string GetName(this MailboxMaxAge maxAge) => maxAge switch
{
MailboxMaxAge.LAST_3_MONTHS => TB("The last 3 months"),
MailboxMaxAge.LAST_6_MONTHS => TB("The last 6 months"),
MailboxMaxAge.LAST_12_MONTHS => TB("The last 12 months"),
MailboxMaxAge.LAST_24_MONTHS => TB("The last 24 months"),
MailboxMaxAge.ALL => TB("All mails"),
_ => TB("Unknown period"),
};
/// <summary>
/// The first day of the period, as the sync asks the server for it.
/// </summary>
/// <param name="maxAge">The period.</param>
/// <param name="now">The current time.</param>
/// <returns>The first day, or null when the period has no end.</returns>
public static DateTimeOffset? GetReceivedSince(this MailboxMaxAge maxAge, DateTimeOffset now) => maxAge switch
{
MailboxMaxAge.LAST_6_MONTHS => now.AddMonths(-6),
MailboxMaxAge.LAST_12_MONTHS => now.AddMonths(-12),
MailboxMaxAge.LAST_24_MONTHS => now.AddMonths(-24),
MailboxMaxAge.ALL => null,
// The shortest period also stands in for a value this version does not know, as it does in the settings:
_ => now.AddMonths(-3),
};
}
@@ -0,0 +1,27 @@
namespace AIStudio.Settings.DataModel;
/// <summary>
/// How the connection to an IMAP server is encrypted. There is deliberately no way without encryption.
/// </summary>
public enum MailboxTransportSecurity
{
/// <summary>
/// A way this version of AI Studio does not know, e.g., one written by a newer version. AI Studio does not connect then.
/// </summary>
/// <remarks>
/// It is the member with the underlying value 0 on purpose: when the settings file holds a value
/// TolerantEnumConverter cannot read, it falls back to that member. Falling back to a way of
/// connecting instead would mean guessing how the password travels.
/// </remarks>
UNKNOWN = 0,
/// <summary>
/// TLS from the first byte on, usually on port 993.
/// </summary>
SSL_ON_CONNECT,
/// <summary>
/// A plain connection which switches to TLS before signing in, usually on port 143. When the server does not offer the switch, AI Studio does not connect.
/// </summary>
STARTTLS,
}
@@ -0,0 +1,38 @@
using AIStudio.Tools.PluginSystem;
namespace AIStudio.Settings.DataModel;
public static class MailboxTransportSecurityExtensions
{
/// <summary>
/// The port IMAP with TLS from the start usually listens on.
/// </summary>
public const int SSL_ON_CONNECT_PORT = 993;
/// <summary>
/// The port IMAP with STARTTLS usually listens on.
/// </summary>
public const int STARTTLS_PORT = 143;
private static string TB(string fallbackEN) => I18N.I.T(fallbackEN, typeof(MailboxTransportSecurityExtensions).Namespace, nameof(MailboxTransportSecurityExtensions));
public static string GetName(this MailboxTransportSecurity transportSecurity) => transportSecurity switch
{
MailboxTransportSecurity.SSL_ON_CONNECT => TB("TLS from the start (usually port 993)"),
MailboxTransportSecurity.STARTTLS => TB("STARTTLS (usually port 143)"),
_ => TB("Unknown encryption"),
};
/// <summary>
/// The port the encryption usually comes with.
/// </summary>
/// <returns>The port, or null when there is no usual one.</returns>
public static int? GetUsualPort(this MailboxTransportSecurity transportSecurity) => transportSecurity switch
{
MailboxTransportSecurity.SSL_ON_CONNECT => SSL_ON_CONNECT_PORT,
MailboxTransportSecurity.STARTTLS => STARTTLS_PORT,
_ => null,
};
}
@@ -0,0 +1,31 @@
namespace AIStudio.Settings.DataModel;
/// <summary>
/// Where a chat may still send data, and which data source demands it.
/// </summary>
/// <remarks>
/// The data source travels along so a tool which is kept from running can name it, and the user
/// learns which mailbox stands in the way. It is its id rather than its name, because a mailbox
/// may be renamed after a chat read from it.
/// </remarks>
/// <param name="Restriction">Where the chat may still send data.</param>
/// <param name="DataSourceId">The id of the data source which demands the restriction, or an empty text when nothing restricts the chat.</param>
public sealed record OutboundDataRequirement(OutboundDataRestriction Restriction, string DataSourceId)
{
/// <summary>
/// Demands nothing: what a chat holds before it read from any mailbox, and what a result
/// demands which brought no content of one in.
/// </summary>
public static readonly OutboundDataRequirement NONE = new(OutboundDataRestriction.UNRESTRICTED, string.Empty);
/// <summary>
/// The stricter of this requirement and another one.
/// </summary>
/// <remarks>
/// On a tie, this one is kept, so the data source which set a level first stays the one which
/// is named, and a chat does not name another mailbox with every search.
/// </remarks>
/// <param name="other">The other requirement.</param>
/// <returns>The requirement whose restriction is the stricter one.</returns>
public OutboundDataRequirement StricterOf(OutboundDataRequirement other) => other.Restriction < this.Restriction ? other : this;
}
@@ -0,0 +1,32 @@
using AIStudio.Tools.PluginSystem;
namespace AIStudio.Settings.DataModel;
public static class OutboundDataRequirementExtensions
{
private static string TB(string fallbackEN) => I18N.I.T(fallbackEN, typeof(OutboundDataRequirementExtensions).Namespace, nameof(OutboundDataRequirementExtensions));
/// <summary>
/// Why a tool cannot run in a chat with this requirement, and how to get it back.
/// </summary>
/// <remarks>
/// One text for the tool selection and for a call which was turned down, so the user reads the
/// same in both places, and the model can pass it on. It names the setting as the mailbox
/// dialog labels it, so the user finds it there. The level is the one the setting had when the
/// chat read the mails: a change made later does not loosen the chat.
/// </remarks>
/// <param name="requirement">What the chat demands.</param>
/// <param name="mailboxes">The configured mailboxes, to find the name of the one which demands it.</param>
/// <returns>The text for the user.</returns>
public static string GetToolBlockedMessage(this OutboundDataRequirement requirement, IEnumerable<DataSourceMailbox> mailboxes)
{
var mailboxName = mailboxes
.Where(mailbox => string.Equals(mailbox.Id, requirement.DataSourceId, StringComparison.Ordinal))
.Select(mailbox => mailbox.Name)
.FirstOrDefault();
return string.IsNullOrWhiteSpace(mailboxName)
? string.Format(TB("This chat read mails from a mailbox which has been removed since, while its setting 'Where a chat may send data after reading mails' was '{0}'. This tool would send data beyond that, so it is not available in this chat. A new chat can use it again."), requirement.Restriction.GetName())
: string.Format(TB("This chat read mails from the mailbox '{0}' while its setting 'Where a chat may send data after reading mails' was '{1}'. This tool would send data beyond that, so it is not available in this chat. A new chat can use it again."), mailboxName, requirement.Restriction.GetName());
}
}
@@ -0,0 +1,31 @@
namespace AIStudio.Settings.DataModel;
/// <summary>
/// Where a chat may still send data, once it has read from a mailbox.
/// </summary>
/// <remarks>
/// The members run from the strictest to the most permissive, so the stricter of two levels is the
/// smaller one. Services which are configured in AI Studio, such as Confluence or the mailbox itself,
/// stay allowed on every level.
/// </remarks>
public enum OutboundDataRestriction
{
// The strictest level is deliberately the member with the underlying value 0: when the settings
// file holds a value TolerantEnumConverter cannot read, it falls back to that member. Falling back
// to a more permissive level would open a way out for mail content the user meant to keep in.
/// <summary>
/// Only the services configured in AI Studio.
/// </summary>
ONLY_CONFIGURED_SERVICES = 0,
/// <summary>
/// The configured services, and the addresses written in the chat, either by the user or in the result of a tool. No searches with third-party services.
/// </summary>
ONLY_LINKS_FROM_CHAT,
/// <summary>
/// No restriction beyond the ones of each tool.
/// </summary>
UNRESTRICTED,
}
@@ -0,0 +1,37 @@
using AIStudio.Tools.PluginSystem;
namespace AIStudio.Settings.DataModel;
public static class OutboundDataRestrictionExtensions
{
private static string TB(string fallbackEN) => I18N.I.T(fallbackEN, typeof(OutboundDataRestrictionExtensions).Namespace, nameof(OutboundDataRestrictionExtensions));
/// <summary>
/// The stricter of two restrictions.
/// </summary>
/// <param name="restriction">The one restriction.</param>
/// <param name="other">The other restriction.</param>
/// <returns>The restriction which allows less.</returns>
public static OutboundDataRestriction StricterOf(this OutboundDataRestriction restriction, OutboundDataRestriction other) => other < restriction ? other : restriction;
public static string GetName(this OutboundDataRestriction restriction) => restriction switch
{
OutboundDataRestriction.ONLY_CONFIGURED_SERVICES => TB("Only services configured in AI Studio"),
OutboundDataRestriction.ONLY_LINKS_FROM_CHAT => TB("Configured services and addresses from the chat"),
OutboundDataRestriction.UNRESTRICTED => TB("No restriction"),
_ => TB("Unknown restriction"),
};
/// <summary>
/// What a chat may still do on this level, once it has read from the mailbox.
/// </summary>
public static string GetDescription(this OutboundDataRestriction restriction) => restriction switch
{
OutboundDataRestriction.ONLY_CONFIGURED_SERVICES => TB("The chat may only use services configured in AI Studio, such as this mailbox or your Confluence. It reads no web pages, and it does not search the web."),
OutboundDataRestriction.ONLY_LINKS_FROM_CHAT => TB("The chat may also read web pages whose addresses stand in the chat, written by you or returned by a tool, exactly as they stand there. It does not search the web, and the AI cannot choose addresses of its own."),
OutboundDataRestriction.UNRESTRICTED => TB("The chat may use every tool you chose, web search and any web page included. Content of your mails can then reach third parties, e.g., inside a search query or the address of a web page."),
_ => TB("This version of AI Studio does not know this restriction, so it applies the strictest one."),
};
}
@@ -17,4 +17,5 @@ public enum PreviewFeatures
PRE_SPEECH_TO_TEXT_2026,
PRE_META_ASSISTANT_V1,
PRE_VISUAL_BRIEFING_ASSISTANT_2026,
PRE_MAILBOXES_2026,
}
@@ -17,7 +17,8 @@ public static class PreviewFeaturesExtensions
PreviewFeatures.PRE_SPEECH_TO_TEXT_2026 => TB("Transcription: Convert recordings and audio files into text"),
PreviewFeatures.PRE_META_ASSISTANT_V1 => TB("Assistant Builder: Generate and install assistant plugins"),
PreviewFeatures.PRE_VISUAL_BRIEFING_ASSISTANT_2026 => TB("Visual Briefing Assistant: Turn source material into an interactive briefing"),
PreviewFeatures.PRE_MAILBOXES_2026 => TB("Mailboxes: Let the AI search and read your e-mails, kept in a local index on your computer"),
_ => TB("Unknown preview feature")
};
@@ -23,8 +23,9 @@ public static class PreviewVisibilityExtensions
if (visibility >= PreviewVisibility.PROTOTYPE)
{
features.Add(PreviewFeatures.PRE_MAILBOXES_2026);
}
if (visibility >= PreviewVisibility.EXPERIMENTAL)
{
features.Add(PreviewFeatures.PRE_WRITER_MODE_2024);
@@ -94,6 +94,33 @@ public static class DataSourceSecurityTrustExtensions
return providerConfidenceLevel >= requiredConfidenceLevel;
}
/// <summary>
/// Whether a mailbox may require this provider confidence level.
/// </summary>
/// <remarks>
/// Only the levels from very low to high qualify. NONE lets every provider through, cf.
/// AllowsDataSourceConfidenceLevel, and UNTRUSTED and UNKNOWN do almost the same: both lie at or
/// below the level of a provider nobody rated, so even such a provider would meet them.
/// </remarks>
public static bool IsAllowedMailboxConfidence(this ConfidenceLevel confidenceLevel) => confidenceLevel is ConfidenceLevel.VERY_LOW or ConfidenceLevel.LOW or ConfidenceLevel.MODERATE or ConfidenceLevel.MEDIUM or ConfidenceLevel.HIGH;
/// <summary>
/// Whether a provider of this confidence level may see the content of a mailbox.
/// </summary>
/// <remarks>
/// Unlike AllowsDataSourceConfidenceLevel, this closes a mailbox without an allowed level to every
/// provider, whether it was never chosen or edited by hand in the settings file. Every provider
/// which sees mail content has to pass it: the chat provider, the embedding provider, and the
/// provider of every agent reading mails.
/// </remarks>
public static bool AllowsMailboxConfidenceLevel(this ConfidenceLevel providerConfidenceLevel, ConfidenceLevel mailboxConfidenceLevel)
{
if (!mailboxConfidenceLevel.IsAllowedMailboxConfidence())
return false;
return providerConfidenceLevel >= mailboxConfidenceLevel;
}
public static ConfidenceLevel GetRequiredConfidenceLevel(this IEnumerable<IDataSource> dataSources)
{
var requiredConfidenceLevel = ConfidenceLevel.NONE;
@@ -2,25 +2,19 @@ using System.Text.Json.Serialization;
using AIStudio.Chat;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.PluginSystem;
using AIStudio.Tools.RAG;
namespace AIStudio.Settings;
/// <summary>
/// The common interface for all data sources.
/// The common interface for the data sources in DataSources, which classic RAG, Semantic Search and the agents read.
/// </summary>
[JsonPolymorphic(TypeDiscriminatorPropertyName = "$type_discriminator")]
[JsonDerivedType(typeof(DataSourceLocalDirectory), nameof(DataSourceType.LOCAL_DIRECTORY))]
[JsonDerivedType(typeof(DataSourceLocalFile), nameof(DataSourceType.LOCAL_FILE))]
[JsonDerivedType(typeof(DataSourceERI_V1), nameof(DataSourceType.ERI_V1))]
public interface IDataSource : IConfigurationObject
public interface IDataSource : IDataSourceBase
{
/// <summary>
/// Which type of data source is this?
/// </summary>
public DataSourceType Type { get; init; }
/// <summary>
/// The maximum number of matches one retrieval returns. Searched page by page, it is the size of a page.
/// </summary>
@@ -0,0 +1,21 @@
using AIStudio.Settings.DataModel;
using AIStudio.Tools.PluginSystem;
namespace AIStudio.Settings;
/// <summary>
/// What every configured data source has, whichever list of the settings it is stored in.
/// </summary>
/// <remarks>
/// Not every data source is an IDataSource. Those are the ones in DataSources, which classic RAG,
/// Semantic Search and the agents read. A data source kept in a list of its own is never handed to
/// them, and the compiler sees to that, because it implements only this interface or
/// IIndexedDataSource.
/// </remarks>
public interface IDataSourceBase : IConfigurationObject
{
/// <summary>
/// Which type of data source is this?
/// </summary>
public DataSourceType Type { get; init; }
}
@@ -0,0 +1,35 @@
using AIStudio.Provider;
namespace AIStudio.Settings;
/// <summary>
/// A data source whose content AI Studio embeds and indexes itself.
/// </summary>
/// <remarks>
/// The embedding signature, the chunking and the embedding provider are worked out from these
/// members alone. That is why the indexer serves every data source implementing this interface,
/// whether it is stored in DataSources or not.
/// </remarks>
public interface IIndexedDataSource : IDataSourceBase
{
/// <summary>
/// Which provider confidence level is required by this data source?
/// </summary>
public ConfidenceLevel ConfidenceLevel { get; init; }
/// <summary>
/// The unique identifier of the embedding provider used by this data source.
/// </summary>
public string EmbeddingId { get; init; }
/// <summary>
/// Optional maximum number of tokens per embedding chunk for this data source.
/// A value of 0 means the embedding provider's setting is used.
/// </summary>
public int MaxChunkTokenLength { get; init; }
/// <summary>
/// Optional number of tokens to overlap between consecutive chunks.
/// </summary>
public int ChunkOverlapTokenLength { get; init; }
}
@@ -1,27 +1,6 @@
using AIStudio.Provider;
namespace AIStudio.Settings;
public interface IInternalDataSource : IDataSource
{
/// <summary>
/// Which provider confidence level is required by this internal data source?
/// </summary>
public ConfidenceLevel ConfidenceLevel { get; init; }
/// <summary>
/// The unique identifier of the embedding method used by this internal data source.
/// </summary>
public string EmbeddingId { get; init; }
/// <summary>
/// Optional maximum number of tokens per embedding chunk for this data source.
/// A value of 0 means the embedding provider's setting is used.
/// </summary>
public int MaxChunkTokenLength { get; init; }
/// <summary>
/// Optional number of tokens to overlap between consecutive chunks.
/// </summary>
public int ChunkOverlapTokenLength { get; init; }
}
/// <summary>
/// A data source in DataSources whose content AI Studio embeds and indexes itself.
/// </summary>
public interface IInternalDataSource : IDataSource, IIndexedDataSource;
@@ -3,7 +3,6 @@ using System.Text.Json;
using AIStudio.Provider;
using AIStudio.Settings.DataModel;
using AIStudio.Tools.ToolCallingSystem;
using AIStudio.Tools.PluginSystem;
using AIStudio.Tools.Services;
@@ -845,22 +844,8 @@ public sealed class SettingsManager
return this.ConfigurationData.ChatTemplates.FirstOrDefault(x => x.Id.Equals(chatTemplateId, StringComparison.OrdinalIgnoreCase)) ?? ChatTemplate.NO_CHAT_TEMPLATE;
}
public HashSet<string> GetDefaultToolIds(AIStudio.Tools.Components component)
{
var key = component.ToString();
if (this.ConfigurationData.Tools.DefaultToolIdsByComponent.TryGetValue(key, out var toolIds))
return ToolSelectionRules.NormalizeSelection(toolIds);
return [];
}
public bool AreToolsEnabled() => this.ConfigurationData.Tools.EnableTools;
public bool IsToolActive(string toolId) =>
this.AreToolsEnabled() &&
!this.ConfigurationData.Tools.DisabledToolIds.Contains(toolId);
/// <remarks>
/// The document analysis is deliberately absent: there its policy names the tools, so the user
/// has nothing to select.
@@ -889,66 +874,85 @@ public sealed class SettingsManager
}
/// <summary>
/// Resolves which provider confidence a tool needs, and where that value came from.
/// Resolves which provider confidence a tool collection needs, and where that value came from.
/// </summary>
/// <remarks>
/// The default is passed in rather than looked up here. It belongs to the tool definition,
/// and the definitions live in the tool registry — which already depends on this class, so
/// asking it back would be a circle. Every caller has the definition at hand anyway.
/// Which IDs stand for the collection and its default are passed in rather than looked up here.
/// Both belong to the definitions, and the definitions live in the tool registry — which already
/// depends on this class, so asking it back would be a circle.<br/><br/>
/// When several of the IDs carry a level, the strictest one wins: an administrator who raised
/// the level of one tool of a collection must not find it lowered by the entry of another one.
/// For the same reason, an invalid managed level anywhere among them asks for HIGH.
/// </remarks>
/// <param name="toolId">The tool to resolve the confidence for.</param>
/// <param name="defaultLevel">The tool's own minimum, used when nothing overrides it.</param>
public ToolMinimumProviderConfidenceResolution GetMinimumProviderConfidenceResolutionForTool(string toolId, ConfidenceLevel defaultLevel)
/// <param name="settingsIds">The IDs which stand for the collection: its own, then those of its tools.</param>
/// <param name="defaultLevel">The collection's own minimum, used when nothing overrides it.</param>
public ToolMinimumProviderConfidenceResolution GetMinimumProviderConfidenceResolution(IReadOnlyList<string> settingsIds, ConfidenceLevel defaultLevel)
{
if (ManagedConfiguration.TryGet(x => x.Tools, x => x.MinimumProviderConfidenceByToolId, out var configMeta) && configMeta.IsLocked)
{
var managedValues = configMeta.GetValue();
if (managedValues.TryGetValue(toolId, out var configuredManagedLevel) &&
Enum.TryParse<ConfidenceLevel>(configuredManagedLevel, true, out var managedConfidenceLevel) &&
Enum.IsDefined(managedConfidenceLevel) &&
managedConfidenceLevel is not ConfidenceLevel.UNKNOWN)
ConfidenceLevel? strictestManagedLevel = null;
foreach (var settingsId in settingsIds)
{
return new(managedConfidenceLevel, "managed config");
if (!managedValues.TryGetValue(settingsId, out var configuredManagedLevel))
continue;
if (!TryParseMinimumProviderConfidence(configuredManagedLevel, out var managedConfidenceLevel))
{
this.logger.LogError(
"Managed minimum provider confidence '{ConfiguredLevel}' for tool '{ToolId}' is invalid. Requiring HIGH as a safe fallback.",
configuredManagedLevel,
settingsId);
return new(ConfidenceLevel.HIGH, "invalid managed config; safe fallback");
}
if (strictestManagedLevel is null || managedConfidenceLevel > strictestManagedLevel)
strictestManagedLevel = managedConfidenceLevel;
}
if (managedValues.ContainsKey(toolId))
{
this.logger.LogError(
"Managed minimum provider confidence '{ConfiguredLevel}' for tool '{ToolId}' is invalid. Requiring HIGH as a safe fallback.",
configuredManagedLevel,
toolId);
return new(ConfidenceLevel.HIGH, "invalid managed config; safe fallback");
}
if (strictestManagedLevel is { } managedLevel)
return new(managedLevel, "managed config");
}
if (this.ConfigurationData.Tools.MinimumProviderConfidenceByToolId.TryGetValue(toolId, out var configuredLevel) &&
Enum.TryParse<ConfidenceLevel>(configuredLevel, true, out var confidenceLevel) &&
Enum.IsDefined(confidenceLevel) &&
confidenceLevel is not ConfidenceLevel.UNKNOWN)
ConfidenceLevel? strictestStoredLevel = null;
foreach (var settingsId in settingsIds)
{
return new(confidenceLevel, "stored override");
if (this.ConfigurationData.Tools.MinimumProviderConfidenceByToolId.TryGetValue(settingsId, out var configuredLevel) &&
TryParseMinimumProviderConfidence(configuredLevel, out var confidenceLevel) &&
(strictestStoredLevel is null || confidenceLevel > strictestStoredLevel))
{
strictestStoredLevel = confidenceLevel;
}
}
if (strictestStoredLevel is { } storedLevel)
return new(storedLevel, "stored override");
return new(defaultLevel, "default fallback");
}
public ConfidenceLevel GetMinimumProviderConfidenceForTool(string toolId, ConfidenceLevel defaultLevel) => this.GetMinimumProviderConfidenceResolutionForTool(toolId, defaultLevel).ConfidenceLevel;
private static bool TryParseMinimumProviderConfidence(string? configuredLevel, out ConfidenceLevel confidenceLevel) =>
Enum.TryParse(configuredLevel, true, out confidenceLevel) &&
Enum.IsDefined(confidenceLevel) &&
confidenceLevel is not ConfidenceLevel.UNKNOWN;
/// <summary>
/// Stores which provider confidence a tool needs.
/// Stores which provider confidence a tool collection needs.
/// </summary>
/// <param name="toolId">The tool to store the confidence for.</param>
/// <remarks>
/// Stored under the ID of the collection. Entries under the IDs of its tools are removed, since
/// the strictest entry wins and an older one would otherwise outvote the level just chosen.
/// </remarks>
/// <param name="settingsIds">The IDs which stand for the collection: its own, then those of its tools.</param>
/// <param name="confidenceLevel">The level the user chose.</param>
/// <param name="defaultLevel">The tool's own minimum. Choosing it again removes the override.</param>
public void SetMinimumProviderConfidenceForTool(string toolId, ConfidenceLevel confidenceLevel, ConfidenceLevel defaultLevel)
/// <param name="defaultLevel">The collection's own minimum. Choosing it again removes the override.</param>
public void SetMinimumProviderConfidence(IReadOnlyList<string> settingsIds, ConfidenceLevel confidenceLevel, ConfidenceLevel defaultLevel)
{
if (confidenceLevel == defaultLevel)
{
this.ConfigurationData.Tools.MinimumProviderConfidenceByToolId.Remove(toolId);
return;
}
foreach (var settingsId in settingsIds)
this.ConfigurationData.Tools.MinimumProviderConfidenceByToolId.Remove(settingsId);
this.ConfigurationData.Tools.MinimumProviderConfidenceByToolId[toolId] = confidenceLevel.ToString();
if (confidenceLevel != defaultLevel)
this.ConfigurationData.Tools.MinimumProviderConfidenceByToolId[settingsIds[0]] = confidenceLevel.ToString();
}
public ConfidenceLevel GetConfiguredConfidenceLevel(LLMProviders llmProvider)
@@ -48,7 +48,7 @@ public static class DataSourceReindexWarning
var candidates = GetDataSourcesUsing(settingsManager, before.Id)
.Where(dataSource => EmbeddingChangeImpact.AffectsStoredIndex(dataSource, before, after))
.Cast<IDataSource>()
.Cast<IDataSourceBase>()
.ToList();
if (candidates.Count == 0)
@@ -69,7 +69,7 @@ public static class DataSourceReindexWarning
/// <param name="token">The cancellation token.</param>
/// <returns>True when the edit may be saved.</returns>
public static async Task<bool> ConfirmDataSourceChangeAsync(IDialogService dialogService, SettingsManager settingsManager, DataSourceEmbeddingService embeddingService,
IInternalDataSource before, IInternalDataSource after, CancellationToken token = default)
IIndexedDataSource before, IIndexedDataSource after, CancellationToken token = default)
{
// Without a provider nothing is embedded at all, so nothing can be lost:
if (!DataSourceEmbeddingProviders.TryResolve(settingsManager, after, out var afterProvider))
@@ -112,7 +112,7 @@ public static class DataSourceReindexWarning
if (embeddingProvider == EmbeddingProvider.NONE)
return string.Empty;
var affected = GetDataSourcesUsing(settingsManager, embeddingProvider.Id).Cast<IDataSource>().ToList();
var affected = GetDataSourcesUsing(settingsManager, embeddingProvider.Id).Cast<IDataSourceBase>().ToList();
if (affected.Count == 0)
return string.Empty;
@@ -129,14 +129,15 @@ public static class DataSourceReindexWarning
}
/// <summary>
/// The data sources which are indexed with a given embedding provider.
/// The data sources and mailboxes which are indexed with a given embedding provider.
/// </summary>
/// <param name="settingsManager">The settings holding the data sources.</param>
/// <param name="settingsManager">The settings holding the data sources and the mailboxes.</param>
/// <param name="embeddingProviderId">The id of the embedding provider.</param>
/// <returns>The data sources pointing at that embedding provider.</returns>
private static IReadOnlyList<IInternalDataSource> GetDataSourcesUsing(SettingsManager settingsManager, string embeddingProviderId) =>
/// <returns>The data sources and mailboxes pointing at that embedding provider.</returns>
private static IReadOnlyList<IIndexedDataSource> GetDataSourcesUsing(SettingsManager settingsManager, string embeddingProviderId) =>
settingsManager.ConfigurationData.DataSources
.OfType<IInternalDataSource>()
.Concat(settingsManager.ConfigurationData.Mailboxes.Cast<IIndexedDataSource>())
.Where(dataSource => embeddingProviderId.Equals(dataSource.EmbeddingId, StringComparison.OrdinalIgnoreCase))
.ToList();
@@ -145,7 +146,7 @@ public static class DataSourceReindexWarning
/// </summary>
/// <param name="dataSources">The data sources to name.</param>
/// <returns>The Markdown list.</returns>
private static string FormatDataSourceNames(IReadOnlyList<IDataSource> dataSources)
private static string FormatDataSourceNames(IReadOnlyList<IDataSourceBase> dataSources)
{
var names = dataSources
.Select(dataSource => dataSource.Name)
@@ -159,7 +160,7 @@ public static class DataSourceReindexWarning
return string.Join(Environment.NewLine, lines);
}
private static async Task<bool> ConfirmAsync(IDialogService dialogService, IReadOnlyList<IDataSource> affected, bool usesCloudEmbedding)
private static async Task<bool> ConfirmAsync(IDialogService dialogService, IReadOnlyList<IDataSourceBase> affected, bool usesCloudEmbedding)
{
if (affected.Count == 0)
return true;
@@ -0,0 +1,73 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <remarks>
/// The mail tools search, count and read mails through these. Each one asks about one mailbox,
/// so a mail of another mailbox never turns up, whatever id a caller hands in.
/// </remarks>
public abstract partial class IndexStoreClient
{
/// <summary>
/// Lists the mails of a mailbox which meet the conditions, the most recently received first.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="filter">The conditions.</param>
/// <param name="offset">How many mails to skip, for the pages after the first one.</param>
/// <param name="limit">How many mails to list at most.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The ids of the mails. The order stays the same between calls, so pages neither repeat nor skip a mail.</returns>
public abstract Task<IReadOnlyList<string>> QueryMailsAsync(string dataSourceId, MailFilter filter, int offset, int limit, CancellationToken token);
/// <summary>
/// Reads the ids of every chunk of the mails which meet the conditions, to restrict a vector search to them.
/// </summary>
/// <remarks>
/// The list can be long: it holds every chunk of every matching mail, and a broad condition
/// matches most of a mailbox. A caller without any condition does better to search the whole
/// store and drop what GetMailSummariesAsync does not know.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="filter">The conditions.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The chunk ids, which are the ids of their points in the vector store.</returns>
public abstract Task<IReadOnlyList<string>> GetMailChunkIdsAsync(string dataSourceId, MailFilter filter, CancellationToken token);
/// <summary>
/// Searches the chunks of the mails which meet the conditions by their words, with BM25.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="query">The words to search for.</param>
/// <param name="filter">The conditions.</param>
/// <param name="maxMatches">How many chunks to return at most.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The chunks, best first; the id of their document is the id of their mail.</returns>
public abstract Task<IReadOnlyList<IndexStoreSearchResult>> SearchMailChunksAsync(string dataSourceId, string query, MailFilter filter, int maxMatches, CancellationToken token);
/// <summary>
/// Counts the mails of a mailbox which meet the conditions.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="filter">The conditions.</param>
/// <param name="grouping">How to break the number down.</param>
/// <param name="maxGroups">How many groups to return at most, the largest ones.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The total and the largest groups.</returns>
public abstract Task<MailCountResult> CountMailsAsync(string dataSourceId, MailFilter filter, MailCountGrouping grouping, int maxGroups, CancellationToken token);
/// <summary>
/// Reads what a list of mails shows about each of the given ones.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="mailIds">The ids of the mails, in the order the list shows them.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The summaries in the given order. A mail this mailbox does not hold, or which lies nowhere any more, is left out.</returns>
public abstract Task<IReadOnlyList<MailSummary>> GetMailSummariesAsync(string dataSourceId, IReadOnlyList<string> mailIds, CancellationToken token);
/// <summary>
/// Finds a mail by its Message-ID, e.g., the one another mail replies to.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="messageId">The Message-ID, without angle brackets.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The id of the mail, the one received first when there are copies; null when the mailbox holds no such mail.</returns>
public abstract Task<string?> FindMailByMessageIdAsync(string dataSourceId, string messageId, CancellationToken token);
}
@@ -0,0 +1,207 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <remarks>
/// Every mail is a document like any other: its row in the files, its chunks, its vectors. What
/// follows is what only a mail has, and it rests on that document. So a mail is stored after its
/// last chunk, and deleting its document takes all of this along.
///
/// A mail is addressed by its id, which is the id of its document. Folders are addressed by their
/// path, and have to be stored before the mails which lie in them.
/// </remarks>
public abstract partial class IndexStoreClient
{
/// <summary>
/// Reads the folders of a mailbox and how far their sync got.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The folders, ordered by their path.</returns>
public abstract Task<IReadOnlyList<MailFolderRecord>> GetMailFoldersAsync(string dataSourceId, CancellationToken token);
/// <summary>
/// Stores a folder of a mailbox, or how far its sync got.
/// </summary>
/// <remarks>
/// A UIDVALIDITY other than the stored one voids every UID stored for the folder, so its
/// locations are dropped. Mails left without any location are orphaned, not deleted: the sync
/// finds them again under their new UIDs and links them anew, without embedding them again.
/// That is why a new UIDVALIDITY has to be stored before the first location under it, not
/// at the end of the pass: by then, it would drop the very locations the pass just linked.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="folder">The folder, identified by its path.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task UpsertMailFolderAsync(string dataSourceId, MailFolderRecord folder, CancellationToken token);
/// <summary>
/// Removes a folder which is gone from the server, or no longer part of the mailbox.
/// </summary>
/// <remarks>
/// Its mails are not deleted along with it, only orphaned when they lie nowhere else: they may
/// just have moved to a folder the sync gets to later.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="folderPath">The path of the folder.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task DeleteMailFolderAsync(string dataSourceId, string folderPath, CancellationToken token);
/// <summary>
/// Stores what the index keeps about a mail beyond its chunks, replacing what it kept before.
/// </summary>
/// <remarks>
/// Only ever after the last chunk of the mail: indexing a document deletes it first, and that
/// takes everything stored here along. When the mail was seen before, the earlier of both
/// first sightings stays. A UID names one mail only, so should the index still hold one of the
/// given locations for another mail, that location moves over, and the other mail is orphaned
/// when it lies nowhere else.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="mail">The mail, with at least one location in a stored folder.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task UpsertMailAsync(string dataSourceId, MailRecord mail, CancellationToken token);
/// <summary>
/// Reads everything the index keeps about a mail beyond its chunks.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="mailId">The id of the mail.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The mail, or null when the mailbox holds no such mail.</returns>
public abstract Task<MailRecord?> GetMailAsync(string dataSourceId, string mailId, CancellationToken token);
/// <summary>
/// Links a mail the index already holds to one more place on the server, without embedding it again.
/// </summary>
/// <remarks>
/// This is how a moved mail and a mail under a new UIDVALIDITY are taken care of. The mail stops
/// being an orphan, and a location held for another mail moves over as with UpsertMailAsync.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="mailId">The id of the mail.</param>
/// <param name="location">The place, in a stored folder.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>True when the mail was linked, false when the mailbox holds no such mail and it has to be indexed.</returns>
public abstract Task<bool> AddMailLocationAsync(string dataSourceId, string mailId, MailLocationRecord location, CancellationToken token);
/// <summary>
/// Forgets places on the server where mails no longer lie.
/// </summary>
/// <remarks>
/// Mails left without any location are orphaned, not deleted, cf. GetOrphanedMailsAsync.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="folderPath">The path of the folder.</param>
/// <param name="uids">The UIDs which are gone from that folder.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task RemoveMailLocationsAsync(string dataSourceId, string folderPath, IReadOnlyCollection<long> uids, CancellationToken token);
/// <summary>
/// Reads which UIDs of a folder the index holds, and their flags.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="folderPath">The path of the folder.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The flags by UID, empty when the folder is not stored.</returns>
public abstract Task<IReadOnlyDictionary<long, MailFlags>> GetMailLocationsAsync(string dataSourceId, string folderPath, CancellationToken token);
/// <summary>
/// Stores the flags of mails in one folder. Flags of UIDs the index does not hold are ignored.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="folderPath">The path of the folder.</param>
/// <param name="flagsByUid">The flags by UID.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task UpdateMailFlagsAsync(string dataSourceId, string folderPath, IReadOnlyDictionary<long, MailFlags> flagsByUid, CancellationToken token);
/// <summary>
/// Reads the mails which lost their last location before the given point in time.
/// </summary>
/// <remarks>
/// A moved mail disappears from one folder before it shows up in the other one, maybe only in
/// the next run. Asking with the start of the current run therefore leaves out every mail
/// which became an orphan during that run, and those get one more run to be found again.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="orphanedBefore">Only mails orphaned before this point in time.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The keys of their documents, to delete them like any other document.</returns>
public abstract Task<IReadOnlyList<string>> GetOrphanedMailsAsync(string dataSourceId, DateTimeOffset orphanedBefore, CancellationToken token);
/// <summary>
/// Reads how the sync of a mailbox as a whole stands.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The state; every value is null for a mailbox which was never synced.</returns>
public abstract Task<MailboxSyncState> GetMailboxSyncStateAsync(string dataSourceId, CancellationToken token);
/// <summary>
/// Holds back a sync which would remove many mails from the index at once, until the user decides.
/// </summary>
/// <remarks>
/// A count other than the one held back so far voids an earlier approval: the user agreed to
/// a number, not to whatever a later sync comes up with.
/// </remarks>
/// <param name="dataSourceId">The mailbox, which has to be stored as a data source.</param>
/// <param name="removalCount">How many mails the sync would remove from the index.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task HoldBackMailRemovalAsync(string dataSourceId, int removalCount, CancellationToken token);
/// <summary>
/// Records that the user agreed to the held back removal.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="removalCount">The count the user was shown and agreed to.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>True when exactly that removal was held back and is now approved. False when nothing or another count is held back by now, and the user has to be asked again.</returns>
public abstract Task<bool> ApprovePendingMailRemovalAsync(string dataSourceId, int removalCount, CancellationToken token);
/// <summary>
/// Records a complete sync of a mailbox. A removal held back before has been dealt with by then.
/// </summary>
/// <param name="dataSourceId">The mailbox, which has to be stored as a data source.</param>
/// <param name="completedUtc">When the sync ended.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task CompleteMailboxSyncAsync(string dataSourceId, DateTimeOffset completedUtc, CancellationToken token);
/// <summary>
/// Reads whether the server of a mailbox refused a sign-in which nobody has dealt with yet.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="token">The cancellation token.</param>
/// <returns>The refused sign-in, or null when AI Studio may sign in.</returns>
public abstract Task<MailboxAuthFailure?> GetMailboxAuthFailureAsync(string dataSourceId, CancellationToken token);
/// <summary>
/// Records a sign-in the server refused. From then on, AI Studio does not sign in on its own,
/// not even after a restart, until the failure is cleared.
/// </summary>
/// <remarks>
/// Kept apart from everything else about the mailbox, so it survives a rebuild of the index,
/// and stored for a mailbox which was never indexed as well: the very first sign-in can fail.
/// </remarks>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="failure">The refused sign-in.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task UpsertMailboxAuthFailureAsync(string dataSourceId, MailboxAuthFailure failure, CancellationToken token);
/// <summary>
/// Clears a refused sign-in, when the user saved a new password, asked for another try, or
/// deleted the mailbox.
/// </summary>
/// <param name="dataSourceId">The mailbox.</param>
/// <param name="token">The cancellation token.</param>
public abstract Task ClearMailboxAuthFailureAsync(string dataSourceId, CancellationToken token);
/// <summary>
/// Lists every mailbox the index keeps something of: an index, a refused sign-in, or both.
/// </summary>
/// <remarks>
/// The refused sign-ins count on their own, since they outlive the index on purpose. What is
/// listed here for a mailbox which is no longer configured is left over, cf.
/// DataSourceEmbeddingService.DeleteOrphanedMailboxIndexesAsync.
/// </remarks>
/// <param name="token">The cancellation token.</param>
/// <returns>The ids of the mailboxes, each once.</returns>
public abstract Task<IReadOnlyCollection<string>> GetStoredMailboxIdsAsync(CancellationToken token);
}
@@ -2,7 +2,7 @@ using AIStudio.Tools.Services;
namespace AIStudio.Tools.Databases.IndexStore;
public abstract class IndexStoreClient(string name, string path) : DatabaseClient(name, path)
public abstract partial class IndexStoreClient(string name, string path) : DatabaseClient(name, path)
{
public abstract Task<DataSourceEmbeddingManifest> GetManifestAsync(string dataSourceId, CancellationToken token);
@@ -19,6 +19,20 @@ internal sealed class IndexStoreDbContext(DbContextOptions<IndexStoreDbContext>
public DbSet<IndexStoreSearchResultEntity> SearchResults => this.Set<IndexStoreSearchResultEntity>();
public DbSet<MailMessageEntity> MailMessages => this.Set<MailMessageEntity>();
public DbSet<MailAddressEntity> MailAddresses => this.Set<MailAddressEntity>();
public DbSet<MailPartEntity> MailParts => this.Set<MailPartEntity>();
public DbSet<MailFolderEntity> MailFolders => this.Set<MailFolderEntity>();
public DbSet<MailLocationEntity> MailLocations => this.Set<MailLocationEntity>();
public DbSet<MailboxSyncStateEntity> MailboxSyncStates => this.Set<MailboxSyncStateEntity>();
public DbSet<MailboxAuthStateEntity> MailboxAuthStates => this.Set<MailboxAuthStateEntity>();
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
var utcDateTimeOffsetConverter = new IndexStoreDateTimeOffsetConverter();
@@ -124,6 +138,171 @@ internal sealed class IndexStoreDbContext(DbContextOptions<IndexStoreDbContext>
entity.Property(result => result.LastWriteUtc).HasConversion(utcDateTimeOffsetConverter);
entity.Property(result => result.EmbeddedAtUtc).HasConversion(utcDateTimeOffsetConverter);
});
modelBuilder.Entity<MailMessageEntity>(entity =>
{
entity.ToTable("mail_messages");
entity.HasKey(mail => mail.ParentFileId);
entity.Property(mail => mail.ParentFileId).HasColumnName("parent_file_id");
entity.Property(mail => mail.DataSourceId).HasColumnName("data_source_id").IsRequired();
entity.Property(mail => mail.MessageId).HasColumnName("message_id").IsRequired();
entity.Property(mail => mail.InReplyTo).HasColumnName("in_reply_to").IsRequired();
entity.Property(mail => mail.ReferenceMessageIds).HasColumnName("reference_message_ids").IsRequired();
entity.Property(mail => mail.SentAtUtc).HasColumnName("sent_at_utc").HasConversion(utcDateTimeOffsetConverter);
entity.Property(mail => mail.ReceivedAtUtc).HasColumnName("received_at_utc").HasConversion(utcDateTimeOffsetConverter).IsRequired();
entity.Property(mail => mail.Importance).HasColumnName("importance").IsRequired();
entity.Property(mail => mail.EncryptionKind).HasColumnName("encryption_kind").IsRequired();
entity.Property(mail => mail.MailHash).HasColumnName("mail_hash").IsRequired();
entity.Property(mail => mail.FirstSeenUtc).HasColumnName("first_seen_utc").HasConversion(utcDateTimeOffsetConverter).IsRequired();
entity.Property(mail => mail.OrphanedAtUtc).HasColumnName("orphaned_at_utc").HasConversion(utcDateTimeOffsetConverter);
entity.HasIndex(mail => new { mail.DataSourceId, mail.ReceivedAtUtc }).HasDatabaseName("idx_mail_messages_data_source_received");
entity.HasIndex(mail => new { mail.DataSourceId, mail.MessageId }).HasDatabaseName("idx_mail_messages_data_source_message_id");
entity
.HasOne<EmbeddingStateFileEntity>()
.WithOne()
.HasForeignKey<MailMessageEntity>(mail => mail.ParentFileId)
.OnDelete(DeleteBehavior.Cascade);
entity
.HasOne<EmbeddingStateDataSourceEntity>()
.WithMany()
.HasForeignKey(mail => mail.DataSourceId)
.OnDelete(DeleteBehavior.Cascade);
entity
.HasMany(mail => mail.Addresses)
.WithOne(address => address.Message)
.HasForeignKey(address => address.ParentFileId)
.OnDelete(DeleteBehavior.Cascade);
entity
.HasMany(mail => mail.Parts)
.WithOne(part => part.Message)
.HasForeignKey(part => part.ParentFileId)
.OnDelete(DeleteBehavior.Cascade);
entity
.HasMany(mail => mail.Locations)
.WithOne(location => location.Message)
.HasForeignKey(location => location.ParentFileId)
.OnDelete(DeleteBehavior.Cascade);
});
modelBuilder.Entity<MailAddressEntity>(entity =>
{
entity.ToTable("mail_addresses");
entity.HasKey(address => address.Id);
entity.Property(address => address.Id).HasColumnName("id").ValueGeneratedOnAdd();
entity.Property(address => address.ParentFileId).HasColumnName("parent_file_id").IsRequired();
entity.Property(address => address.Role).HasColumnName("role").IsRequired();
entity.Property(address => address.Position).HasColumnName("position");
entity.Property(address => address.Address).HasColumnName("address").UseCollation("NOCASE").IsRequired();
entity.Property(address => address.DisplayName).HasColumnName("display_name").IsRequired();
entity.HasIndex(address => new { address.ParentFileId, address.Role, address.Position }).HasDatabaseName("idx_mail_addresses_parent_file_role_position").IsUnique();
entity.HasIndex(address => address.Address).HasDatabaseName("idx_mail_addresses_address");
});
modelBuilder.Entity<MailPartEntity>(entity =>
{
entity.ToTable("mail_parts");
entity.HasKey(part => part.Id);
entity.Property(part => part.Id).HasColumnName("id").ValueGeneratedOnAdd();
entity.Property(part => part.ParentFileId).HasColumnName("parent_file_id").IsRequired();
entity.Property(part => part.Kind).HasColumnName("kind").IsRequired();
entity.Property(part => part.Position).HasColumnName("position");
entity.Property(part => part.Name).HasColumnName("name").IsRequired();
entity.Property(part => part.ContentType).HasColumnName("content_type").IsRequired();
entity.Property(part => part.PartSize).HasColumnName("part_size");
entity.Property(part => part.Text).HasColumnName("text");
entity.Property(part => part.TextState).HasColumnName("text_state").IsRequired();
entity.HasIndex(part => new { part.ParentFileId, part.Kind, part.Position }).HasDatabaseName("idx_mail_parts_parent_file_kind_position").IsUnique();
});
modelBuilder.Entity<MailFolderEntity>(entity =>
{
entity.ToTable("mail_folders");
entity.HasKey(folder => folder.Id);
entity.Property(folder => folder.Id).HasColumnName("id").ValueGeneratedOnAdd();
entity.Property(folder => folder.DataSourceId).HasColumnName("data_source_id").IsRequired();
entity.Property(folder => folder.Path).HasColumnName("path").IsRequired();
entity.Property(folder => folder.SpecialUse).HasColumnName("special_use").IsRequired();
entity.Property(folder => folder.UidValidity).HasColumnName("uid_validity");
entity.Property(folder => folder.UidNext).HasColumnName("uid_next");
entity.Property(folder => folder.HighestModSeq).HasColumnName("highest_mod_seq");
entity.Property(folder => folder.ServerMessageCount).HasColumnName("server_message_count");
entity.Property(folder => folder.ServerUnseenCount).HasColumnName("server_unseen_count");
entity.Property(folder => folder.InitialSyncCompletedUtc).HasColumnName("initial_sync_completed_utc").HasConversion(utcDateTimeOffsetConverter);
entity.HasIndex(folder => new { folder.DataSourceId, folder.Path }).HasDatabaseName("idx_mail_folders_data_source_path").IsUnique();
entity
.HasOne<EmbeddingStateDataSourceEntity>()
.WithMany()
.HasForeignKey(folder => folder.DataSourceId)
.OnDelete(DeleteBehavior.Cascade);
entity
.HasMany(folder => folder.Locations)
.WithOne(location => location.Folder)
.HasForeignKey(location => location.FolderId)
.OnDelete(DeleteBehavior.Cascade);
});
modelBuilder.Entity<MailLocationEntity>(entity =>
{
entity.ToTable("mail_locations");
entity.HasKey(location => location.Id);
entity.Property(location => location.Id).HasColumnName("id").ValueGeneratedOnAdd();
entity.Property(location => location.ParentFileId).HasColumnName("parent_file_id").IsRequired();
entity.Property(location => location.FolderId).HasColumnName("folder_id");
entity.Property(location => location.Uid).HasColumnName("uid");
entity.Property(location => location.IsSeen).HasColumnName("is_seen");
entity.Property(location => location.IsFlagged).HasColumnName("is_flagged");
entity.Property(location => location.IsAnswered).HasColumnName("is_answered");
entity.HasIndex(location => new { location.FolderId, location.Uid }).HasDatabaseName("idx_mail_locations_folder_uid").IsUnique();
entity.HasIndex(location => location.ParentFileId).HasDatabaseName("idx_mail_locations_parent_file");
});
modelBuilder.Entity<MailboxSyncStateEntity>(entity =>
{
entity.ToTable("mailbox_sync_state");
entity.HasKey(state => state.DataSourceId);
entity.Property(state => state.DataSourceId).HasColumnName("data_source_id");
entity.Property(state => state.LastSyncCompletedUtc).HasColumnName("last_sync_completed_utc").HasConversion(utcDateTimeOffsetConverter);
entity.Property(state => state.PendingRemovalCount).HasColumnName("pending_removal_count");
entity.Property(state => state.PendingRemovalApprovedUtc).HasColumnName("pending_removal_approved_utc").HasConversion(utcDateTimeOffsetConverter);
entity
.HasOne<EmbeddingStateDataSourceEntity>()
.WithOne()
.HasForeignKey<MailboxSyncStateEntity>(state => state.DataSourceId)
.OnDelete(DeleteBehavior.Cascade);
});
//
// Without any relationship on purpose, cf. MailboxAuthStateEntity: a refused sign-in has
// to outlive a rebuild of the index.
//
modelBuilder.Entity<MailboxAuthStateEntity>(entity =>
{
entity.ToTable("mailbox_auth_state");
entity.HasKey(state => state.DataSourceId);
entity.Property(state => state.DataSourceId).HasColumnName("data_source_id");
entity.Property(state => state.FailedAtUtc).HasColumnName("failed_at_utc").HasConversion(utcDateTimeOffsetConverter).IsRequired();
entity.Property(state => state.FailureMessage).HasColumnName("failure_message").IsRequired();
});
}
private static string BuildConnectionString(string databasePath) => new SqliteConnectionStringBuilder
@@ -10,6 +10,7 @@ internal static class IndexStoreSchemaMigrator
[DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(Migrations.PermanentIndexingFailures))]
[DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(Migrations.DropFileConfidenceLevel))]
[DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(Migrations.DropDataSourceName))]
[DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(Migrations.MailboxIndex))]
public static async Task MigrateAsync(IndexStoreDbContext context, CancellationToken token)
{
await context.Database.MigrateAsync(token);
@@ -0,0 +1,30 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One address from the header of a mail.
/// </summary>
internal sealed class MailAddressEntity
{
public int Id { get; set; }
public string ParentFileId { get; set; } = string.Empty;
/// <summary>
/// The header the address comes from: FROM, SENDER, REPLY_TO, TO, CC or BCC, stored by name.
/// </summary>
public string Role { get; set; } = string.Empty;
/// <summary>
/// The place of the address within its header, starting at zero.
/// </summary>
public int Position { get; set; }
public string Address { get; set; } = string.Empty;
/// <summary>
/// The name shown next to the address, empty when the header gives none.
/// </summary>
public string DisplayName { get; set; } = string.Empty;
public MailMessageEntity? Message { get; set; }
}
@@ -0,0 +1,11 @@
using AIStudio.Tools.Mail;
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One address from the header of a mail.
/// </summary>
/// <param name="Role">The header the address comes from.</param>
/// <param name="Address">The address itself.</param>
/// <param name="DisplayName">The name shown next to it, empty when the header gives none.</param>
public sealed record MailAddressRecord(MailAddressRole Role, string Address, string DisplayName);
@@ -0,0 +1,9 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// The number of mails in one group, e.g., one folder or one sender.
/// </summary>
/// <param name="Key">The folder path or the sender address.</param>
/// <param name="DisplayName">A name the sender uses, empty for folders and for senders without one.</param>
/// <param name="Count">How many mails belong to the group.</param>
public sealed record MailCountGroup(string Key, string DisplayName, long Count);
@@ -0,0 +1,22 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// How to break down the number of mails in a mailbox.
/// </summary>
public enum MailCountGrouping
{
/// <summary>
/// Only the total.
/// </summary>
NONE,
/// <summary>
/// By the folders the mails lie in. A mail in two folders counts in both.
/// </summary>
FOLDER,
/// <summary>
/// By the address in the From header.
/// </summary>
SENDER,
}
@@ -0,0 +1,8 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// How many mails of a mailbox meet some conditions.
/// </summary>
/// <param name="TotalCount">How many mails meet them, each mail counted once.</param>
/// <param name="Groups">The largest groups, largest first; empty without a grouping.</param>
public sealed record MailCountResult(long TotalCount, IReadOnlyList<MailCountGroup> Groups);
@@ -0,0 +1,69 @@
using AIStudio.Tools.Mail;
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// Conditions a mail has to meet. A condition left out holds for every mail.
/// </summary>
/// <remarks>
/// A mail counts as read, flagged or answered when it is so in any of its folders: whoever read
/// the copy in the inbox read the mail, even when another copy in an archive still says unread.
/// Mails which lost their last place on the server never match, since they are gone or about to
/// show up somewhere else.
/// </remarks>
public sealed record MailFilter
{
/// <summary>
/// A part of the address or the name of the sender, as the From or Sender header gives it.
/// </summary>
public string? From { get; init; }
/// <summary>
/// A part of the address or the name of a recipient, as the To, Cc or Bcc header gives it.
/// </summary>
public string? To { get; init; }
/// <summary>
/// Only mails which arrived at the server at this point in time or later.
/// </summary>
public DateTimeOffset? ReceivedSinceUtc { get; init; }
/// <summary>
/// Only mails which arrived at the server before this point in time.
/// </summary>
public DateTimeOffset? ReceivedBeforeUtc { get; init; }
public bool? IsUnread { get; init; }
public bool? IsFlagged { get; init; }
public bool? IsEncrypted { get; init; }
public MailImportance? Importance { get; init; }
public bool? HasAttachments { get; init; }
/// <summary>
/// Only mails which lie in one of these folders, given by their full paths.
/// </summary>
/// <remarks>
/// An empty collection matches no mail at all. It never turns into "any folder": a caller
/// whose folder names matched no folder must not get the whole mailbox back.
/// </remarks>
public IReadOnlyCollection<string>? FolderPaths { get; init; }
/// <summary>
/// Whether any condition is set at all.
/// </summary>
public bool HasConditions =>
!string.IsNullOrWhiteSpace(this.From)
|| !string.IsNullOrWhiteSpace(this.To)
|| this.ReceivedSinceUtc is not null
|| this.ReceivedBeforeUtc is not null
|| this.IsUnread is not null
|| this.IsFlagged is not null
|| this.IsEncrypted is not null
|| this.Importance is not null
|| this.HasAttachments is not null
|| this.FolderPaths is not null;
}
@@ -0,0 +1,9 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// The flags of a mail at one of its locations.
/// </summary>
/// <param name="IsSeen">Whether the mail was read (\Seen).</param>
/// <param name="IsFlagged">Whether the mail is flagged (\Flagged).</param>
/// <param name="IsAnswered">Whether the mail was answered (\Answered).</param>
public readonly record struct MailFlags(bool IsSeen, bool IsFlagged, bool IsAnswered);
@@ -0,0 +1,58 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One folder of a mailbox and how far its sync got.
/// </summary>
internal sealed class MailFolderEntity
{
public int Id { get; set; }
public string DataSourceId { get; set; } = string.Empty;
/// <summary>
/// The full path of the folder, as the server names it.
/// </summary>
public string Path { get; set; } = string.Empty;
/// <summary>
/// The special use the server announces for the folder, e.g., SENT, or NONE, stored by name.
/// </summary>
public string SpecialUse { get; set; } = string.Empty;
/// <summary>
/// The UIDVALIDITY the stored locations of this folder belong to.
/// </summary>
/// <remarks>
/// When the server reports another one, every UID stored for this folder is void.
/// </remarks>
public long UidValidity { get; set; }
/// <summary>
/// The UIDNEXT at the end of the last complete pass over the folder, or null until one completed.
/// </summary>
public long? UidNext { get; set; }
/// <summary>
/// The HIGHESTMODSEQ at the end of the last complete pass over the folder, or null until one
/// completed, or when the server does not support CONDSTORE.
/// </summary>
public long? HighestModSeq { get; set; }
/// <summary>
/// How many mails the folder holds on the server (STATUS MESSAGES), or null when not read yet.
/// </summary>
public long? ServerMessageCount { get; set; }
/// <summary>
/// How many of them are unread (STATUS UNSEEN), or null when not read yet.
/// </summary>
public long? ServerUnseenCount { get; set; }
/// <summary>
/// When every mail of the folder which belongs into the index was in it for the first time, or
/// null while that first pass is still running.
/// </summary>
public DateTimeOffset? InitialSyncCompletedUtc { get; set; }
public List<MailLocationEntity> Locations { get; set; } = [];
}
@@ -0,0 +1,24 @@
using AIStudio.Tools.Mail;
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One folder of a mailbox and how far its sync got.
/// </summary>
/// <param name="Path">The full path of the folder, as the server names it.</param>
/// <param name="SpecialUse">What the server says the folder is for.</param>
/// <param name="UidValidity">The UIDVALIDITY the stored locations of this folder belong to.</param>
/// <param name="UidNext">The UIDNEXT at the end of the last complete pass, or null until one completed.</param>
/// <param name="HighestModSeq">The HIGHESTMODSEQ at the end of the last complete pass, or null until one completed or without CONDSTORE.</param>
/// <param name="ServerMessageCount">How many mails the folder holds on the server, or null when not read yet.</param>
/// <param name="ServerUnseenCount">How many of them are unread, or null when not read yet.</param>
/// <param name="InitialSyncCompletedUtc">When the first pass over the folder completed, or null while it runs.</param>
public sealed record MailFolderRecord(
string Path,
MailFolderSpecialUse SpecialUse,
long UidValidity,
long? UidNext,
long? HighestModSeq,
long? ServerMessageCount,
long? ServerUnseenCount,
DateTimeOffset? InitialSyncCompletedUtc);
@@ -0,0 +1,33 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One place on the server where a mail lies.
/// </summary>
/// <remarks>
/// A mail can lie in more than one folder, and moving it changes only its locations, never its
/// embedding. The flags are kept here, not with the mail, because IMAP keeps them per folder: a
/// copy in another folder has flags of its own.
/// </remarks>
internal sealed class MailLocationEntity
{
public int Id { get; set; }
public string ParentFileId { get; set; } = string.Empty;
public int FolderId { get; set; }
/// <summary>
/// The UID of the mail within its folder, valid for the UIDVALIDITY of that folder.
/// </summary>
public long Uid { get; set; }
public bool IsSeen { get; set; }
public bool IsFlagged { get; set; }
public bool IsAnswered { get; set; }
public MailMessageEntity? Message { get; set; }
public MailFolderEntity? Folder { get; set; }
}
@@ -0,0 +1,9 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One place on the server where a mail lies.
/// </summary>
/// <param name="FolderPath">The full path of the folder, as the server names it.</param>
/// <param name="Uid">The UID of the mail within that folder.</param>
/// <param name="Flags">The flags of the mail there.</param>
public sealed record MailLocationRecord(string FolderPath, long Uid, MailFlags Flags);
@@ -0,0 +1,87 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// What the index knows about one mail beyond its text, one row per mail and mailbox.
/// </summary>
/// <remarks>
/// The mail itself is a row in embedded_files, whose primary key this row shares. Indexing a mail
/// again deletes that row first, and the cascade takes this one with it. So this row is written
/// after the last chunk of a mail, never before.
/// </remarks>
internal sealed class MailMessageEntity
{
public string ParentFileId { get; set; } = string.Empty;
public string DataSourceId { get; set; } = string.Empty;
/// <summary>
/// The Message-ID header, empty when the mail has none.
/// </summary>
public string MessageId { get; set; } = string.Empty;
/// <summary>
/// The Message-ID the In-Reply-To header names, empty when there is none.
/// </summary>
public string InReplyTo { get; set; } = string.Empty;
/// <summary>
/// The Message-IDs of the References header, oldest first and separated by spaces, the way the
/// header itself lists them. A Message-ID cannot contain a space, so nothing gets lost.
/// </summary>
public string ReferenceMessageIds { get; set; } = string.Empty;
/// <summary>
/// When the sender says the mail was written (Date header), or null when it does not say so readably.
/// </summary>
public DateTimeOffset? SentAtUtc { get; set; }
/// <summary>
/// When the mail arrived at the server (INTERNALDATE).
/// </summary>
/// <remarks>
/// Unlike the Date header, the sender has no say in this one. It is also the date IMAP compares
/// against when the time range of a mailbox gets searched.
/// </remarks>
public DateTimeOffset ReceivedAtUtc { get; set; }
/// <summary>
/// LOW, NORMAL or HIGH, stored by name.
/// </summary>
public string Importance { get; set; } = string.Empty;
/// <summary>
/// How the content of the mail is encrypted, NONE when it is not, stored by name.
/// </summary>
/// <remarks>
/// There is no separate flag for whether a mail is encrypted: it would only be a second copy of
/// this column, free to disagree with it. A kind written by a newer version still is not NONE,
/// so such a mail still counts as encrypted.
/// </remarks>
public string EncryptionKind { get; set; } = string.Empty;
/// <summary>
/// A hash over the mail as the server delivered it, independent of how AI Studio turns it into text.
/// </summary>
public string MailHash { get; set; } = string.Empty;
/// <summary>
/// When AI Studio found this mail for the first time.
/// </summary>
public DateTimeOffset FirstSeenUtc { get; set; }
/// <summary>
/// When the mail lost its last location, or null while it has one.
/// </summary>
/// <remarks>
/// A mail which is moved disappears from one folder before it shows up in the other one, maybe
/// only in the next run. Removing it at once would embed it again when it reappears, so it stays
/// for one more run.
/// </remarks>
public DateTimeOffset? OrphanedAtUtc { get; set; }
public List<MailAddressEntity> Addresses { get; set; } = [];
public List<MailPartEntity> Parts { get; set; } = [];
public List<MailLocationEntity> Locations { get; set; } = [];
}
@@ -0,0 +1,49 @@
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One part of a mail as AI Studio read it: the header block, the body or an attachment.
/// </summary>
/// <remarks>
/// The chunks hold the same text, but cut into overlapping pieces. Reading a mail as a whole, or
/// checking its header block, needs it in one piece, and without asking the server again.
/// </remarks>
internal sealed class MailPartEntity
{
public int Id { get; set; }
public string ParentFileId { get; set; } = string.Empty;
/// <summary>
/// HEADERS, BODY or ATTACHMENT, stored by name.
/// </summary>
public string Kind { get; set; } = string.Empty;
/// <summary>
/// The place of the part among the parts of its kind, starting at zero.
/// </summary>
public int Position { get; set; }
/// <summary>
/// The file name of an attachment, empty for the other parts.
/// </summary>
public string Name { get; set; } = string.Empty;
public string ContentType { get; set; } = string.Empty;
/// <summary>
/// The size of the part on the server, in bytes.
/// </summary>
public long PartSize { get; set; }
/// <summary>
/// The text of the part, or null when there is none. TextState says why.
/// </summary>
public string? Text { get; set; }
/// <summary>
/// Whether the text could be read and, when not, why, stored by name.
/// </summary>
public string TextState { get; set; } = string.Empty;
public MailMessageEntity? Message { get; set; }
}
@@ -0,0 +1,14 @@
using AIStudio.Tools.Mail;
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// One part of a mail as AI Studio read it.
/// </summary>
/// <param name="Kind">The header block, the body or an attachment.</param>
/// <param name="Name">The file name of an attachment, empty for the other parts.</param>
/// <param name="ContentType">The content type of the part.</param>
/// <param name="PartSize">The size of the part on the server, in bytes.</param>
/// <param name="Text">The text of the part, or null when there is none.</param>
/// <param name="TextState">Whether the text could be read and, when not, why.</param>
public sealed record MailPartRecord(MailPartKind Kind, string Name, string ContentType, long PartSize, string? Text, MailPartTextState TextState);
@@ -0,0 +1,38 @@
using AIStudio.Tools.Mail;
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// What the index keeps about one mail beyond its chunks.
/// </summary>
/// <remarks>
/// The addresses come back grouped by their role and the parts by their kind, each group in the
/// order it was stored in. The locations come back ordered by folder and UID.
/// </remarks>
/// <param name="MailId">The id of the mail, which is the id of its document.</param>
/// <param name="MessageId">The Message-ID header, empty when the mail has none.</param>
/// <param name="InReplyTo">The Message-ID the In-Reply-To header names, empty when there is none.</param>
/// <param name="ReferenceMessageIds">The Message-IDs of the References header, oldest first.</param>
/// <param name="SentAtUtc">When the sender says the mail was written, or null when it does not say so readably.</param>
/// <param name="ReceivedAtUtc">When the mail arrived at the server.</param>
/// <param name="Importance">How important the sender marked the mail.</param>
/// <param name="EncryptionKind">How the content of the mail is encrypted, NONE when it is not.</param>
/// <param name="MailHash">A hash over the mail as the server delivered it.</param>
/// <param name="FirstSeenUtc">When AI Studio found the mail for the first time.</param>
/// <param name="Addresses">The addresses from the header of the mail.</param>
/// <param name="Parts">The parts of the mail as AI Studio read them.</param>
/// <param name="Locations">Where the mail lies on the server, at least one place.</param>
public sealed record MailRecord(
string MailId,
string MessageId,
string InReplyTo,
IReadOnlyList<string> ReferenceMessageIds,
DateTimeOffset? SentAtUtc,
DateTimeOffset ReceivedAtUtc,
MailImportance Importance,
MailEncryptionKind EncryptionKind,
string MailHash,
DateTimeOffset FirstSeenUtc,
IReadOnlyList<MailAddressRecord> Addresses,
IReadOnlyList<MailPartRecord> Parts,
IReadOnlyList<MailLocationRecord> Locations);
@@ -0,0 +1,32 @@
using AIStudio.Tools.Mail;
namespace AIStudio.Tools.Databases.IndexStore;
/// <summary>
/// What a list of mails shows about one of them, without its text.
/// </summary>
/// <param name="MailId">The id of the mail.</param>
/// <param name="Subject">The subject of the mail.</param>
/// <param name="ReceivedAtUtc">When the mail arrived at the server.</param>
/// <param name="SentAtUtc">When the sender says the mail was written, or null when it does not say so readably.</param>
/// <param name="MessageId">The Message-ID header, empty when the mail has none.</param>
/// <param name="InReplyTo">The Message-ID the In-Reply-To header names, empty when there is none.</param>
/// <param name="Addresses">The addresses from the header, grouped by their role.</param>
/// <param name="FolderPaths">The folders the mail lies in, ordered by path.</param>
/// <param name="Flags">The flags of the mail; each one is set when it is set in any folder.</param>
/// <param name="Importance">How important the sender marked the mail.</param>
/// <param name="EncryptionKind">How the content of the mail is encrypted, NONE when it is not.</param>
/// <param name="AttachmentNames">The file names of the attachments, in their order.</param>
public sealed record MailSummary(
string MailId,
string Subject,
DateTimeOffset ReceivedAtUtc,
DateTimeOffset? SentAtUtc,
string MessageId,
string InReplyTo,
IReadOnlyList<MailAddressRecord> Addresses,
IReadOnlyList<string> FolderPaths,
MailFlags Flags,
MailImportance Importance,
MailEncryptionKind EncryptionKind,
IReadOnlyList<string> AttachmentNames);
Loaded 100 of 343 files, more files were not shown because too many files have changed in this diff. Show more