Merge branch 'main' into chunk-data

Resolved 29 conflicting files. The notable decisions:

Confidence: main's tool-calling gate (RequiredProviderConfidence) and this
branch's local-RAG gate (DataConfidenceLevel) turned out to be the same rule
on the same axis, so they are now one field. Both tool results and data
sources raise it through RequireProviderConfidence(). The gate checks the
level strictly and no longer exempts providers trusted by configuration:
TrustedProviderIds is documented as applying to data-source security checks
only, and organizations set confidence through DataConfidence
.CustomConfidenceScheme instead. The security axis (DataSecurity, ERI,
IsTrustedForDataSourceSecurityChecks) is unchanged.

Provider creation: main's CreateProvider signature won (hfEndpointKind,
capabilityOverrides, no model parameter); tokenizerPath was added to it and
is set for every provider, including the new Hetzner, IONOS and LiteLLM.
Provider and EmbeddingProvider combine the record parameters, Lua parsing and
Lua serialization of both sides.

File types: main's hierarchy (ODT leaf, WORD parent, PowerPoint without the
legacy .ppt, TABULAR instead of DELIMITED_TABLE) plus this branch's
SPREADSHEET parent with ODS and the xlsm/xlsb/xla/xlam extensions, which the
runtime already reads. Both sides had added a conflicting HTML filter; the
reading family keeps the name, and the export path uses a narrow
HTML_DOCUMENT, following the existing LATEX/TEX split.

Runtime: main's file_data.rs is the base, including the prompt-injection
sanitizer and the extraction routes. Token counting and chunk segmentation
moved into take_released, so they act on the text the filter has released
rather than on text it is still holding. A failed count is logged and left
out instead of ending the extraction, because the app counts such a segment
itself.

Data sources: the participating-provider checks of this branch are kept, and
main's GetAllowedDataSources overload now builds on them. DirectChatService
resolves the launched chat's data source options before the check, so filter
and chat see the same options.

.NET and Rust both build clean; I18N regenerated to 4060 keys.

app/MindWork AI Studio/Tools/ContentStreamSseHandler.cs

@@ -7,8 +7,9 @@ public static class ContentStreamSseHandler
 {
     private static readonly ConcurrentDictionary<string, List<ContentStreamPptxImageData>> CHUNKED_IMAGES = new();
     private static readonly ConcurrentDictionary<string, SlideManager> SLIDE_MANAGERS = new();
+    private static readonly ConcurrentDictionary<string, DocumentManager> DOCUMENT_MANAGERS = new();
 
-    public static string? ProcessEvent(ContentStreamSseEvent? sseEvent, bool extractImages = true)
+    public static ContentStreamProcessedEvent ProcessEvent(ContentStreamSseEvent? sseEvent, bool extractImages = true)
     {
         switch (sseEvent)
         {
@@ -16,16 +17,16 @@ public static class ContentStreamSseHandler
                 switch (sseEvent.Metadata)
                 {
                     case ContentStreamTextMetadata:
-                        return sseEvent.Content;
-                
+                        return ContentStreamProcessedEvent.FromContent(sseEvent.Content);
+
                     case ContentStreamPdfMetadata pdfMetadata:
                         var pageNumber = pdfMetadata.Pdf?.PageNumber ?? 0;
-                        return $"""
+                        return ContentStreamProcessedEvent.FromContent($"""
                                 # Page {pageNumber}
                                 {sseEvent.Content}
-                                
-                                """;
-                    
+
+                                """);
+
                     case ContentStreamSpreadsheetMetadata spreadsheetMetadata:
                         var sheetName = spreadsheetMetadata.Spreadsheet?.SheetName;
                         var rowNumber = spreadsheetMetadata.Spreadsheet?.RowNumber;
@@ -37,38 +38,66 @@ public static class ContentStreamSseHandler
                         }
 
                         spreadSheetResult.Append(sseEvent.Content);
-                        return spreadSheetResult.ToString();
-                
-                    case ContentStreamDocumentMetadata:
+                        return ContentStreamProcessedEvent.FromContent(spreadSheetResult.ToString());
+
+                    //
+                    // Documents which the runtime reads page by page are buffered, so the images of
+                    // a page can follow its Markdown. Documents converted as a whole, e.g. by Pandoc,
+                    // carry no page number and are passed on unchanged.
+                    //
+                    case ContentStreamDocumentMetadata documentMetadata:
+                        if (documentMetadata.Document?.PageNumber is not > 0)
+                            return ContentStreamProcessedEvent.FromContent(sseEvent.Content);
+
+                        var documentManager = DOCUMENT_MANAGERS.GetOrAdd(sseEvent.StreamId!, _ => new());
+                        var documentContent = documentManager.AddPage(documentMetadata, sseEvent.Content, extractImages);
+                        return documentContent is null ? ContentStreamProcessedEvent.NOTHING : ContentStreamProcessedEvent.FromContent(documentContent);
+
                     case ContentStreamImageMetadata:
-                        return sseEvent.Content;
+                        return ContentStreamProcessedEvent.FromContent(sseEvent.Content);
 
                     case ContentStreamPresentationMetadata presentationMetadata:
                         if (!extractImages)
                         {
                             var slideNumber = presentationMetadata.Presentation?.SlideNumber ?? 0;
-                            return slideNumber > 0
+                            return ContentStreamProcessedEvent.FromContent(slideNumber > 0
                                 ? $"# Slide {slideNumber}\n{sseEvent.Content}"
-                                : sseEvent.Content;
+                                : sseEvent.Content);
                         }
 
                         var slideManager = SLIDE_MANAGERS.GetOrAdd(
                             sseEvent.StreamId!,
                             _ => new()
                         );
-                        
+
                         slideManager.AddSlide(presentationMetadata, sseEvent.Content, extractImages);
-                        return null;
-                    
+                        return ContentStreamProcessedEvent.NOTHING;
+
+                    //
+                    // The runtime reported a failure. It must not contribute any content: an empty
+                    // or partial document would otherwise be handed to the AI as if it were the
+                    // real file content.
+                    //
+                    case ContentStreamErrorMetadata errorMetadata:
+                        return ContentStreamProcessedEvent.FromError(errorMetadata.Error);
+
+                    //
+                    // The runtime filtered suspected prompt injections out of the content. The
+                    // content itself already arrived through the events before this one, so this
+                    // only reports what was removed.
+                    //
+                    case ContentStreamPromptInjectionMetadata promptInjectionMetadata:
+                        return ContentStreamProcessedEvent.FromPromptInjection(promptInjectionMetadata.PromptInjection);
+
                     default:
-                        return sseEvent.Content;
+                        return ContentStreamProcessedEvent.FromContent(sseEvent.Content);
                 }
-                
+
             case { Content: not null, Metadata: null }:
-                return sseEvent.Content;
-            
+                return ContentStreamProcessedEvent.FromContent(sseEvent.Content);
+
             default:
-                return null;
+                return ContentStreamProcessedEvent.NOTHING;
         }
     }
 
@@ -87,6 +116,7 @@ public static class ContentStreamSseHandler
             Content = content,
             Segment = segment,
             IsEnd = isEnd,
+            MediaType = contentStreamPptxImageData.MediaType,
         };
         
         CHUNKED_IMAGES.AddOrUpdate(
@@ -118,7 +148,32 @@ public static class ContentStreamSseHandler
         CHUNKED_IMAGES.Remove(id, out _);
         return base64Image;
     }
-    
+
+    /// <summary>
+    /// Assembles the collected segments of an image into a Markdown image.
+    /// </summary>
+    /// <remarks>
+    /// Handing the naked Base64 data to the AI says nothing: it is neither readable text nor an
+    /// image it could look at. Only the data URI makes it one, so every reader must embed its
+    /// images this way.
+    /// </remarks>
+    /// <param name="id">The ID of the image to assemble.</param>
+    /// <param name="mediaType">The media type the runtime reported, if any.</param>
+    /// <returns>The Markdown image, or null when no data was collected for that ID.</returns>
+    public static string? BuildImageMarkdown(string id, string? mediaType)
+    {
+        var base64Image = BuildImage(id);
+        if (string.IsNullOrWhiteSpace(base64Image))
+            return null;
+
+        //
+        // Both readers compress their images, and that compression produces JPEG. A runtime which
+        // does not report the media type therefore delivered JPEG as well.
+        //
+        var imageMediaType = string.IsNullOrWhiteSpace(mediaType) ? "image/jpeg" : mediaType;
+        return $"![Image](data:{imageMediaType};base64,{base64Image})";
+    }
+
     public static string? Clear(string streamId)
     {
         if (string.IsNullOrWhiteSpace(streamId))
@@ -131,8 +186,16 @@ public static class ContentStreamSseHandler
             if (!string.IsNullOrWhiteSpace(result))
                 finalContentChunk.Append(result);
         }
+
+        if (DOCUMENT_MANAGERS.TryGetValue(streamId, out var documentManager))
+        {
+            var result = documentManager.Flush();
+            if (!string.IsNullOrWhiteSpace(result))
+                finalContentChunk.Append(result);
+        }
         
         SLIDE_MANAGERS.TryRemove(streamId, out _);
+        DOCUMENT_MANAGERS.TryRemove(streamId, out _);
         var imageIdPrefix = $"{streamId}-";
         foreach (var key in CHUNKED_IMAGES.Keys.Where(k => k.StartsWith(imageIdPrefix, StringComparison.InvariantCultureIgnoreCase)))
             CHUNKED_IMAGES.TryRemove(key, out _);