23 Commits

Author	SHA1	Message	Date
Thorsten Sommer	d9fe426c81	Accept PDFs whose header is not at the start of the file	2026-09-09 15:17:25 +02:00
Thorsten Sommer	1bf11c0ca7	Fix the Clippy warnings in the runtime	2026-09-06 15:03:21 +02:00
Thorsten Sommer	fe35630eff	Merge branch 'main' into chunk-data ... Resolved 29 conflicting files. The notable decisions: Confidence: main's tool-calling gate (RequiredProviderConfidence) and this branch's local-RAG gate (DataConfidenceLevel) turned out to be the same rule on the same axis, so they are now one field. Both tool results and data sources raise it through RequireProviderConfidence(). The gate checks the level strictly and no longer exempts providers trusted by configuration: TrustedProviderIds is documented as applying to data-source security checks only, and organizations set confidence through DataConfidence .CustomConfidenceScheme instead. The security axis (DataSecurity, ERI, IsTrustedForDataSourceSecurityChecks) is unchanged. Provider creation: main's CreateProvider signature won (hfEndpointKind, capabilityOverrides, no model parameter); tokenizerPath was added to it and is set for every provider, including the new Hetzner, IONOS and LiteLLM. Provider and EmbeddingProvider combine the record parameters, Lua parsing and Lua serialization of both sides. File types: main's hierarchy (ODT leaf, WORD parent, PowerPoint without the legacy .ppt, TABULAR instead of DELIMITED_TABLE) plus this branch's SPREADSHEET parent with ODS and the xlsm/xlsb/xla/xlam extensions, which the runtime already reads. Both sides had added a conflicting HTML filter; the reading family keeps the name, and the export path uses a narrow HTML_DOCUMENT, following the existing LATEX/TEX split. Runtime: main's file_data.rs is the base, including the prompt-injection sanitizer and the extraction routes. Token counting and chunk segmentation moved into take_released, so they act on the text the filter has released rather than on text it is still holding. A failed count is logged and left out instead of ending the extraction, because the app counts such a segment itself. Data sources: the participating-provider checks of this branch are kept, and main's GetAllowedDataSources overload now builds on them. DirectChatService resolves the launched chat's data source options before the check, so filter and chat see the same options. .NET and Rust both build clean; I18N regenerated to 4060 keys.	2026-09-05 21:20:33 +02:00
Thorsten Sommer	7d9a4f5ab1	Reduced memory usage and fixed several memory leaks (#933)	2026-08-23 21:15:37 +02:00
Sabrina-devops	902a01a4d0	Added a prompt injection detection (#857) ... Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>	2026-08-23 11:09:11 +02:00
Thorsten Sommer	e9ce64cf0c	Fixed estimated document pages (#929)	2026-08-18 18:28:40 +02:00
Paul Koudelka	c8c1e38475	made tokenizer concurrent and threadsafe	2026-08-10 21:36:01 +02:00
nilskruthoff	6eab9dc574	Improved docx and odt import (#890) ... Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>	2026-08-10 20:39:31 +02:00
Paul Koudelka	3918b48409	fixed tokenization issue for long files	2026-08-10 18:21:31 +02:00
Thorsten Sommer	8f8f788896	Fixed attached files silently reaching the AI as empty documents (#904)	2026-08-10 16:37:44 +02:00
nilskruthoff	33b0850c71	Improved slide import function (#878)	2026-07-21 12:02:32 +02:00
Paul Koudelka	79fbeb636c	Made file-extensions case insensitive (#858)	2026-07-15 14:47:01 +02:00
Thorsten Sommer	d69eab8807	Migrated to axum (#757)	2026-05-12 20:31:08 +02:00
Thorsten Sommer	0e14a3cb58	Improved PDFium initialization and error handling logic (#649)	2026-02-01 19:51:12 +01:00
Thorsten Sommer	009bb33d83	Added a preview of the document analysis assistant (#561) ... Co-authored-by: Peer Schütt <20603780+peerschuett@users.noreply.github.com>	2025-11-24 12:37:18 +01:00
Thorsten Sommer	68f5bb1512	Finished the function for importing arbitrary documents (#516)	2025-06-30 21:51:02 +02:00
nilskruthoff	aaedf667fe	Allow loading any document in some assistants (#508) ... Co-authored-by: krut_ni <nils.kruthoff@dlr.de> Co-authored-by: Thorsten Sommer <mail@tsommer.org>	2025-06-30 18:56:48 +02:00
nilskruthoff	18b2e22725	Added support for reading PowerPoint files (#506) ... Co-authored-by: krut_ni <nils.kruthoff@dlr.de>	2025-06-23 14:49:36 +02:00
Thorsten Sommer	437da0fd9b	Finished the Pandoc integration (#487)	2025-05-30 22:39:16 +02:00
Thorsten Sommer	b0d4615506	Refactored PDFium initialization (#440)	2025-05-03 12:20:22 +02:00
Thorsten Sommer	c4d9af18cd	Add component to read PDF files into assistants (#439)	2025-05-02 23:09:50 +02:00
Thorsten Sommer	07c1182611	Housekeeping (#379)	2025-04-01 20:44:43 +02:00
nilskruthoff	3ce6e168cb	Read arbitrary data from the file system (#374) ... Co-authored-by: krut_ni <nils.kruthoff@dlr.de> Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>	2025-04-01 19:10:29 +02:00