Added the Model Comparison assistant for blind comparisons of two models (#1009)

Co-authored-by: Thorsten Sommer <SommerEngineering@users.noreply.github.com>
This commit is contained in:
Dominic NeuburgandThorsten Sommer authored and GitHub committed 2026-11-14 14:11:34 +01:00
1 parent 12f26445ba
commit b7255694cd
221 files changed
+19989 -90

No files matched your search

+2
View File
@@ -224,6 +224,8 @@ A call has to pass both the restriction and the free address choice, so with the
`semantic_search` lets the model search the data sources of a chat itself, with a query it works out from the conversation, whenever a question calls for it. The classic RAG process, `AISrcSelWithRetCtxVal`, searches them with every message instead, using the message as the query. One place decides which of the two runs, `ToolRegistry.GetEffectiveRetrievalModeAsync`. Semantic search is the default, and the user can choose the other way per chat through `DataSourceOptions.RetrievalMode`. Whenever the tool cannot be offered — a model without tool calling, tools or this tool switched off, a provider below a confidence the organization set for it — the classic process searches instead. That process steps back only when the answer is semantic search, so a chat never ends up searching nothing.
Chats run in two places: in the chat of the app, and in the comparison series of the Model Comparison assistant, whose chats run as `Components.MODEL_COMPARISON_ASSISTANT`. `semantic_search` is visible in both, see `ToolVisibilityDefinition.AllowedComponents`; anywhere else, `GetEffectiveRetrievalModeAsync` answers `NOT_AVAILABLE_HERE`, and the data sources are searched with every message. The mode Answer Variance of the Model Comparison assistant runs as the same component, but its chats keep the data sources switched off, so the tool offers nothing there.
The tool offers the data sources of the chat which the provider may use. With the automatic selection switched on, those are all data sources the provider may use: the AI which selects is the chat model itself. No agent takes part, so `DataSourceService` counts only the chat provider when it decides what may be used, see `DataSourceRetrievalMode`. The description of the tool lists the offered data sources by ID, name, and kind, with their own description shortened to 500 characters. It names no page size: how many passages a page holds depends on how long they are, so each result tells whether a further page holds more. The description of an ERI data source comes from its server, so it goes through the prompt-injection filter before it gets there. The listed IDs are the only values `data_source_ids` accepts.
An ERI data source has a single page of matches, see `IDataSource.GetLastPage`, and asks its server for exactly as many as the classic process does. ERI v1 knows no offset, so a further page would have to ask for all pages before it as well, and a server which allows only the configured number of matches would refuse it: the model would hear that the data source could not be searched rather than that there is nothing more. When these matches do not fit into one search, the model still turns pages within them, and each page asks the server again.