Read the context window from a provider's model list

This commit is contained in:
Thorsten Sommer 2026-09-13 10:20:17 +02:00
parent d8a30c4e60
commit 1c03a46e2f
Signed by untrusted user who does not match committer: tsommer
GPG Key ID: 371BBA77A02C0108
8 changed files with 467 additions and 15 deletions

View File

@ -853,10 +853,15 @@ public partial class ProviderDialog : MSGComponentBase, ISecretId
private ModelProfile GetCurrentModelProfile() => this.CreateProviderSettings().GetModelProfile();
/// <summary>
/// What the rules alone say about the model, which is what each switch shows as its automatic answer.
/// What holds without anybody switching anything, which is what each field shows as its automatic answer.
/// </summary>
/// <remarks>
/// The rules, plus whatever the provider itself stated when the model list was loaded a moment
/// ago. A self-hosted engine is the case this matters for: it reports the window it was started
/// with, and that is the number a person gets by leaving the field below empty.
/// </remarks>
/// <returns>The profile.</returns>
private ModelProfile GetAutomaticModelProfile() => this.DataLLMProvider.GetModelProfile(this.GetSelectedModel());
private ModelProfile GetAutomaticModelProfile() => this.CreateProviderSettings().GetAutomaticModelProfile();
private string GetCurrentModelApiLabel()
{

View File

@ -0,0 +1,86 @@
using System.Collections.Concurrent;
using System.Collections.Frozen;
namespace AIStudio.Models.Live;
/// <summary>
/// What the configured providers last said about the models they serve.
/// </summary>
/// <remarks>
/// One snapshot per configured provider instance, and reporting replaces the snapshot rather than
/// adding to it. That is the same reason the registry replaces what the plugins declare: a model an
/// installation no longer serves has to stop answering, and a window somebody halved by restarting
/// their engine must not go on being reported alongside its correction.
///
/// Nothing here is written to disk. These are statements about a machine as it is running right
/// now, and the app asks that machine again before every chat round anyway. An instance somebody
/// deleted keeps its snapshot until the app is closed -- a few dozen kilobytes at the very worst,
/// which is not worth a second mechanism to watch the settings for.
/// </remarks>
public sealed class ListedModels
{
/// <summary>
/// The one the app reports into and asks.
/// </summary>
public static ListedModels Shared { get; } = new();
/// <summary>
/// Per configured provider instance, what its model list said about each model.
/// </summary>
/// <remarks>
/// Both keys ignore case. The IDs come back from the same list they were stored under, so
/// ordinal would do -- but a model an organization wrote into a configuration plugin by hand
/// was typed by a person, and the availability check already treats such a name as the same
/// model regardless of case. Being stricter here would leave exactly those people without the
/// numbers.
/// </remarks>
private readonly ConcurrentDictionary<string, FrozenDictionary<string, ModelListing>> byProvider = new(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Takes over what one provider instance said about its models, replacing what it said before.
/// </summary>
/// <remarks>
/// Only ever call this with a whole list in hand. Reporting a filtered part of one would tell
/// this instance that everything left out has stopped existing.
/// </remarks>
/// <param name="configuredProviderId">The instance that was asked. Nothing happens without one.</param>
/// <param name="listings">What its list stated, with the models it stated nothing about left in or out as convenient.</param>
public void Report(string configuredProviderId, IEnumerable<ModelListing> listings)
{
//
// A provider instance nobody has configured yet is not a machine we could ask again later,
// so there is nothing to remember it by. The provider dialog is not such a case: it works
// on a fully built instance from the moment it opens, ID included.
//
if (string.IsNullOrWhiteSpace(configuredProviderId))
return;
var stated = new Dictionary<string, ModelListing>(StringComparer.OrdinalIgnoreCase);
foreach (var listing in listings)
{
if (string.IsNullOrWhiteSpace(listing.ModelId) || !listing.IsKnown)
continue;
stated[listing.ModelId] = listing;
}
this.byProvider[configuredProviderId] = stated.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase);
}
/// <summary>
/// What one provider instance said about one of its models.
/// </summary>
/// <param name="configuredProviderId">The instance serving the model.</param>
/// <param name="modelId">The model, named the way that instance names it.</param>
/// <returns>What it stated, which is nothing when it was never asked or said nothing.</returns>
public ModelListing Of(string configuredProviderId, string modelId)
{
if (string.IsNullOrWhiteSpace(configuredProviderId) || string.IsNullOrWhiteSpace(modelId))
return ModelListing.NOTHING;
if (!this.byProvider.TryGetValue(configuredProviderId, out var stated))
return ModelListing.NOTHING;
return stated.TryGetValue(modelId, out var listing) ? listing : ModelListing.NOTHING;
}
}

View File

@ -0,0 +1,44 @@
namespace AIStudio.Models.Live;
/// <summary>
/// What a provider's own model list says about one of the models it serves.
/// </summary>
/// <remarks>
/// That list is fetched anyway: before every chat round, before every assistant run, and whenever
/// somebody opens the provider dialog. Reading what it already carries therefore costs no request
/// of its own, which is the whole reason these numbers are taken from here and not asked for.
///
/// This describes one installation, never the model as such. Two machines may serve the same
/// weights behind different settings, and a statement about one of them says nothing about the
/// other -- which is why a listing is kept per configured provider instance and is gone with the
/// process. It is also the only source for a self-hosted model: a rule can say what the weights
/// were trained for, but only the engine knows what its operator started it with.
/// </remarks>
/// <param name="ModelId">The model, named the way the provider names it in its list.</param>
/// <param name="Context">The window the provider states for it, or unknown where it states none.</param>
public readonly record struct ModelListing(string ModelId, ContextWindow Context)
{
/// <summary>
/// What we have about a model nobody has reported anything about.
/// </summary>
public static readonly ModelListing NOTHING = new(string.Empty, ContextWindow.UNKNOWN);
/// <summary>
/// Whether this listing states anything at all.
/// </summary>
public bool IsKnown => this.Context.IsKnown;
/// <summary>
/// Puts what the provider stated over what the rules worked out.
/// </summary>
/// <remarks>
/// A stated window replaces the whole window, the ceiling included, for the same reason the
/// expert settings do: what a model card says it could be raised to is a statement about the
/// model, while this is a statement about the installation serving it. Whoever started that
/// engine has already decided, and a ceiling nobody can reach without restarting it is not a
/// number to keep showing.
/// </remarks>
/// <param name="profile">What is known about the model without this listing.</param>
/// <returns>The profile, with what the provider stated in it.</returns>
public ModelProfile ApplyTo(in ModelProfile profile) => this.IsKnown ? profile with { Context = this.Context } : profile;
}

View File

@ -1,3 +1,23 @@
using System.Text.Json.Serialization;
namespace AIStudio.Provider.SelfHosted;
public readonly record struct Model(string Id, string? Object, string? OwnedBy, ModelArchitecture? Architecture);
/// <summary>
/// One model as an OpenAI-compatible engine lists it.
/// </summary>
/// <remarks>
/// The context window is vLLM's addition to that route: it reports the window the operator started
/// the engine with, which is the one number no rule about the weights could ever know. Ollama,
/// LM Studio, and llama.cpp answer the same route without it, so it stays unknown there instead of
/// being guessed.
///
/// vLLM calls that field max_model_len, which reads like a limit on the model rather than on a
/// conversation. The wire keeps their spelling, and this record says what the number means, so that
/// nobody has to remember the translation while reading the code that uses it.
/// </remarks>
/// <param name="Id">The model's ID.</param>
/// <param name="Object">What kind of thing the entry is. Known value: "model".</param>
/// <param name="OwnedBy">Who the engine names as the owner of the model.</param>
/// <param name="Architecture">Which kinds of input and output the model takes, where the engine says.</param>
/// <param name="ContextWindowTokens">The context window the engine was started with, in tokens, where it says.</param>
public readonly record struct Model(string Id, string? Object, string? OwnedBy, ModelArchitecture? Architecture, [property: JsonPropertyName("max_model_len")] int? ContextWindowTokens);

View File

@ -3,6 +3,8 @@ using System.Runtime.CompilerServices;
using System.Text.Json;
using AIStudio.Chat;
using AIStudio.Models;
using AIStudio.Models.Live;
using AIStudio.Provider.OpenAI;
using AIStudio.Settings;
using AIStudio.Tools.PluginSystem;
@ -188,8 +190,23 @@ public sealed class ProviderSelfHosted(Host host, string hostname) : BaseProvide
return FailedModelLoadResult(this.GetModelLoadFailureReason(lmStudioResponse, responseBody), $"Status={(int)lmStudioResponse.StatusCode} {lmStudioResponse.ReasonPhrase}; Body='{responseBody}'");
}
var lmStudioModelResponse = await lmStudioResponse.Content.ReadFromJsonAsync<ModelsResponse>(token);
//
// Read with the shared options, the way every other model list of this app is read.
// This one route did without them, which quietly cost it every field an engine spells
// in snake case: owned_by has been arriving as nothing all along, and the next field
// somebody adds here would have gone the same way without anything failing.
//
var lmStudioModelResponse = await lmStudioResponse.Content.ReadFromJsonAsync<ModelsResponse>(JSON_SERIALIZER_OPTIONS, token);
var models = lmStudioModelResponse.Data ?? [];
//
// What the engine said about its own models, taken from the whole list rather than
// from what is offered below: a model filtered out here as an embedding model is still
// a model somebody may have configured this instance with, and this list is the only
// place its window is ever stated.
//
ListedModels.Shared.Report(this.ConfiguredProviderId, ListingsOf(models));
return SuccessfulModelLoadResult(models.
Where(model => !string.IsNullOrWhiteSpace(model.Id) &&
!ignorePhrases.Any(ignorePhrase => model.Id.Contains(ignorePhrase, StringComparison.InvariantCulture)) &&
@ -302,6 +319,27 @@ public sealed class ProviderSelfHosted(Host host, string hostname) : BaseProvide
}
}
/// <summary>
/// What an engine stated about the models it serves.
/// </summary>
/// <remarks>
/// A window of zero or less is dropped rather than repaired. An engine answering that way is
/// telling us something we cannot interpret, and falling back to what the rules say about the
/// weights is the one answer nobody has to invent.
/// </remarks>
/// <param name="models">The models exactly as the engine listed them.</param>
/// <returns>One listing per model the engine said something usable about.</returns>
private static IEnumerable<ModelListing> ListingsOf(IEnumerable<Model> models)
{
foreach (var model in models)
{
if (string.IsNullOrWhiteSpace(model.Id) || model.ContextWindowTokens is not > 0)
continue;
yield return new(model.Id, ContextWindow.Of(model.ContextWindowTokens.Value));
}
}
private static bool IsMatchingLlamaCppTextModel(Model model, string[] ignorePhrases, string[] filterPhrases)
{
if (string.IsNullOrWhiteSpace(model.Id))

View File

@ -1,4 +1,5 @@
using AIStudio.Models;
using AIStudio.Models.Live;
using AIStudio.Models.Registry;
using AIStudio.Provider;
@ -11,23 +12,42 @@ public static partial class ProviderExtensions
/// </summary>
/// <remarks>
/// The one door to that question. Behind it stand the links of the chain, in the order they
/// win: what the person said about their own installation, then what the rules worked out from
/// the name, then what the app assumes when nothing else said anything.
/// win: what the person said about their own installation, then what the installation itself
/// reported, then what the rules worked out from the name, and last what the app assumes when
/// nothing else said anything.
/// </remarks>
/// <param name="provider">The configured provider.</param>
/// <returns>The profile of the configured model.</returns>
public static ModelProfile GetModelProfile(this Provider provider)
{
var stated = provider.UsedLLMProvider.GetModelProfile(provider.Model);
return provider.CapabilityOverrides?.ApplyTo(stated) ?? stated;
var automatic = provider.GetAutomaticModelProfile();
return provider.CapabilityOverrides?.ApplyTo(automatic) ?? automatic;
}
/// <summary>
/// Everything the rules know about a model at a provider, without anybody's own settings.
/// Everything known about the configured model except what the person themselves switched.
/// </summary>
/// <remarks>
/// What the expert dialog shows next to each switch as the automatic answer, so that a person
/// can see what they are overriding.
/// This is what happens when somebody fills in nothing, which is why the expert dialog shows it
/// as the automatic answer. It has to include what the provider reported: a person who leaves
/// the window empty gets the number their own engine stated, and a placeholder showing them a
/// different one would be a promise the app does not keep.
/// </remarks>
/// <param name="provider">The configured provider.</param>
/// <returns>The profile of the configured model, without that provider's overrides.</returns>
public static ModelProfile GetAutomaticModelProfile(this Provider provider)
{
var stated = provider.UsedLLMProvider.GetModelProfile(provider.Model);
return ListedModels.Shared.Of(provider.Id, provider.Model.Id).ApplyTo(stated);
}
/// <summary>
/// Everything the rules know about a model at a provider, without anybody's own installation.
/// </summary>
/// <remarks>
/// The answer to the model as such, which is the same for everybody who uses that name at that
/// provider -- and therefore the answer the registry caches. What one particular installation
/// says about it is asked one link further up, where the instance is known.
///
/// The assumed profile fills in where no rule stated a single capability. It fills in the
/// capabilities only: a modifier may well have said what the model is made for without any rule

View File

@ -0,0 +1,154 @@
using AIStudio.Models;
using AIStudio.Models.Live;
using AIStudio.Provider;
namespace AIStudio.Tests.Models.Live;
/// <summary>
/// Checks what a running installation is allowed to say about the models it serves.
/// </summary>
/// <remarks>
/// Every test here builds its own store rather than using the shared one. What a provider reported
/// is state which outlives a single question, and a test leaving some of it behind would decide
/// what the next test sees.
/// </remarks>
[TestFixture]
public sealed class ListedModelsTests
{
private const string ONE_MACHINE = "11111111-1111-1111-1111-111111111111";
private const string ANOTHER_MACHINE = "22222222-2222-2222-2222-222222222222";
private const string MODEL = "qwen3-32b";
/// <summary>
/// A model the rules have something to say about, so that a report has something to contradict.
/// </summary>
private static readonly ModelProfile WHAT_THE_RULES_SAY = new()
{
Capabilities = Capability.TEXT_INPUT | Capability.TEXT_OUTPUT | Capability.MULTIPLE_IMAGE_INPUT,
Kind = ModelKind.CHAT,
Context = ContextWindow.Of(131_072, 262_144),
Images = new(null, 20),
};
[Test]
public void AMachineWhichWasNeverAskedSaysNothing()
{
var listed = new ListedModels();
Assert.Multiple(() =>
{
Assert.That(listed.Of(ONE_MACHINE, MODEL), Is.EqualTo(ModelListing.NOTHING));
Assert.That(listed.Of(ONE_MACHINE, MODEL).IsKnown, Is.False);
});
}
[Test]
public void WhatOneMachineSaysIsNotWhatAnotherSays()
{
//
// The same weights behind two engines, each started by somebody who decided for themselves.
// This is the whole reason these numbers are kept per configured instance.
//
var listed = new ListedModels();
listed.Report(ONE_MACHINE, [new(MODEL, ContextWindow.Of(32_768))]);
listed.Report(ANOTHER_MACHINE, [new(MODEL, ContextWindow.Of(8_192))]);
Assert.Multiple(() =>
{
Assert.That(listed.Of(ONE_MACHINE, MODEL).Context.DefaultTokens, Is.EqualTo(32_768));
Assert.That(listed.Of(ANOTHER_MACHINE, MODEL).Context.DefaultTokens, Is.EqualTo(8_192));
});
}
[Test]
public void WhatAMachineNoLongerServesStopsAnswering()
{
var listed = new ListedModels();
listed.Report(ONE_MACHINE, [new(MODEL, ContextWindow.Of(32_768)), new("gemma3-27b", ContextWindow.Of(16_384))]);
listed.Report(ONE_MACHINE, [new("gemma3-27b", ContextWindow.Of(16_384))]);
Assert.Multiple(() =>
{
Assert.That(listed.Of(ONE_MACHINE, MODEL), Is.EqualTo(ModelListing.NOTHING), "The engine was restarted without it, so nothing is known about it any more.");
Assert.That(listed.Of(ONE_MACHINE, "gemma3-27b").Context.DefaultTokens, Is.EqualTo(16_384));
});
}
[Test]
public void AMachineWhichHalvedItsWindowIsBelievedTheSecondTimeToo()
{
var listed = new ListedModels();
listed.Report(ONE_MACHINE, [new(MODEL, ContextWindow.Of(32_768))]);
listed.Report(ONE_MACHINE, [new(MODEL, ContextWindow.Of(16_384))]);
Assert.That(listed.Of(ONE_MACHINE, MODEL).Context.DefaultTokens, Is.EqualTo(16_384));
}
[TestCase("Qwen3-32B")]
[TestCase("qwen3-32b")]
public void AModelSomebodyTypedIsStillTheSameModel(string asConfigured)
{
//
// An organization writes the model of a provider into its configuration plugin by hand,
// and the availability check already treats such a name as the same model whatever case it
// was typed in. Being stricter here would leave exactly those people without the numbers.
//
var listed = new ListedModels();
listed.Report(ONE_MACHINE, [new("qwen3-32b", ContextWindow.Of(32_768))]);
Assert.That(listed.Of(ONE_MACHINE, asConfigured).Context.DefaultTokens, Is.EqualTo(32_768));
}
[Test]
public void AnInstanceWithoutAnIdIsNothingToRemember()
{
var listed = new ListedModels();
listed.Report(string.Empty, [new(MODEL, ContextWindow.Of(32_768))]);
Assert.Multiple(() =>
{
Assert.That(listed.Of(string.Empty, MODEL), Is.EqualTo(ModelListing.NOTHING));
Assert.That(listed.Of(ONE_MACHINE, MODEL), Is.EqualTo(ModelListing.NOTHING), "One nameless report does not become every machine's answer.");
});
}
[Test]
public void AModelTheMachineSaidNothingAboutIsNotStored()
{
var listed = new ListedModels();
listed.Report(ONE_MACHINE, [new(MODEL, ContextWindow.UNKNOWN), new(string.Empty, ContextWindow.Of(32_768))]);
Assert.That(listed.Of(ONE_MACHINE, MODEL), Is.EqualTo(ModelListing.NOTHING));
}
[Test]
public void AReportedWindowReplacesTheWholeWindow()
{
var after = new ModelListing(MODEL, ContextWindow.Of(32_768)).ApplyTo(WHAT_THE_RULES_SAY);
Assert.Multiple(() =>
{
Assert.That(after.Context.DefaultTokens, Is.EqualTo(32_768));
Assert.That(after.Context.RaisableToTokens, Is.Null, "What the weights could be raised to is not a number anybody reaches without restarting this engine.");
});
}
[Test]
public void AWindowSaysNothingAboutAnythingElse()
{
var after = new ModelListing(MODEL, ContextWindow.Of(32_768)).ApplyTo(WHAT_THE_RULES_SAY);
Assert.Multiple(() =>
{
Assert.That(after.Capabilities, Is.EqualTo(WHAT_THE_RULES_SAY.Capabilities));
Assert.That(after.Images, Is.EqualTo(WHAT_THE_RULES_SAY.Images));
Assert.That(after.Kind, Is.EqualTo(WHAT_THE_RULES_SAY.Kind));
});
}
[Test]
public void SayingNothingKeepsEverythingTheRulesWorkedOut()
{
Assert.That(ModelListing.NOTHING.ApplyTo(WHAT_THE_RULES_SAY), Is.EqualTo(WHAT_THE_RULES_SAY));
}
}

View File

@ -1,4 +1,5 @@
using AIStudio.Models;
using AIStudio.Models.Live;
using AIStudio.Provider;
using AIStudio.Settings;
using AIStudio.Tests.Models.Corpus;
@ -10,14 +11,31 @@ namespace AIStudio.Tests.Settings;
/// </summary>
/// <remarks>
/// Behind it stand the links of the chain in the order they win: what a person said about their own
/// installation, then what the rules worked out, then what the app assumes. The rules themselves are
/// measured elsewhere, against the whole corpus. What is measured here is the last link -- the one
/// nothing held to account until now, because a model falling through looked exactly like a model
/// nobody had asked about.
/// installation, then what that installation reported about itself, then what the rules worked out,
/// then what the app assumes. The rules themselves are measured elsewhere, against the whole corpus.
/// What is measured here is everything around them -- the last link, which nothing held to account
/// until now because a model falling through looked exactly like a model nobody had asked about,
/// and the order of the three links above it, each of which can speak about the same number.
///
/// What a provider reported lands in the store the app shares, so these tests must not run next to
/// anything else touching it.
/// </remarks>
[TestFixture]
[NonParallelizable]
public sealed class ModelProfileChainTests
{
private const string MACHINE = "33333333-3333-3333-3333-333333333333";
/// <summary>
/// A window no rule would ever state, so that finding it proves where the answer came from.
/// </summary>
private const int WHAT_THE_MACHINE_REPORTS = 33_333;
private static readonly Model MODEL = new("qwen3-32b", null);
[SetUp]
public void ForgetWhatTheMachineSaidBefore() => ListedModels.Shared.Report(MACHINE, []);
[Test]
public void EveryModelLeftToTheDefaultIsAnsweredByTheAssumption()
{
@ -100,4 +118,71 @@ public sealed class ModelProfileChainTests
Assert.That(profile.Has(Capability.TEXT_INPUT), Is.True, "Everything nobody said anything about stays as the rules had it.");
});
}
[Test]
public void WhatThePersonTypedBeatsWhatTheMachineReported()
{
//
// Somebody who types a window has a reason for it, and the app is not in a position to know
// it better -- they may be working around an engine reporting nonsense.
//
ListedModels.Shared.Report(MACHINE, [new(MODEL.Id, ContextWindow.Of(WHAT_THE_MACHINE_REPORTS))]);
var configured = ProviderWith(new() { ContextWindowTokens = 8_192 });
Assert.That(configured.GetModelProfile().Context.DefaultTokens, Is.EqualTo(8_192));
}
[Test]
public void WhatTheMachineReportedBeatsWhatTheRulesWorkedOut()
{
ListedModels.Shared.Report(MACHINE, [new(MODEL.Id, ContextWindow.Of(WHAT_THE_MACHINE_REPORTS))]);
var configured = ProviderWith(null);
Assert.That(configured.GetModelProfile().Context.DefaultTokens, Is.EqualTo(WHAT_THE_MACHINE_REPORTS));
}
[Test]
public void ASilentMachineLeavesTheRulesStanding()
{
var configured = ProviderWith(null);
Assert.That(configured.GetModelProfile().Context, Is.EqualTo(LLMProviders.SELF_HOSTED.GetModelProfile(MODEL).Context));
}
[Test]
public void TheAutomaticAnswerIsWhatHappensWithoutTheSwitches()
{
//
// This is the number the expert dialog offers as its placeholder. Showing the rules there
// while the chat goes by the reported window would tell a person that emptying the field
// gets them something it does not.
//
ListedModels.Shared.Report(MACHINE, [new(MODEL.Id, ContextWindow.Of(WHAT_THE_MACHINE_REPORTS))]);
var configured = ProviderWith(new() { ContextWindowTokens = 8_192 });
Assert.Multiple(() =>
{
Assert.That(configured.GetAutomaticModelProfile().Context.DefaultTokens, Is.EqualTo(WHAT_THE_MACHINE_REPORTS));
Assert.That(configured.GetModelProfile().Context.DefaultTokens, Is.EqualTo(8_192), "What the person typed is still what counts everywhere else.");
});
}
[Test]
public void WhatOneMachineReportsIsNoAnswerForAnother()
{
ListedModels.Shared.Report(MACHINE, [new(MODEL.Id, ContextWindow.Of(WHAT_THE_MACHINE_REPORTS))]);
var somebodyElse = ProviderWith(null) with { Id = "44444444-4444-4444-4444-444444444444" };
Assert.That(somebodyElse.GetModelProfile().Context, Is.EqualTo(LLMProviders.SELF_HOSTED.GetModelProfile(MODEL).Context));
}
/// <summary>
/// A configured self-hosted provider, the way the settings hold one.
/// </summary>
/// <param name="overrides">What the person switched, or nothing when they switched nothing.</param>
/// <returns>The configured provider.</returns>
private static AIStudio.Settings.Provider ProviderWith(ProviderCapabilityOverrides? overrides) => new(1, MACHINE, "A machine of my own", LLMProviders.SELF_HOSTED, MODEL, IsSelfHosted: true)
{
CapabilityOverrides = overrides,
};
}