namespace AIStudio.Tools; /// /// Content which a reader held back, together with the token count and the page of exactly that /// content. /// /// /// Readers which assemble a page or a slide from several stream events cannot pass their content /// on right away. Its token count has to travel with it: the count describes the content, not the /// event which happened to arrive at the moment the content was released. Keeping the two together /// is what stops a page from being sized by the text of the page after it. The page number travels /// for the very same reason, and because a number the runtime already stated must not be derived /// from the text again further down the line. /// /// The assembled content. /// The number of tokens of that content, or null when it is unknown. /// The page that content came from, or null when it has none. public readonly record struct ContentStreamPendingContent(string Content, int? TokenCount, int? PageNumber = null) { /// /// Adds up two token counts, where an unknown count makes the sum unknown as well. /// /// /// A partial sum would understate the whole and would let the chunking size a chunk by a part /// of what it holds. Reporting the count as unknown is the honest answer, because the caller /// can still count the content itself. /// /// The first count, or null when it is unknown. /// The second count, or null when it is unknown. /// The sum, or null when either count is unknown. public static int? AddTokenCounts(int? left, int? right) => left is null || right is null ? null : left + right; }