Commit Graph

8 Commits

Author SHA1 Message Date
Paul Koudelka
c8c1e38475 made tokenizer concurrent and threadsafe 2026-08-10 21:36:01 +02:00
Paul Koudelka
9ab196c13c ensured prompts fit embedding model 2026-08-10 19:42:39 +02:00
PaulKoudelka
1bf9328fb9 made the chunking process more robust 2026-07-28 16:37:07 +02:00
Paul Koudelka
ea87d79ca9 current main is now successfully merged into branch 2026-05-27 20:02:43 +02:00
Paul Koudelka
aa554d4a53 added possibility to add tokenizer via plugin and refactored earlier changes 2026-04-23 16:29:30 +02:00
Paul Koudelka
4a8ab90173 cleanup to ensure uniform style of code 2026-04-15 10:17:14 +02:00
PaulKoudelka
41573406d5 token count now adapts to chosen provider tokenizer 2026-04-14 16:27:10 +02:00
PaulKoudelka
0854debc00 added functionality to save/move tokenizer file 2026-04-10 18:31:20 +02:00