Paul Koudelka
|
c8c1e38475
|
made tokenizer concurrent and threadsafe
|
2026-08-10 21:36:01 +02:00 |
|
Paul Koudelka
|
9ab196c13c
|
ensured prompts fit embedding model
|
2026-08-10 19:42:39 +02:00 |
|
PaulKoudelka
|
1bf9328fb9
|
made the chunking process more robust
|
2026-07-28 16:37:07 +02:00 |
|
Paul Koudelka
|
ea87d79ca9
|
current main is now successfully merged into branch
|
2026-05-27 20:02:43 +02:00 |
|
Paul Koudelka
|
aa554d4a53
|
added possibility to add tokenizer via plugin and refactored earlier changes
|
2026-04-23 16:29:30 +02:00 |
|
Paul Koudelka
|
4a8ab90173
|
cleanup to ensure uniform style of code
|
2026-04-15 10:17:14 +02:00 |
|
PaulKoudelka
|
41573406d5
|
token count now adapts to chosen provider tokenizer
|
2026-04-14 16:27:10 +02:00 |
|
PaulKoudelka
|
0854debc00
|
added functionality to save/move tokenizer file
|
2026-04-10 18:31:20 +02:00 |
|