master
autoparser: support case of JSON_NATIVE with per-call markers (test case: Reka-Edge) (#21892)
gguf : new file format with flexible meta data (beta) (#2398)
Work on the BPE tokenizer (#3252)
Add more tokenizer tests (#3742)
llama : fix BPE pre-tokenization (#6920)
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
command-r : add BPE pre-tokenization (#7063)
vocab: add gemma4 tokenizer tests, fix edge case (#21534)
tests : improve UGM tokenizer test coverage (#13773)
Per token attributes (#7685)
llama : add BPE pre-tokenization for Qwen2 (#7114)
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)