/

/

karpathy_minbpe

Обзор ДокументацияВойти

/

/

karpathy_minbpe

Аналитика

Безопасность

karpathy_minbpe

Публичный

Описание

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

1.7 MiB

Лицензия

В избранном0

Языки

Python100%

Сообщить о нарушении

Коммиты:

Andrej

Merge pull request #67 from shubham0204/master

23 апр 2024, 01:11

Не верифицирован

23 апр 2024, 01:111acefe8

more work on lecture hehe. this is hard work! not sure an LLM can do this tbh

3 года назад

add small amount of code to write out the gpt4 vocab in the same format as the base class, taking into account the byte shuffle

3 года назад

change the handling of special tokens inside encode of RegexTokenizer, make the default behavior consistent with tiktoken, i.e. the person has to explicitly declare intent to use or not use special tokens in encoding, if they are present

3 года назад

updated .gitignore

3 года назад

3 года назад

add gnp/minbpe-rs as community extensions in README.md

2 года назад

create exercise to rewrite minbpe for learners

3 года назад

fix token count

3 года назад

requirements.txt

create requirements.txt

3 года назад

optimization: allow get_stats to update an existing counts dict. train.py runtime goes from 32s to 22s doing this. ty @gklab for original suggestion in a PR

3 года назад

README.md