main
gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing (#150275)
gh-151461: Fix encoding-related exception handling in file tokenizer (GH-151462)
gh-145234: Normalize decoded CR in string tokenizer (#145281)
gh-104169: Refactor tokenizer into lexer and wrappers (#110684)