TheBloke

/

TinyLlama-1.1B-Chat-v0.3-AWQ

like 4

As a llama tokenizer, trie model was trained with a bug. Following a deprecation cycle and changes in transformers v5, we no longer have legacy flags to fix this so we need to update the file!

The only change is:

        self._tokenizer.pre_tokenizer = None
        self._tokenizer.normalizer = normalizers.Sequence(
            [normalizers.Prepend(prepend="▁"), normalizers.Replace(pattern=" ", content="▁")]
        )

in LlamaTokenizer, then save_pretrained