Birchlabs/llama-13b-stepwise-tokenizer

forked from https://huggingface.co/huggyllama/llama-13b/tree/main

This tokenizer supports Birchlabs/llama-13b-stepwise-adapter.

Adds four new tokens for stepwise reasoning:

<|step_start|>
<|step_end|>
<|answer_start|>
<|answer_end|>

See Birchlabs/llama-13b-stepwise-adapter for details of how all the parts should be used together.