Add a note about padded vocab size during training

#43

by mathemakitten - opened Nov 30, 2022

base: refs/heads/main

←

from: refs/pr/43

Discussion Files changed

-0

mathemakitten

Nov 30, 2022

Per our Slack convo — for newcomers to BLOOM it's unclear why the vocab_size in the config.json is 250880 while the model card says 250680. This helps clarify that the effective vocab size is 250680 but the instantiated matrix will be of shape 250880.

Add a note about padded vocab size during trainingd441d2b6

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Cannot merge

This branch has merge conflicts in the following files:

README.md

· Sign up or log in to comment