openaccess-ai-collective

/

phi-platypus-qlora

like 2

Follow

Open Access AI Collective 168

Seems I'm getting the same error with this model too... adapted gist from here from @TheBloke

@teknium seems to have figured it out? perhaps this issue is some bug in config of qlora?

RuntimeError: Error(s) in loading state_dict for MixFormerSequentialForCausalLM: size mismatch for layers.25.linear.lora_B.default.weight: copying a param with shape torch.Size([50304, 64]) from checkpoint, the shape in current model is torch.Size([51200, 64]).
note: only occurs on the final layer!?....