Workaround for error when starting TheBloke/WizardLM-30B-Uncensored-GPTQ

[aviav]

Hi! First of all, great job on this! Running the 13B model with your repo and LangChain works great for me!

There's now a 30B GPTQ model, and I can start it in Oobabooga by just not specifying --groupsize 128.

As a note for anybody trying the 30B model with this repo/workaround: Commenting out this line allows it to start.

For anybody searching the error message, this is what I get:

Traceback (most recent call last):
  File "/home/tf/abhinand5/gptq_for_langchain/demo.py", line 84, in <module>
    llm_chain = create_gptq_llm_chain(mem_buf)
  File "/home/tf/abhinand5/gptq_for_langchain/demo.py", line 28, in create_gptq_llm_chain
    model, tokenizer = load_quantized_model(MODEL_NAME, args=AttributeDict(args))
  File "/home/tf/abhinand5/gptq_for_langchain/wizardlm_langchain/model.py", line 107, in load_quantized_model
    model = _load_quant(
  File "/home/tf/abhinand5/gptq_for_langchain/wizardlm_langchain/model.py", line 83, in _load_quant
    model.load_state_dict(safe_load(checkpoint), strict=False)
  File "/home/tf/.conda/envs/wizardlm_langchain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
        size mismatch for model.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
        size mismatch for model.layers.0.self_attn.k_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
        size mismatch for model.layers.0.self_attn.o_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
        size mismatch for model.layers.0.self_attn.o_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
        size mismatch for model.layers.0.self_attn.q_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
        size mismatch for model.layers.0.self_attn.q_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
...
...
...
        size mismatch for model.layers.59.mlp.gate_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]).
        size mismatch for model.layers.59.mlp.up_proj.qzeros: copying a param with shape torch.Size([1, 2240]) from checkpoint, the shape in current model is torch.Size([52, 2240]).
        size mismatch for model.layers.59.mlp.up_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]).

[abhinand5]

@aviav Great to hear that you're able to run models successfully. Thanks for letting me know about the groupsize issue in larger models, I'll try to add more models in the coming days.

PRs will be welcomed!