Generation problem after / before instruction fine-tuning

[hxssgaa]

Environment: 6xA6000 48GB with Ubuntu 22.04, Pytorch 1.13.0

I ran into a generation problem after following your instruction to convert LLaMA-7B weight using your attached script.

I simply used the following script to directly test generation after loading the converted LLaMA-7B model:

tokenizer.batch_decode(model.generate(**tokenizer('I want to ', return_tensors="pt")))

The output of above code is:

'I want to acoérницschutzirectorioieckťDEX threshold släktetolasĭüttpiel'

The problem happens both before and after following your README for instruction fine-tuning. (note that I see the loss is decreasing over time during the fine-tuning stage which seems OK)

I have no problem running generation using original code from LLaMA, may I know your generation script so that I can test what caused the problem? Thanks.

[puyuanliu]

I have the same issue.

[Xuan-ZW]

I have a same issue, and the saved model has a 26GB

[puyuanliu]

@Xuan-ZW Do you see any errors during training?

[helloeve]

@puyuanliu do you have the full code-snippet for loading the model and conducting the generation? I suspect there might be something weird happened with loading weights from the fine-tuned model.

[puyuanliu]

@helloeve Yeah I do have. Mentioned in #48 (comment)

[helloeve]

@puyuanliu I was able to conduct prediction without an issue. The only difference comparing to your method is that I didn't use model = model.half()

[puyuanliu]

@helloeve Thanks a lot! I found the issue was the model saving. For some reason, the script runs into a CUDA OOM error when saving the model, and the actually saved model is corrupted. I fixed the issue and my model is now working.

[mlaprise]

I have the same issue. @puyuanliu did figured out why the OOM error is happening ?

[puyuanliu]

I mentioned the solution in #81 (comment) Martin Laprise ***@***.***> 于 2023年3月17日周五 18:19写道：

…

I have the same issue. @puyuanliu <https://github.com/puyuanliu> did figured out why the OOM error is happening ? — Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKN5FWVLLNC3H4IKLZRCMU3W4T5SXANCNFSM6AAAAAAV43ZTFU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hxssgaa]

I manage to resolve the issue finally, I fonnd the issue to be the inconsistancy script I used in a newer version of llama conversion scripts which is incompatible of the current version of LLaMA. After reverting to the correct commit 68d640f7c368bcaaaecfc678f11908ebbd3d6176 and redo the conversion, the issue resolved.

[Hins]

@puyuanliu when generation I loaded model by 'model = model.to("cuda")', I had 8 A100 gpus, but encountered OOM, it seemed pytorch loading the model in 1 A100 gpu, how do you fix this issue?