Lack of half precision as datatype for PTQ.

[hkayann]

I might be missing something very simple, so I wanted to ask for clarification regarding the data type support in ptq.evaluate.py.

Currently, the script includes the following argument for data type selection:

parser.add_argument(
    '--dtype', default='float', choices=['float', 'bfloat16'], help='Data type to use')

Have we considered extending this to support FP16 (i.e. torch.half) by modifying the argument as follows?

parser.add_argument(
    '--dtype',
    default='float',
    choices=['float', 'bfloat16', 'half'],
    help='Data type to use (float for FP32, bfloat16, or half for FP16)')

My question is: Is there a specific reason we don’t include FP16 support by default but include BF16?

[Giuseppe5]

I will look into that. It shouldn't be a problem.

Thanks for this, I'll update once the PR is opened/merged

[hkayann]

Many thanks for the quick feedback @Giuseppe5.

I can put PR asap. I have tested and seen no problems at all.

[Giuseppe5]

Feel free to open a PR, but I might ask please to also update the README?

We have the list with all the options, and we would like to that up-to-date. We generally just use the output of the argparse helper.

Thanks!

[Giuseppe5]

Also, please check the CONTRIBUTING.md file about how to sign off your commit

[hkayann]

Hi @Giuseppe5, I've put the PR dev branch, and seems like ReadME was not updated via argparser for a while as I see some other changes there as well.

Fantastic work btw, many thanks for this API. I needed the custom bit-width implementations for FP16, and this API provides that.

[Giuseppe5]

Sometimes we struggle to keep up with our own rules, but we're working hard to improve :)
Thanks for work and the kind words too!

[Giuseppe5]

Thanks for the PR.

With respect to the comment about float16 and custom precision minifloat, it could be worth opening a separate issue with some examples and we can discuss a bit more in details what's going on there.

[hkayann]

You are welcome, @Giuseppe5.

For now, I am going to do a bit more digging, but I already have a workable setup where I save models in FP32 and still simulate custom bit widths via the embedded quantization metadata. Many thanks for the offer.

[Giuseppe5]

#1174 was merged, closing this :)