[FR]: Olive Quantization

[miaoqiz]

Proposal Summary

Hi,

Thanks very much for the tool and support! I appreciate it!

Two questions with regards to "olive quantize":

1. Is there a way to use it to quantize adapter? I understand "olive convert-adapters" can process "int4" quantization, how about other precisions (e.g., uint4,uint8,uint16,fp4,fp8,fp16,nf4)
2. There seems to be an incorrect argument of "-a, --adapter_path" with "olive quantize" in the documentation, ""olive quantize -h" does not show the argument.

Thanks!

What component(s) does this request affect?

• OliveModels
• OliveSystems
• OliveEvaluator
• Metrics
• Engine
• Passes
• Other

[devang-ml]

The olive quantize command quantizes the entire model or skips adapter weights in case of ONNX model. It does not quantize adapters only.

What is your scenario? What is your master weights precision and intended adapter weights precision?

[miaoqiz]

Thanks very much! I appreciate the information.