Add support for Nemotron-CC quality classifiers

[sarahyurick]

Awaiting Hugging Face releases.

TODO:

• tests/test_classifiers.py
• README.md
• tutorials/distributed_data_classification/README.md
• Create notebook tutorial(s)
• docs/user-guide/api/classifiers.rst
• docs/user-guide/cpuvsgpu.rst
• docs/user-guide/distributeddataclassification.rst
• examples/classifiers/README.md
• Example script(s)
• nemo_curator/classifiers/init.py
• Integration script(s)
• nemo_curator/scripts/classifiers/README.md
• Classifier script(s)
• pyproject.toml

[VibhuJawa]

Did an initial review, mostly looks good.

Have requested changes around naming, class structure etc.

[VibhuJawa]

Interesting that we read FINEWEB_MIXTRAL_IDENTIFIER , FINEWEB_NEMOTRON_IDENTIFIER in bfloat16 now.

Do you know why we cant/dont do it for EDU classifier ?

Can you add a comment stating the reason for this

[sarahyurick]

Not sure, this was how it was in the script that the Nemotron-CC developers used.

[VibhuJawa]

Can we do a quick benchmark with autocast. I think this should happen automatically when we use torch.autocast , which we use/or should use.

If the results on a dataset line up (for both accuracy and throughput) we can probably skip this fork.

[sarahyurick]

Just ran, the results look a bit different without the bfloat16 but still pretty similar to the ones from before. I have removed torch_dtype=torch.bfloat16 for now.

[VibhuJawa]

Minor nits around autocast and tokenizer types

[VibhuJawa]

Can we do a quick benchmark with autocast. I think this should happen automatically when we use torch.autocast , which we use/or should use.

If the results on a dataset line up (for both accuracy and throughput) we can probably skip this fork.

[VibhuJawa]

LGTM