v0.9.2: Fixes for OOM and other errors in Ludwig 0.9.1

What's Changed

• fix: Handle missing and unexpected keys during LLMEncoder state dict load by @jeffkinnison in #3841
• fix: Add name and description classmethods to IA3Config by @jeffkinnison in #3844
• Improve IA3 long description by @arnavgarg1 in #3845
• fix: Handle missing and unexpected keys during LLMEncoder state dict load, part 2 by @jeffkinnison in #3843
• Update description for max_new_tokens to explain the dynamic setting behavior in our docs by @arnavgarg1 in #3847
• Add default LoRA target modules for Mixtral and Mixtral instruct by @arnavgarg1 in #3852
• QOL: Fail config validation if a user tries to use ECD with a text output feature and an LLM encoder. by @justinxzhao in #3792
• Pin minimum transformers to 4.36 for Mixtral and Phi support by @arnavgarg1 in #3854
• Revert hack that leads to OOM during fine-tuning by @arnavgarg1 in #3858
• Add support for exporting models to Carton by @VivekPanyam in #3797
• [Maintenance] Bump minimum tokenizers to 0.15 by @arnavgarg1 in #3856
• fix: correct typo in FeatureCollection by @dennisrall in #3863
• Convert test main script in algorithm_utils to unit test by @dennisrall in #3864
• Allow hyperopt config to be loaded from a file by @arnavgarg1 in #3865
• fix: unify ludwig training set metadata and hf pad token by @geoffreyangus in #3860
• Add a utility to detect LLM usage in a config by @jeffkinnison in #3869
• Early stop training if model weights have nan or inf tensors by @arnavgarg1 in #3740
• Scrub credentials from model_hyperparameters.json and description.json by @Infernaught in #3866
• [Maintenance] Bump minimum torch version to 2.0.0 by @arnavgarg1 in #3873
• [Maintenance] Fix docker images by pinning ray==2.3.1, daft==0.1.20, unpinning proto, and using torch 2.1.1. by @justinxzhao in #3872
• [BUGFIX] Guard against UnicodeEncodeError when saving validation results in Google Colab environment by @alexsherstinsky in #3875
• Docker image fixes part 2: pin to torch==2.1.0, add dependency for urllib<2 by @arnavgarg1 in #3877
• Add custom prepare_for_trianing logic to ECD model for LLM encoder adapter initialization by @jeffkinnison in #3874
• qol: Fix some lints. by @justinxzhao in #3868
• [Maintenance] Docker Image Fix part 3: fix torchaudio 2.1.0 dependencies by installing libsox-dev and update API by @arnavgarg1 in #3879
• Add streaming support for zero shot inference by @arnavgarg1 in #3878
• [Maintenance] Remove torchdata pin for nightly install by @arnavgarg1 in #3855
• Add per-step token utilization to tensorboard and progress tracker. by @justinxzhao in #3867
• Set use_reentrant to False for gradient checkpointing by @arnavgarg1 in #3882
• [BUGFIX] Pinning torch nightly to January 13, 2024 to avoid AttributeError by @alexsherstinsky in #3885

New Contributors

• @VivekPanyam made their first contribution in #3797

Full Changelog: v0.9.1...v0.9.2