Linoy Tsaban
e30d3bf544
* initial commit * initial commit * initial commit * initial commit * initial commit * initial commit * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com> * move prompt embeds, pooled embeds outside * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: hlky <hlky@hlky.ac> * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: hlky <hlky@hlky.ac> * fix import * fix import and tokenizer 4, text encoder 4 loading * te * prompt embeds * fix naming * shapes * initial commit to add HiDreamImageLoraLoaderMixin * fix init * add tests * loader * fix model input * add code example to readme * fix default max length of text encoders * prints * nullify training cond in unpatchify for temp fix to incompatible shaping of transformer output during training * smol fix * unpatchify * unpatchify * fix validation * flip pred and loss * fix shift!!! * revert unpatchify changes (for now) * smol fix * Apply style fixes * workaround moe training * workaround moe training * remove prints * to reduce some memory, keep vae in `weight_dtype` same as we have for flux (as it's the same vae) https://github.com/huggingface/diffusers/blob/bbd0c161b55ba2234304f1e6325832dd69c60565/examples/dreambooth/train_dreambooth_lora_flux.py#L1207 * refactor to align with HiDream refactor * refactor to align with HiDream refactor * refactor to align with HiDream refactor * add support for cpu offloading of text encoders * Apply style fixes * adjust lr and rank for train example * fix copies * Apply style fixes * update README * update README * update README * fix license * keep prompt2,3,4 as None in validation * remove reverse ode comment * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Update examples/dreambooth/train_dreambooth_lora_hidream.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * vae offload change * fix text encoder offloading * Apply style fixes * cleaner to_kwargs * fix module name in copied from * add requirements * fix offloading * fix offloading * fix offloading * update transformers version in reqs * try AutoTokenizer * try AutoTokenizer * Apply style fixes * empty commit * Delete tests/lora/test_lora_layers_hidream.py * change tokenizer_4 to load with AutoTokenizer as well * make text_encoder_four and tokenizer_four configurable * save model card * save model card * revert T5 * fix test * remove non diffusers lumina2 conversion --------- Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com> Co-authored-by: hlky <hlky@hlky.ac> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
4.8 KiB
LoRA
LoRA is a fast and lightweight training method that inserts and trains a significantly smaller number of parameters instead of all the model parameters. This produces a smaller file (~100 MBs) and makes it easier to quickly train a model to learn a new concept. LoRA weights are typically loaded into the denoiser, text encoder or both. The denoiser usually corresponds to a UNet ([UNet2DConditionModel], for example) or a Transformer ([SD3Transformer2DModel], for example). There are several classes for loading LoRA weights:
- [
StableDiffusionLoraLoaderMixin] provides functions for loading and unloading, fusing and unfusing, enabling and disabling, and more functions for managing LoRA weights. This class can be used with any model. - [
StableDiffusionXLLoraLoaderMixin] is a Stable Diffusion (SDXL) version of the [StableDiffusionLoraLoaderMixin] class for loading and saving LoRA weights. It can only be used with the SDXL model. - [
SD3LoraLoaderMixin] provides similar functions for Stable Diffusion 3. - [
FluxLoraLoaderMixin] provides similar functions for Flux. - [
CogVideoXLoraLoaderMixin] provides similar functions for CogVideoX. - [
Mochi1LoraLoaderMixin] provides similar functions for Mochi. - [
AuraFlowLoraLoaderMixin] provides similar functions for AuraFlow. - [
LTXVideoLoraLoaderMixin] provides similar functions for LTX-Video. - [
SanaLoraLoaderMixin] provides similar functions for Sana. - [
HunyuanVideoLoraLoaderMixin] provides similar functions for HunyuanVideo. - [
Lumina2LoraLoaderMixin] provides similar functions for Lumina2. - [
WanLoraLoaderMixin] provides similar functions for Wan. - [
CogView4LoraLoaderMixin] provides similar functions for CogView4. - [
AmusedLoraLoaderMixin] is for the [AmusedPipeline]. - [
HiDreamImageLoraLoaderMixin] provides similar functions for HiDream Image - [
LoraBaseMixin] provides a base class with several utility methods to fuse, unfuse, unload, LoRAs and more.
To learn more about how to load LoRA weights, see the LoRA loading guide.
StableDiffusionLoraLoaderMixin
autodoc loaders.lora_pipeline.StableDiffusionLoraLoaderMixin
StableDiffusionXLLoraLoaderMixin
autodoc loaders.lora_pipeline.StableDiffusionXLLoraLoaderMixin
SD3LoraLoaderMixin
autodoc loaders.lora_pipeline.SD3LoraLoaderMixin
FluxLoraLoaderMixin
autodoc loaders.lora_pipeline.FluxLoraLoaderMixin
CogVideoXLoraLoaderMixin
autodoc loaders.lora_pipeline.CogVideoXLoraLoaderMixin
Mochi1LoraLoaderMixin
autodoc loaders.lora_pipeline.Mochi1LoraLoaderMixin
AuraFlowLoraLoaderMixin
autodoc loaders.lora_pipeline.AuraFlowLoraLoaderMixin
LTXVideoLoraLoaderMixin
autodoc loaders.lora_pipeline.LTXVideoLoraLoaderMixin
SanaLoraLoaderMixin
autodoc loaders.lora_pipeline.SanaLoraLoaderMixin
HunyuanVideoLoraLoaderMixin
autodoc loaders.lora_pipeline.HunyuanVideoLoraLoaderMixin
Lumina2LoraLoaderMixin
autodoc loaders.lora_pipeline.Lumina2LoraLoaderMixin
CogView4LoraLoaderMixin
autodoc loaders.lora_pipeline.CogView4LoraLoaderMixin
WanLoraLoaderMixin
autodoc loaders.lora_pipeline.WanLoraLoaderMixin
AmusedLoraLoaderMixin
autodoc loaders.lora_pipeline.AmusedLoraLoaderMixin
HiDreamImageLoraLoaderMixin
autodoc loaders.lora_pipeline.HiDreamImageLoraLoaderMixin
LoraBaseMixin
autodoc loaders.lora_base.LoraBaseMixin