obscura/diffusers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,728 Commits 711 Branches 93 Tags
06e78d28c807f3dabe07eb075d79051379e401b6
Commit Graph

4728 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Parag Ekbote 06e78d28c8 Move IP Adapter Scripts to research project (#9960) 
* Move files to research-projects.

* docs: add IP Adapter training instructions

* Delete venv

* Update examples/ip_adapter/tutorial_train_sdxl.py

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Cherry-picked commits and re-moved files
to research_projects.

* make style.

* Update toctree and delete ip_adapter.

* Nit Fix

* Fix nit.

* Fix nit.

* Create training script for single GPU and set
model format to .safetensors

* Add sample inference script and restore _toctree

* Restore toctree.yaml

* fix spacing.

* Update toctree.yaml

---------

Co-authored-by: AMohamedAakhil <a.aakhilmohamed@gmail.com>
Co-authored-by: BootesVoid <78485654+AMohamedAakhil@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:18 +05:30
Dhruv Nair 2f509de896 [CI] Unpin torch<2.5 in CI (#9961) 
* update

* update
 2024-12-23 13:02:18 +05:30
Aryan 7b7c5536f0 Make CogVideoX RoPE implementation consistent (#9963) 
* update cogvideox rope implementation

* apply suggestions from review
 2024-12-23 13:02:18 +05:30
Sayak Paul 19dc28f3ea [LoRA] feat: save_lora_adapter() (#9862) 
* feat: save_lora_adapter.
 2024-12-23 13:02:18 +05:30
Linoy Tsaban b97b4c1444 [advanced flux training] bug fix + reduce memory cost as in #9829 (#9838) 
* memory improvement as done here: https://github.com/huggingface/diffusers/pull/9829

* fix bug

* fix bug

* style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:18 +05:30
Parag Ekbote 18216ac4ad Notebooks for Community Scripts-2 (#9952) 
4 Notebooks for Community Scripts and minor
script improvements.
 2024-12-23 13:02:18 +05:30
Yuxuan.Zhang 8c553e0a3c CogVideoX 1.5 (#9877) 
* CogVideoX1_1PatchEmbed test

* 1360 * 768

* refactor

* make style

* update docs

* add modeling tests for cogvideox 1.5

* update

* make fix-copies

* add ofs embed(for convert)

* add ofs embed(for convert)

* more resolution for cogvideox1.5-5b-i2v

* use even number of latent frames only

* update pipeline implementations

* make style

* set patch_size_t as None by default

* #skip frames 0

* refactor

* make style

* update docs

* fix ofs_embed

* update docs

* invert_scale_latents

* update

* fix

* Update docs/source/en/api/pipelines/cogvideox.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/api/pipelines/cogvideox.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/api/pipelines/cogvideox.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/api/pipelines/cogvideox.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update src/diffusers/models/transformers/cogvideox_transformer_3d.py

* update conversion script

* remove copied from

* fix test

* Update docs/source/en/api/pipelines/cogvideox.md

* Update docs/source/en/api/pipelines/cogvideox.md

* Update docs/source/en/api/pipelines/cogvideox.md

* Update docs/source/en/api/pipelines/cogvideox.md

---------

Co-authored-by: Aryan <aryan@huggingface.co>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:18 +05:30
Grant Sherrick 7f236b4975 Add server example (#9918) 
* Add server example.

* Minor updates to README.

* Add fixes after local testing.

* Apply suggestions from code review

Updates to README from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* More doc updates.

* Maybe this will work to build the docs correctly?

* Fix style issues.

* Fix toc.

* Minor reformatting.

* Move docs to proper loc.

* Fix missing tick.

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Sync docs changes back to README.

* Very minor update to docs to add space.

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:18 +05:30
Parag Ekbote 7b28265fed Fixed Nits in Docs and Example Script (#9940) 
Fixed nits in docs and example script.

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:18 +05:30
ちくわぶ dd9a3f2206 Add all AttnProcessor classes in AttentionProcessor type (#9909) 
Add all AttnProcessor in `AttentionProcessor` type
 2024-12-23 13:02:18 +05:30
_ 398a99644e Correct pipeline_output.py to the type Mochi (#9945) 
Correct pipeline_output.py
 2024-12-23 13:02:18 +05:30
Heavenn 08e5177053 Modify apply_overlay for inpainting with padding_mask_crop (Inpainting area: "Only Masked") (#8793) 
* Modify apply_overlay for inpainting

* style

---------

Co-authored-by: root <root@debian>
Co-authored-by: Álvaro Somoza <asomoza@users.noreply.github.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
 2024-12-23 13:02:18 +05:30
高佳宝 c3698c8225 Update ip_adapter.py (#8882) 
update comments of load_ip_adapter function
 2024-12-23 13:02:18 +05:30
Parag Ekbote d27c692917 Move Wuerstchen Dreambooth to research_projects (#9935) 
update file paths to research_projects folder.

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:18 +05:30
Pakkapon Phongthawee 94007c0697 add depth controlnet sd3 pre-trained checkpoints to docs (#9937) 2024-12-23 13:02:18 +05:30
Sam 0f599ee6b9 Update pipeline_flux_img2img.py (#9928) 
* Update pipeline_flux_img2img.py

Added FromSingleFileMixin to this pipeline loader like the other FLUX pipelines.

* Update pipeline_flux_img2img.py

typo

* modified:   src/diffusers/pipelines/flux/pipeline_flux_img2img.py
 2024-12-23 13:02:18 +05:30
Benjamin Paine 8731574e49 Fix Progress Bar Updates in SD 1.5 PAG Img2Img pipeline (#9925) 
fix progress bar updates in SD 1.5 PAG Img2Img pipeline
 2024-12-23 13:02:18 +05:30
Parag Ekbote 1022f6c2db Notebooks for Community Scripts Examples (#9905) 
* Add Notebooks on Community Scripts
 2024-12-23 13:02:18 +05:30
Eliseu Silva aa71132aaf Feature IP Adapter Xformers Attention Processor (#9881) 
* Feature IP Adapter Xformers Attention Processor: this fix error loading incorrect attention processor when setting Xformers attn after load ip adapter scale, issues: #8863 #8872
 2024-12-23 13:02:18 +05:30
Sayak Paul 291db3e538 Revert "[Flux] reduce explicit device transfers and typecasting in flux." (#9896) 
Revert "[Flux] reduce explicit device transfers and typecasting in flux. (#9817)"

This reverts commit 5588725e8e.
 2024-12-23 13:02:18 +05:30
Sayak Paul dbea93cb14 [Advanced LoRA v1.5] fix: gradient unscaling problem (#7018) 
fix: gradient unscaling problem

Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:18 +05:30
SahilCarterr dd3e554b42 [FIX] Fix TypeError in DreamBooth SDXL when use_dora is False (#9879) 
* fix use_dora

* fix style and quality

* fix use_dora with peft version

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:18 +05:30
Michael Tkachuk a27125d589 Enabling gradient checkpointing in eval() mode (#9878) 
* refactored
 2024-12-23 13:02:17 +05:30
SahilCarterr 55ec25ca08 [fix] Replaced shutil.copy with shutil.copyfile (#9885) 
fix shutil.copy
 2024-12-23 13:02:17 +05:30
Dhruv Nair 72e69ca811 Improve downloads of sharded variants (#9869) 
* update

* update

* update

* update

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul cb7016cca3 [Flux] reduce explicit device transfers and typecasting in flux. (#9817) 
reduce explicit device transfers and typecasting in flux.
 2024-12-23 13:02:17 +05:30
Sayak Paul e92bbf47c0 [Core] introduce controlnet module (#8768) 
* move vae flax module.

* controlnet module.

* prepare for PR.

* revert a commit

* gracefully deprecate controlnet deps.

* fix

* fix doc path

* fix-copies

* fix path

* style

* style

* conflicts

* fix

* fix-copies

* sparsectrl.

* updates

* fix

* updates

* updates

* updates

* fix

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:17 +05:30
SahilCarterr 221d6dbeba Updated _encode_prompt_with_clip and encode_prompt in train_dreamboth_sd3 (#9800) 
* updated encode prompt and clip encod prompt


---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Sookwan Han f546404097 Add new community pipeline for 'Adaptive Mask Inpainting', introduced in [ECCV2024] ComA (#9228) 
* Add new community pipeline for 'Adaptive Mask Inpainting', introduced in [ECCV2024] Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models
 2024-12-23 13:02:17 +05:30
Vahid Askari d1c42c626c Fix: Remove duplicated comma in distributed_inference.md (#9868) 
Fix: Remove duplicated comma

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
SahilCarterr 2861fc925b [Fix] Test of sd3 lora (#9843) 
* fix test

* fix test asser

* fix format

* Update test_lora_layers_sd3.py
 2024-12-23 13:02:17 +05:30
Aryan 939bb9e1d2 [core] Mochi T2V (#9769) 
* update

* udpate

* update transformer

* make style

* fix

* add conversion script

* update

* fix

* update

* fix

* update

* fixes

* make style

* update

* update

* update

* init

* update

* update

* add

* up

* up

* up

* update

* mochi transformer

* remove original implementation

* make style

* update inits

* update conversion script

* docs

* Update src/diffusers/pipelines/mochi/pipeline_mochi.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* Update src/diffusers/pipelines/mochi/pipeline_mochi.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* fix docs

* pipeline fixes

* make style

* invert sigmas in scheduler; fix pipeline

* fix pipeline num_frames

* flip proj and gate in swiglu

* make style

* fix

* make style

* fix tests

* latent mean and std fix

* update

* cherry-pick 1069d210e1

* remove additional sigma already handled by flow match scheduler

* fix

* remove hardcoded value

* replace conv1x1 with linear

* Update src/diffusers/pipelines/mochi/pipeline_mochi.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* framewise decoding and conv_cache

* make style

* Apply suggestions from code review

* mochi vae encoder changes

* rebase correctly

* Update scripts/convert_mochi_to_diffusers.py

* fix tests

* fixes

* make style

* update

* make style

* update

* add framewise and tiled encoding

* make style

* make original vae implementation behaviour the default; note: framewise encoding does not work

* remove framewise encoding implementation due to presence of attn layers

* fight test 1

* fight test 2

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
 2024-12-23 13:02:17 +05:30
RogerSinghChugh a820e3a702 Refac training utils.py (#9815) 
* Refac training utils.py

* quality

---------

Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul c1313968fc [feat] add load_lora_adapter() for compatible models (#9712) 
* add first draft.

* fix

* updates.

* updates.

* updates

* updates

* updates.

* fix-copies

* lora constants.

* add tests

* Apply suggestions from code review

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

* docstrings.

---------

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Dorsa Rohani eb25f54a8b Add Diffusion Policy for Reinforcement Learning (#9824) 
* enable cpu ability

* model creation + comprehensive testing

* training + tests

* all tests working

* remove unneeded files + clarify docs

* update train tests

* update readme.md

* remove data from gitignore

* undo cpu enabled option

* Update README.md

* update readme

* code quality fixes

* diffusion policy example

* update readme

* add pretrained model weights + doc

* add comment

* add documentation

* add docstrings

* update comments

* update readme

* fix code quality

* Update examples/reinforcement_learning/README.md

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update examples/reinforcement_learning/diffusion_policy.py

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* suggestions + safe globals for weights_only=True

* suggestions + safe weights loading

* fix code quality

* reformat file

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Leo Jiang 1a70054007 Reduce Memory Cost in Flux Training (#9829) 
* Improve NPU performance

* Improve NPU performance

* Improve NPU performance

* Improve NPU performance

* [bugfix] bugfix for npu free memory

* [bugfix] bugfix for npu free memory

* [bugfix] bugfix for npu free memory

* Reduce memory cost for flux training process

---------

Co-authored-by: 蒋硕 <jiangshuo9@h-partners.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Boseong Jeon 2a9727bdd1 Handling mixed precision for dreambooth flux lora training (#9565) 
Handling mixed precision and add unwarp

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
ScilenceForest 70bfeacc46 Update train_controlnet_flux.py,Fix size mismatch issue in validation (#9679) 
Update train_controlnet_flux.py

Fix the problem of inconsistency between size of image and size of validation_image which causes np.stack to report error.

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
SahilCarterr ea68d7ccf4 Fixes EMAModel "from_pretrained" method (#9779) 
* fix from_pretrained and added test

* make style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Leo Jiang ad754e6182 NPU Adaption for FLUX (#9751) 
* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

---------

Co-authored-by: 蒋硕 <jiangshuo9@h-partners.com>
 2024-12-23 13:02:17 +05:30
Abhipsha Das c538dea8fc [Model Card] standardize advanced diffusion training sd15 lora (#7613) 
* modelcard generation edit

* add missed tag

* fix param name

* fix var

* change str to dict

* add use_dora check

* use correct tags for lora

* make style && make quality

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:17 +05:30
YiYi Xu 1ef46d9d58 Revert "[LoRA] fix: lora loading when using with a device_mapped mode… (#9823) 
Revert "[LoRA] fix: lora loading when using with a device_mapped model. (#9449)"

This reverts commit 41e4779d98.
 2024-12-23 13:02:17 +05:30
Sayak Paul bb6a324577 [LoRA] fix: lora loading when using with a device_mapped model. (#9449) 
* fix: lora loading when using with a device_mapped model.

* better attibutung

* empty

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>

* minors

* better error messages.

* fix-copies

* add: tests, docs.

* add hardware note.

* quality

* Update docs/source/en/training/distributed_inference.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fixes

* skip properly.

* fixes

---------

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul 18a46d12b2 [CI] add a big GPU marker to run memory-intensive tests separately on CI (#9691) 
* add a marker for big gpu tests

* update

* trigger on PRs temporarily.

* onnx

* fix

* total memory

* fixes

* reduce memory threshold.

* bigger gpu

* empty

* g6e

* Apply suggestions from code review

* address comments.

* fix

* fix

* fix

* fix

* fix

* okay

* further reduce.

* updates

* remove

* updates

* updates

* updates

* updates

* fixes

* fixes

* updates.

* fix

* workflow fixes.

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:17 +05:30
Sayak Paul d143ba6478 [Tests] clean up and refactor gradient checkpointing tests (#9494) 
* check.

* fixes

* fixes

* updates

* fixes

* fixes
 2024-12-23 13:02:17 +05:30
Sayak Paul 2094e7a2b5 [training] use the lr when using 8bit adam. (#9796) 
* use the lr when using 8bit adam.

* remove lr as we pack it in params_to_optimize.

---------

Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul dfbe972dd5 [training] fixes to the quantization training script and add AdEMAMix optimizer as an option (#9806) 
* fixes

* more fixes.
 2024-12-23 13:02:17 +05:30
Sayak Paul bbbd1c0f99 [CI] add new runner for testing (#9699) 
new runner.
 2024-12-23 13:02:17 +05:30
Aryan 63c55c0c21 Allegro VAE fix (#9811) 
fix
 2024-12-23 13:02:17 +05:30
Aryan c24688aab0 [core] Allegro T2V (#9736) 
* update

* refactor transformer part 1

* refactor part 2

* refactor part 3

* make style

* refactor part 4; modeling tests

* make style

* refactor part 5

* refactor part 6

* gradient checkpointing

* pipeline tests (broken atm)

* update

* add coauthor

Co-Authored-By: Huan Yang <hyang@fastmail.com>

* refactor part 7

* add docs

* make style

* add coauthor

Co-Authored-By: YiYi Xu <yixu310@gmail.com>

* make fix-copies

* undo unrelated change

* revert changes to embeddings, normalization, transformer

* refactor part 8

* make style

* refactor part 9

* make style

* fix

* apply suggestions from review

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* update example

* remove attention mask for self-attention

* update

* copied from

* update

* update

---------

Co-authored-by: Huan Yang <hyang@fastmail.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30