obscura/diffusers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,439 Commits 711 Branches 93 Tags
095393a5b8bfc5eb1df3f9d93cd2b2139bbf9d6d
Commit Graph

4439 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
林金鹏 095393a5b8 Support SD3 controlnet inpainting (#9099) 
* add controlnet inpainting pipeline

* [SD3] add controlnet inpaint example

* update example and fix code style

* fix code style with ruff

* Update controlnet_sd3.md : add control inpaint pipeline

* Update docs/source/en/api/pipelines/controlnet_sd3.md

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update docs/source/en/api/pipelines/controlnet_sd3.md

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update docs/source/en/api/pipelines/controlnet_sd3.md

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet_inpainting.py

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update __init__.py : add sd3 control pipelines

* Update pipeline : add new param doc & check input reference.

* fix typo

* make style & make quality

* add unittest for sd3 controlnet inpaint

---------

Co-authored-by: 鹏徙 <linjinpeng.ljp@alibaba-inc.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul 0c78d2af0b Update distributed_inference.md to include a fuller example on distributed inference (#9152) 
* Update distributed_inference.md

* Update docs/source/en/training/distributed_inference.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:15 +05:30
Linoy Tsaban 5fd5487f18 [Flux Dreambooth LoRA] - te bug fixes & updates (#9139) 
* add requirements + fix link to bghira's guide

* text ecnoder training fixes

* text encoder training fixes

* text encoder training fixes

* text encoder training fixes

* style

* add tests

* fix encode_prompt call

* style

* unpack_latents test

* fix lora saving

* remove default val for max_sequenece_length in encode_prompt

* remove default val for max_sequenece_length in encode_prompt

* style

* testing

* style

* testing

* testing

* style

* fix sizing issue

* style

* revert scaling

* style

* style

* scaling test

* style

* scaling test

* remove model pred operation left from pre-conditioning

* remove model pred operation left from pre-conditioning

* fix trainable params

* remove te2 from casting

* transformer to accelerator

* remove prints

* empty commit
 2024-12-23 13:02:15 +05:30
Dhruv Nair fc0f4c5eae Update Video Loading/Export to use imageio (#9094) 
* update

* update

* update

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Dibbla! d5c0d5dbba Errata - fix typo (#9100) 2024-12-23 13:02:15 +05:30
Steven Liu 052edeba21 [docs] Resolve internal links to PEFT (#9144) 
* resolve peft links

* fuse_lora
 2024-12-23 13:02:15 +05:30
Daniel Socek e42d61e021 Fix textual inversion SDXL and add support for 2nd text encoder (#9010) 
* Fix textual inversion SDXL and add support for 2nd text encoder

Signed-off-by: Daniel Socek <daniel.socek@intel.com>

* Fix style/quality of text inv for sdxl

Signed-off-by: Daniel Socek <daniel.socek@intel.com>

---------

Signed-off-by: Daniel Socek <daniel.socek@intel.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Linoy Tsaban 6e9c6a298b [Flux] Dreambooth LoRA training scripts (#9086) 
* initial commit - dreambooth for flux

* update transformer to be FluxTransformer2DModel

* update training loop and validation inference

* fix sd3->flux docs

* add guidance handling, not sure if it makes sense(?)

* inital dreambooth lora commit

* fix text_ids in compute_text_embeddings

* fix imports of static methods

* fix pipeline loading in readme, remove auto1111 docs for now

* fix pipeline loading in readme, remove auto1111 docs for now, remove some irrelevant text_encoder_3 refs

* Update examples/dreambooth/train_dreambooth_flux.py

Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com>

* fix te2 loading and remove te2 refs from text encoder training

* fix tokenizer_2 initialization

* remove text_encoder training refs from lora script (for now)

* try with vae in bfloat16, fix model hook save

* fix tokenization

* fix static imports

* fix CLIP import

* remove text_encoder training refs (for now) from lora script

* fix minor bug in encode_prompt, add guidance def in lora script, ...

* fix unpack_latents args

* fix license in readme

* add "none" to weighting_scheme options for uniform sampling

* style

* adapt model saving - remove text encoder refs

* adapt model loading - remove text encoder refs

* initial commit for readme

* Update examples/dreambooth/train_dreambooth_lora_flux.py

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update examples/dreambooth/train_dreambooth_lora_flux.py

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* fix vae casting

* remove precondition_outputs

* readme

* readme

* style

* readme

* readme

* update weighting scheme default & docs

* style

* add text_encoder training to lora script, change vae_scale_factor value in both

* style

* text encoder training fixes

* style

* update readme

* minor fixes

* fix te params

* fix te params

---------

Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul a60eb14a5c Update README.md to include InstantID (#8770) 
Co-authored-by: YiYi Xu <yixu310@gmail.com>
 2024-12-23 13:02:15 +05:30
Monjoy Narayan Choudhury a46c3d7f90 Add Differential Diffusion to HunyuanDiT. (#9040) 
* Add Differential Pipeline.

* Fix Styling Issue using ruff -fix

* Add details to Contributing.md

* Revert "Fix Styling Issue using ruff -fix"

This reverts commit d347de162d.

* Revert "Revert "Fix Styling Issue using ruff -fix""

This reverts commit ce7c3ff216.

* Revert README changes

* Restore README.md

* Update README.md

* Resolved Comments:

* Fix Readme based on review

* Fix formatting after make style

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
David Steinberg d8d8e86924 Fix a dead link (#9116) 
Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
sayantan sadhu 23e204790d fix for lr scheduler in distributed training (#9103) 
* fix for lr scheduler in distributed training

* Fixed the recalculation of the total training step section

* Fixed lint error

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Steven Liu c690fc2635 [docs] Organize model toctree (#9118) 
* toctree

* fix
 2024-12-23 13:02:15 +05:30
zR dbf5d348e6 Add CogVideoX text-to-video generation model (#9082) 
* add CogVideoX

---------

Co-authored-by: Aryan <aryan@huggingface.co>
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:15 +05:30
Dhruv Nair 871d32eecb Freenoise change vae_batch_size to decode_chunk_size (#9110) 
* update

* update
 2024-12-23 13:02:15 +05:30
Aryan fbb294e8e0 [feat] allow sparsectrl to be loaded from single file (#9073) 
* allow sparsectrl to be loaded with single file

* update

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:15 +05:30
latentCall145 f771be1d7b Flux fp16 inference fix (#9097) 
* clipping for fp16

* fix typo

* added fp16 inference to docs

* fix docs typo

* include link for fp16 investigation

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Aryan b6fac9d573 [core] FreeNoise (#8948) 
* initial work draft for freenoise; needs massive cleanup

* fix freeinit bug

* add animatediff controlnet implementation

* revert attention changes

* add freenoise

* remove old helper functions

* add decode batch size param to all pipelines

* make style

* fix copied from comments

* make fix-copies

* make style

* copy animatediff controlnet implementation from #8972

* add experimental support for num_frames not perfectly fitting context length, ocntext stride

* make unet motion model lora work again based on #8995

* copy load video utils from #8972

* copied from AnimateDiff::prepare_latents

* address the case where last batch of frames does not match length of indices in prepare latents

* decode_batch_size->vae_batch_size; batch vae encode support in animatediff vid2vid

* revert sparsectrl and sdxl freenoise changes

* revert pia

* add freenoise tests

* make fix-copies

* improve docstrings

* add freenoise tests to animatediff controlnet

* update tests

* Update src/diffusers/models/unets/unet_motion_model.py

* add freenoise to animatediff pag

* address review comments

* make style

* update tests

* make fix-copies

* fix error message

* remove copied from comment

* fix imports in tests

* update

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul f35bdb6a03 fix train_dreambooth_lora_sd3.py loading hook (#9107) 2024-12-23 13:02:15 +05:30
Álvaro Somoza 3510d0ef5e [Kolors] Add PAG (#8934) 
* txt2img pag added

* autopipe added, fixed case

* style

* apply suggestions

* added fast tests, added todo tests

* revert dummy objects for kolors

* fix pag dummies

* fix test imports

* update pag tests

* add kolor pag to docs

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Dhruv Nair 47874e837d [Single File] Add single file support for Flux Transformer (#9083) 
* update

* update

* update

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Marc Sun 8bdafc6fc4 Fix loading sharded checkpoints when we have variants (#9061) 
* Fix loading sharded checkpoint when we have variant

* add test

* remote print

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Ahn Donghoon (안동훈 / suno) f25823781d add PAG support for Stable Diffusion 3 (#8861) 
add pag sd3


---------

Co-authored-by: HyoungwonCho <jhw9811@korea.ac.kr>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: crepejung00 <jaewoojung00@naver.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
Dhruv Nair 4a91ee80c2 [Docs] Add community projects section to docs (#9013) 
* update

* update

* update
 2024-12-23 13:02:15 +05:30
Dhruv Nair faa0826328 update 2024-12-23 13:02:15 +05:30
Vinh H. Pham 81d58eb03e [Tests] Improve transformers model test suite coverage - Hunyuan DiT (#8916) 
* add hunyuan model test

* apply suggestions

* reduce dims further

* reduce dims further

* run make style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Aryan f33b233789 [bug] remove unreachable norm_type=ada_norm_continuous from norm3 initialization conditions (#9006) 
remove ada_norm_continuous from norm3 list

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul 1616a6372e [Core] add QKV fusion to AuraFlow and PixArt Sigma (#8952) 
* add fusion support to pixart

* add to auraflow.

* add tests

* apply review feedback.

* add back args and kwargs

* style
 2024-12-23 13:02:15 +05:30
Tolga Cangöz 51f45da25f Update CLIPFeatureExtractor to CLIPImageProcessor and DPTFeatureExtractor to DPTImageProcessor (#9002) 
* fix: update `CLIPFeatureExtractor` to `CLIPImageProcessor` in codebase

* `make style && make quality`

* Update `DPTFeatureExtractor` to `DPTImageProcessor` in codebase

* `make style`

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
omahs 49544cc1fb Fix typos (#9077) 
* fix typo
 2024-12-23 13:02:15 +05:30
YiYi Xu 627fd46ab8 add sentencepiece as a soft dependency (#9065) 
* add sentencepiece as  soft dependency for kolors

* up

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul 8881fc9872 [Docs] add stable cascade unet doc. (#9066) 
* add stable cascade unet doc.

* fix path
 2024-12-23 13:02:15 +05:30
Aryan 9dbffc8c60 PAG variant for HunyuanDiT, PAG refactor (#8936) 
* copy hunyuandit pipeline

* pag variant of hunyuan dit

* add tests

* update docs

* make style

* make fix-copies

* Update src/diffusers/pipelines/pag/pag_utils.py

* remove incorrect copied from

* remove pag hunyuan attn procs to resolve conflicts

* add pag attn procs again

* new implementation for pag_utils

* revert pag changes

* add pag refactor back; update pixart sigma

* update pixart pag tests

* apply suggestions from review

Co-Authored-By: yixu310@gmail.com

* make style

* update docs, fix tests

* fix tests

* fix test_components_function since list not accepted as valid __init__ param

* apply patch to fix broken tests

Co-Authored-By: Sayak Paul <spsayakpaul@gmail.com>

* make style

* fix hunyuan tests

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Vinh H. Pham fa55429b04 [Tests] Improve transformers model test suite coverage - Latte (#8919) 
* add LatteTransformer3DModel model test

* change patch_size to 1

* reduce req len

* reduce channel dims

* increase num_layers

* reduce dims further

* run make style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
Sayak Paul 499b7d6dde [FLUX] support LoRA (#9057) 
* feat: lora support for Flux.

add tests

fix imports

major fixes.

* fix

fixes

final fixes?

* fix

* remove is_peft_available.
 2024-12-23 13:02:15 +05:30
Aryan 44a4886771 [refactor] apply qk norm in attention processors (#9071) 
* apply qk norm in attention processors

* revert attention processor

* qk-norm in only attention proc 2.0 and fused variant
 2024-12-23 13:02:15 +05:30
psychedelicious 01829c699a type get_attention_scores as optional in get_attention_scores (#9075) 
`None` is valid for `get_attention_scores`, should be typed as such
 2024-12-23 13:02:15 +05:30
asfiyab-nvidia fce5debd8c Update TensorRT txt2img and inpaint community pipelines (#9037) 
* Update TensorRT txt2img and inpaint community pipelines

Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* update tensorrt install instructions

Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

---------

Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul 0db81141b9 [Flux] minor documentation fixes for flux. (#9048) 
* minor documentation fixes for flux.

* clipskip

* add gist
 2024-12-23 13:02:15 +05:30
Philip Rideout 9c02c40a13 Fix grammar mistake. (#9072) 2024-12-23 13:02:15 +05:30
Aryan 6e5b374630 [refactor] create modeling blocks specific to AnimateDiff (#8979) 
* animatediff specific transformer model

* make style

* make fix-copies

* move blocks to unet motion model

* make style

* remove dummy object

* fix incorrectly passed param causing test failures

* rename model and output class

* fix sparsectrl imports

* remove todo comments

* remove temporal double self attn param from controlnet sparsectrl

* add deprecated versions of blocks

* apply suggestions from review

* update

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:15 +05:30
Tolga Cangöz c6ac793955 Errata: Fix typos & \s+$ (#9008) 
* Fix typos

* chore: Fix typos

* chore: Update README.md for promptdiffusion example

* Trim trailing white spaces

* Fix a typo

* update number

* chore: update number

* Trim trailing white space

* Update README.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update README.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:14 +05:30
Frank (Haofan) Wang 9769fae865 Update transformer_flux.py (#9060) 2024-12-23 13:02:14 +05:30
Dhruv Nair a615e524e5 Fix Nightly Deps (#9036) 
update
 2024-12-23 13:02:14 +05:30
Sayak Paul fdce85ccf1 [Flux] allow tests to run (#9050) 
* fix tests

* fix

* float64 skip

* remove sample_size.

* remove

* remove more

* default_sample_size.

* credit black forest for flux model.

* skip

* fix: tests

* remove OriginalModelMixin

* add transformer model test

* add: transformer model tests
 2024-12-23 13:02:14 +05:30
Sayak Paul c8a236ba5c [Core] Add PAG support for PixArtSigma (#8921) 
* feat: add pixart sigma pag.

* inits.

* fixes

* fix

* remove print.

* copy paste methods to the pixart pag mixin

* fix-copies

* add documentation.

* add tests.

* remove correction file.

* remove pag_applied_layers

* empty
 2024-12-23 13:02:14 +05:30
Sayak Paul 7739beb740 Flux pipeline (#9043) 
add flux!

Signed-off-by: Adrien <adrien@huggingface.co>
Co-authored-by: Adrien <adrien.69740@gmail.com>
Co-authored-by: Anatoly Belikov <abelikov@singularitynet.io>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
 2024-12-23 13:02:14 +05:30
Aryan 6f90bc1a63 [docs] fix pia example (#9015) 
fix pia example docstring
 2024-12-23 13:02:14 +05:30
YiYi Xu ceeaf1d469 fix load sharded checkpoint from a subfolder (local path) (#8913) 
fix

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:14 +05:30
Dhruv Nair 232a967613 Updates deps for pipeline test fetcher (#9033) 
update
 2024-12-23 13:02:14 +05:30