suzukimain
753f0edff2
* [docs] Replace runwayml/stable-diffusion-v1-5 with Lykon/dreamshaper-8 Updated documentation as runwayml/stable-diffusion-v1-5 has been removed from Huggingface. * Update docs/source/en/using-diffusers/inpaint.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Replace with stable-diffusion-v1-5/stable-diffusion-v1-5 * Update inpaint.md --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
94 lines
3.6 KiB
Markdown
94 lines
3.6 KiB
Markdown
## Textual Inversion fine-tuning example
|
|
|
|
[Textual inversion](https://arxiv.org/abs/2208.01618) is a method to personalize text2image models like stable diffusion on your own images using just 3-5 examples.
|
|
The `textual_inversion.py` script shows how to implement the training procedure and adapt it for stable diffusion.
|
|
|
|
## Running on Colab
|
|
|
|
Colab for training
|
|
[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb)
|
|
|
|
Colab for inference
|
|
[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb)
|
|
|
|
## Running locally with PyTorch
|
|
### Installing the dependencies
|
|
|
|
Before running the scripts, make sure to install the library's training dependencies:
|
|
|
|
**Important**
|
|
|
|
To make sure you can successfully run the latest versions of the example scripts, we highly recommend **installing from source** and keeping the install up to date as we update the example scripts frequently and install some example-specific requirements. To do this, execute the following steps in a new virtual environment:
|
|
```bash
|
|
git clone https://github.com/huggingface/diffusers
|
|
cd diffusers
|
|
pip install .
|
|
```
|
|
|
|
Then cd in the example folder and run
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
And initialize an [🤗Accelerate](https://github.com/huggingface/accelerate/) environment with:
|
|
|
|
```bash
|
|
accelerate config
|
|
```
|
|
|
|
|
|
### Cat toy example
|
|
|
|
You need to accept the model license before downloading or using the weights. In this example we'll use model version `v1-5`, so you'll need to visit [its card](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5), read the license and tick the checkbox if you agree.
|
|
|
|
You have to be a registered user in 🤗 Hugging Face Hub, and you'll also need to use an access token for the code to work. For more information on access tokens, please refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).
|
|
|
|
Run the following command to authenticate your token
|
|
|
|
```bash
|
|
huggingface-cli login
|
|
```
|
|
|
|
If you have already cloned the repo, then you won't need to go through these steps.
|
|
|
|
<br>
|
|
|
|
Now let's get our dataset. For this example we will use some cat images: https://huggingface.co/datasets/diffusers/cat_toy_example .
|
|
|
|
Let's first download it locally:
|
|
|
|
```py
|
|
from huggingface_hub import snapshot_download
|
|
|
|
local_dir = "./cat"
|
|
snapshot_download("diffusers/cat_toy_example", local_dir=local_dir, repo_type="dataset", ignore_patterns=".gitattributes")
|
|
```
|
|
|
|
This will be our training data.
|
|
Now we can launch the training using
|
|
|
|
## Use ONNXRuntime to accelerate training
|
|
In order to leverage onnxruntime to accelerate training, please use textual_inversion.py
|
|
|
|
The command to train on custom data with onnxruntime:
|
|
|
|
```bash
|
|
export MODEL_NAME="stable-diffusion-v1-5/stable-diffusion-v1-5"
|
|
export DATA_DIR="path-to-dir-containing-images"
|
|
|
|
accelerate launch textual_inversion.py \
|
|
--pretrained_model_name_or_path=$MODEL_NAME \
|
|
--train_data_dir=$DATA_DIR \
|
|
--learnable_property="object" \
|
|
--placeholder_token="<cat-toy>" --initializer_token="toy" \
|
|
--resolution=512 \
|
|
--train_batch_size=1 \
|
|
--gradient_accumulation_steps=4 \
|
|
--max_train_steps=3000 \
|
|
--learning_rate=5.0e-04 --scale_lr \
|
|
--lr_scheduler="constant" \
|
|
--lr_warmup_steps=0 \
|
|
--output_dir="textual_inversion_cat"
|
|
```
|
|
|
|
Please contact Prathik Rao (prathikr), Sunghoon Choi (hanbitmyths), Ashwini Khade (askhade), or Peng Wang (pengwa) on github with any questions. |