estelleafl
666743302f
* fixed typo * updated doc to be consistent in naming * make style/quality --------- Co-authored-by: Aflalo <estellea@isl-iam1.rr.intel.com>
56 lines
3.1 KiB
Plaintext
56 lines
3.1 KiB
Plaintext
<!--Copyright 2023 The Intel Labs Team Authors and HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
-->
|
|
|
|
# LDM3D
|
|
|
|
LDM3D was proposed in [LDM3D: Latent Diffusion Model for 3D](https://arxiv.org/abs/2305.10853) by Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, Vasudev Lal
|
|
The abstract of the paper is the following:
|
|
|
|
*This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences. A short video summarizing the approach can be found at [this url](https://t.ly/tdi2).*
|
|
|
|
|
|
*Overview*:
|
|
|
|
| Pipeline | Tasks | Colab | Demo
|
|
|---|---|:---:|:---:|
|
|
| [pipeline_stable_diffusion_ldm3d.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py) | *Text-to-Image Generation* | - | -
|
|
|
|
## Tips
|
|
|
|
- LDM3D generates both an image and a depth map from a given text prompt, compared to the existing txt-to-img diffusion models such as [Stable Diffusion](./stable_diffusion/overview) that generates only an image.
|
|
- With almost the same number of parameters, LDM3D achieves to create a latent space that can compress both the RGB images and the depth maps.
|
|
|
|
|
|
Running LDM3D is straighforward with the [`StableDiffusionLDM3DPipeline`]:
|
|
|
|
```python
|
|
>>> from diffusers import StableDiffusionLDM3DPipeline
|
|
|
|
>>> pipe = StableDiffusionLDM3DPipeline.from_pretrained("Intel/ldm3d")
|
|
prompt ="A picture of some lemons on a table"
|
|
output = pipe(prompt)
|
|
rgb_image, depth_image = output.rgb, output.depth
|
|
rgb_image[0].save("lemons_ldm3d_rgb.jpg")
|
|
depth_image[0].save("lemons_ldm3d_depth.png")
|
|
```
|
|
|
|
|
|
## StableDiffusionPipelineOutput
|
|
[[autodoc]] pipelines.stable_diffusion.StableDiffusionPipelineOutput
|
|
- all
|
|
- __call__
|
|
|
|
## StableDiffusionLDM3DPipeline
|
|
[[autodoc]] StableDiffusionLDM3DPipeline
|
|
- all
|
|
- __call__
|