BlobCtrl

😃 This repository contains the implementation of "BlobCtrl: Taming Controllable Blob for Element-level Image Editing".

Keywords: Image Generation, Image Editing, Diffusion Models, Element-level

TL;DR: BlobCtrl enables precise, user-friendly multi-round element-level visual manipulation.
Main Features: 🦉 Element-level Add / Remove / Move / Replace / Enlarge / Shrink.

Authors:
Yaowei Li¹, Lingen Li³, Zhaoyang Zhang^{2 ‡}, Xiaoyu Li², Guangzhi Wang², Hongxiang Li¹, Xiaodong Cun², Ying Shan², Yuexian Zou^{1 ✉
1}Peking University ²ARC Lab, Tencent PCG ³The Chinese University of Hong Kong ^‡Project Lead ^✉Corresponding Author

🌐Project Page | 📜Arxiv | 📹Video | 🤗HuggingFace Demo | 🤗HuggingFace Model | 🤗HuggingFace Data

compress_BlobCtrl-video.mp4

Youtube Introduction Video: Youtube.

📖 Table of Contents

• BlobCtrl
  • 🔥 Update Logs
  • 🛠️ Method Overview
  • 🚀 Getting Started
  • 🏃🏼 Running Scripts
  • 🤝🏼 Cite Us
  • 💖 Acknowledgement
  • ❓ Contact
  • 🌟 Star History

🔥 Update Logs

• [14/11/2025] Update and polish the repo.
• [20/03/2025] Release the inference code.
• [17/03/2025] Release the paper, webpage and gradio demo.

🛠️ Method Overview

BlobCtrl is a framework for element-level image editing based on a probabilistic blob representation. By treating blobs as visual primitives, BlobCtrl disentangles layout from appearance, enabling fine-grained and controllable object-level manipulations such as addition, removal, scaling, and replacement.

🚀 Getting Started

Environment Requirement 🌍

BlobCtrl has been implemented and tested on CUDA121, Pytorch 2.2.0, python 3.10.15.

Clone the repo:

git clone [email protected]:TencentARC/BlobCtrl.git

We recommend you first use conda to create virtual environment, and install needed libraries. For example:

conda create -n blobctrl python=3.10.15 -y
conda activate blobctrl
python -m pip install --upgrade pip
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121
pip install xformers torch==2.2.0 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Then, you can install diffusers and blobctrl (implemented in this repo) with:

pip install -e .
pip install -e diffusers

Download Model Checkpoints 💾

Download the corresponding checkpoints of BlobCtrl.

sh scripts/download_models.sh

The ckpt folder contains

• Our provided BlobCtrl checkpoints (UNet LoRA + BlobNet).
• Pretrained SD-v1.5 checkpoint.
• Pretrained DINOv2 checkpoint.
• Pretrained SAM checkpoint.

The checkpoint structure should be like:

|-- models
    |-- blobnet
        |-- config.json
        |-- diffusion_pytorch_model.safetensors
    |-- dinov2-large
        |-- config.json
        |-- model.safetensors
        ...
    |-- sam
        |-- sam_vit_h_4b8939.pth
    |-- unet_lora
        |-- pytorch_lora_weights.safetensors

🏃🏼 Running Scripts

BlobCtrl demo 🤗

You can run the demo using the script:

sh scripts/run_app.sh

BlobCtrl Inference 🌠

You can run the inference using the script:

sh scripts/inference.sh

🤝🏼 Cite Us

@article{li2025blobctrl,
  title={BlobCtrl: Taming Controllable Blob for Element-level Image Editing},
  author={Li, Yaowei and Li, Lingen and Zhang, Zhaoyang and Li, Xiaoyu and Wang, Guangzhi and Li, Hongxiang and Cun, Xiaodong and Shan, Ying and Zou, Yuexian},
  journal={arXiv preprint arXiv:2503.13434},
  year={2025}
}

💖 Acknowledgement

Our implementation builds upon the diffusers library. We extend our sincere gratitude to all the contributors of the diffusers project!

We also acknowledge the BlobGAN project for providing valuable insights and inspiration for our blob-based representation approach.

❓ Contact

For any question, feel free to email [email protected].

🌟 Star History