Welcome to DiffSynth-Studio’s Documentation

Documentation Introduction

• DiffSynth-Studio Documentation
  • Section 1: Getting Started
  • Section 2: Model Details
  • Section 3: Training Framework
  • Section 4: Model Integration
  • Section 5: API Reference
  • Section 6: Diffusion Templates
  • Section 7: Research Guide
  • Section 8: Frequently Asked Questions

Getting Started

• Installing Dependencies
  • GPU/NPU Support
  • Other Installation Issues
• Model Inference
  • Loading Models
  • Starting Inference
  • Loading LoRA
• Inference Acceleration
  • Efficient Attention Mechanisms
  • Multi-GPU Parallel Inference
  • Computation Graph Compilation
• VRAM Management
  • Basic Inference
  • CPU Offload
  • FP8 Quantization
  • Dynamic VRAM Management
  • Disk Offload
  • More Usage Methods
  • Choosing the Best Inference Solution
• Model Quantization
  • Installation
  • Quick Start
  • Supported Quantization Methods
  • QuantizeConfig in Detail
  • Loading Pre-quantized Weights
  • Saving a Quantized Model
  • Mixed Quantization
  • Quantization + LoRA Training
  • Custom Quantization Backends
  • Combining Quantization with VRAM Management
  • Best Practices
• Model Training
  • Script Parameters
  • Preparing Datasets
  • Loading Models
  • Setting Trainable Modules
  • Starting the Training Program
  • Training Considerations
  • Low VRAM Training
• Environment Variables
  • DIFFSYNTH_SKIP_DOWNLOAD
  • DIFFSYNTH_MODEL_BASE_PATH
  • DIFFSYNTH_ATTENTION_IMPLEMENTATION
  • DIFFSYNTH_DISK_MAP_BUFFER_SIZE
  • DIFFSYNTH_DOWNLOAD_SOURCE
• GPU/NPU Support
  • NVIDIA GPU
  • AMD GPU
  • Ascend NPU
• Inference WebUI
  • Launching the Inference WebUI
  • How It Works
  • Usage Tips

Model Details

• FLUX
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Wan
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Qwen-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Qwen-Video-Edit
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• FLUX.2
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Z-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Anima
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• LTX-2
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• ERNIE-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• JoyAI-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• ACE-Step
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• HiDream-O1-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Stable Diffusion
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Stable Diffusion XL
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Image Quality Evaluation Metrics
  • Installation
  • Quick Start
  • Metrics Overview
  • Important Notes
• Ideogram 4
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Krea-2
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
  • License
• Boogu-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• LingBot-Video
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• MiniMax-H3
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• DiffSynth-Music
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• MiniMax-Music3
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• YuE2
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• SenseNova-U1
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Qwen-Image-2.1
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• FLUX.3
  • Installation
  • Model Access
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• MOSS-Image
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
• Fibo 2.0
  • Installation
  • Quick Start
  • Model Overview
  • Model Inference
  • Model Training
  • License

Training Framework

• Basic Principles of Diffusion Models
  • Introduction
  • How is the noise content at each step defined?
  • How is the iterative denoising computation performed?
  • How to train such Diffusion models?
  • What is the architecture of modern Diffusion models?
  • How does this project encapsulate and implement model training?
• Standard Supervised Training
  • Framework Design Concept
• Enabling FP8 Precision in Training
  • Enabling FP8
  • Training Framework Design Concept
• Distillation Accelerated Training
  • What Is Distillation Accelerated Training?
  • Direct Distill
  • DMD2
  • OPD
  • Future Work
• End-to-End Direct Distillation Acceleration (Deprecated)
  • Algorithm Principle
  • Models Trained with Direct Distillation
  • Using Distillation Accelerated Training in the Training Framework
• Two-Stage Split Training
  • Split Training
  • Enabling Split Training
  • Principles of the Computational Graph Splitting Algorithm
• Differential LoRA Training
  • Training Approach
  • Model Effects
  • Using Differential LoRA Training in the Training Framework
  • Framework Design Concept
• Enabling DeepSpeed
  • Configuring Training Parameters
  • DeepSpeed ZeRO Stage 3
• Offload Training
  • What is Offload Training
  • How It Works
  • Usage
  • Integrating Offload Training Module in Other Codebases

Model Integration

• Integrating Model Architecture
  • Step 1: Integrate Model Architecture Code
  • Step 2: Model File Format Conversion
  • Step 3: Writing Model Config
  • Step 4: Verifying Whether the Model Can Be Recognized and Loaded
  • Step 5: Writing Model VRAM Management Scheme
• Building a Pipeline
  • __init__
  • from_pretrained
  • __call__
  • units
  • model_fn
  • Compilation Acceleration
• Fine-Grained VRAM Management Scheme
  • How Much VRAM Does a 20B Model Need?
  • Writing Fine-Grained VRAM Management Scheme
  • Disk Offload
  • Writing Default Configuration
  • Writing Fine-Grained VRAM Management Scheme
  • Disk Offload
  • Writing Default Configuration
• Integrating Model Training
  • Training-Inference Consistent Pipeline Modification
  • Writing Training Scripts
• Integrating a Quantization Backend
  • Framework Structure
  • Step 1: Write the Quantized Linear
  • Step 2: Write the Backend
  • Step 3: Write the Backend Config
  • Step 4: Register the Quantization Method
  • Step 5: Self-Check
  • Integration Checklist

API Reference

• diffsynth.core.attention: Attention Mechanism Implementation
  • Attention Mechanism
  • More Efficient Implementations
  • Developer Guide
  • Best Practices
• diffsynth.core.data: Data Processing Operators and Universal Dataset
  • Data Processing Operators
  • Universal Dataset
• diffsynth.core.gradient: Gradient Checkpointing and Offload
  • Gradient Checkpointing
  • Enabling Gradient Checkpointing and Its Offload
  • Best Practices
• diffsynth.core.loader: Model Download and Loading
  • ModelConfig
  • Model File Loading
  • Model Hash
  • Model Loading
• diffsynth.core.quant: Model Quantization
  • User Interfaces
  • Extension Interface: Custom Backends
  • Verification Tools
• diffsynth.core.vram: VRAM Management
  • Skipping Model Parameter Initialization
  • State Dict Disk Mapping
  • Replacable Modules for VRAM Management

Diffusion Templates

• Diffusion Templates
  • Model Performance Overview
• Diffusion Templates Architecture Details
  • Framework Structure
  • Model Capability Medium
  • Template Model Format
• Template Model Inference
  • Enabling Template Models on Base Model Pipelines
  • CFG Enhancement for Template Models
  • Low VRAM Support
  • Enabling Multiple Template Models
• Template Model Training
  • Continuing Training from Pretrained Models
  • Building New Template Models

Research Guide

• Training Models from Scratch
  • 1. Building Model Architecture
  • 2. Building Pipeline
  • 3. Preparing Dataset
  • 5. Verifying Training Results
• Inference Optimization Techniques
  • 1. Image Quality Quantification
  • 2. Inference-time Scaling Techniques
• ModelScope AIGC Series Course - Controllable Generation Technology
  • Image Structure Control
  • Attribute Value Control
  • Image Editing
  • Style Control
  • Training Controllable Generation Models

FAQ

• Frequently Asked Questions
  • Why doesn’t the training framework support batch size > 1?
  • Why aren’t redundant parameters removed from certain models?
  • Why does FP8 quantization show no acceleration effect?
  • Why doesn’t the training framework support native FP8 precision training?
  • How to dynamically load LoRA models during inference?
  • How to reduce VRAM usage during training?

Indices and tables

• Index

• Module Index

• Search Page