Z-Image Turbo in ComfyUI: ControlNet, Qwen VL & LoRA Training
Z-Image Turbo (ZIT) is a fast 6-billion-parameter image model that produces both realistic and illustrative outputs. This guide covers model setup, ControlNet for consistent compositions, Qwen VL for detailed prompt generation, upscaling techniques, and online LoRA training for characters and styles—all within ComfyUI.
Special Offer - $5 Credit Included!
When you sign up for RunPod using our affiliate link, you'll receive a $5 credit that can be used to generate up to 9,000 images and 300 videos. This gives you plenty of resources to explore ComfyUI and AI image/video generation without any upfront cost!
What You'll Learn
Prerequisites
Before starting, ensure you have:
- ComfyUI installed – see our ComfyUI Installation Guide if needed
- ComfyUI Manager installed for custom node management
- NVIDIA GPU with at least 12 GB VRAM (FP8 recommended for lower VRAM; BF16 for 16 GB+)
- Fal account for online LoRA training (credit card required, training costs ~$2.26)
- Basic familiarity with ComfyUI workflows and node connections
GPU Requirements
Z-Image Turbo runs on GPUs with 12 GB VRAM or less using FP8 format. For 16 GB or more, BF16 is recommended. The all-in-one checkpoint bundles model, text encoder, and VAE for simpler setup.
Step-by-Step Process
Step 1: Download and Place Model Files
Choose between the split workflow or the all-in-one checkpoint approach.
- Split workflow: Download the Z-Image diffusion model and place it in
ComfyUI/models/diffusion_models. Download the Qwen text encoder and place it inComfyUI/models/text_encoders. You also need a VAE – if you already work with Flux, you may already have it. - All-in-one checkpoint: Download the bundled file and place it in
ComfyUI/models/checkpoints. This includes the model, text encoder, and VAE. - Check workflow notes for any required custom nodes. Install them via ComfyUI Manager if needed.
Model Format Recommendations
- FP8: Smaller file size, recommended for GPUs with 12 GB VRAM or less. May introduce small quality errors on some systems.
- BF16: Recommended for 16 GB VRAM or more. On high-VRAM systems, BF16 and FP8 produce effectively the same images and speed.
Pro Tip
If you already have a Flux VAE, you can reuse it – no need to download another. The split workflow gives you more control over each component.
Custom Nodes
Some workflows require specific custom nodes. Check the workflow’s notes or metadata for a list. Use ComfyUI Manager’s “Install Missing Custom Nodes” feature to automate this.
Step 2: Configure Model Loading and Sampling Settings
Correct node configuration is critical for Z-Image to interpret prompts properly.
- Load CLIP node: Select the Lumina 2 type so the Qwen text encoder is interpreted correctly.
- Model shift: The workflow commonly uses model shift 3.
- Sampling settings:
- Steps: Do not go above 9 steps – higher steps can degrade image quality.
- Sampler and scheduler: Different combinations affect sharpness and skin rendering. Test with a fixed seed.
- CFG: Use CFG 1 when no negative prompt is used – it keeps generation fast. CFG 2 or 3 increases saturation and contrast but slows generation. Lower CFG can look desaturated.
- First run: Expect the first generation to be slower. Subsequent runs typically take 6–7 seconds on a high-VRAM system.
Why CFG 1 Works Best
Z-Image is designed to work without a negative prompt. CFG 1 gives the fastest results and avoids over-saturation. If you need more contrast, try CFG 2 or 3, but expect slower generation.
Pro Tip
Always keep the seed fixed while testing settings. This lets you compare the effect of each parameter change accurately.
Step Count Limit
Going above 9 steps can actually degrade the image. Stick to 6–9 steps for optimal quality. Different sampler/scheduler combos may require slight adjustments.
Step 3: Handle Resolution and Upscaling
Z-Image’s output quality depends heavily on resolution. Use these techniques for Full HD and 2K.
- Full HD (1920x1080): Direct generation can appear smoother or more diffuse. Instead, generate at an intermediate size that preserves detail, then use a Crop node after VAE Decode to remove extra pixels.
- Portrait orientation: Swap width and height in the generation settings.
- 2K output: Direct generation can distort the image. Generate at a suitable smaller resolution, then upscale the decoded image to 2K and pass it through a KSampler again with denoise 0.3 to refine details while preserving similarity.
- Higher denoise (e.g., 0.7) creates a more different result – sometimes a desaturated cinematic matte look.
- General upscale workflow: Use an upscale model to double the image, then refine at ~0.3 denoise. This can make plastic-looking Flux images more realistic, but may introduce artifacts – adjust seed and denoise.
Resolution and Saturation
Image size influences saturation and overall appearance. Test different intermediate resolutions with a fixed seed to find the sweet spot for your desired look.
Pro Tip
For 2K upscaling, start with a resolution like 1024x576 or 1280x720, then upscale to 1920x1080 or 2560x1440. The denoise of 0.3 keeps the original composition while adding detail.
Literal Prompt Interpretation
Z-Image can interpret prompt words too literally. If the model adds unwanted visual elements, remove the problematic term and regenerate.
Step 4: Use Qwen VL for Prompt Generation
Qwen VL can turn a reference image into a detailed prompt, or expand a short text prompt into a rich description.
- Image to prompt: Load an image into the Qwen VL node, select a formula preset, and increase max tokens if the generated description is cut short. Feed the resulting long prompt into Z-Image.
- Expand short text prompt: Concatenate a prompt formula with your simple input, send that to the Qwen VL custom-formula field, and connect the output to the text encoder. This creates richer prompt variation.
How Qwen VL Works
Qwen VL analyzes the reference image and generates a natural language description that captures details like lighting, composition, and style. The output can resemble the reference from text alone, though results still depend on seed and model limitations.
Pro Tip
When the generated description is cut off, increase max tokens in the Qwen VL node. A longer description gives Z-Image more context to work with.
Image Size Still Matters
Even with a detailed prompt, the generation resolution still influences saturation and overall appearance. Keep your resolution settings consistent when comparing prompts.
Step 5: Apply ControlNet for Structural Consistency
ControlNet ensures consistent structure across generations. Z-Image uses the Fun ControlNet Union model.
- Download the Z-Image Fun ControlNet Union model and place it in
ComfyUI/models/model_patches. - Loader node: This is in ComfyUI core. If missing or errors occur, update ComfyUI.
- Preprocessors:
- Canny: Extracts edges – good for keeping an illustration’s composition.
- Depth Anything: Preserves depth and scene layout – may require several attempts with different seeds.
- DW Pose: Extracts a pose skeleton – usually holds body position, but not tested on very complex poses.
- ControlNet strength: 1.0 is too rigid in the demonstration. Use 0.6 to 0.8 for a good balance between control and creativity.
ControlNet Strength Tuning
Lower strength (0.6–0.8) allows the model more freedom while still respecting the structural guide. Higher strength (1.0) can force exact edge matching, which may look unnatural.
Pro Tip
For Depth Anything, try multiple seeds if the first attempt doesn’t preserve the scene layout well. The preprocessor output can vary with different seeds.
Update ComfyUI
If the ControlNet loader node is missing, update ComfyUI to the latest version. The node is part of core ComfyUI, not a custom node.
Step 6: Train a Character LoRA Online with Fal
Train a LoRA to maintain a specific character across different outfits, backgrounds, and resolutions.
- Dataset: Use at least 10 images (the example uses 15 consistent images of the same character from different angles). One image came from Z-Image; additional views were created with Nano Banana.
- Training settings on Fal:
- Steps: 1,000 steps
- Learning rate: Default
- Trigger token: Unique (e.g.,
chara_whitney) - Training type: Content (for character identity)
- Cost: Approximately $2.26; completion time 15–20 minutes.
- Download and load: Download the finished LoRA, name it after the trigger token, place it in
ComfyUI/models/loras, press R to refresh, and load it with a LoRA loader. - Prompt: Put the trigger token plus the character description in the prompt. Explicit hair and eye colors improve identity accuracy.
- LoRA strength: For non-photographic looks, reduce strength. 0.8 produced a much better watercolor result than full strength.
Dataset Requirements for Character LoRA
The dataset must show the same character from multiple angles with consistent appearance (same hair color, eye color, facial features). Avoid mixing different characters or styles.
Pro Tip
Use Nano Banana or similar tools to generate additional views of your character from a single source image. This expands your dataset without manual work.
Training Cost
Fal charges for training. The example cost $2.26 for 1,000 steps. Make sure your dataset is well-curated to avoid wasting credits.
LoRA Strength for Different Styles
Full strength (1.0) works well for photorealistic outputs. For watercolor, anime, or other non-photographic styles, reduce strength to 0.8 or lower to let the base model’s style shine through.
Step 7: Train a Style LoRA
Style LoRAs require a coherent dataset – simply adding many images is not enough.
- Dataset consistency: The first attempt with 40 images containing too many different subjects and styles failed at both 1,000 and 2,000 steps. A second dataset of 15 images restricted to game items worked well.
- Training settings:
- Steps: 1,500 steps
- Learning rate: Default
- Trigger token: Unique
- Training type: Style
- Testing: With the LoRA enabled, a fixed-seed 3D-looking object changed into the intended game-digital-painting style. Disabling the LoRA restored the original 3D appearance.
Why Dataset Consistency Matters
A style LoRA learns the visual patterns common across all images. If your dataset mixes realistic photos, anime, and oil paintings, the LoRA will produce a confused blend. Stick to one visual style with similar subjects.
Pro Tip
Start with a small, highly curated dataset (10–20 images) and increase steps if needed. It’s easier to add more images later than to fix a messy dataset.
Style vs Content Training
Content training preserves a specific character’s identity. Style training captures the artistic look (e.g., watercolor, digital painting, pixel art). Choose the correct type for your goal.
Related Guides
- ComfyUI Installation Guide – Set up ComfyUI from scratch
- Running ComfyUI on RunPod – Run ComfyUI on cloud GPUs for faster generation
- Deploy Your First RunPod – Get started with RunPod cloud computing
Next Steps
Now that you’ve mastered Z-Image Turbo:
- Experiment with different sampler/scheduler combinations for varied skin rendering and sharpness
- Try ControlNet strength between 0.6 and 0.8 for different levels of structural guidance
- Train a style LoRA on a small, consistent dataset of your own artwork
- Combine character and style LoRAs for unique, consistent outputs
