Create Hyperrealistic Consistent AI Characters with ComfyUI
This guide will walk you through creating hyperrealistic, consistent AI characters using the Gwen image edit workflow in ComfyUI. You'll learn how to generate datasets, upscale images, and train LoRA models for consistent character generation and video creation, all locally and for free.
What You'll Learn
Prerequisites
Before starting, ensure you have:
- ComfyUI installed. If not, follow the ComfyUI Installation Guide.
- A single input image of the character you want to create.
- Sufficient GPU VRAM for running the models (at least 12GB recommended).
VRAM Requirements
The amount of VRAM required depends on the model size and the complexity of the workflow. Larger models and more complex workflows require more VRAM. If you run out of VRAM, ComfyUI will likely crash or produce errors.
If your hardware isn't powerful enough or you want to speed up generations, consider using RunPod's cloud GPU service.
Special Offer - $5 Credit Included!
When you sign up for RunPod using our affiliate link, you'll receive a $5 credit that can be used to generate up to 9,000 images and 300 videos. This gives you plenty of resources to explore ComfyUI and AI image/video generation without any upfront cost!
Step-by-Step Process
Step 1: Install Custom Nodes and Download Workflow
Prepare ComfyUI by installing the necessary custom nodes and loading the workflow.
- Download the Workflow: Download the Gwen image edit workflow file from the link in the video description.
- Load Workflow in ComfyUI: Drag and drop the downloaded workflow file into the ComfyUI interface.
- Install Missing Custom Nodes: Go to the ComfyUI Manager, click Install Missing Custom Nodes, select all of them, and click Install. If you don't have the ComfyUI Manager, follow the instructions in the ComfyUI Installation Guide to install it.
- Restart ComfyUI: After the installation is complete, restart ComfyUI.
Pro Tip
Make sure the ComfyUI Manager is up to date for smooth installation of custom nodes. You can update it from within the ComfyUI Manager itself.
Step 2: Download Required Models
Download the required models and place them in the correct ComfyUI folders.
- Identify Models: The workflow displays the required models in yellow boxes with download links and folder locations.
- Download Models: Download each model from the provided links.
- Place Models: Place each model in the specified folder within your ComfyUI directory (e.g.,
ComfyUI/models/). - GGUF Model: For the main model, download the GGUF version that fits your GPU's VRAM. Place the GGUF file in
ComfyUI/models/unet/GGUF/(you may need to create theGGUFfolder). - Reload ComfyUI: After placing the models, press R in the ComfyUI window to reload and refresh the model list.
- Select Models in Workflow: Ensure the correct models are selected in the corresponding nodes within the workflow.
GGUF Versions
GGUF is a way to compress model sizes, making them smaller but potentially sacrificing some quality. Choose a version that comfortably fits within your GPU's VRAM.
Pro Tip
Create separate folders within the models directory to organize your models (e.g., unet/GGUF). This makes it easier to manage and find your models.
Step 3: Install Ultrasharp Model
Install the Ultrasharp model using the ComfyUI Manager.
- Open ComfyUI Manager: Go to Manager in the ComfyUI interface.
- Model Manager: Click on Model Manager.
- Search for Ultrasharp: Search for "Ultrasharp" in the Model Manager.
- Install: Click Install next to the Ultrasharp model.
- Refresh ComfyUI: Refresh ComfyUI after installation.
Step 4: Input Image and Character Details
Configure the input image and character details to start generating images.
- Input Image: Drag and drop your input image into the Load Image node at the top left of the workflow.
- Character Prompt: In the Character Prompt node, create a prompt describing the character's style. The default prompt is "make an image in this style of the character."
- Character Name: Enter a unique name for your character in the Character Name node. This name will be used as the trigger word for LoRA training and as the output folder name.
Pro Tip
Use a unique and descriptive name for your character to avoid conflicts with other LoRAs. A good name might include the character's profession or a key characteristic.
Step 5: Generate Initial Images
Run the workflow to generate the initial set of images.
- Click Run: Click the Run button in ComfyUI.
- Review Turnaround Sheet: Check the character turnaround sheet generated in the top left. Note that GGUF versions may sometimes double the poses.
- Add Clothing Details: Add clothing details to the Character Prompt node to influence the character's overall appearance. For example, "wearing chunky sneakers."
- Modify Prompts: Adjust the prompts in the different view groups to refine the generated images.
Pro Tip
Copy and paste groups to generate more images with different prompts. This allows you to quickly iterate and explore different variations of your character.
Step 6: Customize Emotions and Poses
Customize the character's emotions and poses to create a diverse dataset.
- Adjust Emotions: Use the slider in the Emotions group to change the character's expression.
- Change Image: Optionally, switch the input image in the Get Close-Up group to apply emotions to a different image.
- Additional Poses: Use the Additional Poses group to generate images with specific poses like a tea pose, laying down, or walking.
- Virtual Try-On: In the Virtual Try-On group, drag and drop an image of a clothing item and describe it in the prompt (e.g., "make the character wear the gray winter coat").
- Pose Transfer: In the Pose Transfer group, load a reference image with the desired pose to apply it to your character.
Pro Tip
Use images with monochrome backgrounds for the virtual try-on to get better results. This helps the AI isolate the clothing item and apply it more accurately.
Step 7: Save Selected Images
Save the images you want to use for your dataset.
- Change Character Name: Change the Character Name in the workflow by adding a number (e.g., "CharacterName1").
- Click Run: Click the Run button. This will create a new folder with only the currently displayed images.
Pro Tip
This step is crucial for saving only the images you want to include in your final dataset. Carefully review each image and only save the ones that best represent your character.
Step 8: Create Dataset
Create detailed captions and upscale the images for your LoRA training dataset.
- Click Dataset Creation: Click the Dataset Creation button in the workflow.
- Add Prompt: Add a prompt to the Prompt node in the Dataset Creation group.
- Click Run: Click the Run button.
- Review Images: The workflow will load all generated images, create detailed captions, and upscale them to the specified resolution (default is 2K).
- Compare Before and After: Compare the before and after images to see the upscaling results.
Pro Tip
Adjust the start step in the upscaling process to control the level of detail and consistency. Higher values result in minor changes, while lower values add more detail. Experiment to find the best balance for your character.
Step 9: LoRA Training with AI Toolkit on RunPod
Train a LoRA model using AI Toolkit on RunPod for optimal performance.
- Create RunPod Account: Sign up for a RunPod account.
- Add Funds: Add funds to your RunPod account (at least $10 is recommended).
- Deploy AI Toolkit Pod: Go to Pod Templates and search for AI Toolkit by ostress and click Deploy Pod.
- Select GPU: Choose a GPU (an RTX 3090 or higher is recommended).
- Change Template: Click Change Template, then Edit, and change the password in Environment Variables.
- Deploy On Demand: Click Deploy On Demand.
- Access AI Toolkit: Once deployed, click on HTTP Service to launch AI Toolkit.
- Create Dataset: In AI Toolkit, go to Datasets and click New Dataset. Give it a name and drag and drop your upscaled images into the dataset.
- Remove Unwanted Images: Remove any images that look weird or too similar.
- Create New Job: Click on New Job and name your LoRA.
- Choose Model: Select the sd_v2-1 model.
- Select Dataset: Select the dataset you just created.
- Configure Samples: Set the Sample Steps to 500 and the Video Frames to 1.
- Adjust Prompts: Adjust the example prompts to fit your character.
- Create Job: Click Create Job and then Start.
- Download LoRA: After training (approximately 1.5 hours), download the final LoRA version and the version before it.
- Stop Pod: Stop the RunPod to avoid further charges.
RunPod Cost
Training a LoRA on RunPod will cost a few dollars, but it's faster and more efficient than training locally, especially with limited VRAM.
AI Toolkit
AI Toolkit is an easy-to-use LoRA trainer for most AI models. Check out Ostrus AI's YouTube channel for more tutorials on using AI Toolkit.
LoRA Training Complete!
Congratulations! You have successfully trained a LoRA model for your character. You are now ready to generate images and videos with consistent character representation.
Step 10: Generate Images with the Trained LoRA
Use the trained LoRA in ComfyUI to generate images of your character.
- Download Workflow: Download the image generation workflow.
- Install Missing Nodes: Install any missing custom nodes.
- Download Loras: Download the Lenovo Ultra Real and Insta Real 2.2 Loras.
- Load Character LoRA: Load your trained character LoRA in both the High Noise Model and Low Noise Model nodes.
- Create Detailed Prompt: Create a detailed prompt including your character's trigger word, descriptions, and the trigger words for the other Loras (insta reel and Lenovo).
- Adjust Post-Processing: Adjust the post-processing effects (chromatic aberration, sharpening, bloom, grain) to your liking.
- Generate Images: Change the seed and click Generate to create different variations.
Pro Tip
Use a large language model to help you create detailed and varied prompts. The more descriptive your prompt, the better the results will be.
Step 11: Speed Up Image Generation
Speed up image generation using Light x Loras.
- Download Light x LoRA: Download the Light x LoRA for fast image and video generation.
- Load Light x LoRA: Load the Light x LoRA in both the High Noise Model and Low Noise Model nodes.
- Adjust Sampler Settings: Set the Maximum Steps to 8, CFG to 1, and Start at Step to 3 in both samplers.
- Activate Light x LoRA: Activate the Light x LoRA using Ctrl+B.
Experimental Feature
Light x LoRA is an experimental feature and may not work perfectly with all models and workflows. Experiment with different settings to find what works best for your character.
Step 12: Generate Videos
Generate videos of your character using the same workflow.
- Increase Length: Increase the Length to the desired number of frames (e.g., 41).
- Add Video Combine Node: Add a Video Combine node.
- Rewrite Prompt: Ask a large language model to rewrite the prompt for video generation.
- Generate Video: Run the workflow to generate a video.
- Download Video Workflow: Use the preconfigured video workflow for easier video generation.
Video Generation Considerations
Video generation requires significantly more processing power than image generation. Be prepared for longer generation times and potential memory limitations.
Potential Data Loss
Always back up your ComfyUI workflows and models before making significant changes or updates. Data loss can occur due to unexpected errors or compatibility issues.
Related Guides
- ComfyUI Installation Guide - Complete installation process for ComfyUI
- Running ComfyUI on RunPod - Run ComfyUI on cloud GPUs instead of local hardware
Next Steps
Now that you've created hyperrealistic, consistent AI characters:
- Experiment with different styles and characters.
- Explore advanced workflows for higher quality results.
- Share your creations and tag the creator!
Pro Tip
Join online communities and forums to share your creations, learn from others, and stay up-to-date on the latest advancements in AI image and video generation.
Ethical Considerations
Be mindful of the ethical implications of AI-generated content. Avoid creating content that is harmful, misleading, or infringes on the rights of others.