Stable Diffusion 3 and 3.5 Workflow Guide: Setup, Text Encoders, Prompting and Recommended Settings
How to run Stable Diffusion 3 Medium and the newer 3.5 models locally: choosing a checkpoint and text encoders, ComfyUI and Diffusers setup, natural-language prompting, why negative prompts behave differently, recommended steps, CFG, sampler and shift settings, troubleshooting and licensing.
Stable Diffusion 3 changed how Stability AI’s open-weight image models work, and many habits carried over from SDXL produce worse results with it. SD3 replaced the U-Net of earlier versions with a Multimodal Diffusion Transformer (MMDiT), added a large T5 text encoder alongside two CLIP encoders, and as a result follows long, descriptive prompts and renders short text far better. This guide covers choosing a model, setting it up, prompting it, and the settings recommended in Replicate’s guide to SD3, written with input from Stability AI’s tutorials. It also covers the newer Stable Diffusion 3.5 models, which most new projects should start with.
Which SD3-family model to use
| Model | Released | Size | Notes |
|---|---|---|---|
| SD3 Medium | June 2024 | 2B parameters | First open release of the SD3 series |
| SD3.5 Large | October 22, 2024 | 8B parameters | Highest quality in the family |
| SD3.5 Large Turbo | October 22, 2024 | 8B, distilled | Generates in a few steps |
| SD3.5 Medium | October 29, 2024 | 2.5B parameters | MMDiT-X architecture, designed to run on consumer hardware |
Sources: Stability AI’s SD3.5 announcement and the SD3.5 Large model card.
Stability AI says SD3.5 Medium outperforms other medium-sized models on balance of prompt adherence and image quality. If you have a capable GPU, SD3.5 Large gives the best quality; on more modest hardware, SD3.5 Medium is the usual choice. Most of the prompting advice below applies to the whole family.
How SD3 differs from SDXL
The SD3 research paper describes an architecture that keeps separate weights for image and text representations and lets information flow between them. In practice this produces two noticeable improvements.
First, text rendering. SD 1.5 rarely produced legible words and SDXL was inconsistent, while SD3 can usually render a short phrase correctly when you put it in quotation marks. Longer passages remain less reliable.
Second, prompt adherence. SD3 is much better at following spatial instructions, such as which object sits on the left and which on the right, and at assigning attributes to the correct subject in a scene with several people or objects.
Hands and complex anatomy remain weak points, particularly for SD3 Medium, so plan for inpainting if a project needs accurate hands.
Setting up
Hardware. SD3 Medium and SD3.5 Medium are designed to run on consumer GPUs, and NVIDIA cards have the broadest tool support; more VRAM gives more headroom for higher resolutions and the full T5 encoder. The full 8B SD3.5 Large needs considerably more memory unless you use quantised versions.
Choosing a checkpoint. SD3 Medium was published in several packages that differ in which text encoders they include:
| File | Includes | Trade-off |
|---|---|---|
sd3_medium_incl_clips_t5xxlfp16 | Both CLIP encoders and T5 at fp16 | Best prompt understanding, most memory |
sd3_medium_incl_clips_t5xxlfp8 | Both CLIP encoders and T5 at fp8 | Good balance for most users |
sd3_medium_incl_clips | CLIP only, no T5 | Lowest memory, weaker prompt adherence and text |
sd3_medium | No text encoders | Load the encoders separately |
The T5 encoder is the main reason SD3 understands long prompts and renders text better, but it is also the largest memory cost. Dropping it saves memory at the price of weaker results, especially for text.
ComfyUI. ComfyUI is the most common way to run SD3 locally: update it, place the checkpoint in the checkpoints folder and use an SD3 workflow with the SD3 empty latent node. Model weights are gated on Hugging Face, so you need to accept Stability AI’s licence terms before downloading.
Diffusers. For scripted or batch workflows, the Hugging Face Diffusers SD3 pipeline supports SD3 and SD3.5, including options to drop the T5 encoder, load it in 8-bit, or offload parts of the model to system memory to save VRAM.
Prompting SD3
SD3 rewards descriptive, natural-language prompts rather than comma-separated keyword lists. Prompts can run to 10,000 characters, so there is no need to compress ideas into tags. Describe the scene as you would to a photographer or illustrator, in complete sentences.
A keyword prompt such as “woman, portrait, beautiful, detailed” leaves the model guessing. A descriptive version works better: “A close-up portrait photograph of a woman in her late twenties with wavy auburn hair, wearing a cream turtleneck. Soft window light from the left creates gentle shadows. Shot with a portrait lens at a wide aperture, with a softly blurred background.”
Be explicit about relationships and attributes. “Standing against a solid red backdrop” is clearer than “red background”, and “holding a white cardboard sign with black text” is clearer than “holding sign”. For text in the image, put the words in quotation marks and keep them short.
Negative prompts behave differently. According to Replicate’s guide, SD3 was not trained with negative prompts. Adding one does change the image, but mostly by adding noise to the conditioning, much like changing the seed, rather than reliably removing what you listed. Describe what you do want instead.
Because SD3 has three text encoders, it is technically possible to send a different prompt to each, for example style terms to the CLIP encoders and detailed descriptions to T5. This is experimental and results vary.
Recommended settings
Replicate’s guide suggests starting from these values and experimenting:
| Setting | Starting point | Notes |
|---|---|---|
| Steps | 28 | Very few steps give noisy, incoherent images; changing the step count can also change the content, not just the detail |
| CFG (guidance) | 3.5 to 4.5 | Much lower than SDXL; if images look “burnt” with too much contrast, lower it |
| Sampler and scheduler | dpmpp_2m with sgm_uniform in ComfyUI (dpm++ 2M in Automatic1111); Euler also works | Ancestral and SDE samplers and the Karras scheduler do not work well |
| Shift | 3.0 | 6.0 also scored well in human evaluation; 1.5 to 2.0 gives a rawer look |
| Resolution | About 1 megapixel, dimensions divisible by 64 | 1024x1024 (1:1), 1344x768 (16:9), 1536x640 (21:9), 1216x832 (3:2), 832x1216 (2:3) |
Lower CFG values also reduce the difference between running with and without the T5 encoder, which helps if memory is tight. Stability AI’s own SD3.5 Large examples use 28 steps with guidance of 3.5 to 4.5, while the Turbo model is designed for a handful of steps.
Troubleshooting
Out-of-memory errors. Lower the resolution temporarily, switch to the fp8 or CLIP-only checkpoint, or enable model offloading, which moves parts of the model to system RAM at the cost of speed.
Harsh, over-contrasted images. CFG is probably too high; values carried over from SDXL, such as 7, are a common cause.
Noisy or incoherent output. Check the step count and make sure you are using a compatible sampler and scheduler.
Parts of the prompt ignored. Add specific details. “A golden retriever puppy with floppy ears” works better than “a dog”.
Garbled text. Shorten the phrase, put it in quotation marks and make sure the T5 encoder is loaded.
Odd framing at unusual sizes. Generate at a recommended resolution and crop or upscale afterwards.
Where SD3 fits
SD3 and SD3.5 make sense when you need local control, private workflows or customisation through fine-tuning. Their strengths are prompt adherence, short text and multi-subject compositions. SDXL still has the largest ecosystem of fine-tunes and LoRAs, so a specific community style may only exist for SDXL. Hosted models such as Midjourney or Black Forest Labs’ FLUX.2 [max], covered in our FLUX.2 [max] guide, often give more polished results with less effort but less control.
Licensing
SD3 and SD3.5 weights are released under the Stability AI Community License. It is free for research, non-commercial use and commercial use by organisations or individuals with less than $1 million in annual revenue; above that threshold, an enterprise licence is required. That makes the models open-weight rather than open source in the strict sense, so check the terms before commercial deployment.
Sources and further reading
- Esser et al., Scaling Rectified Flow Transformers (SD3 paper, arXiv:2403.03206)
- Replicate: How to get the best results from Stable Diffusion 3
- Stability AI: Introducing Stable Diffusion 3.5
- Stable Diffusion 3 Medium on Hugging Face
- Hugging Face Diffusers SD3 pipeline
- Stability AI license
Our analysis
The biggest adjustment when moving to SD3 is unlearning SDXL habits: write descriptive sentences, lower the guidance scale and stop relying on negative prompts. For new projects, SD3.5 Large or Medium is the better starting point than the original SD3 Medium. The community licence makes these models attractive for individuals and small businesses who want local control, provided they stay within its revenue threshold.