Stable Diffusion 3 and 3.5 Workflow Guide: Setup, Text Encoders, Prompting and Recommended Settings

How to run Stable Diffusion 3 Medium and the newer 3.5 models locally: choosing a checkpoint and text encoders, ComfyUI and Diffusers setup, natural-language prompting, why negative prompts behave differently, recommended steps, CFG, sampler and shift settings, troubleshooting and licensing.

PUBLISHED // UPDATED
DURATION // 08 MIN READ
Stable Diffusion 3 and 3.5 Workflow Guide: Setup, Text Encoders, Prompting and Recommended Settings
FIG.01 // VISUAL DOCUMENTATION Stable Diffusion 3 and 3.5 Workflow Guide: Setup, Text Encoders, Prompting and Recommended Settings

Stable Diffusion 3 changed how Stability AI’s open-weight image models work, and many habits carried over from SDXL produce worse results with it. SD3 replaced the U-Net of earlier versions with a Multimodal Diffusion Transformer (MMDiT), added a large T5 text encoder alongside two CLIP encoders, and as a result follows long, descriptive prompts and renders short text far better. This guide covers choosing a model, setting it up, prompting it, and the settings recommended in Replicate’s guide to SD3, written with input from Stability AI’s tutorials. It also covers the newer Stable Diffusion 3.5 models, which most new projects should start with.

Which SD3-family model to use

ModelReleasedSizeNotes
SD3 MediumJune 20242B parametersFirst open release of the SD3 series
SD3.5 LargeOctober 22, 20248B parametersHighest quality in the family
SD3.5 Large TurboOctober 22, 20248B, distilledGenerates in a few steps
SD3.5 MediumOctober 29, 20242.5B parametersMMDiT-X architecture, designed to run on consumer hardware

Sources: Stability AI’s SD3.5 announcement and the SD3.5 Large model card.

Stability AI says SD3.5 Medium outperforms other medium-sized models on balance of prompt adherence and image quality. If you have a capable GPU, SD3.5 Large gives the best quality; on more modest hardware, SD3.5 Medium is the usual choice. Most of the prompting advice below applies to the whole family.

How SD3 differs from SDXL

The SD3 research paper describes an architecture that keeps separate weights for image and text representations and lets information flow between them. In practice this produces two noticeable improvements.

First, text rendering. SD 1.5 rarely produced legible words and SDXL was inconsistent, while SD3 can usually render a short phrase correctly when you put it in quotation marks. Longer passages remain less reliable.

Second, prompt adherence. SD3 is much better at following spatial instructions, such as which object sits on the left and which on the right, and at assigning attributes to the correct subject in a scene with several people or objects.

Hands and complex anatomy remain weak points, particularly for SD3 Medium, so plan for inpainting if a project needs accurate hands.

Setting up

Hardware. SD3 Medium and SD3.5 Medium are designed to run on consumer GPUs, and NVIDIA cards have the broadest tool support; more VRAM gives more headroom for higher resolutions and the full T5 encoder. The full 8B SD3.5 Large needs considerably more memory unless you use quantised versions.

Choosing a checkpoint. SD3 Medium was published in several packages that differ in which text encoders they include:

FileIncludesTrade-off
sd3_medium_incl_clips_t5xxlfp16Both CLIP encoders and T5 at fp16Best prompt understanding, most memory
sd3_medium_incl_clips_t5xxlfp8Both CLIP encoders and T5 at fp8Good balance for most users
sd3_medium_incl_clipsCLIP only, no T5Lowest memory, weaker prompt adherence and text
sd3_mediumNo text encodersLoad the encoders separately

The T5 encoder is the main reason SD3 understands long prompts and renders text better, but it is also the largest memory cost. Dropping it saves memory at the price of weaker results, especially for text.

ComfyUI. ComfyUI is the most common way to run SD3 locally: update it, place the checkpoint in the checkpoints folder and use an SD3 workflow with the SD3 empty latent node. Model weights are gated on Hugging Face, so you need to accept Stability AI’s licence terms before downloading.

Diffusers. For scripted or batch workflows, the Hugging Face Diffusers SD3 pipeline supports SD3 and SD3.5, including options to drop the T5 encoder, load it in 8-bit, or offload parts of the model to system memory to save VRAM.

Prompting SD3

SD3 rewards descriptive, natural-language prompts rather than comma-separated keyword lists. Prompts can run to 10,000 characters, so there is no need to compress ideas into tags. Describe the scene as you would to a photographer or illustrator, in complete sentences.

A keyword prompt such as “woman, portrait, beautiful, detailed” leaves the model guessing. A descriptive version works better: “A close-up portrait photograph of a woman in her late twenties with wavy auburn hair, wearing a cream turtleneck. Soft window light from the left creates gentle shadows. Shot with a portrait lens at a wide aperture, with a softly blurred background.”

Be explicit about relationships and attributes. “Standing against a solid red backdrop” is clearer than “red background”, and “holding a white cardboard sign with black text” is clearer than “holding sign”. For text in the image, put the words in quotation marks and keep them short.

Negative prompts behave differently. According to Replicate’s guide, SD3 was not trained with negative prompts. Adding one does change the image, but mostly by adding noise to the conditioning, much like changing the seed, rather than reliably removing what you listed. Describe what you do want instead.

Because SD3 has three text encoders, it is technically possible to send a different prompt to each, for example style terms to the CLIP encoders and detailed descriptions to T5. This is experimental and results vary.

Replicate’s guide suggests starting from these values and experimenting:

SettingStarting pointNotes
Steps28Very few steps give noisy, incoherent images; changing the step count can also change the content, not just the detail
CFG (guidance)3.5 to 4.5Much lower than SDXL; if images look “burnt” with too much contrast, lower it
Sampler and schedulerdpmpp_2m with sgm_uniform in ComfyUI (dpm++ 2M in Automatic1111); Euler also worksAncestral and SDE samplers and the Karras scheduler do not work well
Shift3.06.0 also scored well in human evaluation; 1.5 to 2.0 gives a rawer look
ResolutionAbout 1 megapixel, dimensions divisible by 641024x1024 (1:1), 1344x768 (16:9), 1536x640 (21:9), 1216x832 (3:2), 832x1216 (2:3)

Lower CFG values also reduce the difference between running with and without the T5 encoder, which helps if memory is tight. Stability AI’s own SD3.5 Large examples use 28 steps with guidance of 3.5 to 4.5, while the Turbo model is designed for a handful of steps.

Troubleshooting

Out-of-memory errors. Lower the resolution temporarily, switch to the fp8 or CLIP-only checkpoint, or enable model offloading, which moves parts of the model to system RAM at the cost of speed.

Harsh, over-contrasted images. CFG is probably too high; values carried over from SDXL, such as 7, are a common cause.

Noisy or incoherent output. Check the step count and make sure you are using a compatible sampler and scheduler.

Parts of the prompt ignored. Add specific details. “A golden retriever puppy with floppy ears” works better than “a dog”.

Garbled text. Shorten the phrase, put it in quotation marks and make sure the T5 encoder is loaded.

Odd framing at unusual sizes. Generate at a recommended resolution and crop or upscale afterwards.

Where SD3 fits

SD3 and SD3.5 make sense when you need local control, private workflows or customisation through fine-tuning. Their strengths are prompt adherence, short text and multi-subject compositions. SDXL still has the largest ecosystem of fine-tunes and LoRAs, so a specific community style may only exist for SDXL. Hosted models such as Midjourney or Black Forest Labs’ FLUX.2 [max], covered in our FLUX.2 [max] guide, often give more polished results with less effort but less control.

Licensing

SD3 and SD3.5 weights are released under the Stability AI Community License. It is free for research, non-commercial use and commercial use by organisations or individuals with less than $1 million in annual revenue; above that threshold, an enterprise licence is required. That makes the models open-weight rather than open source in the strict sense, so check the terms before commercial deployment.

Sources and further reading

Our analysis

The biggest adjustment when moving to SD3 is unlearning SDXL habits: write descriptive sentences, lower the guidance scale and stop relying on negative prompts. For new projects, SD3.5 Large or Medium is the better starting point than the original SD3 Medium. The community licence makes these models attractive for individuals and small businesses who want local control, provided they stay within its revenue threshold.