Triple Clip Encoding by CLIP Text Encode for SD3 Family models Might Not Work with Long Prompts #6325
Labels
Potential Bug
User is reporting a bug. This should be tested.
Comments
Mithrillion
added
the
Potential Bug
Mithrillion
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
When encoding text, the encoder should output conditioning tensors of consistent size, as specified by the technical papers. For SD3/SD3.5, the text embedding shape should be
[1, 154, 4096], which is the size we will get when inputting a short prompt. There seems to be a limit as to how long the token dimension can be before the models will break.Actual Behavior
When the prompt is very long,
CLIP Text Encodenode appears to concatenate the individual embeddings directly, leading to seemingly unusual embedding shapes such as[1, 465, 4096]. When passed to SD3.5, the prompt still functions, but the resulting images will contain edge artifacts. It is observed that slightly overshooting 154 tokens might be fine, but there seems to be a limit how high the token numbers can be without causing issues. Around the 400 token mark, broken edges will start to appear on some generated images.glitched:
normal:
Steps to Reproduce
The reproducing workflow (normal):
sd35_glitch.json
To produce the glitch, add the following sample prompt to the positive prompt text box:
best quality, double exposure, realistic, whimsical, fantastic, splash art, intricate detailed, hyperdetailed, maximalist style, psychedelic, post-apocalyptic, photorealistic, concept art, sharp focus, harmony, serenity, tranquility, mysterious glow, ambient occlusion, halation, cozy ambient lighting, dynamic lighting,masterpiece, liiv1, linquivera, metix, mentixis, excellent composition, finest details, highest aesthetics, strong muted Highlighter of the turquoise ocean around, Blue hue, mystical glowing, best quality sharp focus, high contrast, stylized, clear, colorful, surreal, ultra quality, 8k, best quality, a breathtaking masterpiece, award winningObserve the different conditioning tensor shape in the console.
Debug Logs
Other
The text encoder for Flux appears to always return consistent conditioning length regardless of embedding text length. Perhaps a similar implementation is possible for SD3 family as well, as the models apparently do not handle elongated conditioning tensors gracefully.
My initial test appears to suggest that this issue is not related to specific model version / quantisation or the precision selected in the diffusion model loader node, and is not related to the VAE as I initially suspected. It is reproducible across multiple aspect ratios, and may appear along top & left, or bottom & right edges. Using only CLIP encoders and not T5 appears to not result in such issues.
Here are the output shapes of SDXL, SD3 and Flux using clip_l and t5 for encoding, where the two outputs per model correspond to long prompt and short prompt respectively:
(I think it is possible that SD3/SD3.5 models were never intended to work with very long prompts, but architecturally there are no constraints limiting the length of the token sequence, so when given long prompts the model behaves weirdly. It is possible this issue is on Stability's side, but it is still a bit strange when it appears that only SD3 family models have this problem.)
The text was updated successfully, but these errors were encountered: