[New Model] The finished tile model is released

[lllyasviel]

hello everyone, we released the finished tile model. The name is

'control_v11f1e_sd15_tile' (The old "control_v11u_sd15_tile" will be removed)

The desc is at https://github.com/lllyasviel/ControlNet-v1-1-nightly#controlnet-11-tile
Download model here https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main

You have 5 ways to use it:

1. it can do 2x, 4x, or 8x super resolution
2. it can add, or change, or re-generate image details in an image
3. it can fix, refine, and improve bad image details obtained by any other super resolution methods like bad details or blurring from RealESRGAN
4. it can guide SD to diffuse in tiles, "one beautiful girl" will not generate 16 girls if you use 16 tiles and denoising strength 1.0.
5. it can finish unfinished artwork drafts if those drafts are drawn by color blocks

The performance of this model will be between production-ready and experimental. Final quality flag is [e].

You need ControlNet at least v1.1.107
The preprocessor can be none or tile_resample (tile_resample is just a resizing to make your image smaller if you want more variations)

Hint: do not forget to try different control modes - it can produce interesting effects
Hint2: this model supports resolution 768 or 1024 well. You can try it if you find SD's 512 is too small and blur.

[lllyasviel]

If you have any problem, make sure that you are using "control_v11f1e_sd15_tile", not the old "control_v11u_sd15_tile".

Pay attention to the name "v11f1e"

[fyushan]

Nice work!
There is one thing that confuses me. When I read your code, I can't find the tile operation that divides a large image into several tiles. It seems that you only resize the input to a fixed resolution, and the color fix is done as a post-processing.

[lllyasviel]

Beside official ways to use it, for a1111 users, we provide a special functionality: the tile model can be combined with "my prompt is more important" to do arbitary prompt-based transfer

[lllyasviel]

"Silver Armor" test in a1111

[lllyasviel]

some other up scaling tests should be trivial. just try your self

[lllyasviel]

when the detail replacement is large (using "my prompt is more important"), it can interprete the contents and replace with promts while maintaining the overall structure. This method does not rely on image depth or edges

Below is "balanced"

[54681]

Simple poses that only require one image are awesome. It also supports lora well

[lllyasviel]

OK perhaps I need to give an upscale example so that it can be really called "tile" and prove that it is not off topic.

Below is protogen without using any external upscaler (except the native a1111 Lanczos, which is not a super resolution method, just for resampling)

This is from 192x256 to 1536x2048

meta:

beautiful woman in sweater, high-quality, extremely detailed
Negative prompt: long body, low resolution, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality
Steps: 40, Sampler: Euler a, CFG scale: 7, Seed: 1, Size: 512x680, Model hash: 44f90a0972, Model: protogenX34Photorealism_1, Denoising strength: 0.35, Clip skip: 2, ControlNet-0 Enabled: True, ControlNet-0 Module: tile_resample, ControlNet-0 Model: control_v11f1e_sd15_tile [a371b31b], ControlNet-0 Weight: 1, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 1, Hires upscale: 3, Hires steps: 40, Hires upscaler: Lanczos

Note that image content may be changed because I used prompts to specify "in sweater"

screen (note that I am using the more challenging t2i, which is weaker than i2i in upscaling):

non cherry picked, random seed "1":

non cherry picked, random seed "2":

Note that I am using the worst method to do upscaling to test the worst performance. I am not using any XXX upscaler. This is just a basic t2i diffusion. Users may find much better ways. This is just the "naked" results without the help of special things. I am not even using esrgan

this example just show it can draw detailed hair at high resolution.

To really get best results, perhaps you should use I2I + ControlNet or something like high res fix + esrgan, or even other tile extensions.

[torealise]

Hi. I have a problem.

Seems like controlNet tile doesn't work for me.

It doesn't affect an image at all.

Image generated same with and without control net

I am using this versions:

ControlNet v1.1.111

Model
control_v11f1e_sd15_tile [a371b31b]

Preprocessor:

• none
• tile_resample

python: 3.10.6  •  torch: 2.0.0+cu118  •  xformers: N/A  •  gradio: 3.23.0  •  commit: [22bcc7be]

What can be wrong?

Maybe I do something wrong? Do I need to setup big resolution? Use inpaint? Upscaler? Highres fix?
Maybe I need any other extension to make it work?
Or update anything?

Thanks

[foggyflute]

@torealise I dont know if it's your intention to use no preprocessor or not, but does it work with tile_resample preprocessor?

[lllyasviel]

my bad. i forget to rename yaml. now fixed. update and restart completely

[torealise]

my bad. i forget to rename yaml. now fixed. update and restart completely

Thank you. Could I please ask about the chance of contacting you and having a talk? We have been using your ControlNet in our research, and it would be fascinating to have you as our adviser. If it is possible, would you be available for a conversation?

[lllyasviel]

now all 1.1 models and usage codes are finished. We will not release more things unless there is bug fix. The next update will be ControlNet 1.2. But this may depend on what Stability is going to release.

[torealise]

I downloaded control_v11f1e_sd15_tile.pth, and updated control_v11f1e_sd15_tile.yaml, then even restarted pc, but still control net tile doesn't make any difference to generated image.

Maybe I use it wrong way?

• I go to txt2img
• type prompt
• setup resolution 512x768
• enable controlnet 1.1.111
• select none or tile_resampler as a preprocessor
• select control_v11f1e_sd15_tile as a model
• press generate

but result is the same with control net turned on and off

[lllyasviel]

share logs please

[torealise]

is this a log that you asked me?

Unfortunately I don't know what is log exactly

Loaded state_dict from [C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.pth]
Loading config: C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.yaml
Error running process: C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\stable-diffusion-portable-main\modules\scripts.py", line 417, in process
script.process(p, *script_args)
File "C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 959, in process
model_net = self.load_control_model(p, unet, unit.model, unit.low_vram)
File "C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 645, in load_control_model
model_net = self.build_control_model(p, unet, model, lowvram)
File "C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 713, in build_control_model
network = network_module(
File "C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\cldm.py", line 63, in init
self.config = OmegaConf.load(config_path)
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\omegaconf\omegaconf.py", line 188, in load
obj = yaml.load(f, Loader=get_yaml_loader())
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml_init_.py", line 81, in load
return loader.get_single_data()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\composer.py", line 36, in get_single_node
document = self.compose_document()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\composer.py", line 58, in compose_document
self.get_event()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\parser.py", line 118, in get_event
self.current_event = self.state()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\parser.py", line 193, in parse_document_end
token = self.peek_token()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\scanner.py", line 129, in peek_token
self.fetch_more_tokens()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\scanner.py", line 223, in fetch_more_tokens
return self.fetch_value()
File "C:\stable-diffusion-portable-main\venv\lib\site-packages\yaml\scanner.py", line 577, in fetch_value
raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
in "C:\stable-diffusion-portable-main\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.yaml", line 28, column 66

100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 27.43it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 20.60it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 28.55it/s]

[Rkkss]

This is a great test, thank you for doing this -- Are you also using ControlNet is more important like dave?

On that test no. However I did test with ControlNet is more important. Even at denoise strength 0.2 , it changes the face drastically unlike Dave's
https://imgsli.com/MTc1MjEz

May give Dave's settings a try but I think the VAE did this to me. I have bad habit using 84000 mse vae for everything.

[eremtsov]

I found that it gives great results even without Ultimate SD Upscale. It looks like just setting a higher resolution in img2img gives better details and preserves the overall composition of the original picture really well (some color shift may occute at higher denoise settings, though). I immediately thought how awesome it would be if this model could be integrated into the HighresFix. Is this possible?
For the time being I'm just doing the highres fix manually by sending the generated images to img2img and upscaling them there. The results are very good, but this is a bit inconvenient.

Edit: I did some more testing upscaling from 403x716 to 1200x2136 via just img2img (no Ultimate SD upscale).
My workflow is as follows:

1. Generate a lowres image and copy it into img2img.
2. Resolution 3x the original (4x gives me Out of memory) Crop and resize, CFG 7, Denoise 0.75.
3. Control Network - Pixel perfect (not sure if it does anything here) - tile_resample - control_v11f1e_sd15_tile - Controlnet is more important - Crop and Resize.
4. I also put the original image into the ControlNet, but it looks like this is entirely unnecessary, you can just leave it blank to speed up the prep process. Edit2: Yeah, just tested a bit more, looks like it's entirely useless, the results with and w/o the picture are identical.
   Here is the original and the result:

It added some unwanted extra detail, such as the moustache and the pocket, but this is only because I set a very high Denoising value for testing purposes. The generation data is as follows:

old man in his garden, RAW photo, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3
Negative prompt: cropped zombie disfigured ugly grainy low-res Deformed blurry mutation disease amateur childish malformed disgusting mutilated mangled hag surreal text watermark unrecognizable abstract chaotic cluttered noisy unfinished unpolished dated unappealing cliché gory bloody violent disturbing vulgar offensive repulsive, extra limbs, poorly drawn
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 1612931090, Size: 1200x2136, Model hash: e6415c4892, Model: Realistic_vision_v2.0, Denoising strength: 0.75, ControlNet Enabled: True, ControlNet Preprocessor: tile_resample, ControlNet Model: control_v11f1e_sd15_tile [a371b31b], ControlNet Weight: 1, ControlNet Starting Step: 0, ControlNet Ending Step: 1, ControlNet Resize Mode: Crop and Resize, ControlNet Pixel Perfect: True, ControlNet Control Mode: ControlNet is more important, ControlNet Preprocessor Parameters: "(64, 1, 64)"

This is the Highres Fix I have always wnated.

[D-Ogi]

Size: 1200x2136

How to do set such big width? In my case it stops on 2048. Are you using hi-res fix? How do you set it in img2img mode?

ControlNet Preprocessor Parameters: "(64, 1, 64)"

Where do set those?

[lllyasviel]

it seems that Realistic_vision is a preferred model for tile. Protogen is relatively meh in my tests, not sure why

[ssstylusss]

Someone in the reddit forum told we must leave contronet input empty while using it on img2img, is that correct? I am a bit confused now, cause i always put the same image...

[eremtsov]

If I'm not mistaken, it doesn't matter, the result is the same, you can test it yourself, just don't forget to lock the seed.

[ssstylusss]

If I'm not mistaken, it doesn't matter, the result is the same, you can test it yourself, just don't forget to lock the seed.

How can i lock the seed?

[eremtsov]

Press the green recycle ♻️ button. I mean, use any number, but not -1 (random seed).

[ssstylusss]

Thanks! So many details! Have you new examples? Would be great make this discussion bigger :)

[enternalsaga]

Thanks! The tiling upscaler pushed my image to AAA level.
Original

Double Upscaled

[dtxn]

But your upscaled image tiled everywhere. Try to upload to Photoshop, add new curves layer and play with curves. You will see tiles everywhere.

[vinowan]

"Great job! Could you tell me how you trained this model and what is the underlying principle behind it?"

[walnutbread24]

Did you find anything useful? I have the same question and would love to know it too.

[crazygonuts]

Could you provide some details on how this is actually working? I read through as much as I could find on this and I found a lot of examples but without knowing what the controlnet is supposed to be doing it was difficult to figure out what those examples were trying to demonstrate.

I understand that its emphasizing/deemphasizing prompts based on what was in the original tiles, but how is it choosing tile sizes and partitions? Are the tiles arbitrary sizes or is it just something like a 32x32 grid across the whole image? Is there a way for me to control the tile sizes? Or am I misunderstanding the term "tile" here entirely?

Similarly, if the tile_resample preprocessor "down sampling rate" is set to 1, which is its default, is it doing anything at all? That setting seems like its not resampling at all, right? So is that essentially the same as not having a preprocessor? Does a "down sampling rate" of 4 mean that it is sampling the image at 1/4 resolution? And what is the use case where downsampling before passing it to the controlnet is useful?

[lllyasviel]

• I understand that its emphasizing/deemphasizing prompts based on what was in the original tiles, but how is it choosing tile sizes and partitions? Are the tiles arbitrary sizes or is it just something like a 32x32 grid across the whole image?

it seems to work best with image crops from 512x512 to 1024x1024. you can use any tile-based upscaling script to control tile size, like ultimate sd upscale

• if the tile_resample preprocessor "down sampling rate" is set to 1, which is its default, is it doing anything at all?

it copy images

• Does a "down sampling rate" of 4 mean that it is sampling the image at 1/4 resolution?

yes

• what is the use case where downsampling before passing it to the controlnet is useful?

make a bit more variations

[crazygonuts]

Thank you for the swift reply. In your silver armor and anime girl examples, both images were not being upscaled and were only marginally bigger than 512x512 - how does it control the macro composition of the image in those cases when the entire image is just 1 tile anyways?

[lllyasviel]

use weight and control mode

[crazygonuts]

What I mean to ask is, why does the model have any effect on a 512x512 image if the tile size is 512x512? Wouldn't that equate to a single tile with the entire prompt unchanged, which seems like it would have no effect on the output? Obviously I am misunderstanding something.

[Rkkss]

use weight and control mode
how does weight affect tile model exactly?
I figure we use low weight when original image is kinda bad/messy and higher weight to maintain the look as close as possible?

[ssstylusss]

Any tips on how to avoid this strong depth of field? I start with nice images, but the last ones geet this strong DOF and ruined everything

[lllyasviel]

yes this is a problem - a solution is to use Realistic Vision (or any base model trained without the "len blur" concept)

[ssstylusss]

Putting DOF, blur, blurred in the negative prompt helped a lot! The wings get a new detail too. Thanks for such amazing job on this!

[Rkkss]

Anyone knows what causes huge color distortion like this?

https://imgsli.com/MTc3Njky

I compare metadata and and only difference is BREAK keyword ( which non-BREAK is the one cause issue). Is this webUI/Multidiffusion's bug or tile model? Never had this problem using Multidiffusion extension alone so this must be combination of these.
It's not lora, I can reproduce this without using any.

[lllyasviel]

this is because you used "controlnet is more important" and when the token is more than 75, the combo of tiled diffusion and controlnet has a bug to disorder tiles. Removing BREAK will trigger this since it makes token longer. pkuliyi2015/multidiffusion-upscaler-for-automatic1111#153

[lllyasviel]

u can open an issue in tiled diffusion with my explainatio and see what is their opinion of this. make sure to attch your examples and link to pkuliyi2015/multidiffusion-upscaler-for-automatic1111#153

[ssstylusss]

Is just me or after the last updates Tile is completely ruined? I cant run more the denoiser slide than 0.1 and the final image gets completely different and ruined? :(

[jay-azhang]

Agreed

[Guillaume-Fgt]

Not the case for me, I am running the latest version of AUTO1111 and ControlNet. CFG7, denoising 7.5, two times:

[YourFriendlyNeighborhoodMONKE]

hello everyone, we released the finished tile model. The name is

'control_v11f1e_sd15_tile' (The old "control_v11u_sd15_tile" will be removed)

The desc is at https://github.com/lllyasviel/ControlNet-v1-1-nightly#controlnet-11-tile Download model here https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main

You have 5 ways to use it:

1. it can do 2x, 4x, or 8x super resolution

2. it can add, or change, or re-generate image details in an image

3. it can fix, refine, and improve bad image details obtained by any other super resolution methods like bad details or blurring from RealESRGAN

4. it can guide SD to diffuse in tiles, "one beautiful girl" will not generate 16 girls if you use 16 tiles and denoising strength 1.0.

5. **it can finish unfinished artwork drafts if those drafts are drawn by color blocks**

The performance of this model will be between production-ready and experimental. Final quality flag is [e].

You need ControlNet at least v1.1.107 The preprocessor can be none or tile_resample (tile_resample is just a resizing to make your image smaller if you want more variations)

Hint: do not forget to try different control modes - it can produce interesting effects Hint2: this model supports resolution 768 or 1024 well. You can try it if you find SD's 512 is too small and blur.

OP says that the model also supports 768 and 1024 resolutions, but I cannot seem to find how to customize the tile size bigger from 512 pixels for the preprocessor resolution...

To be absolutely clear:
I specifically mean preprosessing tile resolution on the ControlNet model side and not SD Upscaling script setting

Any ideas how I could use bigger tiles?

@lllyasviel

[danielyan86129]

Amazing tool!
Question here: most of the examples here do not enable "High res fix". But w/o it there is no UI to set #times upscale is performed. Do I miss anything? My settings screenshot:

Also, I'm getting 3 images in the result, which is the upscaled one? The first image colors are off:

[danielyan86129]

where can I find Ultimate SD Upscaler?

[Rkkss]

is there something wrong with recent update? Base tile_resampling seems to be overly detail the image @lllyasviel

https://imgsli.com/MTkzMDY3

1girl, water surfing, whirlpool, (masterpiece, best quality:1.3)
Negative prompt: (low quality, worst quality:1.4), (easynegative:1.2), ng_deepnegative_v1_75t, lowres, (greyscale, monochrome:1.1), (bad fingers, bad hands, negative_hand-neg:1.3), jpeg artifacts, extra hands, extra legs, extra fingers, missing limbs, missing legs, bad anatomy, bad proportions, extra digits, fewer digits
Steps: 30, Sampler: DPM++ 2M SDE Karras, CFG scale: 8, Seed: 3729076378, Size: 512x768, Model hash: 72d764a66c, Model: awportrait_v10, Denoising strength: 1, Clip skip: 2, ControlNet 1: "preprocessor: tile_resample, model: control_v11f1e_sd15_tile [a371b31b], weight: 0.7, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 1, 64)", Hires prompt: "1girl, water surfing, whirlpool, (masterpiece, best quality:1.3) <lora:add_detail:1> ", Hires upscale: 2, Hires steps: 24, Hires upscaler: 4x-UltraSharp, Discard penultimate sigma: True, Lora hashes: "add_detail: 7c6bad76eb54", Version: v1.4.1

base image

[sangoi-exe]

I did a test here, I used the tile resample along with the ultimate sd upscale, tested with down sampling 1 and 8, exactly the same settings for both, including the seed, the two images are identical, or practically identical.

So, what's down sampling supposed to do? I understand that down sampling is to reduce an image and use the extra pixels to increase the details, but apparently, this option in CN isn't doing anything at all.

[GoodGoodstudy0daydayup]

i think the tile_colorfix+sharp is very useful, but i can't find it in comfy,how can i use tile_colorfix+sharp in comfy? A1111 run slowly on my laptop.