rank_dict.values() list index out of range in diffuser v0.40

[xalteropsx]

Describe the bug

the bug is not appear in below version of diffuser v0.37 it appear on my current v0.40

i have test it on v38 v39 as well seems like work fine only v40 has problem

when loading lora weight

Reproduction

pipe.load_lora_weights("F:/nuke_lora/glamrous.safetensors",weight_name='lora_glamrous',adapter_name='lora_glamrous') 

Logs

pipe.load_lora_weights("F:/nuke_lora/glamrous.safetensors",weight_name='lora_glamrous',adapter_name='lora_glamrous') 
  File "Z:\software\python11\Lib\site-packages\diffusers\utils\peft_utils.py", line 347, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "Z:\software\python11\Lib\site-packages\diffusers\loaders\lora_pipeline.py", line 224, in load_lora_weights
    self.load_lora_into_text_encoder(
  File "Z:\software\python11\Lib\site-packages\diffusers\loaders\lora_pipeline.py", line 447, in load_lora_into_text_encoder
    _load_lora_into_text_encoder(
  File "Z:\software\python11\Lib\site-packages\diffusers\loaders\lora_base.py", line 397, in _load_lora_into_text_encoder
    lora_config = _create_lora_config(state_dict, network_alphas, metadata, rank, is_unet=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Z:\software\python11\Lib\site-packages\diffusers\utils\peft_utils.py", line 379, in _create_lora_config
    lora_config_kwargs = get_peft_kwargs(
                         ^^^^^^^^^^^^^^^^
  File "Z:\software\python11\Lib\site-packages\diffusers\utils\peft_utils.py", line 158, in get_peft_kwargs
    r = lora_alpha = list(rank_dict.values())[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
❌ Reload failed

System Info

0.40.0 diffuser, 3.11 python

Who can help?

No response

[BeyondBirthday07]

hi @xalteropsx, I'd like to work on this issue.

Since the example uses a local LoRA file, could u share a bit more info to help reproduce it?

• How is pipe initialized? which pipeline/model are you using?
• Which version of transformers and peft are installed?
• If the LoRA is publicly avail, could u share a link to it? Otherwise, a few representative keys from its state dict would be helpful.

I noticed #13984 has a similar failure path, but there isn't enough information yet to tell whether the two issues have the same cause.

Thanks!

[xalteropsx]

i printed rank_dict get blank in v0.40 while below version work fine

https://drive.google.com/file/d/1LVji87eVs6GOe6Tlna7g8DNKCu9oF_ge/view?usp=sharing

use any sd model
if you cant reproduce this error using this lora let me know i will provide more detail since i need to reupdate v0.39 to v0.40 of it then make sure

you need to use - diffuser v0.40.0 peft 0.21.0
i cant tell transformer version currently i using 4.57.6 in diffuser v0.39.0 but i do know that we need to -U latest transformer to make it run on v0.40.0

[asomoza]

Hi @xalteropsx you need to post a minimal snippet of code to reproduce the issue including the exact models you're using so we don't have to make guesses, if you mean SD 1.5 you can use a repo id in the hub.

[BeyondBirthday07]

Thanks @xalteropsx, I could reproduce it with the lora u shared.

Looks like the text encoder keys still have the text_model. prefix, eg. text_model.encoder.layers.0.self_attn.q_proj.lora_B.weight

But with the newer transformers the CLIP module name is: encoder.layers.0.self_attn.q_proj

So the rank lookup doesnt find anything, rank stays empty, and then get_peft_kwargs() crashes with the index error.

I also tried transformers 5.5.0 and it works there because the old text_model.encoder... module names are still used.

Seems related to #13984 / the CLIP namespace change in newer transformers.

I'll try to add a small regression test and fix for this.