A set of training scripts written in python for use in Kohya's SD-Scripts.

Please ensure you have python 3.10.6 and git installed

Gone are the two scripts of the past, and now everything is handled in one simple main.py and the two accompanying bat files run_popup.bat and run_command_line.bat. If you are on a linux system you can run them by activating the venv, which has moved to the sd_scripts folder within this project then run it using the command accelerate launch main.py for the command line, or accelerate launch main.py --popup if you want the popups.

the base version was designed for people who want to directly modify the script themselves, which you can now do so in the file ArgsList.py. to find out all of the commands and what they do you can run the script with the -h command to list every argument or just look at the list of arguments below

the popups are designed for those who are unable or unwilling to directly edit the script, you give up some control this way, but the defaults are generally good, so you won't need to worry too much about the hidden elements. It attempts to keep things simple by asking the user questions through popups, it's slower than the command line but it is a guided experience

there are also a bunch of small scripts meant to do other things that sd-scripts can do, such as resizing and merging LoRA, or down scaling images for datasets.

open up a cmd line in the folder you want to install the scripts to, then type in the following commands, this will clone the correct branch then launch the installer for you.

git clone https://github.com/derrian-distro/LoRA_Easy_Training_Scripts -b old-scripts
cd LoRA_Easy_Training_Scripts
install.bat

Linux doesn't have any installers due to the nature of each distro being different, so follow along as much as possible, it's likely the way you install torch will be different depending on the system

git clone https://github.com/derrian-distro/LoRA_Easy_Training_Scripts -b old-scripts
cd LoRA_Easy_Training_Scripts
git submodule init
git submodule update
cd sd_scripts
python -m venv venv
source venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install --upgrade -r requirements.txt
pip install -U xformers
accelerate config

follow these answers when setting up accelerate

- This machine
- No distributed training
- NO
- NO
- NO
- all
- fp16

If you are using windows, then you can just run any of the scripts by using their respective run_*.bat files.

If you are using Linux, you can run it by activating your venv then running the scripts like so.

source sd_scripts/venv/bin/activate
then
accelerate launch main.py
or
accelerate launch main.py --popup

I believe it also works without accelerate if you just want to run it from the venv and python directly.

To check for updates on windows you can use the new update.bat that will automatically check for updates and apply them when it finds one, it should also update sd_scripts when there is an update as well. Sometimes when you update you will also need to use the upgrade.bat file to make things run, so don't forget to run that once an update as well.

One of the special features of the project is that you can save and load json files.

command_line and popup now handle this the same way as they now use the exact same backend. if you load a json file when the --popup arg is active, it will skip all popups and just load the json in full, rather than only load parts of it.

Additionally, you can load or save JSON files from the command line with their respective commands, --save_json_path "path\to\folder" for saving, and --load_json_path "path\to\json.json" for loading. You can also just set them, in ArgsList.py and it is asked when using --popup. Keep in mind that if you use --save_json_path and --load_json_path it will take precidence over whatever you inserted into the ArgsList or popups.

Finally, I also set up the JSON loading so that it supports the JSON files Kohya_ss generates

I have implemented queues into the project so that it will be able to take in a folder that has a bunch of json files and run through them all. Unlike the previous version, which had the popup version run through the popups for each time you wanted to queue up something, it now just uses the same method as the previous command line version.

you can also supply the path to the folder through the command line argument --multi_run_path "path\to\folder". keep in mind that setting this will override any value set in ArgsList.py or through the popups

if you need to generate json files you can do use through changing the ArgsList.py and setting the variable save_json_only to True, or through the popups by selecting yes on the associated popup. for that popup to appear you must also select yes to generating a json file

With my scripts, you can create a txt file with a list of all tags used during training. The tags can be output in one of two ways, according to most used to least, or alphabetically.

lora_resize.py and it's accompanying bat file lora_resize.bat is a script I wrote to run the resize script that is within SD-Scripts, much like the other two, it has a batch file that can be used to run it. It does things in the popup style, and supports queueing. This script should simplify reducing the dim size of LoRA. I have changed the queue system to better support multiple resizes, either through providing a txt file, or providing multiple values. Because of this, however, it is more prone to failures. the txt file only contains the values used, no the mode of resize or the model to resize. So, if you are planning on doing a fixed resize, you would only need to enter in 1 value per line, but if you are planning on doing a dynamic resize, you want to put two values per line, the dynamic value, and then the max dim size.

I added two new scripts to handle the scripts that sd-scripts added since I created the resizing script. The first one, lora_merge.py and it's accompanying bat file lora_merge.bat can take in any number of loras to merge together, it walks you through the process, so you shouldn't need to know how it works under the hood! I also added support for merging lora into models with this most recent update.

The second one, image_resize.py and it's accompanying bat file image_resize.bat is a script for downscaling your images. This is great if you are training at a high resolution as you can disable bucket upscaling and have a set of images that are "different" to the model, it should improve gens at lower resolutions, and might even help smaller datasets

I also added support for the new D-Adaption which works differently from the other optimizers. It handles the lr by itself, you just need to set the lr values to a value close to or at 1 for each lr. in order to seperate out the lr's for d-adaption though, you must also add the args {"decouple": "True"} to seperate the lr's for d-adaption. using the popups automatically sets this for you

Kohya has introduced a way to weight captions in the txt files, with it is an argument for it, as well as normal tag weight rules. All of the tag weights follow the same setup as auto1111 webui, () for increase, [] for decrease, and (tag:1.2) to increase or decrease according to the weight. As part of this, however, if you have it enabled, make sure to change all () to \(tag\) that aren't weighted as to prevent them from accidentally being weighted during training.

DyLoRA is a new method of training LoRA and LoCon that recently popped up in both LyCORIS and sd-scripts. My scripts are configured to train sd-scripts by default, but it is capable of training both. DyLoRA are activated differently depending on the version you are training. It seems like you cannot use Block Weight Training with it so I have diabled it for the time being.

# for kohya's implementation
self.network_args = {
  'conv_dim': '8', # Optional, must be set to the same as network dim to use kohya's version. conv_alpha is completely ignored if you are doing DyLoRA, instead using the network alpha
  'unit':'4' # Is required if you want to use DyLoRA in my scripts, as that's how I tell what you want to do.
}

# for kohaku's implementation
self.lyco = True
self.network_args = {
  'conv_dim': '8', # Unsure if it must be the same dim size like kohya as I didn't fully test it
  'conv_alpha':'1', # Might end up ignoring it much like kohya does
  'algo':'dylora' # Required to tell the scripts you want to use dylora
}

With the introduction of Block Weight Training by kohya in sd-scripts, I had to think about how to implement everything for this. I have a proper popup in the works, but I didn't have it ready in time for this update. So I opted to create a simpler setup for the time being. The popups will guide you through setting all of the values, or you can take a look at the documentation on it here. I tried my hardest to make it easy to use, however, I had to balance the amount of popups with the complexity. The choice I made meant more work for me in the end, but I think it turned out very well. The block weight training allows you to select a different weight for every layer, as well as dims and alphas. NOTE: If you want to use this outside of the popups, make sure to disable lyco and not have algo in the network_args

I have added support for locon, loha, and ia3 training. You can set the variable lyco in the ArgsList.py to use LyCORIS, which also has a few variables that need to be set. in the networks_args variable, you can set the conv_dim and conv_alpha as well as the algo and if you are using cp_decomp. Typically, an example is like so:

self.network_args = {
  'algo': 'lora', # lora corresponds to locon, loha corresponds to loha, ia3 corresponds to ia3, and lokr corresponds to lokr
  'conv_dim': '8',
  'conv_alpha': '1'
}

The popups handle setting these values for you, if you don't want to set them yourself.

I have added a popup style script for extracting LoCon from models. It does this using an Add Difference style approach, you much have a base model to compare against to extract this data. The process should guide you through the process. you can start it by running the bat file lora_locon_extract.bat. I added a new way to do multiple extractions quickly through either using a txt file or giving multiple values, this is a much better way to set up queues than my original approach, however is more prone to failure. the txt file only contains the values used, not the mode of extract or the models to extract. If you are extracting a LoRA, the file only needs one value per line, LoCon requires two

I have added a popup style script for merging LoCon models into normal models. all you need to do is follow the popups. you can start it by running the bat file locon_loha_merge.bat

I created a rudimentary popup script for training XTI. I have only tested that the arguments are getting properly passed, beyond that I have not tested anything as I don't actually know how to train them. Please submit a bug report if you know how to train them, and it doesn't work for you, as I'd not know otherwise.

• May 20, 2023
  • Updated sd-scripts, added the new arg that came with it.
  • Moved old scripts to branch old-scripts, as they are being deprecated
  • Updated the installers to work within the cloned folder, as all previous installers are now considered deprecated and will not work for this fork.
• May 5, 2023
  • Updated sd-scripts and LyCORIS, added all of the new args added to sd-scripts.
  • re-added the extraction script for kohaku's extraction implementation.
  • Added new pyramid noise args to the argslist, however not to the popups
    • multires_noise_iterations which is the amount of iterations for pyramid noise. It is recommended to use between 6 and 10, I cannot confirm this to be the best option however.
    • multires_noise_discount, which I have literally no clue what this does, seems like it is recommended to be between the values 0.1 and 0.3, I set the default to 0.3
  • Added the new logging args
    • log_with which allows you to specify what logging system to use, tensorboard, wandb, or all
    • log_tracker_name, the name of the tracker, if none then it is set to whatever the specific logging system names it.
    • wandb_api_key, the API key for wandb for use with wandb.
  • Work on the UI update is still on-going, but a bit slow. I recently decided to change out how I'm handling the args, which means I had to remake a lot of the UI elements, but progress is steadily happening.
• Apr 17, 2023
  • Updated sd-scripts and LyCORIS, however no changes to the args have been made
  • Changed the extraction script to kohya's implementation to allow for LoRA extraction.
• Apr 14, 2023
  • Updated sd-scripts and LyCORIS
  • added the weighted captions arguments as well as support for the new DyLoRA.
  • added support for XTI in the form of a popup script. It's not entirely tested beyond making sure it actually passes arguments correctly to the training scripts as well as making assumptions about what arguments are required. It is very rudimentary in comparison to the main scripts, doesn't have a majority of the features the normal scripts have, but at least it has json saving and loading. Since I have no idea how to train TI or XTI, I'm going to need some feedback from people who do so I can fix any bugs that might exist
• Apr 8, 2023
  • Updated sd-scripts and LyCORIS
  • added all of the new huggingface commands to the ArgsList.py that was added recently by sd-scripts, but I didn't include them in the popups.
  • Added support for the new algo LyCORIS created, lokr, in the popups.
  • Added support for the new block weight training that kohya implemented. There is a lot to it, so I suggest you read the documentation here.
  • Added a mode to reading a json that allows you to specify the folders still.
  • XTI implementation will be in the next update
• Apr 4, 2023
  • Updated sd-scripts and LyCORIS, however sd-scripts is one version behind as I still need time to work in the new sd-scripts elements. Note, this next update will disable LyCORIS for the time being until Kohaku can update their code.
  • Added new queue systems to the lora_resize and locon_extract scripts. They allow for much easier queues but it does mean it's more prone to failure. I have included example txt files in the examples folder.
• Mar 28, 2023
  • Updated sd-scripts and LyCORIS
  • Added the new parameters added by sd-scripts for min snr gamma and token warmup
    • min_snr_gamma is an argument that takes in a float, and seemingly improves training
    • token_warmup_min takes in an int, is the smallest amount of tokens that are used when using token warmup
    • token_warmup_step takes in a float, is the last step before all tokens are used in training
  • Added min_snr_gamma to popups as it seems to be a worthwhile addition
• Mar 21, 2023
  • Updated sd-scripts and LyCORIS
  • Added the new parameters added by sd-scripts for a custom scheduler, however I don't do any sort of error checking for it
  • Added support for torch 2.0.0 and 2.1.0 along with built xformers for both
  • Added support for triton for torch 2, however it may not be working as expected.
  • Updated the number of workers to be be 1 by default, as to make it more accessable
  • Changed the main batch files to not use accelerate to launch anymore, instead option to use python directly instead.
  • Added a new batch file for updating to torch 2.0.0 or 2.1.0 with the option to install triton as well.
  • Added new installer that is python based, as to make it easier to check for the correct version of python, as well as check for git being installed. This installer is still only for windows as linux is different for every distro.
• Mar 13, 2023
  • Updated sd-scripts and LyCORIS
  • with it came an update to the resizing scripts, and the locon extraction scripts.
  • Added new parameters for specifying height seperately from width for training at non square resolutions
  • Added new popups to handle CP Decomposition for locon and loha as well as the change to how resolution is handled
• Mar 10, 2023
  • Updated sd-scripts to the newest version which now natively supports LoCon models
  • did a small refactor of the arguments to support the new LyCORIS that replaced the old LoCon repo
  • Removed the old LoCon repo, replacing it with the new LyCORIS repo
  • Added support for Loha models, as well as changed support for LoCon to follow the new pattern that was introduced in LyCORIS
  • depricated the original LoCon variables, in favor of using a singluar variable network_args which works much like optimizer_args.
  • popups were updated to support the new loha and other changes to setup
  • Updated both the LoCon extract and merge script to support the changes, extract still only works with LoCon, but you can merge Loha using the new merge script.
• Mar 7, 2023
  • Updated the locon dependancies to support the new extraction methods, this also includes writing up a new version of locon extraction to follow the update
  • Kohya updated their readme with more details on how to use the image preview system. In the attempt to make this a bit more transparent I'm also adding it here
    • --n is the neg prompt, you want to put it after the main prompt
    • --h is height
    • --w is width
    • --d is the seed of the image
    • --l is the CFG scale
    • --s is the step count.
    • a sample prompt txt document might be like this

    # prompt 1
    masterpiece, best quality, 1girl, white t-shirt, upper body, looking at viewer, simple background --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 768 --h 768 --d 1 --l 7.5 --s 28
    
    # prompt 2
    masterpiece, best quality, 1boy, in business suit, standing at street, looking back --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 576 --h 832 --d 2 --l 5.5 --s 40
    
• Mar, 4, 2023
  • Updated the scripts to support LyCORIS's new algo ia3. Cant guarentee anything at all, have no clue how to train it, and know nothing about it other than that it is seemingly another loha style algo.
  • Updated the scripts to support the new arguments that Kohya introduced
    • dataset config for the toml file support, I haven't actually played with it, so for now it just sets it and that's it
    • all of the sample arguments
      • sample_every_n_steps and sample_every_n_epochs which allows you to generate a sample image every n steps or epochs
      • sample_prompts which is the file you pass in to generate the preview images. you can only specify a positive prompt, there is no support for negative prompts
      • sample_sampler which is the sampler that is used for generating images, it defaults to ddim, but there are a lot of options, I list them all in ArgsList.py
    • tokenizer_cache_dir which, to be honest, I don't really know what this means, it seems to talk about using this for offline training, but I'm pretty sure training is already offline.
    • locon alpha because this was added as a variable once it was fixed
  • Added all of the LoCon args to the popups as they have been tested enough that I feel comfortable adding them in as a whole
  • Added LoCon extracting and merging through their respective scripts locon_extract.bat and locon_merge.bat. look above for a bit more info on them, but the popups will generally guide you enough.
• Feb 27, 2023
  • Updated the LoRA merging scripts to allow for merging LoRA to models, I don't believe this works for locon models for now, I'll look into adding support another time
  • Added LoCon training to the scripts, as well as added the LoCon repo as a submodule. I added two new commands to facilite this
    • locon which is a bool that activates training with LoCon
    • locon_dim which is the dim for the LoCon layers, you must still set the main dim size
• Feb 23, 2023
  • Completely overhauled the scripts, changing everything
  • command line and popup are no longer seperate scripts
    • you can acces the popup version by adding --popup to the command call, or by using the associated batch file for that purpose.
    • you no longer modify the variables directly in the command_line.py script as it no longer exists, instead, there is a script specifically made to house all of the arguments that is seperated from the running logic.
  • changed the entire structure of the file system so that sd_scripts is now a submodule of my scripts, so that things don't break when sd_scripts updates until I update the scripts to support the new updates.
  • Updated all of the smaller side scripts to allow them to work with the new structure.
  • Added the new args, and shuffled around all of the old args to better organize them.
  • Created a new installer to support the new way everything is set up. (soon, gotta get this up first) This is windows only unfortunately
• Feb 20, 2023
  • Updated the scripts to support the new parameters added
    • Lion support added on both scripts, under the param use_lion
    • added the lowram option even though I don't think it will be needed for anybody who actually uses my scripts, as that seems to be geared towards colab users.
  • Added a script for merging LoRA, called lora_merge.py and it's bat file lora_merge.bat
    • It uses the same popup style for the other sub scripts, and will walk you through the process. This script seems to be able to merge more than one LoRA at a time
  • Added a script for downscaling images, called image_resize.py and it's bat file image_resize.bat. This will be a very useful tool for replacing repeats with a better alternative if you use the no_upscale feature.
• Feb 14, 2023
  • Updated the scripts to support the new parameter Kohya added, noise_offset
    • I haven't tested it myself yet, but supposedly having this set will improve generation of very bright and very dark elements. Kohya recommends a value of 0.1
  • Added the verbose option to the lora_resize script, I also wanted to add a way to output this to a text file, but I wan't able to find a way to do so cleanly. I will revisit this another time.
  • Added a queue system to the lora_resize script, it works very similar to my queue system when using the lora_train_popup.py queue system.
  • It seems like the multi-gpu option is baked into the code and will automatically use the gpus you have set when using accelerate config, I suggest that if you want to use multiple gpus to re-run the accelerate config to account for that, as right now, most people have it set up such that only one gpu is used.
• Feb 12, 2023
  • released v4 of the installers
    • had to remove the python and git check because it wasn't working as expected
    • also fixed an issue where ther cudnn wouldn't download because the website had an incorrect certificate
  • Small change to the txt file generation system so that you can output the files alphabetically
  • Added the option to turn on flip_aug for lora_train_popup.py
• Feb 11, 2023
  • released v3 of the installers
    • Added the python check system wide as well as user wide, rather than just user wide
    • Decluttered thing's a bit more for easier reading
    • Added a way to ask for admin so you no longer need to run it as admin
• Fab 10, 2023
  • released v2 of the installers
    • New guards to make sure that python 3.10 and git are installed
    • less cluttered messages so that is makes more sense what is going on
    • the option to install the cudnn patch for higher end 30X0 cards as well as 40X0 cards, using the installer script written by bmaltais. Thanks bmaltais
    • the option to auto install the patch for 10X0 cards, for the sd-scripts installer only
• Feb 9, 2023
  • I modified the bat files to work a bit better on the off chance that bitsandbytes needed to be installed to a different place than expected.
  • I also fixed the bat file for lora_resize.bat because it was looking for the wrong file name.
  • Added new button prompts for some of the popups to make it easier to know what options you have.
  • Added in the new arguments that Kohya introduced, as well as added in the arguments for training on SD2 based models, popups got added to change the ckip skip to 1 if using a realistic models as well. new arguments are as follows:
    • v2
    • v_parameterization
    • caption_dropout_rate
    • caption_dropout_every_n_epochs
    • caption_tag_dropout_rate
  • The new paramaters are added at the bottom of the list of arguments
  • Additionally, I finally decided to rename save_at_n_epochs to save_every_n_epochs internally. This doesn't break the json saving and loading system