A set of training scripts written in python for use in Kohya's SD-Scripts.

Please ensure you have python 3.10.6 and git installed

Gone is the two scripts of the past, and now everything is handled in one simple main.py and the two accompanying bat files run_popup.bat and run_command_line.bat. If you are on a linux system you can run them by activating the venv, which has moved to the sd_scripts folder within this project then run it using the command accelerate launch main.py for the command line, or accelerate launch main.py --popup if you want the popups.

the base version was designed for people who want to directly modify the script themselves, which you can now do so in the file ArgsList.py. to find out all of the commands and what they do you can run the script with the -h command to list every argument or just look at the list of arguments below

the popups are designed for those who are unable or unwilling to directly edit the script, you give up some control this way, but the defaults are generally good, so you won't need to worry too much about the hidden elements. It attempts to keep things simple by asking the user questions through popups, it's slower than the command line but it is a guided experience

there are also a bunch of small scripts meant to do other things that sd-scripts can do, such as resizing and merging LoRA, or down scaling images for datasets.

If you have a windows device I have created batch files here that auto install for you. if you want to specifically use my scripts, then grab the one titled install_sd_scripts.bat which will install SD-Scripts as well as my scripts. Make sure you don't have spaces in your path when you install them

I don't have installers for my scripts for linux, but here is the setup process so you can do it yourself.

git clone https://github.com/derrian-distro/LoRA_Easy_Training_Scripts
cd LoRA_Easy_Training_Scripts
git submodule init
git submodule update
cd sd_scripts
python -m venv venv
source venv/bin/activate
pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu116
pip install --upgrade -r requirements.txt
pip install -U xformers
accelerate config

follow these answers when setting up accelerate

- This machine
- No distributed training
- NO
- NO
- NO
- all
- fp16

If you are using windows, then you can just run any of the scripts by using their respective run_*.bat files.

If you are using Linux, you can run it by activating your venv then running the scripts like so.

source sd_scripts/venv/bin/activate
then
accelerate launch main.py
or
accelerate launch main.py --popup

I believe it also works without accelerate if you just want to run it from the venv and python directly.

To check for updates on windows you can use the new update.bat that will automatically check for updates and apply them when it finds one, it should also update sd_scripts when there is an update as well.

One of the special features of the project is that you can save and load json files.

command_line and popup now handle this the same way as they now use the exact same backend. if you load a json file when the --popup arg is active, it will skip all popups and just load the json in full, rather than only load parts of it.

Additionally, you can load or save JSON files from the command line with their respective commands, --save_json_path "path\to\folder" for saving, and --load_json_path "path\to\json.json" for loading. You can also just set them, in ArgsList.py and it is asked when using --popup. Keep in mind that if you use --save_json_path and --load_json_path it will take precidence over whatever you inserted into the ArgsList or popups.

Finally, I also set up the JSON loading so that it supports the JSON files Kohya_ss generates

I have implemented queues into the project so that it will be able to take in a folder that has a bunch of json files and run through them all. Unlike the previous version, which had the popup version run through the popups for each time you wanted to queue up something, it now just uses the same method as the previous command line version.

you can also supply the path to the folder through the command line argument --multi_run_path "path\to\folder". keep in mind that setting this will override any value set in ArgsList.py or through the popups

if you need to generate json files you can do use through changing the ArgsList.py and setting the variable save_json_only to True, or through the popups by selecting yes on the associated popup. for that popup to appear you must also select yes to generating a json file

With my scripts, you can create a txt file with a list of all tags used during training. The tags can be output in one of two ways, according to most used to least, or alphabetically.

lora_resize.py and it's accompanying bat file lora_resize.bat is a script I wrote to run the resize script that is within SD-Scripts, much like the other two, it has a batch file that can be used to run it. It does things in the popup style, and supports queueing. This script should simplify reducing the dim size of LoRA.

I added two new scripts to handle the scripts that sd-scripts added since I created the resizing script. The first one, lora_merge.py and it's accompanying bat file lora_merge.bat can take in any number of loras to merge together, it walks you through the process, so you shouldn't need to know how it works under the hood! I also added support for merging lora into models with this most recent update.

The second one, image_resize.py and it's accompanying bat file image_resize.bat is a script for downscaling your images. This is great if you are training at a high resolution as you can disable bucket upscaling and have a set of images that are "different" to the model, it should improve gens at lower resolutions, and might even help smaller datasets

I added support for the new Lion optimizer in both scripts. I don't know what the best lr is for them, and still have 8bit adam as the default for now. if you want to know more, take a look at the original github here. They seem to have some insights, namely, use less steps than usual and a smaller lr. stating that lr should be anywhere from 3-10x smaller than normal.

I also added support for the new D-Adaption which works differently from the other optimizers. It handles the lr by itself, you just need to set the lr values to a value close to or at 1 for each lr. in order to seperate out the lr's for d-adaption though, you must also add the args {"decouple": "True"} to seperate the lr's for d-adaption. using the popups automatically sets this for you

I have added support for locon training that was recently released. To that end, I have added two new arguments for activating and managing it, locon for activating it, and locon_dim for setting the dim size. You can also set the alpha through locon_alpha. LoCon has proven it does have use, so it is now part of the popups. The file size is larger, as it stores more info, since the dims are seperate, you can create lower sized models still

I have added a popup style script for extracting LoCon from models. It does this using an Add Difference style approach, you much have a base model to compare against to extract this data. The process should guide you through the process. you can start it by running the bat file locon_extract.bat

I have added a popup style script for merging LoCon models into normal models. all you need to do is follow the popups. you can start it by running the bat file locon_loha_merge.bat

• Mar 10, 2023
  • Updated sd-scripts to the newest version which now natively supports LoCon models
  • did a small refactor of the arguments to support the new LyCORIS that replaced the old LoCon repo
  • Removed the old LoCon repo, replacing it with the new LyCORIS repo
  • Added support for Loha models, as well as changed support for LoCon to follow the new pattern that was introduced in LyCORIS
  • depricated the original LoCon variables, in favor of using a singluar variable network_args which works much like optimizer_args.
  • popups were updated to support the new loha and other changes to setup
  • Updated both the LoCon extract and merge script to support the changes, extract still only works with LoCon, but you can merge Loha using the new merge script.
• Mar 7, 2023
  • Updated the locon dependancies to support the new extraction methods, this also includes writing up a new version of locon extraction to follow the update
  • Kohya updated their readme with more details on how to use the image preview system. In the attempt to make this a bit more transparent I'm also adding it here
    • --n is the neg prompt, you want to put it after the main prompt
    • --h is height
    • --w is width
    • --d is the seed of the image
    • --l is the CFG scale
    • --s is the step count.
    • a sample prompt txt document might be like this

    # prompt 1
    masterpiece, best quality, 1girl, white t-shirt, upper body, looking at viewer, simple background --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 768 --h 768 --d 1 --l 7.5 --s 28
    
    # prompt 2
    masterpiece, best quality, 1boy, in business suit, standing at street, looking back --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 576 --h 832 --d 2 --l 5.5 --s 40
    

• Mar, 4, 2023
  • Updated the scripts to support the new arguments that Kohya introduced
    • dataset config for the toml file support, I haven't actually played with it, so for now it just sets it and that's it
    • all of the sample arguments
      • sample_every_n_steps and sample_every_n_epochs which allows you to generate a sample image every n steps or epochs
      • sample_prompts which is the file you pass in to generate the preview images. you can only specify a positive prompt, there is no support for negative prompts
      • sample_sampler which is the sampler that is used for generating images, it defaults to ddim, but there are a lot of options, I list them all in ArgsList.py
    • tokenizer_cache_dir which, to be honest, I don't really know what this means, it seems to talk about using this for offline training, but I'm pretty sure training is already offline.
    • locon alpha because this was added as a variable once it was fixed
  • Added all of the LoCon args to the popups as they have been tested enough that I feel comfortable adding them in as a whole
  • Added LoCon extracting and merging through their respective scripts locon_extract.bat and locon_merge.bat. look above for a bit more info on them, but the popups will generally guide you enough.
• Feb 27, 2023
  • Updated the LoRA merging scripts to allow for merging LoRA to models, I don't believe this works for locon models for now, I'll look into adding support another time
  • Added LoCon training to the scripts, as well as added the LoCon repo as a submodule. I added two new commands to facilite this
    • locon which is a bool that activates training with LoCon
    • locon_dim which is the dim for the LoCon layers, you must still set the main dim size
• Feb 23, 2023
  • Completely overhauled the scripts, changing everything
  • command line and popup are no longer seperate scripts
    • you can acces the popup version by adding --popup to the command call, or by using the associated batch file for that purpose.
    • you no longer modify the variables directly in the command_line.py script as it no longer exists, instead, there is a script specifically made to house all of the arguments that is seperated from the running logic.
  • changed the entire structure of the file system so that sd_scripts is now a submodule of my scripts, so that things don't break when sd_scripts updates until I update the scripts to support the new updates.
  • Updated all of the smaller side scripts to allow them to work with the new structure.
  • Added the new args, and shuffled around all of the old args to better organize them.
  • Created a new installer to support the new way everything is set up. (soon, gotta get this up first) This is windows only unfortunately
• Feb 20, 2023
  • Updated the scripts to support the new parameters added
    • Lion support added on both scripts, under the param use_lion
    • added the lowram option even though I don't think it will be needed for anybody who actually uses my scripts, as that seems to be geared towards colab users.
  • Added a script for merging LoRA, called lora_merge.py and it's bat file lora_merge.bat
    • It uses the same popup style for the other sub scripts, and will walk you through the process. This script seems to be able to merge more than one LoRA at a time
  • Added a script for downscaling images, called image_resize.py and it's bat file image_resize.bat. This will be a very useful tool for replacing repeats with a better alternative if you use the no_upscale feature.
• Feb 14, 2023
  • Updated the scripts to support the new parameter Kohya added, noise_offset
    • I haven't tested it myself yet, but supposedly having this set will improve generation of very bright and very dark elements. Kohya recommends a value of 0.1
  • Added the verbose option to the lora_resize script, I also wanted to add a way to output this to a text file, but I wan't able to find a way to do so cleanly. I will revisit this another time.
  • Added a queue system to the lora_resize script, it works very similar to my queue system when using the lora_train_popup.py queue system.
  • It seems like the multi-gpu option is baked into the code and will automatically use the gpus you have set when using accelerate config, I suggest that if you want to use multiple gpus to re-run the accelerate config to account for that, as right now, most people have it set up such that only one gpu is used.
• Feb 12, 2023
  • released v4 of the installers
    • had to remove the python and git check because it wasn't working as expected
    • also fixed an issue where ther cudnn wouldn't download because the website had an incorrect certificate
  • Small change to the txt file generation system so that you can output the files alphabetically
  • Added the option to turn on flip_aug for lora_train_popup.py
• Feb 11, 2023
  • released v3 of the installers
    • Added the python check system wide as well as user wide, rather than just user wide
    • Decluttered thing's a bit more for easier reading
    • Added a way to ask for admin so you no longer need to run it as admin
• Fab 10, 2023
  • released v2 of the installers
    • New guards to make sure that python 3.10 and git are installed
    • less cluttered messages so that is makes more sense what is going on
    • the option to install the cudnn patch for higher end 30X0 cards as well as 40X0 cards, using the installer script written by bmaltais. Thanks bmaltais
    • the option to auto install the patch for 10X0 cards, for the sd-scripts installer only
• Feb 9, 2023
  • I modified the bat files to work a bit better on the off chance that bitsandbytes needed to be installed to a different place than expected.
  • I also fixed the bat file for lora_resize.bat because it was looking for the wrong file name.
  • Added new button prompts for some of the popups to make it easier to know what options you have.
  • Added in the new arguments that Kohya introduced, as well as added in the arguments for training on SD2 based models, popups got added to change the ckip skip to 1 if using a realistic models as well. new arguments are as follows:
    • v2
    • v_parameterization
    • caption_dropout_rate
    • caption_dropout_every_n_epochs
    • caption_tag_dropout_rate
  • The new paramaters are added at the bottom of the list of arguments
  • Additionally, I finally decided to rename save_at_n_epochs to save_every_n_epochs internally. This doesn't break the json saving and loading system