Skip to content

Commit

Permalink
Revert "Merge pull request bmaltais#466 from jstayco/consolidated_ins…
Browse files Browse the repository at this point in the history
…tall_scripts"

This reverts commit b7a719b, reversing
changes made to 538752c.
  • Loading branch information
bmaltais committed Apr 1, 2023
1 parent b7a719b commit 97b004e
Show file tree
Hide file tree
Showing 12 changed files with 184 additions and 722 deletions.
3 changes: 1 addition & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,4 @@ wd14_tagger_model
.DS_Store
locon
gui-user.bat
gui-user.ps1
.idea
gui-user.ps1
155 changes: 45 additions & 110 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,21 @@ If you run on Linux and would like to use the GUI, there is now a port of it as

### Table of Contents

- [Tutorials](#tutorials)
- [Required Dependencies](#required-dependencies)
- [Linux/macOS](#linux-and-macos-dependencies)
- [Installation](#installation)
- [Linux/macOS](#linux-and-macos)
- [Default Install Locations](#install-location)
- [Windows](#windows)
- [CUDNN 8.6](#optional--cudnn-86)
- [Upgrading](#upgrading)
- [Windows](#windows-upgrade)
- [Linux/macOS](#linux-and-macos-upgrade)
- [Launching the GUI](#starting-gui-service)
- [Windows](#launching-the-gui-on-windows)
- [Linux/macOS](#launching-the-gui-on-linux-and-macos)
- [Direct Launch via Python Script](#launching-the-gui-directly-using-kohyaguipy)
- [Dreambooth](#dreambooth)
- [Finetune](#finetune)
- [Train Network](#train-network)
- [LoRA](#lora)
- [Troubleshooting](#troubleshooting)
- [Page File Limit](#page-file-limit)
- [No module called tkinter](#no-module-called-tkinter)
- [FileNotFoundError](#filenotfounderror)
- [Change History](#change-history)
- [Tutorials](https://github.com/bmaltais/kohya_ss#tutorials)
- [Required Dependencies](https://github.com/bmaltais/kohya_ss#required-dependencies)
- [Installation](https://github.com/bmaltais/kohya_ss#installation)
- [CUDNN 8.6](https://github.com/bmaltais/kohya_ss#optional-cudnn-86)
- [Upgrading](https://github.com/bmaltais/kohya_ss#upgrading)
- [Launching the GUI](https://github.com/bmaltais/kohya_ss#launching-the-gui)
- [Dreambooth](https://github.com/bmaltais/kohya_ss#dreambooth)
- [Finetune](https://github.com/bmaltais/kohya_ss#finetune)
- [Train Network](https://github.com/bmaltais/kohya_ss#train-network)
- [LoRA](https://github.com/bmaltais/kohya_ss#lora)
- [Troubleshooting](https://github.com/bmaltais/kohya_ss#troubleshooting)
- [Page File Limit](https://github.com/bmaltais/kohya_ss#page-file-limit)
- [No module called tkinter](https://github.com/bmaltais/kohya_ss#no-module-called-tkinter)
- [FileNotFoundError](https://github.com/bmaltais/kohya_ss#filenotfounderror)
- [Change History](https://github.com/bmaltais/kohya_ss#change-history)

## Tutorials

Expand All @@ -48,74 +39,43 @@ If you run on Linux and would like to use the GUI, there is now a port of it as
- Install [Git](https://git-scm.com/download/win)
- Install [Visual Studio 2015, 2017, 2019, and 2022 redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe)

### Linux and macOS dependencies

These dependencies are taken care of via `setup.sh` in the installation section. No additional steps should be needed unless the scripts inform you otherwise.

## Installation

### Runpod
Follow the instructions found in this discussion: https://github.com/bmaltais/kohya_ss/discussions/379

### Linux and macOS
### MacOS
In the terminal, run

```
git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss
# May need to chmod +x ./setup.sh if you're on a machine with stricter security.
# There are additional options if needed for a runpod environment.
# Call 'setup.sh -h' or 'setup.sh --help' for more information.
./setup.sh
```

Setup.sh help included here:

```bash
Kohya_SS Installation Script for POSIX operating systems.

The following options are useful in a runpod environment,
but will not affect a local machine install.

Usage:
setup.sh -b dev -d /workspace/kohya_ss -g https://mycustom.repo.tld/custom_fork.git
setup.sh --branch=dev --dir=/workspace/kohya_ss --git-repo=https://mycustom.repo.tld/custom_fork.git

Options:
-b BRANCH, --branch=BRANCH Select which branch of kohya to check out on new installs.
-d DIR, --dir=DIR The full path you want kohya_ss installed to.
-g REPO, --git_repo=REPO You can optionally provide a git repo to check out for runpod installation. Useful for custom forks.
-h, --help Show this screen.
-i, --interactive Interactively configure accelerate instead of using default config file.
-n, --no-update Do not update kohya_ss repo. No git pull or clone operations.
-p, --public Expose public URL in runpod mode. Won't have an effect in other modes.
-r, --runpod Forces a runpod installation. Useful if detection fails for any reason.
-s, --skip-space-check Skip the 10Gb minimum storage space check.
-v, --verbose Increase verbosity levels up to 3.
bash macos_setup.sh
```

#### Install location
During the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions.

The default install location for Linux is where the script is located if a previous installation is detected that location.
Otherwise, it will fall to `/opt/kohya_ss`. If /opt is not writeable, the fallback is `$HOME/kohya_ss`. Lastly, if all else fails it will simply install to the current folder you are in (PWD).
### Ubuntu
In the terminal, run

On macOS and other non-Linux machines, it will first try to detect an install where the script is run from and then run setup there if that's detected.
If a previous install isn't found at that location, then it will default install to `$HOME/kohya_ss` followed by where you're currently at if there's no access to $HOME.
You can override this behavior by specifying an install directory with the -d option.
```
git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss
bash ubuntu_setup.sh
```

If you are using the interactive mode, our default values for the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions.
These are the same answers as the Windows install.
then configure accelerate with the same answers as in the MacOS instructions when prompted.

### Windows
In the terminal, run:
In the terminal, run

```
git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss
setup.bat
```

Then configure accelerate with the same answers as in the MacOS instructions when prompted.
then configure accelerate with the same answers as in the MacOS instructions when prompted.

### Optional: CUDNN 8.6

Expand All @@ -133,78 +93,53 @@ Run the following commands to install:
python .\tools\cudann_1.8_install.py
```

Once the commands have completed successfully you should be ready to use the new version. MacOS support is not tested and has been mostly taken from https://gist.github.com/jstayco/9f5733f05b9dc29de95c4056a023d645
## Upgrading MacOS

## Upgrading
When a new release comes out, you can upgrade your repo with the following commands in the root directory:

The following commands will work from the root directory of the project if you'd prefer to not run scripts.
These commands will work on any OS.
```bash
git pull
upgrade_macos.sh
```

.\venv\Scripts\activate
Once the commands have completed successfully you should be ready to use the new version. MacOS support is not tested and has been mostly taken from https://gist.github.com/jstayco/9f5733f05b9dc29de95c4056a023d645

pip install --use-pep517 --upgrade -r requirements.txt
```
## Upgrading Windows

### Windows Upgrade
When a new release comes out, you can upgrade your repo with the following commands in the root directory:

```powershell
upgrade.bat
```

### Linux and macOS Upgrade
You can cd into the root directory and simply run
```bash
# Refresh and update everything
./setup.sh
# This will refresh everything, but NOT clone or pull the git repo.
./setup.sh --no-git-update
```
Once the commands have completed successfully you should be ready to use the new version.

# Starting GUI Service
## Launching the GUI using gui.bat or gui.ps1

The script can be run with several optional command line arguments:

The following command line arguments can be passed to the scripts on any OS to configure the underlying service.
```
--listen: the IP address to listen on for connections to Gradio.
--username: a username for authentication.
--password: a password for authentication.
--server_port: the port to run the server listener on.
--inbrowser: opens the Gradio UI in a web browser.
--username: a username for authentication.
--password: a password for authentication.
--server_port: the port to run the server listener on.
--inbrowser: opens the Gradio UI in a web browser.
--share: shares the Gradio UI.
```
### Launching the GUI on Windows

The two scripts to launch the GUI on Windows are gui.ps1 and gui.bat in the root directory.
You can use whichever script you prefer.
To launch the Gradio UI, run the script in a terminal with the desired command line arguments, for example:
These command line arguments can be passed to the UI function as keyword arguments. To launch the Gradio UI, run the script in a terminal with the desired command line arguments, for example:

`gui.ps1 --listen 127.0.0.1 --server_port 7860 --inbrowser --share`

or

`gui.bat --listen 127.0.0.1 --server_port 7860 --inbrowser --share`

## Launching the GUI on Linux and macOS
Run the launcher script with the desired command line arguments similar to Windows.
`gui.sh --listen 127.0.0.1 --server_port 7860 --inbrowser --share`
## Launching the GUI directly using kohya_gui.py
## Launching the GUI using kohya_gui.py

To run the GUI directly bypassing the wrapper scripts, simply use this command from the root project directory:
To run the GUI, simply use this command:

```
.\venv\Scripts\activate
python .\kohya_gui.py
python.exe .\kohya_gui.py
```

## Dreambooth
Expand Down
22 changes: 0 additions & 22 deletions config_files/accelerate/default_config.yaml

This file was deleted.

12 changes: 8 additions & 4 deletions gui.sh
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
#!/usr/bin/env bash
#!/bin/bash

# Activate the virtual environment
source ./venv/bin/activate
source venv/bin/activate

# If the requirements are validated, run the kohya_gui.py script with the command-line arguments
if python tools/validate_requirements.py; then
# Validate the requirements and store the exit code
python tools/validate_requirements.py
exit_code=$?

# If the exit code is 0, run the kohya_gui.py script with the command-line arguments
if [ $exit_code -eq 0 ]; then
python kohya_gui.py "$@"
fi
13 changes: 13 additions & 0 deletions gui_macos.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

# Activate the virtual environment
source venv/bin/activate

# Validate the requirements and store the exit code
python tools/validate_requirements.py --requirements requirements_macos.txt
exit_code=$?

# If the exit code is 0, run the kohya_gui.py script with the command-line arguments
if [ $exit_code -eq 0 ]; then
python kohya_gui.py "$@"
fi
38 changes: 38 additions & 0 deletions macos_setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/bin/bash
# The initial setup script to prep the environment on macOS
# xformers has been omitted as that is for Nvidia GPUs only

if ! command -v brew >/dev/null; then
echo "Please install homebrew first. This is a requirement for the remaining setup."
echo "You can find that here: https://brew.sh"
exit 1
fi

# Install base python packages
echo "Installing Python 3.10 if not found."
brew ls --versions python@3.10 >/dev/null || brew install python@3.10
echo "Installing Python-TK 3.10 if not found."
brew ls --versions python-tk@3.10 >/dev/null || brew install python-tk@3.10

if command -v python3.10 >/dev/null; then
python3.10 -m venv venv
source venv/bin/activate

# DEBUG ONLY
#pip install pydevd-pycharm~=223.8836.43

# Tensorflow installation
if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp; then
python -m pip install tensorflow==0.1a3 -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl
rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl
fi

pip install torch==2.0.0 torchvision==0.15.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html
python -m pip install --use-pep517 --upgrade -r requirements_macos.txt
accelerate config
echo -e "Setup finished! Run ./gui_macos.sh to start."
else
echo "Python not found. Please ensure you install Python."
echo "The brew command for Python 3.10 is: brew install python@3.10"
exit 1
fi
9 changes: 3 additions & 6 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,12 @@ diffusers[torch]==0.10.2
easygui==0.98.3
einops==0.6.0
ftfy==6.1.1
gradio==3.19.1; sys_platform != 'darwin'
gradio==3.23.0; sys_platform == 'darwin'
gradio==3.19.1
lion-pytorch==0.0.6
opencv-python==4.7.0.68
pytorch-lightning==1.9.0
safetensors==0.2.6
tensorboard==2.10.1 ; sys_platform != 'darwin'
tensorboard==2.12.1 ; sys_platform == 'darwin'
tensorboard==2.10.1
tk==0.1.0
toml==0.10.2
transformers==4.26.0
Expand All @@ -25,8 +23,7 @@ requests==2.28.2
timm==0.6.12
# tensorflow<2.11
huggingface-hub==0.13.0
tensorflow==2.10.1; sys_platform != 'darwin'
tensorflow-macos==2.12.0; sys_platform == 'darwin'
tensorflow==2.10.1
# For locon support
lycoris-lora @ git+https://github.com/KohakuBlueleaf/LyCORIS.git@c3d925421209a22a60d863ffa3de0b3e7e89f047
# lycoris_lora==0.1.4
Expand Down
32 changes: 32 additions & 0 deletions requirements_macos.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
accelerate==0.15.0
albumentations==1.3.0
altair==4.2.2
bitsandbytes==0.35.0
dadaptation==1.5
diffusers[torch]==0.10.2
easygui==0.98.3
einops==0.6.0
ftfy==6.1.1
gradio==3.19.1; sys_platform != 'darwin'
gradio==3.23.0; sys_platform == 'darwin'
lion-pytorch==0.0.6
opencv-python==4.7.0.68
pytorch-lightning==1.9.0
safetensors==0.2.6
tensorboard==2.10.1
tk==0.1.0
toml==0.10.2
transformers==4.26.0
voluptuous==0.13.1
# for BLIP captioning
fairscale==0.4.13
requests==2.28.2
timm==0.6.12
# tensorflow<2.11
huggingface-hub==0.12.0; sys_platform != 'darwin'
huggingface-hub==0.13.0; sys_platform == 'darwin'
tensorflow==2.10.1; sys_platform != 'darwin'
# For locon support
lycoris_lora==0.1.2
# for kohya_ss library
.
Loading

0 comments on commit 97b004e

Please sign in to comment.