Inpaint Anything (Inpainting with Segment Anything)

Inpaint Anything performs stable diffusion inpainting on a browser UI using any mask selected from the output of Segment Anything.

Using Segment Anything enables users to specify masks by simply pointing to the desired areas, instead of manually filling them in. This can increase the efficiency and accuracy of the mask creation process, leading to potentially higher-quality inpainting results while saving time and effort.

Extension version for AUTOMATIC1111's Web UI

Installation

Please follow these steps to install the software:

• Create a new conda environment:

conda create -n inpaint python=3.10
conda activate inpaint

• Clone the software repository:

git clone https://github.com/Uminosachi/inpaint-anything.git
cd inpaint-anything

• For the CUDA environment, install the following packages:

pip install -r requirements.txt

• If you are using macOS, please install the package from the following file instead:

pip install -r requirements_mac.txt

Running the application

python iasam_app.py

• Open http://127.0.0.1:7860/ in your browser.
• Note: If you have a privacy protection extension enabled in your web browser, such as DuckDuckGo, you may not be able to retrieve the mask from your sketch.

Options

• --save-seg: Save the segmentation image generated by SAM.
• --offline: Execute inpainting using an offline network.
• --sam-cpu: Perform the Segment Anything operation on CPU.

Downloading the Model

• Launch this application.
• Click on the Download model button, located next to the Segment Anything Model ID. This includes the SAM 2, Segment Anything in High Quality Model ID, Fast Segment Anything, and Faster Segment Anything (MobileSAM).
  • Please note that the SAM is available in three sizes: Base, Large, and Huge. Remember, larger sizes consume more VRAM.
• Wait for the download to complete.
• The downloaded model file will be stored in the models directory of this application's repository.

Usage

• Drag and drop your image onto the input image area.
  • Outpainting can be achieved by the Padding options, configuring the scale and balance, and then clicking on the Run Padding button.
  • The Anime Style checkbox enhances segmentation mask detection, particularly in anime style images, at the expense of a slight reduction in mask quality.
• Click on the Run Segment Anything button.
• Use sketching to point the area you want to inpaint. You can undo and adjust the pen size.
  • Hover over either the SAM image or the mask image and press the S key for Fullscreen mode, or the R key to Reset zoom.
• Click on the Create mask button. The mask will appear in the selected mask image area.

Mask Adjustment

• Expand mask region button: Use this to slightly expand the area of the mask for broader coverage.
• Trim mask by sketch button: Clicking this will exclude the sketched area from the mask.
• Add mask by sketch button: Clicking this will add the sketched area to the mask.

Inpainting Tab

• Enter your desired Prompt and Negative Prompt, then choose the Inpainting Model ID.
• Click on the Run Inpainting button (Please note that it may take some time to download the model for the first time).
  • In the Advanced options, you can adjust the Sampler, Sampling Steps, Guidance Scale, and Seed.
  • If you enable the Mask area Only option, modifications will be confined to the designated mask area only.
• Adjust the iteration slider to perform inpainting multiple times with different seeds.
• The inpainting process is powered by diffusers.

Tips

• You can directly drag and drop the inpainted image into the input image field on the Web UI. (useful with Chrome and Edge browsers)

Model Cache

• The inpainting model, which is saved in HuggingFace's cache and includes inpaint (case-insensitive) in its repo_id, will also be added to the Inpainting Model ID dropdown list.
  • If there's a specific model you'd like to use, you can cache it in advance using the following Python commands:

  python

  from diffusers import StableDiffusionInpaintPipeline
  pipe = StableDiffusionInpaintPipeline.from_pretrained("Uminosachi/dreamshaper_5-inpainting")
  exit()

• The model diffusers downloaded is typically stored in your home directory. You can find it at /home/username/.cache/huggingface/hub for Linux and MacOS users, or at C:\Users\username\.cache\huggingface\hub for Windows users.
  • When executing inpainting, if the following error is output to the console, try deleting the corresponding model from the cache folder mentioned above:

  An error occurred while trying to fetch model name...
  

Cleaner Tab

• Choose the Cleaner Model ID.
• Click on the Run Cleaner button (Please note that it may take some time to download the model for the first time).
• Cleaner process is performed using Lama Cleaner.

Mask only Tab

• Gives ability to just save mask without any other processing, so it's then possible to use the mask in other graphic applications.
• Get mask as alpha of image button: Save the mask as RGBA image, with the mask put into the alpha channel of the input image.
• Get mask button: Save the mask as RGB image.

Auto-saving images

• The inpainted image will be automatically saved in the folder that matches the current date within the outputs directory.

Development

With the Inpaint Anything library, you can perform segmentation and create masks using sketches from other applications.

License

The source code is licensed under the Apache 2.0 license.

References

• Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädel, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., & Feichtenhofer, C. (2024). SAM 2: Segment Anything in Images and Videos. arXiv preprint.
• Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W-Y., Dollár, P., & Girshick, R. (2023). Segment Anything. arXiv:2304.02643.
• Ke, L., Ye, M., Danelljan, M., Liu, Y., Tai, Y-W., Tang, C-K., & Yu, F. (2023). Segment Anything in High Quality. arXiv:2306.01567.
• Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., & Wang, J. (2023). Fast Segment Anything. arXiv:2306.12156 [cs.CV].
• Zhang, C., Han, D., Qiao, Y., Kim, J. U., Bae, S-H., Lee, S., & Hong, C. S. (2023). Faster Segment Anything: Towards Lightweight SAM for Mobile Applications. arXiv:2306.14289.