The storage.repo-archive and storage.packages configs do not accept custom STORAGE_TYPE

[mmeier86]

Description

Hello,

I am using S3 storage provided by my Ceph cluster for Gitea's LFS, Avatar, attachment, repo-archive and packages storage. My storage configuration currently looks like this:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
[storage]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
STORAGE_TYPE = my_minio

[lfs]
STORAGE_TYPE = my_minio

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
[storage.repo-archive]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

[storage.packages]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.mei-home.net:7481
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

[storage.my_minio]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

This configuration works without problems. But I need to repeat my S3 configuration three times. Once for the default storage config used by attachments and so on, and then once each for the package and repo-archive storage.

Now to the bug itself. I had originally hoped that the two new storage items, packages and repo-archive would work similar to the lfs section above, by only setting the STORAGE_TYPE option to my_minio, avoiding the multiple repeats of the S3 configs.
But with the following config, I'm getting errors during Gitea startup:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
[storage]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; storage type
STORAGE_TYPE = my_minio

[lfs]
STORAGE_TYPE = my_minio

; customize storage
[storage.my_minio]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

This seems to working in so far as the package and repo-archive storage now try to use my_minio, but trying to use that then fails with this log output:

2022/08/07 19:10:45 cmd/web.go:106:runWeb() [I] Starting Gitea on PID: 20
2022/08/07 19:10:45 ...s/setting/setting.go:1256:CreateOrAppendToCustomConf() [I] Settings saved to: "/data/gitea/conf/app.ini"
2022/08/07 19:10:45 ...s/setting/setting.go:989:loadFromConf() [I] Default UI Location is Europe/Berlin
2022/08/07 19:10:45 cmd/web.go:157:runWeb() [I] Global init
2022/08/07 19:10:45 ...s/setting/setting.go:989:loadFromConf() [I] Default UI Location is Europe/Berlin
2022/08/07 19:10:45 routers/init.go:104:GlobalInitInstalled() [I] Git Version: 2.36.2, Wire Protocol Version 2 Enabled (home: /data/gitea_data/home)
2022/08/07 19:10:45 routers/init.go:107:GlobalInitInstalled() [I] AppPath: /usr/local/bin/gitea
2022/08/07 19:10:45 routers/init.go:108:GlobalInitInstalled() [I] AppWorkPath: /app/gitea
2022/08/07 19:10:45 routers/init.go:109:GlobalInitInstalled() [I] Custom path: /data/gitea
2022/08/07 19:10:45 routers/init.go:110:GlobalInitInstalled() [I] Log path: /data/gitea_data/logs
2022/08/07 19:10:45 routers/init.go:111:GlobalInitInstalled() [I] Configuration file: /data/gitea/conf/app.ini
2022/08/07 19:10:45 routers/init.go:112:GlobalInitInstalled() [I] Run Mode: Prod
2022/08/07 19:10:45 ...dules/setting/log.go:288:newLogService() [I] Gitea v1.17.0 built with GNU Make 4.3, go1.18.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2022/08/07 19:10:45 ...dules/setting/log.go:335:newLogService() [I] Gitea Log Mode: Console(Console:info)
2022/08/07 19:10:45 ...dules/setting/log.go:249:generateNamedLogger() [I] Router Log: Console(console:info)
2022/08/07 19:10:45 ...les/setting/cache.go:76:newCacheService() [I] Cache Service Enabled
2022/08/07 19:10:45 ...les/setting/cache.go:91:newCacheService() [I] Last Commit Cache Service Enabled
2022/08/07 19:10:45 ...s/setting/session.go:73:newSessionService() [I] Session Service Enabled
2022/08/07 19:10:45 ...es/setting/mailer.go:127:newMailService() [I] Mail Service Enabled
2022/08/07 19:10:45 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path attachments/
2022/08/07 19:10:45 ...s/storage/storage.go:170:initAvatars() [I] Initialising Avatar storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path avatars/
2022/08/07 19:10:45 ...s/storage/storage.go:188:initRepoAvatars() [I] Initialising Repository Avatar storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path repo-avatars/
2022/08/07 19:10:45 ...s/storage/storage.go:182:initLFS() [I] Initialising LFS storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path lfs/
2022/08/07 19:10:45 ...s/storage/storage.go:194:initRepoArchives() [I] Initialising Repository Archive storage with type: my_minio
2022/08/07 19:10:45 routers/init.go:57:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: Unsupported storage type: my_minio

What I would expect to happen: Like for the lfs config, the default storage config, in turn using my customized my_minio storage is used, instead of having to configure the same options three times.

Gitea Version

1.17.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

N/A

Operating System

Run via Docker

How are you running Gitea?

I'm using the official Gitea Docker container, with one slight adaption, namely completely disabling the external SSH daemon by not even starting it.

The container is deployed on a Nomad cluster.

The most important difference perhaps: Instead of MinIO, I'm using S3 provided by Ceph.

Database

PostgreSQL

[lunny]

As a workround, you can remove all the related configuration and only left below:

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

[mmeier86]

And that did the trick. Gitea config considerably shortened. Thanks @lunny!

For me this is an even better solution than solving the actual bug would be. But should we still leave the issue open for a future fix enabling package and repo archive storage to use a custom storage config?

[viceice]

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

Does this work for all storage types? so i don't need to configure attachments / lfs / picture ... manually?

[lunny]

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

Does this work for all storage types? so i don't need to configure attachments / lfs / picture ... manually?

Yes, except you have special configuration which will override it.

[navanshu]

As a workround, you can remove all the related configuration and only left below:

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

Does this also work with [attachment] configuration.

[lunny]

I will add some test and possible fix after #22405 merge.