After upgrading to 1.13.0, can no longer login

[sseneca]

• Gitea version (or commit ref): 1.13.0
• Git version: 2.29.2
• Operating system: Parabola GNU/Linux
• I'm using Gitea via the Helm chart, on my server running k3s. This is the deploy file.
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No (n/a)
• Log gist:

[Macaron] 2020-12-03 22:12:20: Started POST /user/login for 10.42.0.1
2020/12/03 22:12:20 ...m.io/xorm/core/db.go:286:afterProcess() [I] [SQL] SELECT "id", "type", "name", "is_actived", "is_sync_enabled", "cfg", "created_unix", "updated_unix" FROM "login_source" WHERE (is_actived = $1 and type = $2) [true 6] - 795.229µs
2020/12/03 22:12:20 ...m.io/xorm/core/db.go:286:afterProcess() [I] [SQL] SELECT "id", "type", "name", "is_actived", "is_sync_enabled", "cfg", "created_unix", "updated_unix" FROM "login_source" WHERE (is_actived = $1 and type = $2) [true 7] - 579.519µs
2020/12/03 22:12:20 ...m.io/xorm/core/db.go:286:afterProcess() [I] [SQL] SELECT "id", "lower_name", "name", "full_name", "email", "keep_email_private", "email_notifications_preference", "passwd", "passwd_hash_algo", "must_change_password", "login_type", "login_source", "login_name", "type", "location", "website", "rands", "salt", "language", "description", "created_unix", "updated_unix", "last_login_unix", "last_repo_visibility", "max_repo_creation", "is_active", "is_admin", "is_restricted", "allow_git_hook", "allow_import_local", "allow_create_organization", "prohibit_login", "avatar", "avatar_email", "use_custom_avatar", "num_followers", "num_following", "num_stars", "num_repos", "num_teams", "num_members", "visibility", "repo_admin_change_team_access", "diff_view_style", "theme", "keep_activity_private" FROM "user" WHERE "lower_name"=$1 LIMIT 1 [sseneca] - 860.604µs
[Macaron] 2020-12-03 22:12:20: Completed POST /user/login 200 OK in 29.759793ms
2020/12/03 22:12:20 routers/user/auth.go:177:SignInPost() [I] Failed authentication attempt for sseneca from 10.42.0.1

Description

After updating to v1.13.0, I can no longer login to to my account on my two-account Gitea instance. The account is the only admin account and also has 2FA enabled. The other account logs in fine, is not an admin, and does not have 2FA enabled.

Whilst it didn't appear in the above logs, other times when I've tried I've also seen this line:

Unable to negotiate with 10.42.0.1 port 62090: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1 [preauth]

Not sure if it's relevant though because like I said it didn't appear when I tried in the above logs.

Also note that I've read this thread but after checking the account in the database is_active is t. Forum post here.

[birkb]

Same for me. It only affects the local admin user. The LDAP users are working fine.

[kisaragi-hiu]

Does it help if you upgrade the Gitea Helm Chart to the latest version, 2.1.0? (at https://git.ssene.ca/sseneca/k8s-server/src/branch/master/gitea/release.yaml#L23)

I don't actually know Kubernetes, but I can see that there seems to be a version mismatch.

[sseneca]

Just tried it but unfortunately version 2.0.7+ of the Chart is broken on Kubernetes 1.18, which I'm stuck on for now. I'll follow that issue and once it's resolved try to upgrade the Chart version again.

[birkb]

I use https://dl.gitea.io/gitea/1.13.0/gitea-1.13.0-linux-amd64 in my custom Dockerfile with Postgres 12.5 running in another container. Before the upgrade the login with the local admin user was working fine.

[Antoine2tt]

Same here for me
I'm using docker, last version of Gitea, and i'm running it with SQLite.

(Going back to 1.12.4 solved my login problem)

[onedr0p]

I am having the same issue! I just upgraded to 2.1.0 of the helm chart. Wish I saw this issue beforehand :(

[luhahn]

Hi there i will have a look at this issue tomorrow

[onedr0p]

Thanks @luhahn you know where to reach me on Discord if you need any info :D It looks like it might be more of an issue with Gitea 1.13.0 than the chart. But I dunno.

[luhahn]

yeah, i dont know, the init container looks fine. But i can't spend any time today for looking at the chart

[techknowlogick]

This appears related specifically to the helm chart, could you open an issue in the helm chart https://gitea.com/gitea/helm-chart so we can track it there?

[birkb]

@techknowlogick The problem also affects the downloaded gitea binary. Should we track that in a separate case?

[luhahn]

Hi all, please see https://gitea.com/gitea/helm-chart/issues/79 for more information on this issue.

[sseneca]

I'm confused. Both @birkb and @Antoine2tt are facing this same bug, and neither use the Helm chart. Why is this bug being tracked there?

[luhahn]

Well I'm sure I need to adjust some stuff on the init container for the helm chart. So I also created an issue there.
At least the admin creation changed.

However since this also affects the downloaded binary we should not close this issue here. If I find something useful I will also post it here.

[luhahn]

Well, I deleted the admin user via the gitea cli inside my container and created it again. Which seems to work even after reinstalling the Chart.
However it is now required to set the following even for admin accounts, if you don't want to change the password after creation:

--must-change-password=false

# full command
gitea admin create-user --username  myAdmin --password 'myPassword123 --email my@admin.com --admin --must-change-password=false 

Edit: you can also create an additional admin, if you don't want to delete the old one

[sseneca]

For both the users on my instance, must_change_password is already false:

gitea=> SELECT "must_change_password" FROM "user";
 must_change_password
----------------------
 f
 f
(2 rows)

even though one can login and the other can't.

[luhahn]

Yes I know, that's now for newly created admins.

I did not find a way to recover the faulty admins. But I have a workaround, posted in the helm chart issue.

[birkb]

I update the admin password from within the entrypoint.sh and it does not fix the problem.

${GITEA_BIN} ${GITEA_OPTS} admin change-password --username ${GITEA_ADMIN_USERNAME} --password ${GITEA_ADMIN_PASS} 

I will try the delete/create workaround.

[zeripath]

First of all so I've taken so long to look at or notice this - but let's go through this systematically.

@sseneca

After updating to v1.13.0, I can no longer login to to my account on my two-account Gitea instance. The account is the only admin account and also has 2FA enabled. The other account logs in fine, is not an admin, and does not have 2FA enabled.

I think the 2FA here is most likely the biggest hint as to what the issue is - and it would be good to know if you even get asked to provide a 2FA or if that doesn't occur. From your provided logs it looks like that doesn't happen - but it's helpful to know for definite.

@luhahn

I did not find a way to recover the faulty admins. But I have a workaround, posted in the helm chart issue.

It would be helpful to check what the difference is for these users in the user table as it might easily be something wrong with a default value.

In the next reply I'll walk through the provided log.

[sseneca]

Yep, I don't get asked to provide the 2FA code. I get Username or password is incorrect. once I press Sign In.

[zeripath]

OK, let's go through and find the ultimate place that calls and causes this log and think about what we can do to get some more information - or understanding.

[Macaron] 2020-12-03 22:12:20: Started POST /user/login for 10.42.0.1

Comes from:

gitea/routers/routes/routes.go

Line 106 in e39ed0b

_ = log.GetLogger("router").Log(0, level, "Started %s %s for %s", log.ColoredMethod(ctx.Req.Method), ctx.Req.URL.RequestURI(), ctx.RemoteAddr())

and matches:

gitea/routers/routes/routes.go

Line 366 in e39ed0b

m.Post("/login", bindIgnErr(auth.SignInForm{}), user.SignInPost)

gitea/routers/user/auth.go

Lines 151 to 232 in e39ed0b

	// SignInPost response for sign in request
	func SignInPost(ctx *context.Context, form auth.SignInForm) {
	ctx.Data["Title"] = ctx.Tr("sign_in")
	orderedOAuth2Names, oauth2Providers, err := models.GetActiveOAuth2Providers()
	if err != nil {
	ctx.ServerError("UserSignIn", err)
	return
	}
	ctx.Data["OrderedOAuth2Names"] = orderedOAuth2Names
	ctx.Data["OAuth2Providers"] = oauth2Providers
	ctx.Data["Title"] = ctx.Tr("sign_in")
	ctx.Data["SignInLink"] = setting.AppSubURL + "/user/login"
	ctx.Data["PageIsSignIn"] = true
	ctx.Data["PageIsLogin"] = true
	ctx.Data["EnableSSPI"] = models.IsSSPIEnabled()
	if ctx.HasError() {
	ctx.HTML(200, tplSignIn)
	return
	}
	u, err := models.UserSignIn(form.UserName, form.Password)
	if err != nil {
	if models.IsErrUserNotExist(err) {
	ctx.RenderWithErr(ctx.Tr("form.username_password_incorrect"), tplSignIn, &form)
	log.Info("Failed authentication attempt for %s from %s", form.UserName, ctx.RemoteAddr())
	} else if models.IsErrEmailAlreadyUsed(err) {
	ctx.RenderWithErr(ctx.Tr("form.email_been_used"), tplSignIn, &form)
	log.Info("Failed authentication attempt for %s from %s", form.UserName, ctx.RemoteAddr())
	} else if models.IsErrUserProhibitLogin(err) {
	log.Info("Failed authentication attempt for %s from %s", form.UserName, ctx.RemoteAddr())
	ctx.Data["Title"] = ctx.Tr("auth.prohibit_login")
	ctx.HTML(200, "user/auth/prohibit_login")
	} else if models.IsErrUserInactive(err) {
	if setting.Service.RegisterEmailConfirm {
	ctx.Data["Title"] = ctx.Tr("auth.active_your_account")
	ctx.HTML(200, TplActivate)
	} else {
	log.Info("Failed authentication attempt for %s from %s", form.UserName, ctx.RemoteAddr())
	ctx.Data["Title"] = ctx.Tr("auth.prohibit_login")
	ctx.HTML(200, "user/auth/prohibit_login")
	}
	} else {
	ctx.ServerError("UserSignIn", err)
	}
	return
	}
	// If this user is enrolled in 2FA, we can't sign the user in just yet.
	// Instead, redirect them to the 2FA authentication page.
	_, err = models.GetTwoFactorByUID(u.ID)
	if err != nil {
	if models.IsErrTwoFactorNotEnrolled(err) {
	handleSignIn(ctx, u, form.Remember)
	} else {
	ctx.ServerError("UserSignIn", err)
	}
	return
	}
	// User needs to use 2FA, save data and redirect to 2FA page.
	if err := ctx.Session.Set("twofaUid", u.ID); err != nil {
	ctx.ServerError("UserSignIn: Unable to set twofaUid in session", err)
	return
	}
	if err := ctx.Session.Set("twofaRemember", form.Remember); err != nil {
	ctx.ServerError("UserSignIn: Unable to set twofaRemember in session", err)
	return
	}
	if err := ctx.Session.Release(); err != nil {
	ctx.ServerError("UserSignIn: Unable to save session", err)
	return
	}
	regs, err := models.GetU2FRegistrationsByUID(u.ID)
	if err == nil && len(regs) > 0 {
	ctx.Redirect(setting.AppSubURL + "/user/u2f")
	return
	}
	ctx.Redirect(setting.AppSubURL + "/user/two_factor")
	}

Now let's check the SQL

2020/12/03 22:12:20 ...m.io/xorm/core/db.go:286:afterProcess() [I] [SQL] SELECT "id", "type", "name", "is_actived", "is_sync_enabled", "cfg", "created_unix", "updated_unix" FROM "login_source" WHERE (is_actived = $1 and type = $2) [true 6] - 795.229µs
2020/12/03 22:12:20 ...m.io/xorm/core/db.go:286:afterProcess() [I] [SQL] SELECT "id", "type", "name", "is_actived", "is_sync_enabled", "cfg", "created_unix", "updated_unix" FROM "login_source" WHERE (is_actived = $1 and type = $2) [true 7] - 579.519µs

are caused by:

gitea/routers/user/auth.go

Line 155 in e39ed0b

orderedOAuth2Names, oauth2Providers, err := models.GetActiveOAuth2Providers()

So we get in to the meat of this function. Which is performed in:

gitea/routers/user/auth.go

Line 173 in e39ed0b

u, err := models.UserSignIn(form.UserName, form.Password)

gitea/models/login_source.go

Lines 734 to 819 in e39ed0b

	// UserSignIn validates user name and password.
	func UserSignIn(username, password string) (*User, error) {
	var user *User
	if strings.Contains(username, "@") {
	user = &User{Email: strings.ToLower(strings.TrimSpace(username))}
	// check same email
	cnt, err := x.Count(user)
	if err != nil {
	return nil, err
	}
	if cnt > 1 {
	return nil, ErrEmailAlreadyUsed{
	Email: user.Email,
	}
	}
	} else {
	trimmedUsername := strings.TrimSpace(username)
	if len(trimmedUsername) == 0 {
	return nil, ErrUserNotExist{0, username, 0}
	}
	user = &User{LowerName: strings.ToLower(trimmedUsername)}
	}
	hasUser, err := x.Get(user)
	if err != nil {
	return nil, err
	}
	if hasUser {
	switch user.LoginType {
	case LoginNoType, LoginPlain, LoginOAuth2:
	if user.IsPasswordSet() && user.ValidatePassword(password) {
	// Update password hash if server password hash algorithm have changed
	if user.PasswdHashAlgo != setting.PasswordHashAlgo {
	user.HashPassword(password)
	if err := UpdateUserCols(user, "passwd", "passwd_hash_algo"); err != nil {
	return nil, err
	}
	}
	// WARN: DON'T check user.IsActive, that will be checked on reqSign so that
	// user could be hint to resend confirm email.
	if user.ProhibitLogin {
	return nil, ErrUserProhibitLogin{user.ID, user.Name}
	}
	return user, nil
	}
	return nil, ErrUserNotExist{user.ID, user.Name, 0}
	default:
	var source LoginSource
	hasSource, err := x.ID(user.LoginSource).Get(&source)
	if err != nil {
	return nil, err
	} else if !hasSource {
	return nil, ErrLoginSourceNotExist{user.LoginSource}
	}
	return ExternalUserLogin(user, user.LoginName, password, &source)
	}
	}
	sources := make([]*LoginSource, 0, 5)
	if err = x.Where("is_actived = ?", true).Find(&sources); err != nil {
	return nil, err
	}
	for _, source := range sources {
	if source.IsOAuth2() || source.IsSSPI() {
	// don't try to authenticate against OAuth2 and SSPI sources here
	continue
	}
	authUser, err := ExternalUserLogin(nil, username, password, source)
	if err == nil {
	return authUser, nil
	}
	log.Warn("Failed to login '%s' via '%s': %v", username, source.Name, err)
	}
	return nil, ErrUserNotExist{user.ID, user.Name, 0}
	}

The next SQL emitted :

2020/12/03 22:12:20 ...m.io/xorm/core/db.go:286:afterProcess() [I] [SQL] SELECT "id", "lower_name", "name", "full_name", "email", "keep_email_private", "email_notifications_preference", "passwd", "passwd_hash_algo", "must_change_password", "login_type", "login_source", "login_name", "type", "location", "website", "rands", "salt", "language", "description", "created_unix", "updated_unix", "last_login_unix", "last_repo_visibility", "max_repo_creation", "is_active", "is_admin", "is_restricted", "allow_git_hook", "allow_import_local", "allow_create_organization", "prohibit_login", "avatar", "avatar_email", "use_custom_avatar", "num_followers", "num_following", "num_stars", "num_repos", "num_teams", "num_members", "visibility", "repo_admin_change_team_access", "diff_view_style", "theme", "keep_activity_private" FROM "user" WHERE "lower_name"=$1 LIMIT 1 [sseneca] - 860.604µs

is caused by:

gitea/models/login_source.go

Line 758 in e39ed0b

hasUser, err := x.Get(user)

So what is the result of this query? (You may need to change it slightly to match postgreSQL variant.)

Then we get no logs but:

[Macaron] 2020-12-03 22:12:20: Completed POST /user/login 200 OK in 29.759793ms
2020/12/03 22:12:20 routers/user/auth.go:177:SignInPost() [I] Failed authentication attempt for sseneca from 10.42.0.1

The first of which is:

gitea/routers/routes/routes.go

Line 112 in e39ed0b

_ = log.GetLogger("router").Log(0, level, "Completed %s %s %v %s in %v", log.ColoredMethod(ctx.Req.Method), ctx.Req.URL.RequestURI(), log.ColoredStatus(status), log.ColoredStatus(status, http.StatusText(rw.Status())), log.ColoredTime(time.Since(start)))

and the counterpart of the first log. But the second bit comes from:

gitea/routers/user/auth.go

Line 177 in e39ed0b

log.Info("Failed authentication attempt for %s from %s", form.UserName, ctx.RemoteAddr())

'

Now that implies that we have received a models.ErrUserNotExist{}

How do we get a models.ErrUserNotExist?

First of all. Let's assume that

gitea/models/login_source.go

Line 758 in e39ed0b

hasUser, err := x.Get(user)

returned hasUser := true - You can check this and should have checked the result of the query that this is associated with already. What is the result of that query?

To obtain a models.ErrUserNotExist if hasUser == true we have two options:

gitea/models/login_source.go

Lines 764 to 765 in e39ed0b

	switch user.LoginType {
	case LoginNoType, LoginPlain, LoginOAuth2:

(this is the most likely path)

gitea/models/login_source.go

Line 766 in e39ed0b

if user.IsPasswordSet() && user.ValidatePassword(password) {

The most likely problem is that this fails and interestingly if we look at the line below it we get a hint as to what might really be the problem:

gitea/models/login_source.go

Line 769 in e39ed0b

if user.PasswdHashAlgo != setting.PasswordHashAlgo {

I recall on 1.13 we changed the default hashing algorithm to argon2. Is it possible that hashing algorithm is detected incorrectly for postgres? The presumption was that the default would have been stored in the database at the time of insertion and was previously pbkdf2.

I'm gonna stop here because I bet that that is the issue.

[sseneca]

I recall on 1.13 we changed the default hashing algorithm to argon2. Is it possible that hashing algorithm is detected incorrectly for postgres? The presumption was that the default would have been stored in the database at the time of insertion and was previously pbkdf2.

gitea=> SELECT "lower_name", "passwd_hash_algo" FROM "user";
 lower_name | passwd_hash_algo 
------------+------------------
 jonnobrow  | argon2
 sseneca    | pbkdf2
(2 rows)

sseneca is the user that cannot login, jonnobrow is the user that can. So that makes sense.

[zeripath]

OK the other routes should definitely cause more SQL to be emitted so those are ruled out. The issue is definitely at:

gitea/models/login_source.go

Line 766 in e39ed0b

if user.IsPasswordSet() && user.ValidatePassword(password) {

[zeripath]

gitea/models/user.go

Lines 431 to 434 in e39ed0b

	// IsPasswordSet checks if the password is set or left empty
	func (u *User) IsPasswordSet() bool {
	return !u.ValidatePassword("")
	}

just tests that the user's password is not empty - and although there are a few changes here I think it should make no difference.

gitea/models/user.go

Lines 418 to 429 in e39ed0b

	// ValidatePassword checks if given password matches the one belongs to the user.
	func (u *User) ValidatePassword(passwd string) bool {
	tempHash := hashPassword(passwd, u.Salt, u.PasswdHashAlgo)
	if u.PasswdHashAlgo != algoBcrypt && subtle.ConstantTimeCompare([]byte(u.Passwd), []byte(tempHash)) == 1 {
	return true
	}
	if u.PasswdHashAlgo == algoBcrypt && bcrypt.CompareHashAndPassword([]byte(u.Passwd), []byte(passwd)) == nil {
	return true
	}
	return false
	}

calls hashPassword to check the user's passwd:

gitea/models/user.go

Lines 392 to 410 in e39ed0b

	func hashPassword(passwd, salt, algo string) string {
	var tempPasswd []byte
	switch algo {
	case algoBcrypt:
	tempPasswd, _ = bcrypt.GenerateFromPassword([]byte(passwd), bcrypt.DefaultCost)
	return string(tempPasswd)
	case algoScrypt:
	tempPasswd, _ = scrypt.Key([]byte(passwd), []byte(salt), 65536, 16, 2, 50)
	case algoArgon2:
	tempPasswd = argon2.IDKey([]byte(passwd), []byte(salt), 2, 65536, 8, 50)
	case algoPbkdf2:
	fallthrough
	default:
	tempPasswd = pbkdf2.Key([]byte(passwd), []byte(salt), 10000, 50, sha256.New)
	}
	return fmt.Sprintf("%x", tempPasswd)
	}

So the question is is u.PasswdHashAlgo being set correctly? I worry that it's being defaulted to argon2 all the time but I don't understand how or why that could happen. The other possibility is that the algo is incorrectly stored and the stored hash has been updated to argon2. One other possibility is that the salt has got renewed but I don't understand how that could happen.

I guess the only answer here is to add more debugging statements in to a locally built copy of gitea. I don't see how helm could be responsible for this - but neither do I understand how this could have become so broken.

[6543]

I'll reopen since it looks not solved to me

[Antoine2tt]

@zeripath i don't know if it can helps you, but i have the same issue (only 1 user, local admin, can not login after 1.13 upgrade, error 500), and i'm not running Gitea with Helm (i don't even know what Helm is) i'm just running Gitea with Docker (database = SQLite)

i've been back to 1.12.4 and it's working again.

If you need anything, i'll be glad to help you.

[sseneca]

The other possibility is that the algo is incorrectly stored and the stored hash has been updated to argon2. One other possibility is that the salt has got renewed but I don't understand how that could happen.

Yeah, both of these accounts were created before I upgraded Gitea to 1.13: I created sseneca on 2020-10-20, when I was running Gitea 1.12.5, and jonnobrow was created about a month later on 2020-11-13, when I was running the same version. So I doubt they were different before I upgraded Gitea to 1.13; it makes sense that something went wrong with the accounts in the upgrade to 1.13.

Like others have said I really doubt this is anything to do with Helm.

[zeripath]

I think we need one of you to build a gitea with a some added logging to the ValidatePassword function.

[zeripath]

So I can't replicate this on my testing system. I wonder is this possibly a schema issue?

from 1.13 we're a lot more strict with the schema - is it possible that the user table is in a different schema?

[sseneca]

I don't have time to debug things but the workaround posted by @luhahn worked for me. Now I can login again after changing my password with the other admin account. I checked the DB and the password hash algorithm has also been updated:

gitea=> SELECT "lower_name", "passwd_hash_algo" FROM "user";
 lower_name | passwd_hash_algo 
------------+------------------
 jonnobrow  | argon2
 sseneca    | argon2
(2 rows)

[zeripath]

@sseneca did the passwd and/or salt change? Is it possible that the stored passwd hash is in the incorrect hash_algo?

[fblaese]

I have a similar issue. Users that tried to reset their password using account recovery were unable to login using the new password. Setting the password from command line had the same effect.

Only when setting the password using the web interface, they could finally log in again. I checked the user table in the database and noticed, that passwd_hash_also was only updated when the password was set using the webui, so I have the feeling that the password is hashed using the new algorithm but the passwd_hash_algo is not updated in some cases.

[fblaese]

Could the issue be, that passwd_hash_algo is not updated here when the password is changed?

gitea/routers/user/setting/account.go

Line 71 in f8e9a3d

if err := models.UpdateUserCols(ctx.User, "salt", "passwd"); err != nil {

[zeripath]

Could the issue be, that passwd_hash_algo is not updated here when the password is changed?

gitea/routers/user/setting/account.go

Line 71 in f8e9a3d

if err := models.UpdateUserCols(ctx.User, "salt", "passwd"); err != nil {

That could very well be it.