Add support for different journal modes #4272

jonasfranz · 2018-06-18T13:44:40Z

Fixes #2040 (partially)

Add the ability to set the journal mode used by SQLite3. The new default journal mode is WAL, since it produces more less database locked errors (nearly no errors).

Learn more about WAL: https://www.sqlite.org/wal.html

mattn/go-sqlite3#569 (comment)

Maybe should we also add SetMaxOpenConnection(1). But I'm not sure about that.

Signed-off-by: Jonas Franz <info@jonasfranz.software>

lunny · 2018-06-18T14:18:58Z

modules/indexer/issue.go

@@ -64,7 +64,7 @@ func InitIssueIndexer(populateIndexer func() error) {
 		log.Fatal(4, "InitIssuesIndexer: create index, %v", err)
 	}
 	if err = populateIndexer(); err != nil {
-		log.Fatal(4, "InitIssueIndexer: populate index, %v", err)
+		//log.Fatal(4, "InitIssueIndexer: populate index, %v", err)


why comment this line?

lunny · 2018-06-18T14:19:32Z

@JonasFranzDEV Since that, could you enable make test-sqlite?

codecov-io · 2018-06-18T14:27:49Z

Codecov Report

Merging #4272 into master will increase coverage by 0.11%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #4272      +/-   ##
==========================================
+ Coverage   19.97%   20.09%   +0.11%     
==========================================
  Files         153      153              
  Lines       30524    30697     +173     
==========================================
+ Hits         6097     6168      +71     
- Misses      23513    23587      +74     
- Partials      914      942      +28

Impacted Files	Coverage Δ
models/models.go	`33.19% <0%> (+4%)`	⬆️
models/org_team.go	`47.22% <0%> (-5.75%)`	⬇️
models/repo_collaboration.go	`64.22% <0%> (-3.3%)`	⬇️
models/repo_watch.go	`61.62% <0%> (-1.89%)`	⬇️
models/notification.go	`66.84% <0%> (-0.95%)`	⬇️
routers/user/setting/account.go	`12.5% <0%> (-0.31%)`	⬇️
models/org.go	`68.18% <0%> (-0.04%)`	⬇️
routers/api/v1/repo/issue.go	`0% <0%> (ø)`	⬆️
routers/user/setting/security_openid.go	`0% <0%> (ø)`	⬆️
routers/repo/topic.go	`0% <0%> (ø)`	⬆️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 85414d8...c58e2c5. Read the comment docs.

Bwko · 2018-06-18T15:56:19Z

custom/conf/app.ini.sample

@@ -213,6 +213,8 @@ SSL_MODE = disable
 PATH = data/gitea.db
 ; For "sqlite3" only. Query timeout
 SQLITE_TIMEOUT = 500
+; For "sqlite3" only. Journal mode
+JOURNAL_MODE = WLA


WLA -> WAL (https://www.sqlite.org/pragma.html#pragma_journal_mode)

Bwko · 2018-06-18T15:56:37Z

docs/content/doc/advanced/config-cheat-sheet.en-us.md

@@ -129,6 +129,9 @@ Values containing `#` or `;` must be quoted using `` ` `` or `"""`.
 - `PASSWD`: **\<empty\>**: Database user password. Use \`your password\` for quoting if you use special characters in the password.
 - `SSL_MODE`: **disable**: For PostgreSQL only.
 - `PATH`: **data/gitea.db**: For SQLite3 only, the database file path.
+- `JOURNAL_MODE` **WLA**: For SQLite3 only, the journal mode. Use `DELETE` mode if you're running Gitea on a network file system


Bwko · 2018-06-18T15:56:40Z

docs/content/doc/advanced/config-cheat-sheet.en-us.md

@@ -129,6 +129,9 @@ Values containing `#` or `;` must be quoted using `` ` `` or `"""`.
 - `PASSWD`: **\<empty\>**: Database user password. Use \`your password\` for quoting if you use special characters in the password.
 - `SSL_MODE`: **disable**: For PostgreSQL only.
 - `PATH`: **data/gitea.db**: For SQLite3 only, the database file path.
+- `JOURNAL_MODE` **WLA**: For SQLite3 only, the journal mode. Use `DELETE` mode if you're running Gitea on a network file system
+or your operating system does not support VFS. Please checkout the [WAL documentation](https://www.sqlite.org/wal.html) to learn more. WLA


lafriks · 2018-06-19T13:12:20Z

Info about journal mode should be also added in admin settings display page

lunny · 2018-06-23T06:02:16Z

The wal mode maybe an enforce mode not an option.

Upgrade to latest go-sqlite3 because WAL is only supported with newer versions Signed-off-by: Jonas Franz <info@jonasfranz.software>

techknowlogick · 2018-06-26T00:27:07Z

@JonasFranzDEV I've made the changes above and put them in a PR for you to be able to merge them into this PR: jonasfranz#2

yasuokav · 2018-06-26T01:47:16Z

Some files are missing in the last commit (c58e2c5) e.g. sqlite3_func_crypt.go

In addition, I think the database connection need be closed before Gitea exits. Otherwise, gitea.db-shm and gitea.db-wal files never gets deleted.

https://www.sqlite.org/wal.html#avoiding_excessively_large_wal_files

When the last connection to a database closes, that connection does one last checkpoint and then deletes the WAL and its associated shared-memory file, to clean up the disk.

https://www.sqlite.org/tempfiles.html#write_ahead_log_wal_files

However, if the last connection does not shutdown cleanly, the WAL file will remain in the filesystem and will be automatically cleaned up the next time the database is opened.

But I don't know why it hasn't been deleted or reset to 0KB.

jonasfranz · 2018-07-05T16:06:39Z

I've tried it locally and it does not improve the tests at any point. I think I should close this PR since it does not offer benefits to the user. Also setting max open connections to 1 results in the test hanging up.

jonasfranz added 2 commits June 18, 2018 15:15

Add support for different journal modes

f06e9c6

Signed-off-by: Jonas Franz <info@jonasfranz.software>

Move journal mode to connection string

c5e50c4

Signed-off-by: Jonas Franz <info@jonasfranz.software>

lunny reviewed Jun 18, 2018

View reviewed changes

bkcsoft added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 18, 2018

lunny added this to the 1.6.0 milestone Jun 18, 2018

lunny added the type/bug label Jun 18, 2018

Bwko reviewed Jun 18, 2018

View reviewed changes

Replace WLA with WAL

c58e2c5

Upgrade to latest go-sqlite3 because WAL is only supported with newer versions Signed-off-by: Jonas Franz <info@jonasfranz.software>

jonasfranz closed this Jul 5, 2018

lunny removed this from the 1.6.0 milestone Jul 6, 2018

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for different journal modes #4272

Add support for different journal modes #4272

jonasfranz commented Jun 18, 2018 •

edited

Loading

lunny Jun 18, 2018

lunny commented Jun 18, 2018

codecov-io commented Jun 18, 2018 •

edited

Loading

Bwko Jun 18, 2018

Bwko Jun 18, 2018

Bwko Jun 18, 2018

lafriks commented Jun 19, 2018

lunny commented Jun 23, 2018 •

edited

Loading

techknowlogick commented Jun 26, 2018

yasuokav commented Jun 26, 2018 •

edited

Loading

jonasfranz commented Jul 5, 2018

Add support for different journal modes #4272

Add support for different journal modes #4272

Conversation

jonasfranz commented Jun 18, 2018 • edited Loading

lunny Jun 18, 2018

Choose a reason for hiding this comment

lunny commented Jun 18, 2018

codecov-io commented Jun 18, 2018 • edited Loading

Codecov Report

Bwko Jun 18, 2018

Choose a reason for hiding this comment

Bwko Jun 18, 2018

Choose a reason for hiding this comment

Bwko Jun 18, 2018

Choose a reason for hiding this comment

lafriks commented Jun 19, 2018

lunny commented Jun 23, 2018 • edited Loading

techknowlogick commented Jun 26, 2018

yasuokav commented Jun 26, 2018 • edited Loading

jonasfranz commented Jul 5, 2018

jonasfranz commented Jun 18, 2018 •

edited

Loading

codecov-io commented Jun 18, 2018 •

edited

Loading

lunny commented Jun 23, 2018 •

edited

Loading

yasuokav commented Jun 26, 2018 •

edited

Loading