Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to add a database migration #23793

Closed
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion docs/content/doc/contributing/guidelines-backend.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ To maintain understandable code and avoid circular dependencies it is important
- `cmd`: All Gitea actual sub commands includes web, doctor, serv, hooks, admin and etc. `web` will start the web service. `serv` and `hooks` will be invoked by Git or OpenSSH. Other sub commands could help to maintain Gitea.
- `tests`: Common test utility functions
- `tests/integration`: Integration tests, to test back-end regressions
- `tests/e2e`: E2e tests, to test test front-end <> back-end compatibility and visual regressions.
- `tests/e2e`: E2e tests, to test test front-end \<\> back-end compatibility and visual regressions.
silverwind marked this conversation as resolved.
Show resolved Hide resolved
- `models`: Contains the data structures used by xorm to construct database tables. It also contains functions to query and update the database. Dependencies to other Gitea code should be avoided. You can make exceptions in cases such as logging.
- `models/db`: Basic database operations. All other `models/xxx` packages should depend on this package. The `GetEngine` function should only be invoked from `models/`.
- `models/fixtures`: Sample data used in unit tests and integration tests. One `yml` file means one table which will be loaded into database when beginning the tests.
Expand Down Expand Up @@ -106,6 +106,46 @@ i.e. `services/user`, `models/repository`.
Since there are some packages which use the same package name, it is possible that you find packages like `modules/user`, `models/user`, and `services/user`. When these packages are imported in one Go file, it's difficult to know which package we are using and if it's a variable name or an import name. So, we always recommend to use import aliases. To differ from package variables which are commonly in camelCase, just use **snake_case** for import aliases.
i.e. `import user_service "code.gitea.io/gitea/services/user"`

### Data Migrations

Whenever you change the database structure, you **must** add a migration as well.
Migrations are located under `models/migrations/v<next_gitea_version>/<version ID>.go`, and should be called in `models/migrations/migrations.go` using `NewMigration("<human readable title of what the migration does>", <migration function>)`

#### Adding data

If you only add data (structures), your migration function will look like the following example that actually exists:
```go
func AddVersionToActionRunner(x *xorm.Engine) error {
type ActionRunner struct {
Version string `xorm:"VARCHAR(64)"`
}

return x.Sync(new(ActionRunner))
```
As you can see, in this case you simply declare a function that takes `x *xorm.Engine` as parameter, and returns an `error` whether the migration was successful.
Then, you declare a local migration type that shows what changes.
Lastly, you call `return x.Sync(new(<migration type>))` to commit the migration and return the error it may produce.
If your table didn't exist previously, it will be created.
All columns present in your new type will also be created if they didn't exist yet.
Existing columns that don't exist in your migration type are unaffected.
If your migration type uses custom types, i.e. a new `int` type to simulate an enum, declare your migration type with `int` instead of `<your enum type>`. This helps to keep migrations consistent, even if the base type of your custom type is changed later.

#### Deleting data

In case you delete a column or table, please adapt the original migration that added it (and all subsequent migrations that use the deleted value) as well so that the column or table isn't added in the first place for new instances.
Keep the original migration as a no-op (with a comment) in case it becomes empty.
Comment on lines +135 to +136
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this what actually happens? I was always under the impression that past migrations should be left alone. There's less chance that an intermediate migration is missed that would cause some odd potential bug.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well… It's at least what I've observed with previous deletions, see for example #23605
we can also decide to use another process, but we should decide this now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, tricky topic. I think leaving the original as-is may be safer than no-oping it. Treat it kind of like a git history that is never rewritten.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, I agree.
However, as you saw in #23605, it sometimes needs to be done.
What about a paragraph such as

If you delete a column or table, do not adapt the original migration adding it and all migrations inbetween.
There is only one exception: If the original migration was broken and didn't work on some databases, you can convert the original migration and every change in between referencing the deleted content to a no-op.

?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sounds good. General rule should be to not alter old migrations except in the rare case they are broken, in wich case it is warranted to fix it and "rewrite history".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@delvh will you update with it?


#### Updating data

When you need to update data, i.e. by moving data from one column to another, you need to do the most:
1. Declare a session that you use to commit your migration
2. Make sure all data is updated
3. Make sure that your migration is only run once:
- if it wasn't applied yet, apply it
- if it was applied already, it should be a no-op
- you cannot use the migration number as a way of identifying if this migration has already been run
- this is needed as it can happen that a previously applied migration can be applied again, and leads to bug reports otherwise

### Important Gotchas

- Never write `x.Update(exemplar)` without an explicit `WHERE` clause:
Expand Down