Support environment variables in migration files using go templates

[ninthclowd]

This PR adds a -template command line option that, when enabled, allows you to substitute any environment variables into your migration file with go templating. i.e. If you set the LOCAL_WAREHOUSE environment variable to MY_DB and have a migration file with the following contents:

                       INSERT INTO {{.LOCAL_WAREHOUSE}}.INVENTORY.RECORDS ('foo') VALUES ('bar');

it will be transformed into the following before being executed:

                       INSERT INTO MY_DB.INVENTORY.RECORDS ('foo') VALUES ('bar');

See https://pkg.go.dev/text/template for more information on supported template formats.

As mentioned in #747, this functionality could be done with envsubt, but that requires you to build custom scripts to do so. It ended up being just as much effort to add the functionality with go template and kept our pipelines and local development workflow more simple.

[coveralls]

Coverage decreased (-0.04%) to 57.732% when pulling 3ffb53b on ninthclowd:feature/templating into 03613f1 on golang-migrate:master.

[makeavish]

@dhui Can you please review this?

[Bessonov]

@dhui it would be nice to have this functionality. Is there a chance to get it into main?

[francescorivola]

This could be a very nice feature. An use case: it will allow to customize migrations files for testings pipelines that run in parallel and that requires to have different databases for each test thread to avoid conflicts.

We currently have to trick migration files with find/replace to obtain a similar feature. The template support will be very interesting.

[EricThompson-PeopleReign]

Yes, this would be a great feature to have merged into main. I'm having to do the envsubst trick right now and I think that's rather hacky.

[dorfire]

+1

[smitpatelx]

I made this script file to run sql-migrate with env variables present in .env file:

• This will work for mac os.
• For linux, replace #!/bin/sh with #!/bin/bash

migrate.sh

#!/bin/sh

# Import env variables from ./.env
set -a
[ -f .env ] && . .env

echo "DB_NAME = $DB_NAME | DB_USER = $DB_USER | DB_HOST = $DB_HOST | DB_PORT = $DB_PORT"
read -p "⚡️ (Sql-Migrate) Type: <up | d | s> " command

if [[ $command == 'up' ]]
then
    sql-migrate up -env="development" sslmode=disable
elif [[ $command == 'd' ]]
then
    sql-migrate down -env="development" sslmode=disable
elif [[ $command == 's' ]]
then
    sql-migrate status -env="development" sslmode=disable
else
    read -p "Run your own command: " custom_command
    sql-migrate $custom_command -env="development" sslmode=disable
fi

[dhawal1248]

+1

[dorsegal]

@dhui Do you guys have a plan to approve it or we should stick with other solutions?

[yauhen-sobaleu]

Can we merge it into main?

[nekinie]

You can wrap the file when using iofs to support templating

type templateFile struct {
	data any
	fs.File
}

func (t *templateFile) Read(p []byte) (int, error) {
	_, err := t.File.Read(p)
	if err != nil {
		return 0, err
	}

	tmpl, err := template.New("").Parse(string(p))
	if err != nil {
		return 0, err
	}

	buf := new(bytes.Buffer)

	err = tmpl.Execute(buf, t.data)
	if err != nil {
		return 0, err
	}

	return copy(p, buf.Bytes()), nil
}

type templateFS struct {
	data any
	embed.FS
}

func (t *templateFS) Open(name string) (fs.File, error) {
	file, err := t.FS.Open(name)
	if err != nil {
		return nil, err
	}

	return &templateFile{t.data, file}, nil
}

hope this helps someone

[y0j0]

+1

[g-getsov]

+1

[offzale]

@ninthclowd thanks for the contribution! I have just tested it in the latest version and it works great. It has been extremely helpful to inject some static environment values in the migrations.

[VinceHIH]

Hi, is it possible to merge this wonderful PR asap, please?

[mhthnz]

will it be merged? may be in feature branch? just to use in my go.mod

[jeevanragula]

Can this PR be merged?