[Docker][database] the maker does not work because of the YamlSourceManipulator

[lyrixx]

I used https://github.com/dunglas/symfony-docker but I edit the files to use .yaml instead of .yml (see the PR).

But I get an error when running the maker:

>…m/lyrixx/SFLive-Paris2016-Workflow(docker +) bin/console make:docker:data -v

- Docker Compose Setup-
-----------------------

 We found your existing docker-compose.yaml: Let's update it!

 Which database service will you be creating?:
  [0] MySQL
  [1] MariaDB
  [2] Postgres
 > 2

 For a list of supported versions, check out https://hub.docker.com/_/postgres

 What version would you like to use? [alpine]:
 > 



In YamlSourceManipulator.php line 868:
                                                                     
  [Symfony\Bundle\MakerBundle\Util\YamlManipulationFailedException]  
  Cannot find the original value "80:80"                             
                                                                     

Exception trace:
  at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:868
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->findEndPositionOfValue() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:568
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->advanceBeyondValue() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:215
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->updateData() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:207
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->updateData() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:207
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->updateData() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:207
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->updateData() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/YamlSourceManipulator.php:88
 Symfony\Bundle\MakerBundle\Util\YamlSourceManipulator->setData() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Util/ComposeFileManipulator.php:72
 Symfony\Bundle\MakerBundle\Util\ComposeFileManipulator->addDockerService() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Maker/MakeDockerDatabase.php:131
 Symfony\Bundle\MakerBundle\Maker\MakeDockerDatabase->generate() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/vendor/symfony/maker-bundle/src/Command/MakerCommand.php:104
 Symfony\Bundle\MakerBundle\Command\MakerCommand->execute() at /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Command/Command.php:259
 Symfony\Component\Console\Command\Command->run() at /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Application.php:938
 Symfony\Component\Console\Application->doRunCommand() at /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:96
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Application.php:266
 Symfony\Component\Console\Application->doRun() at /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:82
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Application.php:142
 Symfony\Component\Console\Application->run() at /home/gregoire/dev/github.com/lyrixx/SFLive-Paris2016-Workflow/bin/console:43

make:docker:database [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-e|--env ENV] [--no-debug] [--] <command>

ping @dunglas

[jrushlow]

Hi at @lyrixx can you post your docker-compose.yaml before attempting to run make:docker:database ?

[dunglas]

Maybe should we deprecate this command? Now that Flex has Docker and Docker Compose configurators, it shouldn't be necessary to use it anymore.

[jrushlow]

Now that Flex has Docker and Docker Compose configurators, it shouldn't be necessary to use it anymore.

How do those configurators work? From what I gather on the src - we can set

//composer.json
....
"extra": {
    "symfony": {
        "docker":  true
            .....

and it will add say a database service to docker-compose.yaml?

I guess the better question to ask is, if I'm spinning up a new symfony project, how do I get flex to give me a / update an existing docker-compose.yaml

[weaverryan]

Maybe should we deprecate this command? Now that Flex has Docker and Docker Compose configurators, it shouldn't be necessary to use it anymore

We definitely need to find a single path forward :). But I actually prefer the makers over the Flex recipe style. The maker commands give us the ability to customize what the user wants and they can run the command at any time plus the ability for us to educate them on the files we modified and what they should do next. The automatic nature of Flex is super great for DX (just composer require foo and docker-compose up and done!), but not as flexible or good for clarity.

My current thinking is that we should keep pushing forward with both experiments (I almost never like having 2 things) to see if either gain significant traction. We could, however, keep them in sync (or near sync) with each other: there's no reason the commands should generate significantly different code than the recipes. For example, right now, the recipe adds a volume to keep data persistent and the MakerBundle command doesn't. Probably it could/should :).

Btw, fixing the underlying issue for this should be easy - this is a new edge case for YamlSourceManipulator that we can squash easily.

[dunglas]

The maker commands give us the ability to customize what the user wants

Sure, but it will also be a real pain to maintain. For instance, the command currently doesn't work: you need more than a service in Docker-Compose. You also need the appropriate PHP extension, and to do so you have to edit the Dockerfile.

IMHO it's way better, and easier to maintain, to generate good defaults that can be tweaked by the user (it's the spirit of Flex), than supporting a lot of (hard to maintain) possibilities.

For instance, defaulting to Postgres is a conscious choice and has a lot of benefits. Example: the PG LISTEN/NOTIFY mode for Messenger is automatically supported and available. We'll not be able to capitalize on good defaults with a maker.

Also, the Flex integration scales well. It already supports Doctrine, Messenger, Panther, Mercure, and we can add easily as many services as we want. For Maker, we'll need a command per service to install, and the user will have to discover it and to execute it. It's less straightforward imho.

By the way, shouldn't the command also generate data volumes?

[dunglas]

and it will add say a database service to docker-compose.yaml? - @jrushlow

Exactly!

This can also be scripted like this: https://github.com/dunglas/symfony-docker/blob/main/docker/php/docker-entrypoint.sh#L21

To support the Dockerfile Configurator, you'll also need markers in your Dockerfile: https://github.com/dunglas/symfony-docker/blob/main/Dockerfile#L85-L86

[weaverryan]

For instance, the command currently doesn't work: you need more than a service in Docker-Compose. You also need the appropriate PHP extension, and to do so you have to edit the Dockerfile.

Well, that's not fair to say "it doesn't work" - I use it all the time! :) The command currently follows the "Symfony 5: The Fast Track" Docker idea. That's why it doesn't add/edit a Dockerfile. But we have discussed allowing support for both workflows (php in the container and php locally).

By the way, shouldn't the command also generate data volumes?

I said exactly this above :). I was saying how the Flex recipe does, and so the command probably should too :). Or did I misunderstand and you're talking about something else?

For instance, defaulting to Postgres is a conscious choice and has a lot of benefits. Example: the PG LISTEN/NOTIFY mode for Messenger is automatically supported and available. We'll not be able to capitalize on good defaults with a maker.

We can absolutely still make Postgres the default choice in MakerBundle as well (and even educate people on why they should prefer it).

Also, the Flex integration scales well. It already supports Doctrine, Messenger, Panther, Mercure, and we can add easily as many services as we want. For Maker, we'll need a command per service to install, and the user will have to discover it and to execute it. It's less straightforward imho.

Very true! But I don't see this as a problem - a post-install script can educate the user about the commands. Yes, it's less automatic than the Flex version - that is definitely a benefit of the Flex version. But the automatic nature comes at a price: complex Docker setup is dumped into my project without any opportunity to educate me on what was done or ask questions to customize things. So this automatic vs less-automatic point has pros/cons - I'm not sure which is better (other than my own opinion).

[weaverryan]

We will fix this, but after talking more with @dunglas, I think that the Flex/recipe approach is very robust and we should pursue that (and then when we're ready, we could deprecate this command).

Cheers!