"providers": true will break installation of packages with a replace package definition

[andygrunwald]

We run our own instance of Medusa with the configuration:

{
    "repositories": [],
    "require": [
        "symfony/symfony",
        "symfony/filesystem"
    ],
    "repodir": "/appdata/git-mirror",
    "satisurl": "http://my.url/git-mirror",
    "satisconfig": "satis.json"
}

So Medusa is mirroring the repositories.
After this is done we kickoff satis do generate the usual Satis html with the configuration:

{
    "name": "php package mirror",
    "homepage": "http://my.url/external/",
    "require-all": true,
    "require-dependencies": true,
    "providers": true,
    "archive": {
        "directory": "dist",
        "format": "tar",
        "prefix-url": "http://my.url/external/",
        "skip-dev": true
    },
    "repositories": [
        ...
    ]
}

The important part here is "providers": true,.
The providers feature was introduced in #295.

If providers is set to true this composer json will not be able to install both packages and leads to a conflict.

{
    "name": "foo/test",
    "authors": [
        {
            "name": "foo",
            "email": "foo@bar.com"
        }
    ],
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "composer",
            "url": "http://my.url/external"
        }
    ],
    "require": {
           "symfony/filesystem": "3.1.6",
           "symfony/symfony": "3.1.6"
    }
}

This composer.json will succeed, but it will install filesystem in 3.1.5 and symfony in 3.1.6 or vice versa:

{
    "name": "foo/test",
    "authors": [
        {
            "name": "foo",
            "email": "foo@bar.com"
        }
    ],
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "composer",
            "url": "http://my.url/external"
        }
    ],
    "require": {
           "symfony/filesystem": "^3.0",
           "symfony/symfony": "^3.0"
    }
}

The reason is the usage of replace in symfony.
@aaukt mentioned this in the pull request.

From having build this myself i can tell you, that special care has to be taken of "replace" packages. The package that is replaced has to include the version information of its "replacee" in the json file.
As an example you can see how it is done in https://github.com/composer/composer/tree/master/doc/fixtures/repo-composer-with-providers (in detail this )

The sentence "From having build this myself..." is refering to the PHP package aaukt/accelerando.

This is doing (nearly) the same like the provider feature, but with taking care of replacing packages.
See

• protected function findReplacee($packages, $replaced)
• Call to findReplacee

Any chance to add support for the replace functionality?

/cc @Fleshgrinder @aaukt @copitz

[stof]

the providers implementation in the dumper indeed does not take care of identifying all providers. It takes care only of the main package name, not of provided and replaced packages. This is a bug in the implementation

[copitz]

Hey Andy,

could you please try out the fork - if it's working I'll open a PR with that.

@stof Do you know, how provided packages should be added? Same as replacements? If the provided packages are not in the satis selection - which seems to be most likely the case - which information should be dumped for the provided package?

[stof]

@copitz as far as the repository is concerned, replace and provide needs to be treated the same, yes. replace actually means provide+conflict in composer, and the repository side does not care about conflict rules.

[andygrunwald]

@copitz I will test it. Gimme a little bit more time, ok?

[andygrunwald]

@copitz Sadly not successful :(

Composer.json

{
    "name": "foo/test",
    "authors": [
        {
            "name": "foo",
            "email": "foo@bar.com"
        }
    ],
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "composer",
            "url": "http://my.ip/external/"
        }
    ],
    "require": {
           "symfony/filesystem": "3.1.6",
           "symfony/symfony": "3.1.6"
    }
}

Error message after an composer install:

You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading composer repositories with package information
Warning: Accessing 10.1.13.17 over http which is an insecure protocol.
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for symfony/filesystem 3.1.6 -> satisfiable by symfony/filesystem[v3.1.6].
    - don't install symfony/symfony v3.1.6|don't install symfony/filesystem v3.1.6
    - Installation request for symfony/symfony 3.1.6 -> satisfiable by symfony/symfony[v3.1.6].

[andygrunwald]

Here is a log of my satis run:

...
You are running Composer with SSL/TLS protection disabled.
Scanning packages
Warning: Accessing 1.2.3.4 over http which is an insecure protocol.
wrote packages to /my/path/external/p/alb/oauth2-php$8089519d443ba8c1aac111e744035b04b87fedc57b995a4cf7f5f7a82429819d.json
wrote packages to /my/path/external/p/doctrine/annotations$2705cc3933bdffa92049559b2d174005f8475eaa32578f84d494e68f72254871.json
wrote packages to /my/path/external/p/doctrine/cache$c91f641bd5878e519361df0bcba9752e89c17bc9265c6e74ac2665681516bb21.json
wrote packages to /my/path/external/p/doctrine/collections$c386cadfb745c096f605d2f48a203484385949d9417ed06d732ba4d5ded4a6f9.json
...

[/my/path/external]$ cat packages.json
{
    "packages": [],
    "providers-url": "/external/p/%package%$%hash%.json",
    "providers": {
        "alb/oauth2-php": {
            "sha256": "8089519d443ba8c1aac111e744035b04b87fedc57b995a4cf7f5f7a82429819d"
        },
        "doctrine/annotations": {
            "sha256": "2705cc3933bdffa92049559b2d174005f8475eaa32578f84d494e68f72254871"
        },
        "doctrine/cache": {
            "sha256": "c91f641bd5878e519361df0bcba9752e89c17bc9265c6e74ac2665681516bb21"
        },
...

And the generated content is there:

[/my/path/external]$ ls -al p/alb/
total 13
drwxr-xr-x+  2 user  group     3 Nov  8 20:28 .
drwxr-xr-x+ 21 user  group    21 Nov  8 20:28 ..
-rw-r--r--+  1 user  group  1240 Nov  8 20:28 oauth2-php$8089519d443ba8c1aac111e744035b04b87fedc57b995a4cf7f5f7a82429819d.json

[copitz]

@andygrunwald Thank you for testing.

Are you really sure, your satis package is exactly at 196bbaf?

Because when I try with a satis configuration like this:

{
    "name": "php package mirror",
    "require-all": true,
    "require-dependencies": true,
    "providers": true,
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "package",
            "package": {
                "name": "symfony/symfony",
                "version": "3.1.6",
                "type": "metapackage",
                "replace": {
                    "symfony/filesystem": "self.version"
                }
            }
        },
        {
            "type": "package",
            "package": {
                "name": "symfony/filesystem",
                "version": "3.1.6",
                "type": "metapackage"
            }
        }
    ]
}

composer install happily succeeds and composer show gives me the expected

symfony/symfony 3.1.6

When I do the same with the current satis master, I get the very same error as you do.

[andygrunwald]

Thx @copitz.

I invested a little bit more work to test it. And now it works.
I don`t know what i did wrong the last time.
I would be happy if you would apply the PR.
For people who will review this PR here is a detailed description how i tested this patch:

I setted up a small test case that everyone is able to reproduce this issue here. I used cedvan/docker-satis as a basement and forked it into andygrunwald/docker-satis (see branch satis-issue-367).

First we will reproduce the original issue.
For this i prepared a prebuild image andygrunwald/docker-satis:satis-aa25081-providers.
This image is bootstrapped with the command php composer.phar create-project composer/satis satis dev-master --stability=dev --keep-vcs.
At the time where the docker image was build the aa250815f3a40313d01630f5d9b2c3dc6f5a4185 was the up to date commit.

If you clone my fork (andygrunwald/docker-satis) and switch to the cloned directory and fire (and replace Path/To/Clone) ...

docker run --name satis -d \
    -p 8080:80 \
    -v /Path/To/Clone/docker-satis/examples/satis.json:/var/www/satis.json \
    andygrunwald/docker-satis:satis-aa25081-providers

it will start a docker container, with a satis and a satis.json with all needed packages.
Take in mind: When the container starts, satis needs to crawl. You can check the progress by docker logs CONTAINER-ID. Should be less than two minutes.
Next step is to create a composer.json somewhere and add this content:

{
    "name": "foo/test",
    "authors": [
        {
            "name": "foo",
            "email": "foo@bar.com"
        }
    ],
    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "composer",
            "url": "http://192.168.99.100:8080/"
        }
    ],
    "require": {
        "symfony/filesystem": "3.1.6",
        "symfony/symfony": "3.1.6"
    }
}

Please replace 192.168.99.100 with your docker host / docker-machine IP address.
Fire a composer clear-cache and a composer install and you should see this error:

➜  composer-test composer install
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading composer repositories with package information
Warning: Accessing 192.168.99.100 over http which is an insecure protocol.
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for symfony/filesystem 3.1.6 -> satisfiable by symfony/filesystem[v3.1.6].
    - don't install symfony/symfony v3.1.6|don't install symfony/filesystem v3.1.6
    - Installation request for symfony/symfony 3.1.6 -> satisfiable by symfony/symfony[v3.1.6].

Next step would be to test the fork by @copitz.
For this i provided another pre compiled docker container: andygrunwald/docker-satis:netresearch-satis-196bbaf-providers.
The only difference between the second and the first container is that i replaced one line in the Dockerfile: php composer.phar create-project composer/satis satis dev-master --stability=dev --keep-vcs with php composer.phar create-project --repository='{"type": "git","url": "https://github.com/netresearch/satis.git"}' composer/satis satis dev-master --stability=dev --keep-vcs.
This was necessary to load the fork.

Now lets test it:
First lets kill the previous container via docker kill satis && docker rm satis.
Next one boot up the new one:

docker run --name satis -d \
    -p 8080:80 \
    -v /Path/To/Clone/docker-satis/examples/satis.json:/var/www/satis.json \
    andygrunwald/docker-satis:netresearch-satis-196bbaf-providers

Same as above.
Satis starts crawling. Take a minute or two. Check progress via docker logs CONTAINER-ID.
After it we go back to our composer.json example.

Hit composer clear-cache and composer install and it works.

[copitz]

Thanks @andygrunwald for the help and extensive testing!

It think this can be closed then :)

[andygrunwald]

@copitz Great. That was quick. I will ship the new version in production in our setup, test it with a way bigger app and then i will close it ok?

[alcohol]

@andygrunwald fine with me. Please let us know if this resolves it :-)

[andygrunwald]

Shipped to our production system. The firsts tests worked quite well.
I will close this ticket so far. Once more problems occur, i will create a new ticket.

Thanks a lot @copitz and @alcohol!
Especially for this quick merge!