pipelined plugin css ordering and static naming?

[the4thamigo-uk]

I have a problem with pipeline css when we load balance multiple instances of grav. It seems that the ordering of sections of the pipelined css can be different (although the content is exactly the same). Hence the uri generated from the hash of the css is different and this breaks the load balancer since the css asset has a different name on each server.

In particular it seems like sometimes the langswitcher plugin css gets pipelined before the simplesearch plugin css (both have priority 10) and sometimes the other way around. Is there a way to force the ordering of the plugin css?

Firstly, is this a bug, or is the order of plugins in the pipelined css non-deterministic? i.e. in consecutive startups are you guaranteed to get the same pipelined css?

I have looked at the docs and it seems inconclusive :

If pipelining is turned on in the configuration, assets not excluded from pipelining are combined in the order in which assets were added, then processed according to the pipeline configuration. The combined pipeline result is then rendered before or after non-pipelined assets depending on the setting of css_pipeline_before_excludes.

Secondly, Im looking for a workaround. Is there a way to set the name of the pipelines css file to something static e.g. style.css rather than the uid. Unfortunately, Ive looked at the code and it seems like the uid is enforced.

protected function pipelineCss($group = 'head', $returnURL = true)
    {
        // temporary list of assets to pipeline
        $temp_css = [];
        // clear no-pipeline assets lists
        $this->css_no_pipeline = [];
        // Compute uid based on assets and timestamp
        $uid = md5(json_encode($this->css) . $this->css_minify . $this->css_rewrite . $group);
        $file =  $uid . '.css';
        $inline_file = $uid . '-inline.css';

[the4thamigo-uk]

Ive done some more debugging on our setup, and I can confirm that the 'order' of the plugins is different on the two servers Both servers are running identical docker containers that spin the grav site up behind nginx. No css files are modified at startup time, it seems like the load order is different for the plugins. How can we control this?

[the4thamigo-uk]

Also, am I correct in assuming that the following code is the code that determines the order in which the plugins are loaded :

class Plugins extends Iterator
{
    public $formFieldTypes;

    public function __construct()
    {
        parent::__construct();

        /** @var UniformResourceLocator $locator */
        $locator = Grav::instance()['locator'];

        $iterator = $locator->getIterator('plugins://');
        foreach ($iterator as $directory) {
            if (!$directory->isDir()) {
                continue;
            }

            $plugin = $directory->getBasename();

            $this->add($this->loadPlugin($plugin));
        }
    }

If so, is there a chance that different operating systems might return different order of plugins in the iterator?

[rhukster]

It's definitely possible that any DirectoryIterator could produce a different order on different platforms. Usually the order is alphabetically based on filename so it's generally not an issue.

Are you running two different platforms in your load balanced solution? I thought you mentioned they were identical docker containers, but is the underlying platform different?

[the4thamigo-uk]

I checked that, and the docker hosts are both ubuntu 14.04. The docker image is based off phusion/baseimage:0.9.19. The entrypoint doesnt do anything other than start the nginx server up.

[the4thamigo-uk]

Also, digging deeper, the order for the asset is actually set in the call to addCss(). This is called by each plugin in each plugin's onTwigSiteVariables() handler, and the handlers are added by the following code :

    public function init()
    {
        $grav = Grav::instance();

        /** @var Config $config */
        $config = $grav['config'];

        /** @var EventDispatcher $events */
        $events = $grav['events'];

        foreach ($this->items as $instance) {
            // Register only enabled plugins.
            if ($config["plugins.{$instance->name}.enabled"] && $instance instanceof Plugin) {
                $instance->setConfig($config);
                $events->addSubscriber($instance);
            }
        }

        return $this->items;
    }

The order that the handlers are added is the order of the associative array 'items', and the order that the events are firedis the order in which the handlers are stored in 'RocketTheme\Toolbox\Event\EventDispatcher (hence Symfony\Component\EventDispatcher\EventDispatcher). So Im really strugglling to understand how this can be different in the two containers.

[the4thamigo-uk]

I have added extra logging. In the two containers, the order of loading the plugins is very different

container 1	container 2
problems	shortcode-ui
shortcode-ui	breadcrumbs
shortcode-core	error
breadcrumbs	langswitcher
anchors	problems
simplesearch	anchors
error	highlight
highlight	simplesearch
langswitcher	addsubs shortcode-core

This is also the order in which addSubscriber() is called.

The order in which addCss is called (via the EventDispatcher calling onTwigSiteVariables() ) is completely different :

container 1	container 2
plugin://breadcrumbs/css/breadcrumbs.css	plugin://breadcrumbs/css/breadcrumbs.css
plugin://simplesearch/css/simplesearch.css	plugin://langswitcher/css/langswitcher.css
plugin://langswitcher/css/langswitcher.css	plugin://simplesearch/css/simplesearch.css
plugin://highlight/css/default.css	plugin://highlight/css/default.css

[rhukster]

That's good info. What are the timestamps on the files like in the two containers? Do they line up with the ordering?

[the4thamigo-uk]

in container1 I have a log line in addTo() that outputs the following on container 1:

addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/breadcrumbs\/css\/breadcrumbs.css","remote":false,"priority":10,"order":0,"pipeline":true,"loading":"","group":"head","modified":1503320330}
addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/simplesearch\/css\/simplesearch.css","remote":false,"priority":10,"order":1,"pipeline":true,"loading":"","group":"head","modified":1503320330}
addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/langswitcher\/css\/langswitcher.css","remote":false,"priority":10,"order":2,"pipeline":true,"loading":"","group":"head","modified":1503320330}
addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/highlight\/css\/default.css","remote":false,"priority":10,"order":3,"pipeline":true,"loading":"","group":"head","modified":1503320330}

and on container 2

addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/breadcrumbs\/css\/breadcrumbs.css","remote":false,"priority":10,"order":0,"pipeline":true,"loading":"","group":"head","modified":1503320330}
addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/langswitcher\/css\/langswitcher.css","remote":false,"priority":10,"order":1,"pipeline":true,"loading":"","group":"head","modified":1503320330}
addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/simplesearch\/css\/simplesearch.css","remote":false,"priority":10,"order":2,"pipeline":true,"loading":"","group":"head","modified":1503320330}
addTo{"asset":"\/user\/sites\/mcl\/ssodocs\/plugins\/highlight\/css\/default.css","remote":false,"priority":10,"order":3,"pipeline":true,"loading":"","group":"head","modified":1503320330}

[the4thamigo-uk]

Suspicious that modified is the same for all?

[rhukster]

Yah, i'm wondering if its related to your container setup or docker in general. I need to try some tests to see if i can replicate the order changing.

[the4thamigo-uk]

What determines the ordering in Symfony\Component\EventDispatcher\EventDispatcher, I think this :

    /**
     * Sorts the internal list of listeners for the given event by priority.
     *
     * @param string $eventName The name of the event
     */
    private function sortListeners($eventName)
    {
        krsort($this->listeners[$eventName]);
        $this->sorted[$eventName] = array();
        foreach ($this->listeners[$eventName] as $priority => $listeners) {
            foreach ($listeners as $k => $listener) {
                if (is_array($listener) && isset($listener[0]) && $listener[0] instanceof \Closure) {
                    $listener[0] = $listener[0]();
                    $this->listeners[$eventName][$priority][$k] = $listener;
                }
                $this->sorted[$eventName][] = $listener;
            }
        }
    }

However in my ./vendor/symfony/event-dispatcher/EventDispatcher.php file the equivalent function is :

    /**
     * Sorts the internal list of listeners for the given event by priority.
     *
     * @param string $eventName The name of the event
     */
    private function sortListeners($eventName)
    {
        krsort($this->listeners[$eventName]);
        $this->sorted[$eventName] = call_user_func_array('array_merge', $this->listeners[$eventName]);
    }

[the4thamigo-uk]

I will update. As well as Grav, I can see SimpleSearch and LangSwitcher are out of date.

$ ./bin/grav --version
Grav CLI Application version 1.2.4

01. SimpleSearch    [v1.10.2 -> v1.13.0]
02. LangSwitcher    [v1.3.0 -> v1.4.0]

[the4thamigo-uk]

(Accidentally closed issue sorry)

[the4thamigo-uk]

The problem is still there after updating

[mahagr]

The issue is that in Linux many filesystems return the files by order of inodes or generally without applying for any particular order than in which they have been stored into the disk index. The ordering of the plugins is not well defined. I believe that the addCss() just uses whatever order the files come in, which of course causes a few issues if the files are combined together.

The only way to enforce some well-defined ordering is to order the items after foreach'ing through them. Actually, I just looked into this code few days ago and was hoping that nobody depended on the ordering. :)

[rhukster]

Can you please try editing your system/src/Grav/Common/Plugins.php file constructor with this:

class Plugins extends Iterator
{
    public $formFieldTypes;

    public function __construct()
    {
        parent::__construct();

        /** @var UniformResourceLocator $locator */
        $locator = Grav::instance()['locator'];

        $iterator = $locator->getIterator('plugins://');

        $plugins = [];
        foreach($iterator as $directory) {
            if (!$directory->isDir()) {
                continue;
            }
            $plugins[] = $directory->getBasename();
        }

        natsort($plugins);

        foreach ($plugins as $plugin) {
            $this->add($this->loadPlugin($plugin));
        }
    }

This basically gets a list of the plugin names, natural sorts the, then runs through and adds the plugins. It should ensure the plugins are loaded in the correct order.

[the4thamigo-uk]

I am doing something very similar right now as it happens.

[the4thamigo-uk]

Im pretty sure that is working. I assume that although the containers are based off the same image, aufs gives them different inodes then.

The Events are dispatched in the order that Symfony determines, but this is an incomplete sort and does depend on the order of calls to addSubscriber(). So I think this explains it.

I am suprised nobody else has hit this issue. It seems that no one else is running css pipelining behind a load balancer...?

[rhukster]

ok cool, we're looking into a better long-term fix though.

[rhukster]

At least not with docker containers where the order is so random probably because everything has the same createdate but different inode ordering ( I assume).

[the4thamigo-uk]

Thanks for the support on this guys! Grav is a great tool.