Specifying target context doesn't work unless format also specified

[rossburton]

First, thanks for forking j2cli. I filed an issue with the upstream repo but that is dead now so I thought I'd file it here :)

A lot of the times that I could use j2cli the input JSON isn't a dictionary but is a plain array, for example the output from GitLab's pipeline API is an array of dictionaries.

Currently, this causes j2cli to error out. Maybe j2cli could detect that the top-level object is a list not a dict and pass it as a named variable (data?) in the environment?

[m000]

Hi Ross. This sounds like a case I hadn't accounted for. Could you provide some extra input so I can have this fixed?

One of the features I have added is dictionary squashing. I.e. you can provide multiple json (dict-like) files, which will be squashed into a single dictionary to be used during rendering. E.g. if you supply two files containing {"a": 1, "b": 2} and {"a": 10, "c": 3}, the template will be rendered with {"a": 10, "b": 2, "c": 3}.

This works fine as long as all json input is dict-like. But what would be the expected behaviour if json input was:

• only lists (this should be close to your case)
• mixed lists and dicts (this may as well result in an error, but there may be use cases where this could make sense)

Of course, a PRs would also be welcome if you have already worked something out.

[rossburton]

So the GitLab JSON I'm using looks like this:

[
   {
      "allow_failure" : false,
      "artifacts" : [
         {
            "file_format" : null,
            "file_type" : "trace",
            "filename" : "job.log",
            "size" : 591952
         }
      ],
    ...

My straw-man proposal would be to probe the parsed JSON and if the top-level object is a list then transform that to a dict with a data key pointing at the list.

[m000]

I believe I have already added support for this case, but I haven't updated the docs:

% j2 --help
usage: j2 [-h] [-V] [-v] [-f {ini,json,yaml,env}] [--filters python-file [python-file ...]]
          [--tests python-file [python-file ...]] [--customize python-file] [--no-compact]
          [-U {strict,normal,debug}] [-I] [-o outfile]
          template data [data ...]

Command-line interface to Jinja2 for templating in shell scripts.

positional arguments:
  template              Template file to process.
  data                  Input data specification. Multiple sources in different formats can be
                        specified. The different sources will be squashed into a singled dict.
                        The format is <source>:<context_dest>:<format>. Parts of the
                        specification that are not needed can be ommitted. See examples at the
                        end of the help.

I.e.:

% j2 foo.tpl list.json:data

Would this work for you?

[rossburton]

Oh, that looks promising. I'll try it shortly.

[rossburton]

So my own jinja CLI does this:

args.output.write(template.render({"data": json.load(args.json)}))

But using :data doesn't work with j2:

$ j2 gitlab-pipeline-timeline-template.html  pipeline-4.json:data
[ ... ]
  File "/home/ross/Mess/gitlab-pipeline-timeline-template.html", line 22, in <module>
    {% for job in data|rejectattr("started_at", "none") %}
jinja2.exceptions.UndefinedError: 'str object' has no attribute 'stage'

I'll be able to dig further later.

[m000]

Are you sure you're using my fork of j2? This feature shouldn't be available in kolypto's version.

Btw, I will probably be spinning-off my fork as a new pypi package. I have a fair amount of new features plus some more coming, that spinning off starts to make more sense than attempting to sync with the upstream.

[rossburton]

Yes, this is definitely your fork.

[m000]

Ok. Will take a look later in the day.

[m000]

In the meantime, can you check different settings for the -U option? IIRC, the upstream has a different and non-configurable default.

[rossburton]

I threw in a bit more logging.debug:

DEBUG: Got context {'data': {'{"id": 32964, "status": "success", "stage": "build", "name": "tc0", "ref": "dunfell", "tag": false, "coverage": null, "allow_failure": false, "created_at": "2021-05-06T21:13:03.620Z", "started_at": "2021-05-06T21:31:23.744Z", "finished_at": "2021-05-06T21:36:32.620Z", "duration": 308.876465, "user": {"id": 14, "name": "Ross Burton", "username": "Ross.Burton", "state": "active", "avatar_url": "", "web_url": "", "created_at": "2020-10-29T17:27:38.102Z", "bio": "", "bio_html": "", "location": "", "public_email": "ross.burton@arm.com", "skype": "", "linkedin": "", "twitter": "", "website_url": "", "organization": "", "job_title": "", "work_information": null}, "commit": {"id": "d11487b64e1afb9ec51f25350ced2472526ace95", "short_id": "d11487b6", "created_at": "2021-04-30T13:32:22.000-04:00", "parent_ids": ["9545c366da6c2bd65b1fb833043e36561589b00d"': {}}}

It didn't slurp up the entire list, but just put the first element of the array in the dictionary as data.

[m000]

Omitting the format of the input file in spec doesn't seem to work as expected.

Explicitly specifying the format seems to work though:

j2 foo.tpl list.json:data:json

I'll add to my todo list to make the input spec parsing more robust.

[rossburton]

Confirmed that adding the format fixes the problem, thanks.