concourse vs code extension not recognise YAML anchors, references, extend

[chaoyangnz]

anchor &
reference *
extend <<

[kdvolder]

I think I know what you mean by 'anchor' and 'reference'. But can you please explain a bit more about 'extend'. What is that exactly?

An example of something that uses these (especially the 'extend <<' one since I'm not familiar with it) would be very welcome as well.

[kdvolder]

Actually.. I found this: https://blog.daemonl.com/2016/02/yaml.html
That seems to explain all three. Adding this link for future reference.

If you have an example pipeline file that uses these features it would be still be welcome. It is allways nice to use a 'real' example as a test case instead of making up something totally artificial by myself.

[kdvolder]

Talk is cheap, show me the code.

Quoting your own moto.... Show me the code... means. Show me your pipeline example please :-)

[chaoyangnz]

I anchor a reference:

---
dcind: &dcind
  type: docker-image
  source:
    repository: kiwiops/stuff-mem-dcind
    tag: latest
...

I got unknow property 'dcind' for type 'Pipeline'.

- task: deploy-ssp-devint
    privileged: true
    config:
      platform: linux
      image_resource: 
        <<: *dcind
      inputs:
      - name: kms
...

Here, I got Unknow property '<<' for type ImageResource.

[kdvolder]

Hmmm... interesting. Technically, I think the error about unknown property 'dcind' is actually not incorrect. Because... your yaml does define a property called 'dcind' and it isn't in the schema. But I suppose it is probably a common way to define something with anchor. You see... this is why a real example matters. I probably wouldn't thought of that without the example.

In other words... thanks for the nice example. If you have more interesting cases of 'stuff' you would like to have working don't hesitate to attach as well.

PS: It may be some time before I get to implementing support for this. It doesn't seem straightforward so its not something I can just pick off when I have some spare time between other chores. But have every intention of fixing this eventually.

[marco-m]

@kdvolder another YAML documentation that I found useful to understand anchors and extend/merge is https://camel.readthedocs.io/en/latest/yamlref.html. I will provide another example of anchors and extend from my own pipelines. BTW I appreciate your kindness!

Another article explicitly on using YAML anchors and merge for Concourse pipelines: https://www.digitalocean.com/community/tutorials/how-to-set-up-continuous-integration-pipelines-with-concourse-ci-on-ubuntu-16-04

[marco-m]

here is a full pipeline:

     1	pool-template: &pool-template
     2	  uri: ((pool-git-backing-store-uri))
     3	  branch: master
     4	  pool: OVERRIDEME
     5	  private_key: ((pool-git-backing-store-private-key))
     6	
     7	sleep: &sleep
     8	  config:
     9	    platform: linux
    10	    image_resource:
    11	      type: docker-image
    12	      source:
    13	        repository: alpine
    14	        tag: latest
    15	    run:
    16	      path: sh
    17	      args:
    18	      - -exc
    19	      - sleep 60
    20	
    21	##########
    22	
    23	resource_types:
    24	
    25	- name: pool
    26	  type: docker-image
    27	  source:
    28	    repository: ((pool-resource-docker-repo))
    29	    tag: ((pool-resource-tag))
    30	
    31	##########
    32	
    33	resources:
    34	
    35	- name: acquire-pool
    36	  type: pool
    37	  source:
    38	    <<: *pool-template
    39	    pool: acquire-pool
    40	
    41	- name: claim-pool
    42	  type: pool
    43	  source:
    44	    <<: *pool-template
    45	    pool: claim-pool
    46	
    47	##########
    48	
    49	jobs:
    50	
    51	- name: acquire-1
    52	  plan:
    53	    - put: acquire-pool
    54	      params: {acquire: true}
    55	    - task: sleep
    56	      <<: *sleep
    57	  ensure:
    58	    put: acquire-pool
    59	    params: {release: acquire-pool}
    60	
    61	- name: claim-1
    62	  plan:
    63	    - put: claim-pool
    64	      params: {claim: slot-1}
    65	    - task: sleep
    66	      <<: *sleep
    67	  ensure:
    68	    put: claim-pool
    69	    params: {release: claim-pool}
    70	

The extension reports the following (wrong) errors:

[kdvolder]

Perhaps as a first 'modest' goal we could take the given examples and just try and make the 'false positives' go away. In other words, you won't get all the content assist etc as you use anchors and references, but at least we could have the validation be smart enough to recognize you are doing something it doesn't understand and stop complaining about that. That probably isn't too hard especially given these excellent examples to use for crafting some test cases :-)

[marco-m]

The 'modest' goal would be good for me :-)

[marco-m]

Hello @kdvolder, I was wondering if you had any chance/plan/interest to work on this feature ?

[kdvolder]

Interest, yes. Chance... not yet. Plan... nothing concrete. Meaning this ticket has not been added to one of our milestones or short term todo lists. Unfortunately there's a lot of things me and the other folks on the tooling team are working and that are competing for our attention.

Sorry I don't have better news for you. I do have every intention of trying to fix this eventually, because I think its an important feature. But because its not something simple I can just fix in a couple hours between other tasks, it will only happen once it gets put into a milestone or todo list during one of our team formal planning sessions.

[marco-m]

@kdvolder I understand perfectly; thanks for the answer!