Question on using `.instances()`

[steeling]

We have a setup where we have many specs.Containers objects spread across many files.

I'd like to add a script, where I can run kcl render.k, which consumes all the containers, and outputs a single object (either to stdout, or to a file). My issue is that to get this to work, I'm using globbing all kcl files such that the script calls:

kcl file1.k file2.k file3.k ... fileN.k render.k

This seems to have some issues:

1. It's very slow, and I have suspicion it's evaluating some things twice.
2. File scoped variables become global scoped, ie: 2 separate files in 2 separate directories with a top level myField variable, will now result in a conflict
3. specs.Containers.instances() works within render.k, but if I have another file that is not part of the glob, but maybe indirectly imported that also attempts to find all containers, it doesn't. Which leads me to believe there are some further scoping oddities.

I might be abusing the intent of running kcl, is there a better way to do what I want to accomplish? Let me know if I can provide more info on this :)

[spastorclovr]

+1 on instances reporting the double amount of instances actually created.
This is what I am experimenting.

[Peefy]

cc @zong-zhe @He1pa

[He1pa]

For question 2, according to the design of kcl, all files in kcl file1.k file2.k file3.k ... fileN.k render.k are regarded as the entry of compilation. So their variables are all glob,

For question 3， can you provide more details? like example code

[steeling]

Gotcha, I'll try to get an example out tomorrow.

In the meantime, is there a better way to get all instances in a set of directories without globbing everything together, and without explicitly importing each file? I'd like a way where someone can add in a new specs.Container, and we automatically consume it

[He1pa]

Gotcha, I'll try to get an example out tomorrow.

In the meantime, is there a better way to get all instances in a set of directories without globbing everything together, and without explicitly importing each file? I'd like a way where someone can add in a new specs.Container, and we automatically consume it

put the .k files into a directory and import it in main.k?

[steeling]

looked into this a bit more. Let's say I have 100+ files. It's not very desirable to put this into a single directory, as then I need to coordinate a bunch of name conflicts, and I've noticed rendering slows down a lot.

Another option is to autogenerate an imports file, like

import configs.specs
import configs.package1
import configs.package2
import configs.package3
import configs.package4
...

containers = specs.Containers.instances()

However instances only searches the local package. It seems there's no mechanism to auto retrieve/glob instances of an object type without putting them all into one directory, which becomes somewhat of an anti pattern at a larger scale.

Any recommendations on how to proceed here?

[cblkwell]

+1 on instances reporting the double amount of instances actually created. This is what I am experimenting.

I'm also seeing this -- strangely, for only one resource, even though multiple ones are defined in the same exact files. You can see some example code here: https://github.com/cblkwell/kcl-prototype/tree/duplicate-appskeleton

If you look at example/redox-prototype-redux, if you do a kcl run -D image="foo" in one of the sectors directories, you can see that it will show two AppSkeleton instances, but only a single one of each Deployment instance, which is very bizarre. I've tested this and it does not happen if I remove the sector/my-app-prod-aws-use1/main.k from the chain and just use the base and prod files, but as soon as I add the sector file (even if prod is not included), it duplicates that resource. It's something I can work around by de-duping the resources, but I don't understand why this behavior is happening -- especially why it is happening for the AppSkeleton instances but not the Deployment instances.