-
-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generator can create duplicate RegisterSchema functions #490
Comments
Thanks for the report. This one's on me -- I never put more than one schema in a package so I always forget this use case. We should add some tests that do this. The solution you propose sounds reasonable. |
Is there a recommended workaround for the interim? Downgrade to alpha.24? |
@hspaay I think the workaround would be to merge your files into one. Note that you can still import types from separate packages; I do this quite a bit, and it has prevented me from ever requiring mutli-file packages. |
@lthibault thanks for the very quick reply :)
Not sure if I follow. I've got 13 capnp files, one per micro-service and they go into the 'api' package. Are you recommending to put all of these into a single file? I love how compact capnp files are but putting all in a single file might be a bit too much. |
I tend to put each .capnp file in its own package (which is basically why this hasn't come up); I agree dumping everything into the same file could get unwieldly, and furthermore, you'd have to make the interface IDs explicit to avoid changing them, since those are a function of the parent namespace's ID (the root of which is the file ID) and the name of the type/interface. See e.g. https://github.com/zenhack/go.sandstorm/tree/master/capnp |
@zenhack thanks for that. For now I think I'll wait a bit using alpha.24 and hope for a fix. Its all still alpha (including my stuff) so no rush. Again, I much appreciate the awesome support you guys are giving. |
@hspaay I use the same approach as @zenhack. Correspondingly, here is my typical application structure. The Hope this helps.
It's truly our pleasure! 😃 |
Just chiming in to say that this is affecting us at Cloudflare too. We're pinning to |
@alexforster Any chance you might be able to submit a PR? @zenhack recently passed away and we're spread very thin right now. Realistically, I don't think I'll be getting to this one for a while. If you are able to contribute, I am more than happy to lend my support. P.S.: thank you for your feedback, btw! |
@lthibault wrt "zenhack recently passed away". |
@hspaay OMG, I'm so sorry to drop the news on you so suddenly. For some reason I thought you already knew! Yes, Ian was truly an exceptional guy, and it was all very sudden. Go-capnp will be fine, but Ian's presence is truly missed :( |
Well I've been away for a bit so my own fault for not keeping up to date. |
You'll be in good company. I've poured a few out for Ian. |
I'm very sorry to hear that. My condolences. I'll put some thought into this... Looking back at #479, it seems the broader goal of the change was to facilitate dead-code elimination (edit: and to get rid of magic static globals), so going back to the With that in mind, I think there are really only three choices: Option 1: automatically disambiguate the
|
@alexforster Thank you for these detailed thoughts; they're super helpful when diving back into this issue. 😃 I agree Option 2 sounds like the better one. I'd like to get @davidhubbard's thoughts on this as well, since he has recently worked with the code-generator on a loosely-related issue, and will have the freshest recollection of how everything works! David, what do you think? BTW, Alex, feel free to join us in the matrix channel. Most of the dev work goes on in there, and we'd love to include you in the discussion. |
Fixes capnproto#490 by only generating one `RegisterSchema()` function per `$Go.package()`. For example, the unit tests persistent-simple.capnp and persistent-samepkg.capnp now both have the annotation: `$Go.package("persistent_simple");` Their generated go code thus starts with: `package persistent_simple` but only one of them will include a `RegisterSchema()`.
Thinking out loud, #479 took a step toward schema "belonging" to the package. This issue then asks the question, how do capnp files map to the Go concept of packages. I'd like to think any Golang-isms aren't implicit constraints on how capnp works. That's why my first preference is Option 3 or, in my own words, the go code-gen is tasked with making the right decisions for Golang-isms. Option 2 does have an appeal but wouldn't users then have to deal with the problem, instead of avoiding it automatically by keeping track of the presence of a I just sent #544 with Option 3 to noodle around with. @alexforster can you please check if the |
Nice! From an ergonomics perspective, option 3 is the clear winner. I'll run this over our schemas tomorrow and see how it looks. |
I added a command line flag There may be a fairly remote problem if capnp is invoked for a single file at a time. It does not consider any capnp source files that could somehow show up later in the same package. I think a more complete solution might involve searching the output dir for any existing |
A cleaner (but also probably more controversial) solution could be to just generate a single output file per Go module. I think mapping input files to output files is kind of an abstraction leak. If the output conceptually is "multiple Go modules" then trying to maintain the input file structure doesn't make sense. |
We're moving toward invalidating the cache of generated go code and naming the output files, that's encouraging if hard. I'd like to give this some careful thought. Hopefully an immediate improvement will buy some time to land on a satisfactory solution. |
Fixes capnproto#490 by only generating one `RegisterSchema()` function per `$Go.package()`. For example, the unit tests persistent-simple.capnp and persistent-samepkg.capnp now both have the annotation: `$Go.package("persistent_simple");` Their generated go code thus starts with: `package persistent_simple` but only one of them will include a `RegisterSchema()`. Note: if `capnp` invokes `capnpc-go` to generate go code but only includes one .capnp file at a time on the command line, this will not detect any *other* .capnp files that might have the same $Go.package. In that situation ask the user to take responsibility for setting `--schemas true` and `--schemas false` for individual invocations of `capnp`. e.g. This will work:
@alexforster I just pushed a fix and hopefully it hasn't caused confusion -
And I verified it with: |
Fixes capnproto#490 by only generating one `RegisterSchema()` function per `$Go.package()`. For example, the unit tests persistent-simple.capnp and persistent-samepkg.capnp now both have the annotation: `$Go.package("persistent_simple");` Their generated go code thus starts with: `package persistent_simple` but only one of them will include a `RegisterSchema()`. Note: if `capnp` invokes `capnpc-go` to generate go code but only includes one .capnp file at a time on the command line, this will not detect any *other* .capnp files that might have the same $Go.package. In that situation ask the user to take responsibility for setting `--schemas true` and `--schemas false` for individual invocations of `capnp`. e.g. This will work: `capnpc -o go persistent-simple.capnp persistent-samepkg.capnp` This will not work: `capnpc -o go persistent-simple.capnp; capnpc -o go persistent-samepkg.capnp`
So, it did only generate a single Here's what I see:
It only has six "nodes" (whatever those are), whereas before each individual I don't actually know what these nodes are, so maybe this is fine. |
@alexforster you are correct. Let me rework the pull request to include the schemas of all the files in the module. |
Fixes capnproto#490 by only generating one `RegisterSchema()` function per `$Go.package()`. For example, the unit tests persistent-simple.capnp and persistent-samepkg.capnp now both have the annotation: `$Go.package("persistent_simple");` Their generated go code thus starts with: `package persistent_simple` but only one of them will include a `RegisterSchema()`. Note: if `capnp` is invoked with only one .capnp file at a time on its command line, `capnpc-go` will not detect any *other* .capnp files that might have the same $Go.package. For that use case, the user might be able to cope by running `capnpc -o - persistent-simple.capnp | capnpc-go --forceschemasalways` and editing the generated code to rename all but one `RegisterSchema()` and add calls to the renamed code inside the one remainig `RegisterSchema()`... though that is not a supported solution. e.g. Use this: `capnpc -o go persistent-simple.capnp persistent-samepkg.capnp` Because the following will fail to compile due to duplicate `RegisterSchema()` definitions: ``` capnpc -o go persistent-simple.capnp capnpc -o go persistent-samepkg.capnp ```
@lthibault can you please re-review, since this is a rewrite of #544 ? @alexforster this collects all the schemas first before writing any files. Would you be able to recheck this against your workflow? |
Sure, I'll run it tomorrow. I really appreciate your work on this! |
@davidhubbard LGTM 👍 |
Fixes #490 by only generating one `RegisterSchema()` function per `$Go.package()`. For example, the unit tests persistent-simple.capnp and persistent-samepkg.capnp now both have the annotation: `$Go.package("persistent_simple");` Their generated go code thus starts with: `package persistent_simple` but only one of them will include a `RegisterSchema()`. Note: if `capnp` is invoked with only one .capnp file at a time on its command line, `capnpc-go` will not detect any *other* .capnp files that might have the same $Go.package. For that use case, the user might be able to cope by running `capnpc -o - persistent-simple.capnp | capnpc-go --forceschemasalways` and editing the generated code to rename all but one `RegisterSchema()` and add calls to the renamed code inside the one remainig `RegisterSchema()`... though that is not a supported solution. e.g. Use this: `capnpc -o go persistent-simple.capnp persistent-samepkg.capnp` Because the following will fail to compile due to duplicate `RegisterSchema()` definitions: ``` capnpc -o go persistent-simple.capnp capnpc -o go persistent-samepkg.capnp ```
…odes Fixes capnproto#490 after capnproto@2a85fea collected the node IDs incorrectly. This uses the list of IDs after `resolveName()` has recursively scanned the nodeMap and gathered all the IDs in the capnp file. The previous commit used only the top-level file ID. For example, https://github.com/capnproto/go-capnp/blob/main/flowcontrol/internal/test-tool/writer.capnp.go now gets generated with: ``` func RegisterSchema(reg *schemas.Registry) { reg.Register(&schemas.Schema{ String: schema_aca73f831c7ebfdd, Nodes: []uint64{ 0xf82e58b4a78f136b, }, Compressed: true, }) } ``` This commit fixes the above generated code, and includes the following: `0x80b8cd5f44e3c477,` `0xd939de8c6024e7f8,`
I created #545 There are missing nodes in RegisterSchemas. For instance, https://github.com/capnproto/go-capnp/blob/main/flowcontrol/internal/test-tool/writer.capnp.go is generated with only the top-level ID in the list of Nodes:
|
Sorry for falling off, lots of crises at work. Let me know if I can help now that things are calming down for me |
@alexforster no worries! Want to try #545 on your workflow? @lthibault I found and fixed a bug with this just now, hence the reason I reopened this issue. |
Thanks for working on this! |
Another Cloudflarian here 👋 Unfortunately I don't know the fix in place covers all cases. Similarly to @alexforster we're looking to pull multiple capnp into a single module. However, these files are sourced from different locations, so when doing the generation we make heavy use of
AFAICT there's not a way to combine these into a single generation command, which is what the current fix requires. Looking at other projects to get a sense of patterns: prometheus metric registration stands out as something in a similar vein: we have a bunch of Collectors we need to iterate over and make available to a backend. Perhaps there's an option where we wrap a node's schema in a struct that implements a simple
And then look at option 3 as proposed in #490 (comment). Of course, if there's a way to make it work with the current solution that's be amazing! |
#490 (comment) - This comment makes sense. I'm hesitating (more like -- stalling) to have the go-capnp codegen intelligently scan what has happened before it was invoked. Especially with the files being in different locations. Golang isn't really patterned that way: files in different locations all getting compiled into a single package is definitely not a golang-ism... Again, this is not me saying No, Never. But basically, Not Right Now...? |
@davidhubbard, If @ndisidore were able to contribute a patch, would we be able to merge it in? Or are we blocked by the codegen work you mentioned? @ndisidore Are you at all interested in contributing a patch? We could use the help. 🙂 |
Agree, high quality patches welcome. In terms of "Not Right Now," go-capnp can't be everything to everybody, and if it were me (it's not, see below) I'd really try to make the gathering of the files in different directories a build step separate from invoking the go-capnp codegen. Thus "Not Right Now" to me is my way of trying to keep all of us pushing on the work that's broadly needed for go-capnp to keep growing. I don't mean to make it sound like I'd block a motivated contributor from adding and maintaining a feature that's super meaningful to them. |
After the change for #479, if you have multiple capnp files in a package, e.g.
and
then
There will be duplicate functions in the same package. Maybe the register function needs to be generated in a separate file per-package with one
RegisterSchema
for all?The text was updated successfully, but these errors were encountered: