Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Import runtime Call / Storage metadata without import the whole runtime #8158

Open
xlc opened this issue Feb 18, 2021 · 14 comments
Open

Import runtime Call / Storage metadata without import the whole runtime #8158

xlc opened this issue Feb 18, 2021 · 14 comments
Labels
A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it.

Comments

@xlc
Copy link
Contributor

xlc commented Feb 18, 2021

In order to use XCM::Transact, we need to know how to encode the call on destination chain. The best way should be just import the Call enum from runtime. But we cannot do it because last time we tried, it tries to import the whole Rococo runtime into our parachain runtime, which obviously wouldn't work.

We should have some way to allow others to import some generated types from runtime, without import the whole thing.

Not exactly sure how it can be implemented though, all the generic makes thing harder. One potential solution is reconstruct the enum from metadata using polkadot.js.

@bkchr
Copy link
Member

bkchr commented Feb 18, 2021

Cc @ascjones @tomusdrw

@shaunxw
Copy link
Contributor

shaunxw commented Feb 18, 2021

I'll post more details here.

We'd like to implement helpers in orml-xcm-utility, to help init/accept HRMP open channel request (It's helpful to all parachains who want to send Xcm). Like this:

#[pallet::call]
impl<T: Config> Pallet<T> {
    //TODO: weight
    #[pallet::weight(10)]
    #[transactional]
    pub fn hrmp_init_open_channel(
        origin: OriginFor<T>,
        recipient: ParaId,
        proposed_max_capacity: u32,
        proposed_max_message_size: u32,
    ) -> DispatchResultWithPostInfo {
        ensure_root(origin)?;

        let call = rococo_runtime::Call::Hrmp(HrmpCall::hrmp_init_open_channel(
            recipient,
            proposed_max_capacity,
            proposed_max_message_size,
        )).encode();

        let msg = Xcm::Transact {
            origin_type: OriginKind::Native,
            call,
        };

        T::XcmSender::send_xcm(Parent.into(), msg).map_err(|_|
        Error::<T>::SendXcmFailed)?;

        Ok(().into())
    }
}

After integrated this pallet into Mandala parachain runtime, build fails with error:

warning: Linking globals named 'Core_version': symbol multiply defined!
error: failed to load bc of "rococo_runtime-ddaa0aa381fdea09.rococo_runtime.504vydv1-cgu.10.rcgu.o":
error: aborting due to previous error; 1 warning emitted
error: could not compile mandala-runtime-wasm

It seems rococo runtime would be included into our runtime. Even if this error was fixable, the build wasm would be too large.

We're currently doing the encoding via PolkadotJS (I wrote a guide), though it doesn't return the exact encoded call and needs workaround.

I guess this issue could be fixed in the similar way how PolkadotJS do it with metadata.

@ascjones
Copy link
Contributor

ascjones commented Feb 19, 2021

We are currently working on producing rich type metadata using scale-info for substrate runtimes, which will allow generation of SCALE compatible types for interacting with a chain.

However this won't be available in the short term, but what you could do in the meantime is to manually construct a Call enum with just the pallets/calls you want to use, making sure the types match up, and the index of the variant matches with the target runtime e.g.

#[derive(Encode, Decode)]
pub enum Call {
    #[codec(index = "10")] // the index should match the position of the module in `construct_runtime!`
    Hrmp(HrmpCall),
}

#[derive(Encode, Decode)]
pub enum HrmpCall {
    #[codec(index = "1")] // the index should match the position of the dispatchable in the target pallet
    hrmp_init_open_channel(u32, u32, u32),
}

This obviously would become cumbersome the more calls and pallets you use, but I think it should allow you to make progress while waiting for the new metadata format.

@tomusdrw
Copy link
Contributor

We were also discussing generating such standalone (no dependencies) enum as part of runtime compilation. Perhaps with frame-v2 macros this would be easy enough to implement? CC @thiolliere ?

@ascjones
Copy link
Contributor

My thinking is that the metadata should contain enough information to generate SCALE compatible runtime types in any language, e.g. like in this (out of date) prototype for generating Rust types.

@tomusdrw
Copy link
Contributor

My thinking is that the metadata should contain enough information to generate SCALE compatible runtime types in any language, e.g. like in this (out of date) prototype for generating Rust types.

Yes, that would be awesome! I was thinking of a temporary solution before we add scale-info to the metadata, but indeed chameleon approach is the way to go long term.

@olanod
Copy link
Contributor

olanod commented Feb 20, 2021

@ascjones what are the main blockers to not have the type info available in the short term? I saw the frame-metadata crate already has the V13 version of the metadata that includes the type registry and was happy it could land soon to use in a project that would rely on it. I thought it would only be matter of updating the macros to include the Derive(TypeInfo) for all runtime types. Could there be a workaround? Runtimes manually implementing/deriving the type info and expose it via RPC or something?

@gui1117
Copy link
Contributor

gui1117 commented Feb 22, 2021

We were also discussing generating such standalone (no dependencies) enum as part of runtime compilation. Perhaps with frame-v2 macros this would be easy enough to implement? CC @thiolliere ?

I think the scale-info is the only solution I see. Call enum has field with types defined outside of the pallet macro.

@ascjones
Copy link
Contributor

@ascjones what are the main blockers to not have the type info available in the short term?

On the scale-info side we still need paritytech/scale-info#44, which is done now just needs some testing and reviewing from my side.

I thought it would only be matter of updating the macros to include the Derive(TypeInfo) for all runtime types.

There is that and adding various bounds to traits, and also producing the new metadata format which I have a prototype of in https://github.com/paritytech/substrate/compare/aj-metadata-vnext. However that is dependent on converting all the pallets to the new framev2 macros, see #7882. If you run that branch then you should be able to produce the v13 metadata for all the modules which have been migrated already.

It would be possible of course to produce the new style metadata with the old macros too, but I don't currently plan to do so.

@tomusdrw
Copy link
Contributor

tomusdrw commented Feb 22, 2021

I think the scale-info is the only solution I see. Call enum has field with types defined outside of the pallet macro.

The question is how much do we want to cover - obviously scale-info will produce all the typing, but I'd argue that the main use case is to be able to call into an external runtime and majority of the types would be quite simple. So as a middle-ground solution we could just accept generic parameters T: Encode for any non-standard type.

If full scale-info is around the corner already, then probably it's best to focus on this though, instead of wasting efforts on some workarounds.

@ascjones
Copy link
Contributor

ascjones commented Feb 22, 2021 via email

@gavofyork
Copy link
Member

In order to use XCM::Transact, we need to know how to encode the call on destination chain

Transact is meant for cases where the sender already knows the format of the destination chain. The point of (non-Transact) XCM is to abstract over the chain's capabilities and encodings to allow inter-chain interaction without the need to know its metadata.

@xlc
Copy link
Contributor Author

xlc commented Mar 11, 2021

polkadot-fellows/xcm-format#22 will address the module index issue.

@stale
Copy link

stale bot commented Jul 7, 2021

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it.
Projects
None yet
Development

No branches or pull requests

8 participants