Parse valid JSON array of objects and insert each object into a substrate pallet's StorageMap

[ltfschoen]

@xlc i've been trying to use lite-json, and i noticed that you actually created it!

i'm want to use off-chain workers to query an external API endpoint in our code here where i've added the off-chain workers example code https://github.com/DataHighway-DHX/node/blob/luke/rewards-allowance-new-offchain/pallets/mining/rewards-allowance/src/lib.rs#L2684

i want to retrieve data in the following valid JSON format from the body of the http request response:

{
  "data": [
  	{
	    "acct_id": "1234",
	    "mpower": "1",
	    "date_last_updated": "1636096907000"
	},
  	{
	    "acct_id": "1234",
	    "mpower": "1",
	    "date_last_updated": "1636096907000"
	}
  ]
}

then iterate through each object in the array and whilst doing so i'll populate a data structure with each object's values and insert it into the pallet's storage where for each object in the that array i create a key (which is a tuple that contains both the start of the current date that we received the response, and the account_id that was in the response), and a value (which just contains the other two values received including the account_id's mining power mpower and the date that date was last updated off-chain)

    /// Recently submitted mPower data.
    #[pallet::storage]
    #[pallet::getter(fn mpower_of_account_for_date)]
    pub(super) type MPowerForAccountForDate<T: Config> = StorageMap<_, Blake2_128Concat,
        (
            Date, // converted to start of date
            T::AccountId,
        ),
        (
            u128, // mPower
            Date, // date last updated off-chain
            T::BlockNumber, // block receive using off-chain workers
        ),
    >;

i'd also like to know how to serialize and deserialize that kind of object.

how may i do this with lite-json?

[ltfschoen]

i was able to do it with serde_json here https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=7a066487d37870330444b0908115cacf

use serde::{Deserialize, Serialize};

#[cfg_attr(feature = "std", derive(Debug))]
#[derive(Default, Clone, PartialEq, Eq)]
pub struct MPowerPayload {
    acct_id: String,
    mpower: String,
}

#[derive(Debug, Serialize, Deserialize)]
struct DataValue {
    acct_id: String,
    mpower: String,
}

#[derive(Debug, Serialize, Deserialize)]
struct JData {
    data: Vec<DataValue>,
}

fn main() {
    let json_str = r#"{
        "data": [
            { "acct_id": "50000000000000001", "mpower": "1" },
            { "acct_id": "50000000000000002", "mpower": "2" }
        ]
    }"#;

    let mpower_json_data: JData = match serde_json::from_str(json_str) {
        Err(e) => {
            println!("Couldn't parse JSON :( {:?}", e);
            return;
        },
        Ok(data) => data,
    };

    println!("{:?}", mpower_json_data);

    let mut mpower_data_vec = vec![];
    for (i, v) in mpower_json_data.data.into_iter().enumerate() {
        println!("i v {:?} {:?}", i, v);
        let mpower_data_elem = MPowerPayload {
            acct_id: v.acct_id.clone(),
            mpower: v.mpower.clone(),
        };

        mpower_data_vec.push(mpower_data_elem);
    }
}

but serde_json doesn't work in my pallet as it causes error:

error: duplicate lang item in crate `std` (which `serde` depends on): `oom`.

[xlc]

There are no any automatic serialize / deserialize support so you will need to construct JsonValue / match the generated JsonValue to work with the data.

For serde_json, you should just need to disable std somewhere.

[ltfschoen]

i'm using lite-json again now because i can't figure out how to disable std to overcome that error with serde_json,
if i use lite-json, i can get it to work when the value of the key in the JSON is just a string or number, but not when it's an vector.

for example, if i do the below, where the value of the key in the JSON file is an vector:

/// Payload used to hold mPower data required to submit a transaction.
#[cfg_attr(feature = "std", derive(Debug))]
#[derive(Encode, Decode, Default, Clone, PartialEq, Eq)]
pub struct MPowerPayload<U, V> {
    pub account_id_registered_dhx_miner: U,
    pub mpower_registered_dhx_miner: V,
}

type MPowerPayloadData<T> = MPowerPayload<
    <T as frame_system::Config>::AccountId,
    u128,
>;

let mpower_data = r#"{
    "data": [
        { "acct_id": "50000000000000001", "mpower": "1" },
        { "acct_id": "50000000000000002", "mpower": "1" }
    ]
}"#;

let mpower_json_data = lite_json::parse_json(mpower_data);

let mut mpower_data_vec: Vec<MPowerPayloadData<T>> = vec![];
let mpower_array = match mpower_json_data.ok()? {
    JsonValue::Object(obj) => {
        let (_, v) = obj.into_iter().find(|(k, _)| k.iter().copied().eq("data".chars()))?;
        match v {
            JsonValue::Array(vec) => vec,
            _ => return None,
        };
    },
    _ => return None,
};

for (i, obj) in mpower_array.into_iter().enumerate() {
    println!("mpower_array obj {:?} {:?}", i, obj);

    let mpower_data_elem: MPowerPayloadData<T> = MPowerPayload {
        account_id_registered_dhx_miner: obj.acct_id.clone(),
        mpower_registered_dhx_miner: obj.mpower.clone(),
    };

    mpower_data_vec.push(mpower_data_elem);
}

it appears to assign a value of type std::vec::Vec<lite_json::JsonValue> to mpower_array and then gives the following error instead of returning the vector that i can then iterate

error[E0599]: the method `into_iter` exists for unit type `()`, but its trait bounds were not satisfied
    --> pallets/mining/rewards-allowance/src/lib.rs:2761:42
     |
2761 |             for (i, obj) in mpower_array.into_iter().enumerate() {
     |                                          ^^^^^^^^^ method cannot be called on `()` due to unsatisfied trait bounds
     |
     = note: the following trait bounds were not satisfied:
             `(): Iterator`
             which is required by `(): IntoIterator`
             `&(): Iterator`
             which is required by `&(): IntoIterator`
             `&mut (): Iterator`
             which is required by `&mut (): IntoIterator`

[xlc]

Can you create a snippet with https://play.rust-lang.org to reproduce the error?

[ltfschoen]

Can you create a snippet with https://play.rust-lang.org to reproduce the error?

i've reproduced the error in the latest commit in this DataHighway-DHX/node#238 (branch 'luke/rewards-allowance-new-offchain')

unfortunately i can't replicate it using https://play.rust-lang.org/, because you can only use the top % of most used crates there, and lite_json doesn't appear to be in that and thus isn't available on the playground.
i wonder if it'd be worth requesting funding from the Kusama treasury to increase the amount of crates you can use there in that Rust-specific playground. an alternative is to use https://playground.substrate.dev/, but it takes can take much longer to replicate just a Rust error.