[Bug]: `Semantic(None, "invalid type: bytes, expected bytes")`

[RCasatta]

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

I am getting the error Semantic(None, "invalid type: bytes, expected bytes")

Expected Behaviour

deserialization works

Environment Information

Linux 6.5.4-76060504-generic #202309191142169599894322.04~070916d SMP PREEMPT_DYNAMIC Fri S x86_64 x86_64 x86_64 GNU/Linux

Steps To Reproduce

The error is launched from here

ciborium/ciborium/src/de/mod.rs

Line 379 in 2ac91ce

header => Err(header.expected("bytes")),

because the header is bytes but the len of scratch is not enough

I can reproduce the error in a downstream crate with a type implementing custom serde Serialize/Deserialize: ElementsProject/rust-elements@45bd2bf
This type is longer than 4k bytes

When I try to reproduce without the downstream crate using something like Wrap(vec) where vec is bigger than 4k bytes, things work because in this line:

ciborium/ciborium/src/de/mod.rs

Line 369 in 2ac91ce

Header::Bytes(Some(len)) if len <= self.scratch.len() => {

the scratch space has grown to accommodate the len of the bytes, (it is 8k even though it is initialized to 4k) while in the case of the error it's probably not growing for some reason I could not grasp yet.

[RCasatta]

Following code could reproduce:

#[cfg(test)]
mod tests {
    use std::fmt;

    use serde::{de, Deserialize, Serialize};

    #[derive(PartialEq, Eq, Debug)]
    struct Wrap(Vec<u8>);

    impl Wrap {
        fn from_slice(slice: &[u8]) -> Result<Wrap, String> {
            Ok(Wrap(slice.to_vec()))
        }
        fn serialize(&self) -> Vec<u8> {
            self.0.clone()
        }
    }

    impl Serialize for Wrap {
        fn serialize<S: ::serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
            s.serialize_bytes(&self.serialize())
        }
    }

    impl<'de> Deserialize<'de> for Wrap {
        fn deserialize<D: ::serde::Deserializer<'de>>(d: D) -> Result<Wrap, D::Error> {
            d.deserialize_bytes(BytesVisitor::new("a bytestring", Wrap::from_slice))
        }
    }

    pub struct BytesVisitor<F> {
        expectation: &'static str,
        parse_fn: F,
    }

    impl<F, T, Err> BytesVisitor<F>
    where
        F: FnOnce(&[u8]) -> Result<T, Err>,
        Err: fmt::Display,
    {
        pub fn new(expectation: &'static str, parse_fn: F) -> Self {
            BytesVisitor {
                expectation,
                parse_fn,
            }
        }
    }

    impl<'de, F, T, Err> de::Visitor<'de> for BytesVisitor<F>
    where
        F: FnOnce(&[u8]) -> Result<T, Err>,
        Err: fmt::Display,
    {
        type Value = T;

        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            formatter.write_str(self.expectation)
        }

        fn visit_bytes<E: de::Error>(self, v: &[u8]) -> Result<Self::Value, E> {
            (self.parse_fn)(v).map_err(E::custom)
        }
    }

    #[test]
    fn ciborium() {
        let a = Wrap(vec![0u8; 5000]);
        let bytes = cib_vec(&a).unwrap();
        let b: Wrap = ciborium::from_reader(&bytes[..]).unwrap();
        assert_eq!(a, b);
    }

    fn cib_vec<S: Serialize>(s: &S) -> Result<Vec<u8>, ciborium::ser::Error<std::io::Error>> {
        let mut vec = vec![];
        ciborium::ser::into_writer(s, &mut vec)?;
        Ok(vec)
    }
}

[johnlees]

I am also running into this when deserialising objects above a certain size. Increasing the buffer with from_reader_with_buffer() doesn't help.

Object:

#[derive(Serialize, Deserialize, Default, Clone, PartialEq)]
pub struct Inverted {
    index: Vec<HashMap<u16, RoaringBitmap>>,
    sample_names: Vec<String>,
    kmer_size: usize,
    sketch_version: String,
    rc: bool,
    hash_type: HashType,
}

Seems to error when Vec becomes large

[sdbondi]

Also ran into this, looks like it's this code

ciborium/ciborium/src/de/mod.rs

Line 395 in b1daf78

Header::Bytes(Some(len)) if len <= self.scratch.len() => {

The buffer has to be as big as the data.

EDIT: found a workaround:
I had a custom deserializer, using deserialize_bytes. You can use deserialize_byte_buf instead, which allows for arbitrary-sized bytes.

pub fn deserialize<'de, D, T>(d: D) -> Result<T, D::Error>
where
    D: serde::Deserializer<'de>,
    T: From<Box<[u8]>>,
{
    let bytes = d.deserialize_byte_buf(BytesVisitor::new())?; // My ByteVisitor returns Box<[u8]>, but this is not necessary
    Ok(bytes.into())
}

[sdbondi]

Also noticed that it should be fixed in this PR #54