BitArray I/O is awkward

[quinnj]

b = falses(1)
file = tempname()
open(file,"w") do f
    write(f,b)
end
filesize(file) == 8

So the length of the original BitArray is lost in I/O.

Not sure we can really do anything about this since we can't (I think) write a single bit to a file, but this can be a bit surprising.

I wonder though if this calls for a Bit type that would be the eltype(::BitArray). That would at least allow you to do

b = read(io, Bit, 1)

I'm pretty sure this has been discussed elsewhere, so I'll try to dig around to see what was or if there was anything concluded. Feel free to close if this is old news.

[johnmyleswhite]

The Bit type came up before when discussing rand.

[carlobaldassi]

The write method for BitArrays mimics the one for Arrays, in that it only dumps the raw data. It is paired with read!(::IO, ::BitArray) to read it back in. So in the example above instead of

b = read(io, Bit, 1)

you just need:

b = BitArray(1)
read!(io, b)

write could be made a little more granular and write multiple of 8 bits rather than 64. I'm not sure what that would gain us though.
(read! should also be polished a little, e.g. so that it returns the BitArray rather than the chunks; that's a very minor improvement of course. I just noticed that it also should be documented — probably after the doc system has settled a bit.)
Note that when writing an Array to a stream, you also lose information, such as the dimensionality of the array.
So it seems to me that the write method is inherently low level and implicitly requires that you save the metadata information somewhere else anyway. After all, its documented purpose is to "write the canonical binary representation of a value to the given stream" according to the documentation, which is what it does now.
@quinnj: what's your use case here? e.g. why not using HDF5/JLD?

About the Bit type which comes up periodically, I'm still very much convinced it's an exceptionally bad idea. It's not what you get out of a BitArray: that's a Bool. So it can not be the eltype of a BitArray. It conflates the container type with the element type without being either of those (it's basically a flag). It's also not a reasonably extensible mechanism. So its proposed usage (in this case and in all others when it came up) is just a way to pun on the type system for the sake of passing a flag to some functions which normally would return an Array.
On top of this, In the particular case of read, you wouldn't be able to read one bit at a time anyway, only multiples of 8 at best, which seems only confusing to me.

[carlobaldassi]

I see now that this probably came out in the context of #11280, where one would like to have a version of mmap with default arguments, i.e. infer the vector length from the file size.

With BitArrays, this is currently impossible. We need the extra information about the length, or at the very least about the length of the last chunk, which could fit in a byte.

We could in principle write that extra byte (or that extra Int64 with the length) at the beginning of the stream. I don't particularly like this:

1. seems to stretch the definition of what write does
2. requires to pre-parse the stream somehow, which you can't really do in a default function argument in mmap the same way you do this for Arrays
3. does not really work in the use-case of reading the stream in steps (rather than in one go, or rather then using mmap) which to me seems the only use-case for wanting the read(::IO, ::Type, ::Int) interface — or similar — to work

I could be convinced otherwise though. Thoughts?

[quinnj]

@carlobaldassi, I really appreciate the writeup, even though it's probably getting redundant for you.

Totally makes sense on the "lossy"-ness of writing arrays (and bitarrays) to disk. It's unfortunate that writing a bitarray requires some understanding of its internals, but as you said, it's just inherently a low-level operation, so users just need to handle the remembering of length/dimension themselves.

No, after thinking about it more last night, I ended up taking out the "default" length argument for the mmap for BitArrays. Basically, for other types of arrays, we'll guess the array length from the filesize, but for BitArrays, you need to supply your own dimensions. I think that's fair and doesn't try to do anything hand-wavy.