RFC/WIP: Modularize strpack

[pao]

Looks like strpack may needs some "esc"s in it. I may get to this myself, but in the meantime I thought I'd report it.

module MyMod
import Base.*

load("strpack.jl")

type TestStruct
    int1::Int32
    float1::Float32
end
TestStruct(i, f) = TestStruct(convert(Int32, i), convert(Float32, f))

function test()
    io = IOString()
    t = TestStruct(2, 3.2)
    pack(io, t)
end

export test
end # module

julia> import MyMod.*

julia> test()
in anonymous: DataAlign not defined
 in anonymous at no file
 in pack at /home/tim/src/julia/extras/strpack.jl:334
 in test at /tmp/strpmod.jl:15

Commenting out the 3 lines that create this as a module lets everything work properly.

Presumably, strpack itself should be converted to a module. It also depends on iostring.jl and lru.jl, so I wouldn't want to make drastic changes without consulting @pao .

[pao]

Yeah, not surprised. I was wanting to get a couple smaller modules done (hence monads, QuickCheck) before tackling this beast.

I've got through Wednesday to work on this before I disappear through the weekend. Will post here with patches once they start flowing.

[timholy]

Meanwhile, I've figured out how to solve the immediate problem. (Of course, it would probably still be a good idea to turn strpack into a module.) It turns out not to be a case of strpack needing esc, but instead an issue of when the @eval and -> inside strpack (functions struct_unpack and struct_pack) run. The workaround is to force these to run while still inside the defining module: in this example, right after defining the TestStruct default constructor, just add the line

pack(TestStruct(0, 0))

and all is well. Without that line, the pack and unpack functions don't get defined until the user calls test(), and by that point DataAlign is not available.

[pao]

Ah, I see. You can use the Struct constructor directly to do that more cleanly: Struct(TestStruct) should accomplish the same thing.

[timholy]

Good, thanks.

[pao]

Just starting on this, but getting patches up as promised.

Up to this point this drops the s"" syntax (as previously threatened) and creates generic pack/unpack functions per type instead of anonymous functions, since the anonymous functions didn't buy any module encapsulation.

The tricky bit so far is referring to non-exported functions in the quoted blocks using their fully-qualified names since the block must be @evaled to bring the new methods into existence.

[timholy]

Cool! Are you looking for testing/feedback at this stage, or should I wait until you declare completion?

[pao]

I'm happy to take feedback any time, but I do still have work to do. I did quickly check that the basics work, but my original test code won't work because I killed the macro it uses. I would not be surprised if certain specific things fail right now.

Unless I get something done tonight, though, I don't expect to get back to this until next week.

[ViralBShah]

Another candidate for a package? strpack is one of those things that I think would migrate back into Base, once stabilized.

[StefanKarpinski]

I don't really see why this would go into Base.

[ViralBShah]

I was under the impression that this would eventually make it possible to pass structs to C code. If not, then perhaps, we can close this.

[timholy]

It already does make it possible to pass structs to C code, and it is indeed very useful. But I think that this aspect of strpack (and strpack does more than just this) will eventually be superseded by built-in immutable types. The advantage of the latter is that it will be zero-copy, which strpack manifestly is not.

If extras/ is being emptied, then definitely strpack should become a package.

[dmbates]

@ViralBShah See the code at the end of https://github.com/dmbates/JSparse.jl/blob/master/src/JSparse.jl The idiom
pack(struct(object)).data where struct creates the desired C struct allows you to pass structs through ccall

@timholy In your documentation you may want to note that the sequence

iostr = IOString()
pack(iostr, struct)

can be condensed to pack(struct)

[pao]

Yes, to be clear, the ability to construct C-compatible structures is a side effect. This will still have its uses for serializing/deserializing other binary data, particularly when dealing with oddball formats or when a purpose-built serializer/deserializer isn't worth it.

I'll close this with the intent to move to its own package.

[timholy]

@dmbates re the documentation: it's true that it can be condensed, but I was worried that the global iostr variable might look like magic, and I felt the longer version would be clearer.