JuliaCall

[lxvm]

Hello, I am using Julia v1.8.5, Python v3.11.2, and juliacall v0.9.12. I am linking a Julia package to a Python web app and running into some issues with interoperability of StringIO objects. A MWE to demonstrate this goes as follows:

>>> import io
>>> from juliacall import Main as jl
>>> s = "a\nb\nc\n" # test string representing multi-line file
>>> with io.StringIO(s) as i:
...     jl.readline(i)
...     jl.readline(i)
...     jl.readline(i)
... 
'a'
''
''

I would have expected to get the second and third lines as well, but they are empty strings.
If I do the same on the Python side, I get the behavior I would expect

>>> with io.StringIO(s) as i:
...     i.readline()
...     i.readline()
...     i.readline()
... 
'a\n'
'b\n'
'c\n'

I also get the behavior I would expect if I do it all on the Julia side

>>> ss = jl.IOBuffer(s)
>>> jl.readline(ss)
'a'
>>> jl.readline(ss)
'b'
>>> jl.readline(ss)
'c'

It's not clear to me where the problem begins, but the net effect is that in my application I cannot pass a StringIO object from Python to a parser written in Julia and the obvious workaround is to pass an IOBuffer instead.

I also tested on Python v3.10.8 and got the same results

[cjdoris]

What's happening I think is that the Julia object wrapping the Python IO object adds a buffering layer for efficiency.

So when you do jl.readline(i), then i gets wrapped and readline causes the wrapped object to read not only one line but a larger buffer of data. This is so that subsequent readline calls on the same wrapped object do not need to call out to Python each time.

However if you do jl.readline(i) again, another wrapper gets made, but all the data has already been read from the underlying stream, and so you get nothing back.

As you've observed, doing the Pythonic thing from Python and the Julian thing from Julia works fine (because the wrapper gets created at most 1 time) - do you really need to do the other version?

[lxvm]

Thanks for the reply! You're right that I don't really need the other version since jl.IOBuffer works, so I'll close the issue.