Unable to use pykafka's TopicDict

[tk3369]

In an attempt to use pykafka via PyCall, we encountered an issue that the Python TopicDict class is considered a Dict in julia. Because of the laziness implementation, we're getting nothing from dereferencing the key, rather than triggering a call to the __getitem__ Python method. Any suggestion?

You can find the TopicDict source code here:

https://github.com/Parsely/pykafka/blob/65993546478c60d3de0760eea93deb97b87cbb7a/pykafka/cluster.py

[tkf]

You can call whatever the function creating a TopicDict instance with PyDict(py"function(*arguments)"o) or pycall(function, PyDict, arguments...). Example:

using PyCall

py"""
class MyDict(dict):
    def __getitem__(self, key):
        print("[Python] Got:", key)
        return super().__getitem__(key)
"""

a = py"MyDict({1: 2})"
@show typeof(a)

b = py"MyDict({1: 2})"o
@show typeof(b)
@show typeof(PyDict(b))
@show PyDict(b)[1]  # invokes __getitem__

MyDict = py"MyDict"
c = pycall(MyDict, PyDict, Dict(1=>2))
@show typeof(c)
@show c[1]  # invokes __getitem__

Output:

typeof(a) = Dict{Any,Any}
typeof(b) = PyObject
typeof(PyDict(b)) = PyDict{PyAny,PyAny,true}
[Python] Got: 1
(PyDict(b))[1] = 2
typeof(c) = PyDict{PyAny,PyAny,true}
[Python] Got: 1
c[1] = 2

[tk3369]

Thanks for the workaround although it doesn't work for us yet... TopicDict is actually created from the Cluster class but it seems that it requires another argument...

julia> c1 = pycall(cluster.TopicDict, PyDict, client.topics)
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/csubrama/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError("__init__() missing 1 required positional argument: 'exclude_internal_topics'")

Stacktrace:
[1] pyerr_check at /home/csubrama/.julia/packages/PyCall/ttONZ/src/exception.jl:60 [inlined]
[2] pyerr_check at /home/csubrama/.julia/packages/PyCall/ttONZ/src/exception.jl:64 [inlined]
[3] macro expansion at /home/csubrama/.julia/packages/PyCall/ttONZ/src/exception.jl:84 [inlined]
[4] __pycall!(::PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyObject, ::Ptr{Nothing}) at /home/csubrama/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:44
[5] _pycall!(::PyObject, ::PyObject, ::Tuple{Dict{Any,Any}}, ::Int64, ::Ptr{Nothing}) at /home/csubrama/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:29
[6] #pycall#109 at /home/csubrama/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:11 [inlined]
[7] pycall(::PyObject, ::Type{PyDict}, ::Dict{Any,Any}) at /home/csubrama/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:83
[8] top-level scope at none:0

[tkf]

Maybe PyDict(cluster."topics") wroks?

cluster."topics" is a way to suppress auto-conversion which would happen if you did cluster.topics. https://github.com/JuliaPy/PyCall.jl#pyobject

[tk3369]

Great solution! My prelim test seems to work. Will do more extensive testing later.