Skip to content
This repository was archived by the owner on Aug 30, 2020. It is now read-only.

flubstep/thrift_client

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 

Repository files navigation

How to use:

This package thrift_client is intended to be used as a wrapper around
the default client generated by Thrift's gen-py. There are a couple
classes inside this package that are intended to be wrapped around the
service classes to add some additional features for Thrift.

Explanations are boring so let's go into some quick examples. Let's
say that we are implementing a Thrift service 'test_service'. You can
also find the files for this test case in the test/ folder. These
examples are relevant as of thrift-0.2.0, though you may want to use a
more recent version.

test_if.thrift:

    #!/usr/bin/thrift --gen py

    service test_service {
      string ping(),
      void setk(1:string key, 2:string value),
      string getk(1:string key),
      list<string> keys()
    }

This generates a folder called gen-py, which contains the Python
module test_if, with a Thrift-generated module
test_if.test_service. In order to create a simple Thrift client that
can talk to a test_service server, we do:

    import thrift_client
    import test_if.test_service

    if __name__ == '__main__':
        client = thrift_client.SimpleClient(test_if.test_service,
                     host = 'localhost', # this should be the correct host
                     port = 9988,        # this should be the correct port
                     frame = False,      # frame must match server settings
                     timeout = 60000     # timeout in ms
                 )

        # We can now use any function defined in test_if.thrift.
        client.ping()
        client.setk("key", "value1")
        print client.getk("key")
        print client.keys()

If we have multiple servers we want to connect to as a pool, we can
use a variety of options. One of the most encompassing ones is
ThreadedHashClient. To describe what it does, first we should
initialize the pool:

    import thrift_client
    import test_if.test_service

    if __name__ == '__main__':
        pool = thrift_client.ThreadedHashClient(test_if.test_service, frame=True)

        # Initialize a pool of 4 servers.
        pool.add_server('localhost:9901', name='server-1')
        pool.add_server('localhost:9902', name='server-2')
        pool.add_server('localhost:9903', name='server-3')
        pool.add_server('localhost:9904', name='server-4')

The service is initialized with a ThreadedHashClient, and 4 servers
are added to the pool. To demonstrate why it is called a HashClient,
we extend this pool by specifying a hash function for each thrift
call.

        # Set a hash function to shard this service by key.
        pool.setk.set_hash(lambda k, _: k)
        pool.getk.set_hash(lambda k: k)

What this means is that when we call pool.setk("key", "value"), it
will take the parameters ("key", "value"), pass it through the hash
"lambda k, _: k", hash that particular value modulo the number of
servers, and use that server in the pool to serve the
request. Likewise, when we call pool.getk("key"), it uses the hash
function "lambda k: k" to choose a server. Since both getk and setk
are sharded by key, they will both choose the same server to serve a
request.

There is one large difference between return values from a ThriftPool
and SimpleClient: the values returned are not the actual return types
from the server, but are instead ThriftResponse objects. This is so
that if you need more meta-information about the response, e.g. what
server served the request, you can extract that from the
ThriftResponse object.

- Most importantly: to get the actual value returned by the server
  respose r, use r.value(). If the server threw an exception, this
  function call will raise an exception.

- To determine if the server returned an exception without .value(),
  use r.is_error().

- To get the server that served the request r, use r.server. This
  returns a SimpleClient. To get its name, use r.server.name. To get
  its host, use r.server.host, etc.

If the hashing abstraction is too complicated and you wish to just
choose a server yourself, use the get_server(name) method of the
ThriftPool to get a SimpleClient out of the pool.

        pool.get_server(name='server-1').ping()

This will behave as a normal SimpleClient will.

ThreadedHashClient also has a property 'all' that will let you run a
particular method over all the servers in the pool. In this example,
we will try to get all the keys() that were added by setk() in all the
servers.

        responses = pool.all.keys()

This will return a list of ThriftResponse objects from all the servers
in the pool. The Thrift calls are made via a threadpool to optimize
the I/O (thus, the Threaded in ThreadedHashClient). For example, to
use it:

        for response in pool.all.keys():
            print "All the keys in %s" % response.server.name
            print response.value()

The 'all' property in a ThreadedHashClient is actually a
ThreadedReplicatedClient, another ThriftPool implementation that
always excutes every function call over all the clients in its
pool. If you intend on only using replicated function calls, use a
ThreadedReplicatedClient instead of a ThreadedHashClient object's
'all' property.

If you prefer a code example for both initializing Python clients and
servers, check out the example code in test/.

About

Python interfaces for easy creation of Thrift clients, replicated, threaded, and otherwise

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages