This repository was archived by the owner on Aug 30, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
flubstep/thrift_client
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
How to use: This package thrift_client is intended to be used as a wrapper around the default client generated by Thrift's gen-py. There are a couple classes inside this package that are intended to be wrapped around the service classes to add some additional features for Thrift. Explanations are boring so let's go into some quick examples. Let's say that we are implementing a Thrift service 'test_service'. You can also find the files for this test case in the test/ folder. These examples are relevant as of thrift-0.2.0, though you may want to use a more recent version. test_if.thrift: #!/usr/bin/thrift --gen py service test_service { string ping(), void setk(1:string key, 2:string value), string getk(1:string key), list<string> keys() } This generates a folder called gen-py, which contains the Python module test_if, with a Thrift-generated module test_if.test_service. In order to create a simple Thrift client that can talk to a test_service server, we do: import thrift_client import test_if.test_service if __name__ == '__main__': client = thrift_client.SimpleClient(test_if.test_service, host = 'localhost', # this should be the correct host port = 9988, # this should be the correct port frame = False, # frame must match server settings timeout = 60000 # timeout in ms ) # We can now use any function defined in test_if.thrift. client.ping() client.setk("key", "value1") print client.getk("key") print client.keys() If we have multiple servers we want to connect to as a pool, we can use a variety of options. One of the most encompassing ones is ThreadedHashClient. To describe what it does, first we should initialize the pool: import thrift_client import test_if.test_service if __name__ == '__main__': pool = thrift_client.ThreadedHashClient(test_if.test_service, frame=True) # Initialize a pool of 4 servers. pool.add_server('localhost:9901', name='server-1') pool.add_server('localhost:9902', name='server-2') pool.add_server('localhost:9903', name='server-3') pool.add_server('localhost:9904', name='server-4') The service is initialized with a ThreadedHashClient, and 4 servers are added to the pool. To demonstrate why it is called a HashClient, we extend this pool by specifying a hash function for each thrift call. # Set a hash function to shard this service by key. pool.setk.set_hash(lambda k, _: k) pool.getk.set_hash(lambda k: k) What this means is that when we call pool.setk("key", "value"), it will take the parameters ("key", "value"), pass it through the hash "lambda k, _: k", hash that particular value modulo the number of servers, and use that server in the pool to serve the request. Likewise, when we call pool.getk("key"), it uses the hash function "lambda k: k" to choose a server. Since both getk and setk are sharded by key, they will both choose the same server to serve a request. There is one large difference between return values from a ThriftPool and SimpleClient: the values returned are not the actual return types from the server, but are instead ThriftResponse objects. This is so that if you need more meta-information about the response, e.g. what server served the request, you can extract that from the ThriftResponse object. - Most importantly: to get the actual value returned by the server respose r, use r.value(). If the server threw an exception, this function call will raise an exception. - To determine if the server returned an exception without .value(), use r.is_error(). - To get the server that served the request r, use r.server. This returns a SimpleClient. To get its name, use r.server.name. To get its host, use r.server.host, etc. If the hashing abstraction is too complicated and you wish to just choose a server yourself, use the get_server(name) method of the ThriftPool to get a SimpleClient out of the pool. pool.get_server(name='server-1').ping() This will behave as a normal SimpleClient will. ThreadedHashClient also has a property 'all' that will let you run a particular method over all the servers in the pool. In this example, we will try to get all the keys() that were added by setk() in all the servers. responses = pool.all.keys() This will return a list of ThriftResponse objects from all the servers in the pool. The Thrift calls are made via a threadpool to optimize the I/O (thus, the Threaded in ThreadedHashClient). For example, to use it: for response in pool.all.keys(): print "All the keys in %s" % response.server.name print response.value() The 'all' property in a ThreadedHashClient is actually a ThreadedReplicatedClient, another ThriftPool implementation that always excutes every function call over all the clients in its pool. If you intend on only using replicated function calls, use a ThreadedReplicatedClient instead of a ThreadedHashClient object's 'all' property. If you prefer a code example for both initializing Python clients and servers, check out the example code in test/.
About
Python interfaces for easy creation of Thrift clients, replicated, threaded, and otherwise
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published