http_listener default thread count - 40 threads

[zerodefect]

After trying the cpprest library out (on Linux), specifically the http_listener, I have quickly discovered that the default number of threads that are created is 40 threads. Nonetheless, I've examined the code, and I have been unable to ascertain why a value of 40 has been chosen.

A hard-coded thread count is likely to go against a popular school of thought that the number of threads should match the number of CPU cores (or at the most: thread_count <= (2 x cpu cores)). For example, my development system contains 4 cores.

Having thought about this further, there should be two (2) potential changes - the first one is easier than the second:

1. The thread count should be configurable via a parameter during intialisation of the http_listener.
2. Applicable classes should enable developers to instantiate instances of an object specifying a custom thread pool.

Environment:
Ubuntu Zesty 17.04
Boost 1.62
GCC 6.3
cpprest 2.9

[zerodefect]

In Release/src/pplx/threadpool.cpp, I added the following code to detect/set the default thread count.

namespace
{
    
inline size_t default_thread_count()
{
    // Hardware_concurrency may return 0. Nonetheless, need at least two threads.
    return std::max<size_t>(size_t{2},std::thread::hardware_concurrency());
}
    
// ...

// initialize the static shared threadpool
threadpool& threadpool::shared_instance()
{
    static threadpool_impl s_shared(default_thread_count());
    return s_shared;
}

`Unfortunately, I note that a few of the unit tests fail (timeout). Specifically:

• libhttplistener_test.so : request_handler_tests:multiple_clients_multiple_requests
• libpplx_test.so : pplxtask_tests:TestTaskOperators_and_or5

It looks as the test code expects to have more threads at its disposal.

[cmazakas]

Well, there is something to be said for using more threads than there are physical processors.

As task-based as PPL is, each time a thread blocks on something, it's free to be rescheduled. By using context switches at the kernel level, you can avoid a lot of callback management at the software level.

I'm not sure if this is the case for the cpprestsdk but it is something to be said for the threaded model.

[zerodefect]

It swings both ways. Context switches are not free. During a context switch, no useful work is done.

My point comes from someone who is trying to maximise CPU throughput for a given piece of software in the field; minimising thread context switches is imperative.

[mobileben]

I concur that there should be a means to configure the threadpool. 40 is pretty arbitrary and a hardcoded value too.

threadpool& threadpool::shared_instance()
{
    static threadpool_impl s_shared(40);
    return s_shared;
}

It's not just http_listener that gets this. I'm just running client code and it also has the 40 threads because the client will invariably call crossplat::thread pool::static_instance().service().

[zerodefect]

I'd be happy to take this on, but I wouldn't mind some insight from someone at Microsoft on how best to resolve this. I'm just aware that this issue is in the pplx library and not cpprest itself.

@mobileben - I see that you have created a fork. Are you trying to address this issue? If so, I'll stand aside :)

[mobileben]

@zerodefect the main intent of my fork was for updating the iOS build process since it no longer works (at least with latest version of Xcode). But I also would probably make some adjustment to the threads because I don't want all those threads running in my game.

Perhaps we should discuss some approaches. It doesn't matter who does this in the end. But it would probably be preferable to come to some consensus as to a reasonable approach. I started using this relatively recently, so certainly don't want to break anything. And I'm also working in Mac/iOS, so wouldn't really be able to check stuff against other platforms.

One thing that is somewhat concerning to me right now is it isn't incredibly clear to me on the actual interest level MS has on maintaining this or improving this code base. My only basis for this is that I don't seen a lot of issues getting traffic. Some issues I get where they are just asking how to do X, Y, or Z. But there are some that seem legit. I also saw one comment about how they emailed askcasablanca email, but haven't heard back. And here is one where someone has volunteered to do some changes, but crickets.

In either case, while the issue is within pplx, the "issue" becomes how to adjust it? What I mean is one doesn't want to rebuild the library to change the value. As you suggested you can add a function to affect the default. It seems helpful if there is baked in default number, but the user could pick an arbitrary value. It would also be ideal to minimize the steps to the user (as well as make it easy to do).

One thought I had was to use http_listener_config and http_client_config as the means to pass the value through. This has pros and cons. And one would have to think about what happens if each config has a different value. The way the thread pool is built, it is first come first serve. Part of why I thought about doing this is that the thread pools are created upon the creation of the client or listener. So it is sort of the "gatekeeper" in preventing things from being done out of order. I haven't examined the code enough yet to see if there are any side effects from doing this.

Of course we could always add a static method to threadpool to set the value. This is a very direct and easy approach. However, would need to determine what to do if there user already created the thread pool (like maybe throw an exception so they know about it).

[mobileben]

@zerodefect I implemented this. I'll probably commit it later today or tomorrow. I went for adding a static function onto thread pool.

static void set_num_threads(size_t num_threads);

So usage would look something like (doing what you originally wanted to do):

    auto numThreads = std::max<size_t>(size_t{2},std::thread::hardware_concurrency());
    crossplat::threadpool::set_num_threads(numThreads);

I also added in a flag to indicate if the thread pool has been created. This will throw an exception is set_num_threads is called after the thread pool has been created.

[mcmindcoder]

Similar story: I noticed on Linux cpprest creates 40 threads, which feels weird.
Like @zerodefect after some investigation I realized that 40 is hardcoded value:
https://github.com/Microsoft/cpprestsdk/blob/6b72090ae080e6bfc75a554ef221bbf1d2850cb3/Release/src/pplx/threadpool.cpp

After some extra investigation I also found a way to change it:
https://microsoft.github.io/cpprestsdk/classpplx_1_1task__options.html#a59e0a10faf1b2b35bf7e45f53df20d5a
Methods

pplx::task_options::task_options(std::shared_ptr< _SchedType > 	_Scheduler)	
pplx::task_options::task_options(scheduler_interface &_Scheduler)	
pplx::task_options::task_options(scheduler_ptr _Scheduler)	

allow to set custom scheduler. Default scheduler use pplx threadpool with hardcoded values.

I haven't tried it yet, but it's straightforward and looks like what we need, isn't it?

[zerodefect]

@mc-android-developer, thanks for this. I had a quick look at the code upon noticing your recommendation, but I couldn't quite grasp how this would work in my scenario.

From looking at the API, my understanding is that the 'task_options' must be specified on a task-by-task basis at the time of task creation (constructor)? In my case, I'm using the http_listener - the listener abstracts away from the internals of creating tasks. In this case, I don't believe that task_options would be an option in my scenario.

Thanks.

[zerodefect]

I've had the opportunity to visit this again since originally raising the issue. I found a way to circumvent some of the issues, however, I quickly found myself encountering new issues.

In pplx.h :

void _pplx_cdecl set_ambient_scheduler(std::shared_ptr<pplx::scheduler_interface>_Scheduler);
std::shared_ptr<pplx::scheduler_interface> _pplx_cdecl get_ambient_scheduler();

Effectively, you need to implement your own scheduler from the pplx::scheduler_interface (pplxinterface.h). My specific implementation uses an existing boost::asio threadpool that I have already created (which scales according to the number of CPUs).

This resolves issues with tasks created from pplx. The next problems comes with the http listeners because...

What I was then expecting (upon running my application) was to see a reduction in the number of threads by 40. This was not the case.

If you take a look at the likes of hostport_listener::start() in http_server_asio.cpp, there is code which is hard-coded to create an instance of the crossplatform shared threadpool irrespective of the ambient scheduler (which makes sense from a design point).

auto& service = crossplat::threadpool::shared_instance().service();

It feels like we need a factory pattern so that developers can optionally specify their own boost io_service (which I currently create for my own threadpool) when using linux variants.

@ras0219-msft would you be able to kindly offer any suggestions? I'm happy to go away and do the hard work.

[oprince]

How can I access the default_thread_count() method from a code which is making use of http_listener ?

[zerodefect]

@oprince - I'm not sure there is a way this can be done (at the moment). You need this information to be accessible via a getter in threadpool. This doesn't appear to be the case.

[oprince]

@zerodefect - Thanks for your reply !
We have a service based on cpprest http_listener.
It is freezing when active threads count reaches 40, and we are now looking for a solution such as replying with "busy" response if active threads count reaches the limit.
To implement that solution we need to (programmatically) get the limit value...
Is there an alternate solution you would recommend ?
Thanks!