add method to query HW size #630

jeffhammond · 2022-05-03T06:13:48Z

This is my first step towards addressing the oversubscription encountered in #588.

This adds a utility to figure out how many CPUs (hardware threads) are available in the affinity mask of the calling context. This is a stronger restriction than the number of available software threads.

@egaudry

Signed-off-by: Jeff Hammond jehammond@nvidia.com

Open questions (not all need to be addressed in this PR):

1) What about windows?
- HELP! @isuruf
2) What should happen when called in OMP threaded region?
- Respect nested OMP settings (e.g. omp_get_max_threads())?
- Single-threaded?
- Configurable?
3) What should happen when n_threads > n_hw_cores?
- Abort? @devinamatthews does not like
- Round-robin?
- Dynamic?
- Configurable?
4) When maintaining a pthreads thread pool (also TODO), also use CPU mask from user?
- No easy way to know when we're in a user parallel region. Should the user be responsible for setting a CPU mask?
5) What if the user doesn't set the CPU mask appropriately when running multiple user threads? (Could potentially be addressed in pthreads mode)
- Threads for all HW cores could be created and pinned at initialization and "checked out" when needed.

Signed-off-by: Jeff Hammond <jehammond@nvidia.com>

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

this is gross but i need #define GNU_SOURCE before any other headers, or the CPU_ISSET macro isn't defined. Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

devinamatthews · 2022-05-03T20:59:25Z

Sorry @jeffhammond I hijacked the PR description to flesh out some ideas related to the larger problem of thread pinning.

jeffhammond · 2022-05-04T07:52:08Z

To be clear, final version will not abort. But I could not figure out how to set BLIS threading variables correctly. That's why there's a preprocessing warning saying "please help me".

devinamatthews · 2022-05-04T14:45:10Z

@jeffhammond once we're in the OMP region, it's kind of too late to do anything except, as in bli_l3_thread_decorator_thread_check, detect that we have only one thread when we expected more (e.g. if nested OMP is disabled). I think this check would have to come before the OMP region and initialization of the global communicator.

devinamatthews · 2022-05-04T14:47:26Z

@jeffhammond can you walk me through the logic of the PR as implemented? I'm not sure how the CPU mask alone can reliably be used to detect oversubscription.

jeffhammond · 2022-05-04T14:47:55Z

I agree. We can do slightly better than serialization if nested is enabled but I don't think that's an important thing to spend time on.

jeffhammond · 2022-05-04T14:49:54Z

frame/thread/bli_l3_decor_openmp.c

@@ -199,6 +200,12 @@ void bli_l3_thread_decorator_thread_check
     )
 {
 	dim_t n_threads_real = omp_get_num_threads();
+        dim_t n_threads_hwmask;
+        if ( omp_in_parallel() ) {


@devinamatthews here, we look for the number of HW threads available to the thread if in OpenMP already, or the process, if not in OpenMP already, and if the user is trying to use more SW threads than HW threads, we abort.

Right, but this requires the user to have set the CPU mask (or OMP affinity?) correctly. What if I just run BLIS in OpenMP (with nesting) and set both OMP_NUM_THREADS and BLIS_NUM_THREADS equal to the number of cores? Or is this only meant to guard against a more specific use case?

jeffhammond · 2022-05-04T14:50:54Z

frame/thread/bli_l3_decor_openmp.c

+		bli_thrcomm_init( n_threads_hwmask, gl_comm );
+		bli_rntm_set_num_threads_only( n_threads_hwmask, rntm );
+#warning HELP ME HERE
+		bli_rntm_set_ways_only( 1, 1, 1, 1, 1, rntm );


@devinamatthews this is the part where i need help. i can just do serialization, but i was thinking about trying to do slightly better, e.g. if i have 40 cores and the user wants to run on 80 threads, i can set things up to use 40 threads properly.

Reducing the number of active threads after thread creation is possible, but not easy. If this logic can go before the OpenMP region then things are much easier.

jeffhammond · 2022-05-04T14:51:55Z

the affinity mask HW thread count is compared to the user-specified SW thread count.

devinamatthews · 2022-05-04T14:54:20Z

Ah OK, so the user could in principle reduce the number of HW cores "visible" to BLIS by setting a mask but if no action is taken then BLIS just sees all cores. OK.

devinamatthews · 2022-05-04T14:59:28Z

@jeffhammond sorry to be dense but I still don't see how this check fixes the oversubscription problem in #588. Each "instance" of BLIS would still pass the check, but the total number of threads is more than the number of cores. Detecting nested OMP and dropping down to one core could work, but would have to be configurable. Otherwise, maybe we could check omp_get_max_threads?

jeffhammond · 2022-05-04T15:02:42Z

I've asked @egaudry to test with his application, which may be settings affinity masks via MPI or HWLOC.

But I can say that it works to detect oversubscription for me. If I set OMP_NUM_THREADS=16 on an Intel CPU with 8 HW threads, it aborts. When I set OMP_NUM_THREADS=8, it's fine. I added a test that can do other things.

devinamatthews · 2022-05-04T15:05:28Z

If I set OMP_NUM_THREADS=16 on an Intel CPU with 8 HW threads, it aborts. When I set OMP_NUM_THREADS=8, it's fine. I added a test that can do other things.

Yes that should work fine, but setting OMP_NUM_THREADS=BLIS_NUM_THREADS=4 and nesting should not abort and yet oversubscribe. Isn't this the more pressing problem?

devinamatthews · 2022-05-04T15:07:28Z

Here are my suggested solutions:

Maintain a process-wide atomic int which counts the number of active threads. Abort (or warn etc.) if this exceeds the number of HW cores.
Prepare a set of tokens, one for each HW core. Threads check out these tokens when created. Abort if no tokens are available. This approach can be extended to address thread pinning since each token can indicate a particluar physical core.

devinamatthews · 2022-05-04T15:09:59Z

Also, I suggest an environment variable BLIS_OVERSUBSCRIBE. The values ABORT and WARN would do what you expect and any other value ignores oversubscription. Maybe also a SERIALIZE option?

jeffhammond · 2022-05-05T18:35:20Z

The token idea doesn't work if BLIS runs alongside other things that use threads. But I'll see what I can do.

I disagree that nesting is the higher priority. The bug that motivated all of this was not because of nested OpenMP.

devinamatthews · 2022-05-05T18:44:26Z

@egaudry @jeffhammond the issue in #588 was explained here. How is that not nested OMP?

jeffhammond · 2022-05-05T18:53:26Z

Okay it was nested open but also affinity. I can fix them both.

egaudry · 2022-05-10T12:30:36Z

Sorry, I haven't had the time to check on my side.

@jeffhammond is correct that the issue was seen because an affinity mask was set
@devinamatthews is also correct that the issue originated from the oversubscription

to clarify, the slowdown occurred because more thread than available physical cores where used in BLIS within a region protected with an openmp lock.
this could happen as well with oversubscription in case of a nested omp loop, with more thread being used than physically available cores.

jeffhammond added the enhancement label May 3, 2022

jeffhammond requested a review from devinamatthews May 3, 2022 06:13

jeffhammond self-assigned this May 3, 2022

add method to query HW size

8bd9160

Signed-off-by: Jeff Hammond <jehammond@nvidia.com>

jeffhammond force-pushed the affinity_mask_size branch from 19de193 to 8bd9160 Compare May 3, 2022 06:38

jeffhammond added 7 commits May 3, 2022 09:41

use Linux macro

f6e1f56

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

move headers around

b3844fd

this is gross but i need #define GNU_SOURCE before any other headers, or the CPU_ISSET macro isn't defined. Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

add copyright because of changes to bli_l3_thread_decorator_thread_check

3193c90

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

make affinity problem into a fatal error for now

629ef19

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

hack to fix the affinity macro business

3697489

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

add a test of the affinity check

5906e3f

copyright

5550527

Signed-off-by: Jeff Hammond <jeff.science@gmail.com>

jeffhammond commented May 4, 2022

View reviewed changes

rkamd mentioned this pull request Aug 29, 2024

Hotfix : Add a note about thread oversubscription in AOCL ROCm/rocBLAS#1472

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add method to query HW size #630

add method to query HW size #630

jeffhammond commented May 3, 2022 •

edited by devinamatthews

Loading

devinamatthews commented May 3, 2022

jeffhammond commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

jeffhammond commented May 4, 2022

jeffhammond May 4, 2022

devinamatthews May 4, 2022

jeffhammond May 4, 2022

devinamatthews May 4, 2022

jeffhammond commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

jeffhammond commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

jeffhammond commented May 5, 2022

devinamatthews commented May 5, 2022

jeffhammond commented May 5, 2022

egaudry commented May 10, 2022

add method to query HW size #630

Are you sure you want to change the base?

add method to query HW size #630

Conversation

jeffhammond commented May 3, 2022 • edited by devinamatthews Loading

devinamatthews commented May 3, 2022

jeffhammond commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

jeffhammond commented May 4, 2022

jeffhammond May 4, 2022

Choose a reason for hiding this comment

devinamatthews May 4, 2022

Choose a reason for hiding this comment

jeffhammond May 4, 2022

Choose a reason for hiding this comment

devinamatthews May 4, 2022

Choose a reason for hiding this comment

jeffhammond commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

jeffhammond commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

devinamatthews commented May 4, 2022

jeffhammond commented May 5, 2022

devinamatthews commented May 5, 2022

jeffhammond commented May 5, 2022

egaudry commented May 10, 2022

jeffhammond commented May 3, 2022 •

edited by devinamatthews

Loading