Occasional deadlock issue in tests that delete tasks

[jphickey]

Describe the bug
When running the unit tests repeatedly, occasionally some tests are getting into a deadlock. These tests are ones that:

1. use sub-tasks to do various work
2. those sub-tasks use OS_printf()
3. use an asynchronous OS_TaskDelete during their cleanup/teardown

In the event that the sub-task was in the midst of an OS_printf() call when OS_TaskDelete was invoked, the underlying BSP lock will not get released.

Observed in mutex-test, but others may have similar patterns.

To Reproduce
Run mutex-test repeatedly, may deadlock at some runs. (it is a race condition, not 100% reproducible)

Expected behavior
Should run consistently.

System observed on:
Ubuntu

Additional context
This really just a symptom of a generic/known issue with OS_TaskDelete, in that other resources held by that task are not necessarily tracked or freed, depending on what it was doing at the time it was deleted.

Linux/Pthreads does have a workaround but the issue is likely to exist on all OS's

Reporter Info
Joseph Hickey, Vantage Systems, Inc.

[jphickey]

Did some more digging on this one, because I am using the async console/utility task enabled for OS_printf, this should protect against any issues with the OS_printf calls.

In this case the issue is a little deeper:

1. When OS_TaskDelete is called by the main thread, the task record/ID is invalidated internally
2. But on Pthreads/Linux the task (task 2 in this case) keeps running until it hits a cancellation point
3. Next cancellation point will be during the clock_nanosleep() call (part of OS_TaskDelay at the top of the loop)
4. So Task 2 will still attempt to give the mutex one more time (OS_MutSemGive).
5. Inside OS_MutSemGive, this calls OS_TaskGetId as part of the debug check to confirm that this task owns the mutex
6. But since the task is pending deletion, OS_TaskGetId actually returns OS_OBJECT_ID_UNDEFINED here:

   osal/src/os/shared/src/osapi-task.c

   Line 351 in 1d183e9

   task_id = OS_OBJECT_ID_UNDEFINED;

7. This in turn causes it to fail the test and generate a debug message here:

   osal/src/os/shared/src/osapi-mutex.c

   Line 175 in 1d183e9

   OS_DEBUG("WARNING: Task %lu giving mutex %lu while owned by task %lu\n", OS_ObjectIdToInteger(self_task),

8. The debug message is output synchronously, and will directly invoke the BSP to print the message on the console. This will lock the BSP, then call OS_BSP_ConsoleOutput_Impl, which in turn invokes write().
9. But write() is also cancellation point - so this is where the cancellation will happen, and it will leave the BSP locked.

So in the end, this particular sequence of events is really a product of the deferred thread cancellation that exists on POSIX, and the fact that the write system call is a cancellation point, and that the implementation now calls this function with the mutex locked.

It also wouldn't happen if OS_CONFIG_DEBUG_PRINTF is disabled, and also related to the fact that the OS_TaskGetId doesn't work once deletion is pending. Basically if any one of these is changed, the issue doesn't happen.