Can I safely unload DLL after the detach hook?

[sunjal0816-design]

Currently, I inject a DLL into a running process. Inside the DLL, I use the Detours Hook API, and it works very well. Now I want to unload this DLL without interrupting the process, so I exported an unload function from the DLL for remote invocation. This function performs the following calls:

DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
// some DetourDetach calls
DetourTransactionCommit();
FreeLibraryAndExitThread();

After I call this function, the DLL is indeed unloaded, but shortly afterward the target process crashes. It seems that the target process is still trying to access the DLL code that has already been unloaded.

What should I do to safely unload the DLL without affecting the target process?