[lldb-dap] Addressing the order of events during disconnect to flush output.

[ashgti]

The TestDAP_ouput test is flaky due to the order of events during shutdown. We were stopping the output and error handle redirection after we finished the disconnect request, which can cause us to miss output events due to timing. Moving when we stop the redirection to ensure we have consistent output prior to disconnect responding.

Fixes #128567

[llvmbot]

@llvm/pr-subscribers-lldb

Author: John Harrison (ashgti)

Changes

The TestDAP_ouput test is flaky due to the order of events during shutdown. We were stopping the output and error handle redirection after we finished the disconnect request, which can cause us to miss output events due to timing. Moving when we stop the redirection to ensure we have consistent output prior to disconnect responding.

Fixes #128567

---

Full diff: https://github.com/llvm/llvm-project/pull/128583.diff

3 Files Affected:

• (modified) lldb/test/API/tools/lldb-dap/output/TestDAP_output.py (+4)
• (modified) lldb/tools/lldb-dap/DAP.cpp (+6-5)
• (modified) lldb/tools/lldb-dap/DAP.h (+2-2)

diff --git a/lldb/test/API/tools/lldb-dap/output/TestDAP_output.py b/lldb/test/API/tools/lldb-dap/output/TestDAP_output.py
index eae7629409844..fe54511d1e21f 100644
--- a/lldb/test/API/tools/lldb-dap/output/TestDAP_output.py
+++ b/lldb/test/API/tools/lldb-dap/output/TestDAP_output.py
@@ -16,6 +16,7 @@ def test_output(self):
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(
             program,
+            disconnectAutomatically=False,
             exitCommands=[
                 # Ensure that output produced by lldb itself is not consumed by the OutputRedirector.
                 "?script print('out\\0\\0', end='\\r\\n', file=sys.stdout)",
@@ -33,6 +34,9 @@ def test_output(self):
 
         self.continue_to_exit()
 
+        # Disconnecting from the server to ensure any pending IO is flushed.
+        self.dap_server.request_disconnect()
+
         output += self.get_stdout(timeout=lldbdap_testcase.DAPTestCaseBase.timeoutval)
         self.assertTrue(output and len(output) > 0, "expect program stdout")
         self.assertIn(
diff --git a/lldb/tools/lldb-dap/DAP.cpp b/lldb/tools/lldb-dap/DAP.cpp
index 9b22b60a68d94..7a5cd39535617 100644
--- a/lldb/tools/lldb-dap/DAP.cpp
+++ b/lldb/tools/lldb-dap/DAP.cpp
@@ -223,10 +223,7 @@ llvm::Error DAP::ConfigureIO(std::FILE *overrideOut, std::FILE *overrideErr) {
   return llvm::Error::success();
 }
 
-void DAP::StopIO() {
-  out.Stop();
-  err.Stop();
-
+void DAP::StopEventHandlers() {
   if (event_thread.joinable()) {
     broadcaster.BroadcastEventByType(eBroadcastBitStopEventThread);
     event_thread.join();
@@ -853,13 +850,17 @@ lldb::SBError DAP::Disconnect(bool terminateDebuggee) {
 
   SendTerminatedEvent();
 
+  // Stop forwarding the debugger output and error handles.
+  out.Stop();
+  err.Stop();
+
   disconnecting = true;
 
   return error;
 }
 
 llvm::Error DAP::Loop() {
-  auto stop_io = llvm::make_scope_exit([this]() { StopIO(); });
+  auto cleanup = llvm::make_scope_exit([this]() { StopEventHandlers(); });
   while (!disconnecting) {
     llvm::json::Object object;
     lldb_dap::PacketStatus status = GetNextObject(object);
diff --git a/lldb/tools/lldb-dap/DAP.h b/lldb/tools/lldb-dap/DAP.h
index 18de39838f218..d0e1217aaeb8e 100644
--- a/lldb/tools/lldb-dap/DAP.h
+++ b/lldb/tools/lldb-dap/DAP.h
@@ -224,8 +224,8 @@ struct DAP {
   llvm::Error ConfigureIO(std::FILE *overrideOut = nullptr,
                           std::FILE *overrideErr = nullptr);
 
-  /// Stop the redirected IO threads and associated pipes.
-  void StopIO();
+  /// Stop event handler threads.
+  void StopEventHandlers();
 
   // Serialize the JSON value into a string and send the JSON packet to
   // the "out" stream.

[JDevlieghere]

LGTM

[labath]

It looks like it's still flaky: https://lab.llvm.org/buildbot/#/builders/18/builds/11957

I think there's still at least one possible way this can race (return without reading all data):

1. read thread returns from the read call (reads some random data) and gets suspended
2. we begin shutting down: set the flag, and write the sentinel to the pipe
3. read thread wakes up, checks the flag and decides to exit (without reading our sentinel, or any other remaining data)

I think that one way to improve this would be to do a non-blocking read to drain the pipe before returning from the read thread.. which reminds me that I wanted to fix the problem which prevented you from using the Pipe class ~~> #128719