Pipe blocking when input too large? (Mac OS) #218

Activity

1. snoyberg commented on Oct 13, 2021

   

   snoyberg

   on Oct 13, 2021

   Collaborator

   More actions

   The library is behaving correctly in this case. hPutStr is filling up an OS buffer, and blocking when it's full. If the amount of data is small enough, the blocking never occurs and the second process can start. Once the data is large enough to fill the buffer, we block before less can be spawned and cannot drain the pipe.

   When you use shell, the shell itself handles the asynchronous aspect of things.
2. andreasabel commented on Oct 13, 2021

   

   andreasabel

   on Oct 13, 2021

   Last edited by andreasabel

   MemberAuthor

   More actions

   Thanks for the explanation! Makes perfect sense now.

   Consequently, here is the correct implementation of the pipe, which indeed works as expected:

   main = do
   
     (Just inp, Just out, _, ph1)  <- createProcess $
       (proc "cat" [])
         { std_in  = CreatePipe
         , std_out = CreatePipe
         }
     (_, _, _, ph2) <- createProcess $
       (proc "less" [])
         { std_in = UseHandle out }
   
     -- After the pipe is set up, we can pour the input without blocking:
     hPutStr inp longInput
     hClose inp
   
     waitForProcess ph1
     waitForProcess ph2
     putStrLn "Program terminated successfully."
   
     where
     longInput = unlines $ replicate 10000 $ replicate 63 'A'

   Alternatively, writing to inp can be done concurrently to not block the creation of the pipe:

   import Control.Concurrent.Async
   import System.IO
   import System.Process
   
   main = do
   
     (Just inp, Just out, _, ph1)  <- createProcess $
       (proc "cat" [])
         { std_in  = CreatePipe
         , std_out = CreatePipe
         }
   
     withAsync (hPutStr inp longInput >> hClose inp) $ \_ -> do
   
       (_, _, _, ph2) <- createProcess $
         (proc "less" [])
           { std_in = UseHandle out }
   
       waitForProcess ph1
       waitForProcess ph2
       putStrLn "Program terminated successfully."
   
     where
     longInput = unlines $ replicate 10000 $ replicate 63 'A'

   I wonder whether this example could be added as tutorial somewhere to the documentation (I can do this if it is welcome). Previously, I had googled a lot but did not find enough examples for CreatePipe.
3. snoyberg commented on Oct 13, 2021

   

   snoyberg

   on Oct 13, 2021

   Collaborator

   More actions

   I’m certainly up for such a doc addition. At the very least, a “make sure you don’t deadlock by filing buffers” with a link to this issue would be great.