worked on tests,code readability,... #1

kiarash8112 · 2023-11-01T17:14:17Z

used concurrency pattern in master.go
improved grpc error handling
added tests for master
create interfaces to reduce duplication ...

…o mock rpc call in tests

…oved rpc calls error handling

BWbwchen · 2023-11-04T06:18:44Z

Hi @kiarash8112 ,

Thank you for your contribution.

But when I executed your PR, I got an error, and this is the log:

go run -race cmd/main.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 8
INFO[0000] [Master] Master gRPC server start
TRAC[0000] [Master] Wait for enough workers
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker gRPC server start
INFO[0000] Worker load plugin finish
INFO[0000] Worker load plugin finish
INFO[0000] Worker load plugin finish
TRAC[0000] With Time out
TRAC[0000] Start RPC call
INFO[0000] Worker load plugin finish
TRAC[0000] With Time out
TRAC[0000] Start RPC call
INFO[0000] Worker load plugin finish
TRAC[0000] With Time out
INFO[0000] Worker load plugin finish
TRAC[0000] With Time out
TRAC[0000] With Time out
INFO[0000] Worker load plugin finish
TRAC[0000] With Time out
TRAC[0000] Start RPC call
TRAC[0000] With Time out
TRAC[0000] Start RPC call
TRAC[0000] Start RPC call
TRAC[0000] Start RPC call
INFO[0000] Worker load plugin finish
TRAC[0000] With Time out
TRAC[0000] Start RPC call
TRAC[0000] Start RPC call
INFO[0000] [Master] Worker register success
INFO[0000] [Master] Worker register success
INFO[0000] [Master] Worker register success
INFO[0000] [Master] Worker register success
INFO[0000] [Master] Worker register success
INFO[0000] [Master] Worker register success
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
TRAC[0000] End RPC call
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
INFO[0000] Worker register itself finish
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
INFO[0000] [Master] Worker register success
INFO[0000] [Master] Worker register success
TRAC[0000] [Master] Enough workers!
TRAC[0000] [Master] Start distribute workload
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
TRAC[0000] End RPC call
INFO[0000] Worker register itself finish
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] Add map input file
TRAC[0000] [Master] End distribute workload
TRAC[0000] [Master] Start Map task
INFO[0000] [Master] Finding available workers to execute 8/8
INFO[0000] [Worker] Start Map
TRAC[0000] [Worker] Start Mapping
INFO[0000] [Worker] Start Map
TRAC[0000] [Worker] Start Mapping
INFO[0000] [Worker] Start Map
TRAC[0000] [Worker] Start Mapping
INFO[0000] [Worker] Start Map
INFO[0000] [Worker] Start Map
INFO[0000] [Worker] Start Map
TRAC[0000] [Worker] Start Mapping
TRAC[0000] [Worker] Start Mapping
INFO[0000] [Worker] Start Map
INFO[0000] [Worker] Start Map
TRAC[0000] [Worker] Start Mapping
TRAC[0000] [Worker] Start Mapping
TRAC[0000] [Worker] Start Mapping
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] Finish Mapping
TRAC[0001] [Worker] Start partition intermediate kv
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 2daf75fb-c3a9-4323-8f11-b6f3a5825b41 update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 076c4d7f-039e-4045-bb7d-a6f30ef2afe7 update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 7c92f8b4-6acf-45b8-a161-daee183e5b11 update IMD info success
TRAC[0001] [Worker] End partition intermediate kv
TRAC[0001] [Worker] Write intermediate kv to file
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 3885ddb1-ea03-4b06-a0e8-de939e1adabe update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 66569a3b-8794-4120-b1ed-d847c4c9be81 update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 95e7ea72-5ba2-4298-b0fe-a61156e2043c update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 85b240ae-444a-4820-b178-2ad3a9d125a1 update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Worker] End Write intermediate kv to file
TRAC[0001] [Worker] Tell Master the intermediate info
INFO[0001] [Master] 142ed721-6307-449d-8869-972503a29989 update IMD info success
TRAC[0001] [Worker] Finish Tell Master the intermediate info
INFO[0001] [Worker] Finish Map Task
TRAC[0001] [Master] End Map task
TRAC[0001] [Master] Start Reduce task
INFO[0001] [Master] Finding available workers to execute 4/8
INFO[0001] [Worker] Start Reduce
TRAC[0001] [Worker] Get intermediate file
INFO[0001] [Worker] Start Reduce
TRAC[0001] [Worker] Get intermediate file
INFO[0001] [Worker] Start Reduce
INFO[0001] [Worker] Start Reduce
TRAC[0001] [Worker] Get intermediate file
TRAC[0001] [Worker] Get intermediate file
PANI[0001] unknown service Worker
panic: (*logrus.Entry) 0xc0003d5e30

goroutine 304 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003d5570, 0x0, {0xc00016c1b0, 0x16})
        /home/bwbwchen/work/MapReduce/vendor/github.com/sirupsen/logrus/entry.go:259 +0x986
github.com/sirupsen/logrus.(*Entry).Log(0xc0003d5570, 0x0, {0xc0006775f8, 0x1, 0x1})
        /home/bwbwchen/work/MapReduce/vendor/github.com/sirupsen/logrus/entry.go:293 +0x8c
github.com/sirupsen/logrus.(*Logger).Log(0xc0001c4070, 0x0, {0xc0006775f8, 0x1, 0x1})
        /home/bwbwchen/work/MapReduce/vendor/github.com/sirupsen/logrus/logger.go:198 +0x85
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /home/bwbwchen/work/MapReduce/vendor/github.com/sirupsen/logrus/logger.go:247
github.com/sirupsen/logrus.Panic(...)
        /home/bwbwchen/work/MapReduce/vendor/github.com/sirupsen/logrus/exported.go:129
github.com/BWbwchen/MapReduce/worker.(*masterClient).GetIMDData(0xc000384078, {0x6?, 0xc000677730?}, {0xc000118640, 0x37})
        /home/bwbwchen/work/MapReduce/worker/rpcClient.go:98 +0x2f4
github.com/BWbwchen/MapReduce/worker.(*Worker).Reduce(0xc0003d4000, {0xc000146640?, 0xc00067d818?}, 0xc000146640)
        /home/bwbwchen/work/MapReduce/worker/worker.go:198 +0x298
github.com/BWbwchen/MapReduce/rpc._Worker_Reduce_Handler({0xed29e0?, 0xc0003d4000}, {0x1008bb0, 0xc000211fb0}, 0xc000120d20, 0x0)
        /home/bwbwchen/work/MapReduce/rpc/worker_grpc.pb.go:165 +0x25b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003d8000, {0x100c5d8, 0xc000449200}, 0xc0001b4900, 0xc0003824e0, 0x13851f8, 0x0)
        /home/bwbwchen/work/MapReduce/vendor/google.golang.org/grpc/server.go:1282 +0x13eb
google.golang.org/grpc.(*Server).handleStream(0xc0003d8000, {0x100c5d8, 0xc000449200}, 0xc0001b4900, 0x0)
        /home/bwbwchen/work/MapReduce/vendor/google.golang.org/grpc/server.go:1616 +0xff9
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/bwbwchen/work/MapReduce/vendor/google.golang.org/grpc/server.go:921 +0xed
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/bwbwchen/work/MapReduce/vendor/google.golang.org/grpc/server.go:919 +0x4de
exit status 2

kiarash8112 · 2023-11-04T09:18:29Z

Hi @BWbwchen thank you for your review
i fixed this bug and tested multi node manually
but when i run make test_multi_node i get data race warning even in base branch

master/master.go

BWbwchen · 2023-11-07T05:07:44Z

LGTM. Merged.

@kiarash8112 , Thank you for your contribution!

BWbwchen · 2023-11-07T05:25:44Z

And for the potential race conditional warning that you mentioned before, I think that will be the time-lagging issue. I solved it in this commit.

When the worker receives the End grpc, it will signal the EndChan immediately, and the worker will close its grpc server and the master can't receive the response from the worker.

kiarash8112 · 2023-11-07T09:21:14Z

when i run make test_multi_node it get following error

rm output/*
rm: cannot remove 'output/*': No such file or directory
make: [Makefile:10: clean] Error 1 (ignored)
rm cmd/*.so
rm: cannot remove 'cmd/*.so': No such file or directory
make: [Makefile:11: clean] Error 1 (ignored)
go build -race -buildmode=plugin -o cmd/wc.so ./mrapps/wc.go
go build -race -buildmode=plugin -o cmd/merge.so ./mrapps/merge.go
rm output/*
rm: cannot remove 'output/*': No such file or directory
make: [Makefile:27: test_multi_node] Error 1 (ignored)
rm mr-out-*
rm: cannot remove 'mr-out-*': No such file or directory
make: [Makefile:28: test_multi_node] Error 1 (ignored)
go run -race cmd/master.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 8 &
sleep 1
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 1 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 2 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 3 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 4 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 5 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 6 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 7 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 8 
INFO[0000] [Master] Master gRPC server start            
TRAC[0000] [Master] Wait for enough workers             
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0000] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
TRAC[0001] [Master] Enough workers!                     
INFO[0001] [Master] Worker register success             
TRAC[0001] [Master] Start distribute workload           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] [Master] End distribute workload             
TRAC[0001] [Master] Start Map task                      
INFO[0001] [Master] Finding available workers to execute 8/8 
INFO[0001] [Worker] Start Map                           
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0001] [Worker] Start Mapping                       
TRAC[0001] [Worker] Finish Mapping                      
TRAC[0001] [Worker] Start partition intermediate kv     
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0001] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] Finish Mapping                      
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0001] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0001] [Worker] End Write intermediate kv to file   
TRAC[0001] [Worker] Tell Master the intermediate info   
INFO[0001] [Master] dfbe266c-095f-43c0-ba22-f03c1a3192dc update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0001] [Master] 52800c38-289e-40ac-98c8-e95b01ba7f8e update IMD info success 
INFO[0001] [Master] 4744d331-a214-4015-abbb-f32e2239dd5d update IMD info success 
INFO[0001] [Master] 84a20685-69ef-4b76-b41d-046ab8fa17ea update IMD info success 
INFO[0001] [Master] 8be49474-e777-48ee-a1cb-4ca5f5b88c30 update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0001] [Worker] Finish Tell Master the intermediate info 
INFO[0001] [Worker] Finish Map Task                     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] End Write intermediate kv to file   
INFO[0001] [Master] 09a01a3b-2239-4c63-9428-17bad0ea82f9 update IMD info success 
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0001] [Master] 07e24826-d4d9-44dd-98ed-0e704c6e2a85 update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0001] [Master] 40d0408b-cacd-41c1-b79d-fec6a7de044f update IMD info success 
INFO[0000] [Worker] Finish Map Task                     
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
TRAC[0001] [Master] End Map task                        
TRAC[0001] [Master] Start Reduce task                   
INFO[0001] [Master] Finding available workers to execute 4/8 
INFO[0001] [Worker] Start Reduce                        
TRAC[0001] [Worker] Get intermediate file               
INFO[0000] [Worker] Start Reduce                        
TRAC[0000] [Worker] Get intermediate file               
INFO[0000] [Worker] Start Reduce                        
TRAC[0000] [Worker] Get intermediate file               
INFO[0000] [Worker] Start Reduce                        
TRAC[0000] [Worker] Get intermediate file               
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0001] [Worker] Sort intermediate KV                
TRAC[0001] [Worker] Start Reducing                      
TRAC[0001] [Worker] End Reducing                        
INFO[0001] [Worker] End Reduce                          
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0000] [Worker] Sort intermediate KV                
TRAC[0000] [Worker] Start Reducing                      
TRAC[0000] [Worker] End Reducing                        
INFO[0000] [Worker] End Reduce                          
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0000] [Worker] Sort intermediate KV                
TRAC[0000] [Worker] Start Reducing                      
TRAC[0000] [Worker] End Reducing                        
INFO[0000] [Worker] End Reduce                          
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0000] [Worker] Sort intermediate KV                
TRAC[0000] [Worker] Start Reducing                      
TRAC[0000] [Worker] End Reducing                        
INFO[0000] [Worker] End Reduce                          
TRAC[0001] [Master] End Reduce task                     
TRAC[0001] [Master] End Workers Start                   
INFO[0001] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
TRAC[0001] [Master] End Workers done                    
go run -race cmd/master.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 4 &
sleep 1
INFO[0000] [Master] Master gRPC server start            
TRAC[0000] [Master] Wait for enough workers             
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 1 &
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 2 &
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 3 &
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 4
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0002] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0002] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0002] [Master] Worker register success             
TRAC[0002] [Master] Enough workers!                     
TRAC[0002] [Master] Start distribute workload           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] [Master] End distribute workload             
TRAC[0002] [Master] Start Map task                      
INFO[0002] [Master] Finding available workers to execute 4/4 
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] [Worker] Start Map                           
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Start Mapping                       
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] Start Mapping                       
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
panic: runtime error: index out of range [1] with length 1

goroutine 27 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x0?, 0x0?}, {0x0, 0x0}, {0xc0001808a0})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc0001948c0, {0x0, 0x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 35
        /home/user/Desktop/MapReduce/worker/worker.gopanic: runtime error: index out of range [1] with length 1

goroutine 10 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x0?, 0x0?}, {0x0, 0x0}, {0xc00028e480})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc000282690, {0x0, :600x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
 +created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 6
0x185   /home/user/Desktop/MapReduce/worker/worker.go:60 +0x185

INFO[0002] [Master] 9503430e-b944-4d68-8531-1fd401a48b07 update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
panic: runtime error: index out of range [1] with length 1

goroutine 28 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x0?, 0x0?}, {0x0, 0x0}, {0xc000094d20})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc000096eb0, {0x0, 0x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 25
        /home/user/Desktop/MapReduce/worker/worker.go:60 +0x185
exit status 2
exit status 2
WARN[0002] [Master]: error reading from server: EOF     
WARN[0002] [Master]: error reading from server: EOF     
INFO[0002] [Master] Finding available workers to execute 1/4 
INFO[0002] [Master] Re-execute Map Task from 9c0bc0cd-5305-49f7-87ec-c60dd346ab94 To 9503430e-b944-4d68-8531-1fd401a48b07 
INFO[0002] [Master] Finding available workers to execute 1/4 
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
make: *** [Makefile:44: test_multi_node] Error 1
panic: runtime error: index out of range [1] with length 1                                                                                                                                

goroutine 70 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x1?, 0x1?}
, {0x0, 0x0}, {0xc000180420})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc000194aa0, {0x0, 0x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 66
        /home/user/Desktop/MapReduce/worker/worker.go:60 +0x185
exit status 2
WARN[0002] [Master]: error reading from server: EOF     
exit status 2
==================
WARNING: DATA RACE
Write at 0x00c000172170 by goroutine 33:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).SetState()
      /home/user/Desktop/MapReduce/master/workerInfo.go:36 +0x4d
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func1()
      /home/user/Desktop/MapReduce/master/master.go:220 +0x584
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func4()
      /home/user/Desktop/MapReduce/master/master.go:228 +0xa6

Previous read at 0x00c000172170 by goroutine 7:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).Broken()
      /home/user/Desktop/MapReduce/master/workerInfo.go:41 +0x49e
  github.com/BWbwchen/MapReduce/master.(*Master).availableWorkers()
      /home/user/Desktop/MapReduce/master/master.go:144 +0x450
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:233 +0x60e
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 33 (running) created at:
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:215 +0x512
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 7 (running) created at:
  github.com/BWbwchen/MapReduce.startMaster()
      /home/user/Desktop/MapReduce/util.go:101 +0x2a4
  github.com/BWbwchen/MapReduce.StartMaster()
      /home/user/Desktop/MapReduce/masterNode.go:4 +0x27
  main.main()
      /home/user/Desktop/MapReduce/cmd/master.go:8 +0x18
==================
WARN[0002] [Master]: error reading from server: EOF     
==================
WARNING: DATA RACE
Write at 0x00c0001720e0 by goroutine 58:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).SetState()
      /home/user/Desktop/MapReduce/master/workerInfo.go:36 +0x4d
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func2()
      /home/user/Desktop/MapReduce/master/master.go:248 +0x5ac
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func5()
      /home/user/Desktop/MapReduce/master/master.go:255 +0x91

Previous read at 0x00c0001720e0 by goroutine 7:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).Broken()
      /home/user/Desktop/MapReduce/master/workerInfo.go:41 +0x49e
  github.com/BWbwchen/MapReduce/master.(*Master).availableWorkers()
      /home/user/Desktop/MapReduce/master/master.go:144 +0x450
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:233 +0x60e
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 58 (running) created at:
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:242 +0x5b9
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 7 (running) created at:
  github.com/BWbwchen/MapReduce.startMaster()
      /home/user/Desktop/MapReduce/util.go:101 +0x2a4
  github.com/BWbwchen/MapReduce.StartMaster()
      /home/user/Desktop/MapReduce/masterNode.go:4 +0x27
  main.main()
      /home/user/Desktop/MapReduce/cmd/master.go:8 +0x18
==================
PANI[0002] No enough worker!0 1                         
panic: (*logrus.Entry) 0xc0000ac9a0

goroutine 6 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0000ac930, 0x0, {0xc00009e120, 0x14})
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/entry.go:259 +0x97f
github.com/sirupsen/logrus.(*Entry).Log(0xc0000ac930, 0x0, {0xc00019bb80, 0x3, 0x3})
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/entry.go:293 +0x8c
github.com/sirupsen/logrus.(*Logger).Log(0x1305240, 0x0, {0xc00019bb80, 0x3, 0x3})
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/logger.go:198 +0x7a
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/logger.go:247
github.com/sirupsen/logrus.Panic(...)
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/exported.go:129
github.com/BWbwchen/MapReduce/master.(*Master).availableWorkers(0xc000024870, 0x1)
        /home/user/Desktop/MapReduce/master/master.go:149 +0x269
github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask(0xc000024870)
        /home/user/Desktop/MapReduce/master/master.go:233 +0x60f
github.com/BWbwchen/MapReduce/master.StartMaster({0xc000078c40, 0x4, 0x4}, 0x0?, 0x0?, {0xc00001569a, 0x6})
        /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x331
github.com/BWbwchen/MapReduce.startMaster.func1()
        /home/user/Desktop/MapReduce/util.go:102 +0x8a
created by github.com/BWbwchen/MapReduce.startMaster in goroutine 1
        /home/user/Desktop/MapReduce/util.go:101 +0x2a5
exit status 2

BWbwchen · 2023-11-07T12:29:04Z

Ummm.... That's weird. On my machine, it didn't fail.
Let's move this problem into an issue and try to solve it :)

kiarash8112 added 8 commits October 29, 2023 14:50

add orDone pattern to improve readability

f425cdd

create rpcClient interface for master to reduce duplication and use t…

62ccbe4

…o mock rpc call in tests

WIP: adding tests for master

7ec134d

[FIXBUG]fix orDone func bug

35e5a57

[Featue] improve master rpcClient error handling

6393c89

[tests] complete master tests

54df8b3

[Refactor] create RpcClient for worker to reduce duplication and impr…

4045185

…oved rpc calls error handling

[Tests] add MasterClient mock

52c436a

kiarash8112 closed this Nov 1, 2023

kiarash8112 reopened this Nov 1, 2023

kiarash8112 force-pushed the master branch from 4bede68 to 52c436a Compare November 1, 2023 18:47

kiarash8112 marked this pull request as draft November 1, 2023 19:01

kiarash8112 marked this pull request as ready for review November 1, 2023 19:02

add image to README.md

28a119d

[Fix] fix worker GetIMDData calling other ip

58b9aca

kiarash8112 and others added 2 commits November 6, 2023 19:21

Merge branch 'BWbwchen:master' into master

2b5d4cc

[refactor] close master client at end

8de6543

BWbwchen reviewed Nov 7, 2023

View reviewed changes

master/master.go Show resolved Hide resolved

master/master.go Show resolved Hide resolved

master/master.go Show resolved Hide resolved

BWbwchen merged commit 1e7d2b5 into BWbwchen:master Nov 7, 2023

BWbwchen mentioned this pull request Nov 7, 2023

data racing when run make test_multi_node #2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

worked on tests,code readability,... #1

worked on tests,code readability,... #1

kiarash8112 commented Nov 1, 2023

BWbwchen commented Nov 4, 2023

kiarash8112 commented Nov 4, 2023

BWbwchen commented Nov 7, 2023

BWbwchen commented Nov 7, 2023 •

edited

Loading

kiarash8112 commented Nov 7, 2023

BWbwchen commented Nov 7, 2023

worked on tests,code readability,... #1

worked on tests,code readability,... #1

Conversation

kiarash8112 commented Nov 1, 2023

BWbwchen commented Nov 4, 2023

kiarash8112 commented Nov 4, 2023

BWbwchen commented Nov 7, 2023

BWbwchen commented Nov 7, 2023 • edited Loading

kiarash8112 commented Nov 7, 2023

BWbwchen commented Nov 7, 2023

BWbwchen commented Nov 7, 2023 •

edited

Loading