Bugs with workers(), rmprocs()

[alanedelman]

workers() gives 1 when nprocs() is 1 and non-ones when nprocs() >1

Consistency dictates that workers() should return [] when nprocs() is 1

... though perhaps it's better to include 1 all the time?

Right now rmprocs([1]) and rmprocs({]) crashes julia when there is 1 processor

[amitmurthy]

This is intentional. pmap and @parallel only execute on the worker processes.

When processes are added using addprocs() or -p n on the command line, process id 1, is deemed to be the driver process and the others workers.

When nporcs()=1, all "work" is carried out on pid 1 itself - it is both the driver and a worker.

procs() lists all processes and workers() lists only the workers.

rmprocs([1]) is equivalent to removing the self process, hence Julia exits.

rmprocs({}) is working fine on my machine - i.e., it does nothing. Which platform are you on?

[JeffBezanson]

When there is only one processor, it might as well be considered a worker. Otherwise it wouldn't be possible to write code that uses workers() yet still works with one processor. You would instead have to write nprocs()==1 ? [1] : workers() over and over again.

rmprocs(1) indeed causes processor 1 to exit. We could make 1 a special case and throw an error, but how is that consistent?

[alanedelman]

the documentation makes it seem that processor number 1 is never a worker

so workers() returning a [1] is inconsistent

I may say a bit more about all this in a moment.

[amitmurthy]

The immediate next line is When no additional processes are started, the driving process is also deemed to be a worker.

We could however improve the documentation and make it much more explicit.

[alanedelman]

Ok touche

Once could write, when no additional processes are started, the driving process is deemed the worker.
With additional processes, the presumption is that the driving process is already busy driving.

I'm betting burned a bit because I'm changing numbers of processors and doing timings, but I think
I can cope with this.

[alanedelman]

I guess if one takes this point of view and you have a code that you want to check on P processors
and you know you have N=nprocs() going in, you have to add

 addprocs(  P - N + (N==1) )

which seems strange at first, but one can get used to it. Maybe this could be remedied with
a command

setworkers(k)

which at least has the property that

setworkers(k); nprocs() returns k+1 if k>1 and 1 if k==1

[JeffBezanson]

The docs should certainly not say that workers are defined as those with id != 1.

I do think it would be better to do away with the procs/workers distinction. For example, we don't need to support processor 1 being your laptop with the rest of the processors being cluster nodes. All the processors can be cluster nodes, with you connecting to them via notebook, ssh, or a separate julia process "attached" to one of the remote processes. However, to support this we need a preemptively-threaded work queue so tasks can be scheduled while all CPUs are busy. I think it would be worthwhile to hack this in, independent of whatever other support for threading we may or may not have.

[alanedelman]

i was afraid that you were telling me to always use 2:(N+1) everytime i have n processors, so I'm glad you
are saying this

[JeffBezanson]

rmprocs(1) killing your interactive session might be an example of being "too consistent" :)

[alanedelman]

I would say that rmprocs(v) when v contains a 1 should return the warning
"process 1 not removed"