Parallelise the Multicolony functions - action

[janaobsteter]

The package is painfully slow when it comes to MultiColony processing so I'm parallelising it with foreach package. The problem is when all loops try to update the same SP slot at the same time - hence, not a problem if updating information for different colonies/individuals - such as in reQueen or supersede ...

• createMultiColony is now parallelised:

  • the problem is that updateLastColonyId doesn't work within foreach, no idea why. So I preasign the colonyId and update the lastColonyId in SP at the end

• reQueen is paralelised

• next essential one is createCastePop (then used by addCastePop, replaceCastePop ...)

  • the problem is that lastId in SP is not udpated - we need to preasign the Ids OR change them subsequently - the latter would be easier and changing the Id of a Pop also change Ids in geno and haplo BUT doesn't change the ID in the pedigree

[janaobsteter]

Update:

• I've parallelised createCastePop function. The problem of foreach is not that all the loops are simultanesouly updating SP, but that none of them are - which is actually better for us. They do however each update the "local" version of the SP that is preserved within the loop only. I manually update the lastId, lastColonyId and pedigree slots in the SP AFTER creating individuals. The only thing we can not do is to create an exact number of workers - here, we had a while loop that kept creating individuals until we got the desired number (because of inbreeding) - this is not super realistic anyways and I think we can survive without this. The only problem I still have with it is to check how many of the created workers are homozygous within the function to populate the nHomBrood slot.

[janaobsteter]

Update:

• I now update caste information after the creation of individuals as well
• the track rec doesn't work because we need to manually update the hist information, but I am not sure what this consists of (@gregorgorjanc did write it out for one of the functions )
• the isCsdHomozygous still doesn't work - need to retest

[gregorgorjanc]

• the track rec doesn't work because we need to manually update the hist information, but I am not sure what this consists of (@gregorgorjanc did write it out for one of the functions )

@janaobsteter see gaynorr/AlphaSimR#34

[gregorgorjanc]

Linking to previous issue #513, where we have explored what parallel frameworks are/were available for R at the time.

[gregorgorjanc]

@janaobsteter I won't be looking at this soon, but I assume you have a local repo that you will open as PR as some point for review?

[janaobsteter]

@janaobsteter I won't be looking at this soon, but I assume you have a local repo that you will open as PR as some point for review?

Yeah, it's not meant to be looked at yet - there's quite a lot of work. I am just writing here so that I know what I still need to do.

[janaobsteter]

Update:

• have parallelised addCastePop function and buildUp function - however, the case where we request less individuals that already in the colony should be handled by addCastePop, not buildUp (otherwise, we cannot parallelise) - it calls getCastePop (but maybe it's not neccesary, if we require less, nothing happens)

[janaobsteter]

This is testing of the swarm function. I still have to paralellise the "cross" function, some are looking more promising than the others but the benefit kicks in will larger number of colonies

[gregorgorjanc]

How many cores do you have on your machine? You are getting close to 3x improvement!

[gregorgorjanc]

Well done!!!!!!

[janaobsteter]

np = non-parallelised, p = parallelised
This is run on the server with 16 cores (this is 10 replicated each function). Some get a bit more improvement than the rest, still need to investigate (supersede for example shouldn't be this demanding).
The major functions are all parallelised now except for cross (and tracking IBD). I am also guessing that we would benefit from parallelising some auxiliary functions such as nCaste or similar - this is really easy to parallelise since it doesn't change SP. The major hurdle was parallelising creating new individuals since it needs to be done in one go not to overwrite SP. But it's working now :)

[janaobsteter]

Separate scale for each to see better

[gregorgorjanc]

Do we need to ensure tests run in this parallel mode too, so we are sure all is correct?