Safely handle parallel/concurrent invocations

[everythingfunctional]

Description

At present, if you invoke fpm on the same project multiple times in parallel, the compiler will trip over itself as it tries to compile the same files at the same time, and can leave the project in an inconsistent state such that subsequent invocations do not try to rebuild when they should. If possible, fpm should avoid trying to build a project multiple times in parallel.

Possible Solution

A quick and dirty solution would be to create a "lock file" on fpm startup and delete it on program shutdown. If the "lock file" is already present, just wait until it's removed. It's not flawless, but could be sufficient.

Additional Information

No response

[milancurcic]

I can reproduce. Lock file sounds reasonable to me. I'm only concerned about situations where fpm hits an unhandled exception and leaves the lock file behind. Is there a way to handle that?

[perazz]

Interesting enhancement @everythingfunctional.

I think the proposed solution looks very reasonable. Could that be done on a per-dependency basis? (think something similar to cache.toml). We now have the option for a local registry, so, that has to be dealt with as well.

I think we also want to understand what causes this:

and can leave the project in an inconsistent state such that subsequent invocations do not try to rebuild

because we could figure out a fix and force a full rebuild if something went corrupt.

Edit: also see rust-lang/cargo#354

[everythingfunctional]

I'm only concerned about situations where fpm hits an unhandled exception and leaves the lock file behind.

Yep, that's the main danger. If you wanted to get fancy the lock file could contain the PID of the process that created, and then check if that process is still executing, if not delete it and create a new one. Basically, there are solutions.

we also want to understand what causes this

The compiler stepping on its own toes, but updating the timestamp of files anyways. I.e. it looks like the object/mod file got produced after the source code was updated, but it didn't get produced correctly.

[everythingfunctional]

FYI, my use case is trying to get ctest to run my tests in parallel, but there's no way to ensure they're actually compiled without invoking fpm test.

[emmabastas]

Hey, I started working on this issue as a way to get familiar with Fortran and fpm because I'm interested in working on fpm as part of GSoC2025.

My work thus far is in this commit emmabastas@301e17b
I've implemented locking of package directories (the lock file way) for fpm run (i.e. I acquire the lock in the cmd_run subroutine).
On my machine it makes it so that concurrent fpm run's wait for one another.

I just wanted to check-in about this issue, is it still relevant, and am I heading in the right direction with my implementation? I'm not asking for a proper code-review or anything like that, I just don't want to spend time on this only to find out that the issue isn't relevant any more, or that I have the complete wrong idea of how to fix it, etc :-)

[everythingfunctional]

I'm no longer working on the project that needed this, but I imagine it will still be useful. It looks like you've made a promising start down the path I suggested.