"bad" plt and checkpoint files

[n8xm]

While running Nyx using AMReX 25.09, I'm having some issues with "bad" plot and checkpoint files when running on Perlmutter. E.g. I'm crashing and seeing some of the following output directories:

chk00300.bad00
chk00300.bad01
chk00300.bad02

And also these:

plt00286.bad00
plt00286.bad01
plt00286.bad02
plt00286.bad03

There is a backtrace file but nothing is in it. But I did find the amrex::StreamRetry::TryFileOutput function, which is where ".bad" gets appended to file names.

Is there a problem with my code or a problem with the filesystem on Perlmutter? (I am not aware of any issues specific to their scratch filesystem at the moment.) I've already tried running twice and I've run into the same problem, but in different places.

Is there a way to determine where the original "stream error" comes from? Perhaps that could be helpful for debugging?

[atmyers]

I think this is likely a problem with the filesystem - is there anything in stdout or stderr?

[n8xm]

I do not see any useful output in stdout or stderr, but it turns out that one one of the runs that crashed did produce a non-empty backtrace, which I have attached.

Backtrace.0.txt

From the backtrace, if I look at Src/Particle/AMReX_WriteBinaryParticleData.H:678, I see:

    if (ParallelDescriptor::IOProcessor())                                                                                                                                                                                                                                           
    {                                                                                                                                                                                                                                                                                
        HdrFile.flush();                                                                                                                                                                                                                                                             
        HdrFile.close();                                                                                                                                                                                                                                                             
        if ( ! HdrFile.good())                                                                                                                                                                                                                                                       
        {                                                                                                                                                                                                                                                                            
            amrex::Abort("amrex::WriteBinaryParticleDataSync(): problem writing HdrFile");                                                                                                                                                                                           
        }                                                                                                                                                                                                                                                                            
    } 

I do not see the error message above in stdout or stderr, but I'm assuming that it's failing to write a binary particle checkpoint file.

[n8xm]

I believe the root cause is a lack of storage space - i.e. me maxing out my storage quota. In which case it is not an AMReX problem.