EECS Hub: No Space left on Device [BLOCKER]

[balajialg]

Bug description

@Quizenger raised this issue in the Slack channel!

Environment & setup

• Hub: EECS Hub
• Language: Python

How to reproduce

Try accessing a notebook in the EECS hub

[balajialg]

@felder 5xx Error logs

EECS Hub Memory Usage:

@felder Could this be similar to this error?

[balajialg]

@felder @ryanlovett @yuvipanda

Would increasing the storage here to 8Gi help?

[ryanlovett]

@balajialg That is the size of storage for the hub's database volume, not user storage. The NFS disk is full:

rylo@nfsserver-01:~$ df -h /export/homedirs-other-2020-07-29/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        5.0T  5.0T  180K 100% /export/homedirs-other-2020-07-29

I'll have to lookup how the storage has been setup and if it is possible to grow it. I'll also see if some unnecessary things can be removed.

[balajialg]

@ryanlovett Thanks for jumping in! Your support would be crucial, considering an EECS 16A lab is happening right now, and it is a blocker for every student.

[ryanlovett]

@felder replied on slack that he is handling it

[balajialg]

@felder replied on slack that he is handling it

Thanks, @ryanlovett!

[felder]

The disk on the nfs server used by this particular hub was full. I increased it from 5TB to 7.5TB.

[balajialg]

@felder Data 8 also had the same issue today morning! Is the NFS file storage issues common across all the hubs?

[yuvipanda]

data8 and eecs hubs shared NFS disks - see #2815

[balajialg]

@yuvipanda that's helpful!