by Paul Edmon October 20, 2026

Scratch Filesystems

High performance computing (HPC) shops, like FASRC, provide multiple options for storage to serve different needs. One of the unique offerings for HPC is the scratch filesystem. Scratch filesystems get their name from paper scratch pads which are used for throw away notes or math that you need for the moment but do not need forever. Scratch filesystems are set up for a similar purpose, to store temporary data you need for now but do not intend to keep long term.

Due to the nature of workflows on the cluster, this scratch space is frequently in high demand. Users need a temporary space to write data from jobs; data that they then need to postprocess to get to the data they actually want to keep. Users need a space to keep data they intend to feed into their models; but once that is done do not need to keep that data around. Users need to store checkpoint files to ensure they do not lose their progress; but once the run is done they do not require that data anymore.

These requirements mean that scratch filesystems are built for speed and parallel access. Due to the temporary nature of the data, snapshots and backups are not included because users should be copying data they need long term to other more resilient storage. Finally to keep the size of the data on the filesystem under control, retention policies are put in place to clean up old data and prevent users from squatting.

Netscratch

For FASRC the filesystem that serves this purpose is netscratch. This filesystem is built with size, speed, and parallelism in mind. It was sized and architected based on our analysis of previous cluster usage, budgeting in estimates for growth in demand. In addition we have applied our traditional 90 day retention policy.

After years of stability in scratch usage, starting in April/May of 2026 we noticed a concerning upward trend in filesystem size on netscratch:

Graph showing growth of netscratch

This growth in usage is not attributed to any specific lab. Labs have a 50 TiB quota on netscratch which prevents any given group from taking over the filesystem. When we looked at the distribution of usage it was pretty much even over all the labs. This is real increased demand from the community not attributable to any specific cause.

Scratch Retention

This growth is also unsustainable. Getting to 100% utilization would create a host of problems for all users on the cluster as well as possibly damaging the filesystem. Expanding the filesystem is a possibility but would incur expense and take time. This leaves the option of changing the retention period.

Compared to other HPC groups, the 90 day retention period is exceedingly generous. The general community standard is anywhere from 30 days to 60 days. This means that lowering our retention period would not put us out of step with other sites.

Making a graph of the data over time we can see how much of the filesystem would would get back depending on what we set retention to:

Graph of how much retention would capture depending on time.

Interestingly the distribution of usage over time is constant and fits almost perfectly into 90 days. So for every 30 days we decrease the retention period, we would collect roughly a third of the filesystem. Thus for a 60 day we would retain 34%, while for 30 days we would get 63% back.

With all this in mind, FASRC is changing its retention period to 60 days starting at the December 7th, 2026 maintenance. This will allow us to reclaim roughly 1 PiB of storage on netscratch, bringing it down to a more acceptable usage level of ~60%. Users should plan accordingly. As a final note, circumventing our retention policy to retain data on netscratch longer than our retention window is a bannable offense (i.e. your FASRC account will be disabled and you will lose access to FASRC resources). Do not do it.

Our recommendations for optimal storage workflow can be found on our Data Storage website, along with options for purchased storage. If you need assistance with adjusting your workflow swing by our weekly Office Hours or reach out to our Data Manager. If you have further questions about this change please submit a ticket.