#

Announcing Netscratch: Our New High-Performance Scratch Filesystem 

This was sent as an email to all cluster users on Nov 1st, 2024. It has been edited here to provide additional information about moving data.

We’re excited to introduce netscratch, a high-performance scratch filesystem powered by Vast Data. After thorough evaluation, Vast Data’s advanced flash technology stood out for its performance and scalability, making it our solution of choice.

 

Data Transfer from Holyscratch01 to Netscratch

Please copy/move any needed data to /n/netscratch/[your lab name] (example: /n/netscratch/jharvard_lab ) as soon as possible and begin using netscratch for your jobs. Not only do we want to encourage rapid adoption of netscratch, but we are seeing signs of degradation on holyscratch01 that increases the sense of urgency.

See our Data Transfer documentation for info on moving data. All labs who have directories on holyscratch01 should have directories on netscratch. If your lab does not, please contact FASRC.

Holy scratch01 will be made read-only on December 2nd, 2024.

 

Key Benefits

The netscratch system leverages NVMe and SSD technologies, offering:

  • Read Throughput: Up to 250 GB/s
  • Write Throughput: Up to 55 GB/s
  • IOPS: 1,220,000 reads and 500,000 writes This system is optimized for the Cannon cluster, allowing efficient job execution that maximizes both CPU and GPU resources.

Directory Structure for Enhanced Usability ↵

netscratch features a streamlined directory structure that will enhance usability

  • Each lab will have a top-level directory.
  • Subdirectories for “Everyone” and “Lab” will be included for common and lab-specific data. The "Users" directory is deprecated, please see this doc for info.

Technical Details ↵

  • Filesystem Path/n/netscratch (accessible on Cannon and FASSE).
  • Quotas and Data Retention: Remain unchanged.
  • $SCRATCH Variable Update: Will point to netscratch starting December 2nd 2024.
  • External Repos Support: Users of /n/holyscratch01/external_repos should migrate their data as external repos will no longer be supported.

Launch Timeline

  • Availability: Netscratch goes live on Monday, November 4th 2024 after maintenance.
  • Transition Period:
    • Write Access on the current scratch filesystem will continue until December 2nd 2024.
    • After that, it will switch to read-only mode until February 1st, 2025 (data retention policies remain active). We encourage users to start utilizing netscratch for upcoming jobs and to begin migrating data from the old scratch filesystem. For more on data transfers, please refer to the the data transfer documentation.

 

Thank you,
FAS Research Computing
https://docs.rc.fas.harvard.edu/
https://www.rc.fas.harvard.edu/upcoming-training/