Save Your Progress: Checkpointing On FASRC Clusters (virtual)

Description: When a cluster job runs out of time or stops unexpectedly, you can lose hours or days of computation. Learn to save your calculation’s progress and resume from the last saved point instead of starting over. Checkpointing allows a running program to periodically save its computational state so that, if interrupted, it can resume from the last saved point rather than starting over. Similar to saving a document while working, checkpointing protects your computational progress and helps avoid wasting valuable computing time and resources.

In this virtual training session, participants will learn practical techniques for implementing checkpointing and restarting computational jobs on FASRC clusters. We will explore different checkpointing approaches, including application-level checkpointing, transparent checkpointing without modifying source code, and checkpointing capabilities built into scientific applications and machine-learning frameworks.

Hands-on Examples

• Resume a Python or C calculation after an interruption.
• Continue a GROMACS molecular simulation across multiple jobs.
• Resume PyTorch model training from a saved checkpoint.
• Save and resume some programs you didn’t write, including parallel (MPI) applications, without adding checkpoint code.

You’ll also learn to save progress before a job’s time limit, automate restarts, and use FASRC’s requeue partitions.

No prior experience with checkpointing is required.

Who should attend?

Researchers running long calculations, simulations, or model training on FASRC clusters

Date: Thursday, Nov 5, 2026
Time: 1:00 – 2:30 PM
Level: Intermediate
Cost: Free
Presenter: Plamen Krastev
Facilitators: Nathan Weeks, Brad Spitzbart, & Manasvita Joshi

Prerequisites: 

• FASRC cluster account. If you do not have an account, see Request a FAS Research Computing Account well in advance 
• Previous experience submitting batch jobs on a FASRC cluster
• Basic familiarity with Slurm job scripts
• Basic programming experience in Python, C/C++, or Fortran. Participants interested in the AI/ML examples should have basic familiarity with at least one supported machine-learning framework

Registration: Register on Zoom for this session: REGISTER
After registering you will receive a link to the Zoom meeting for that date.