Over the next several months, Research Computing will be making a number of improvements to the Odyssey compute environment. This third generation of high performance computing will include increases in raw compute power, a new CentOS 7-based application stack, and the introduction of the Singularity container system. We are targeting December 2017 for the initial roll out of new hardware, and an early 2018 transition to O3 and CentOS 7. All of these changes are designed to provide a solid foundation for the amazing research being done on Odyssey.
These improvements will impact all Odyssey Slurm compute and login nodes. Virtual machines used for web applications, hybrid nodes, and storage will not be impacted.
Over the next 6 months, you can follow the progress outlined below in the Status section.
Prior to deployment of the new system, RC will host seminars to explain how to use the new application stack and Singularity containers and provide help with migrating existing tools.
- Initial communication, July 2017. Completed.
- Base CentOS 7 image, September 2017. Completed.
- New build environment available, October 2017. Completed.
- Core compiler and libraries built, February 2018. Completed.
- Module migration list published
- CentOS 7 re-imaging
- CentOS 7 modules default, February 2018. Completed.
- New Intel Xeon Broadwell cores added, November 2017. Completed.
- New queues added. Completed. (details)
- Singlularity containers enabled
15,000 new Intel Xeon Broadwell cores
The general compute will be refreshed with nearly 15,000 new Intel Xeon Broadwell cores. Our benchmarking shows that they are 3-4 times faster than the current AMD Abu Dhabi cores. Seriously. Very fast.
Find out more about the new queues and underlying architecture one our Running Jobs page.
CentOS 7 migration
The operating system will be updated from CentOS 6.8 to the latest release of CentOS 7.
There are several advantages to the more modern OS:
- Support to launch modern container at runtime (i.e. Singularity )
- Support for new technologies such as Xeon Phi, GPU's, 3DXpoint Memory, etc.
- Performance and power improvements
- Security enhancements
- Support for newer code bases
Much of the software on Odyssey is delivered via a LMOD module system that enables software built for CentOS 6.8. In many cases, this software can be migrated to CentOS 7 without any changes. However, some code and libraries, especially those that depend heavily on glibc, can be sensitive to architecture or system packages and will likely need to be recompiled. To support this transition, the modulepaths of the current system will be split into CentOS 6 and CentOS 7 branches with the default modules transitioning to CentOS 7 at a specified time.
"legacy" modules will no longer be available on Odyssey. These modules typically start with prefixes like "hpc/", "math/", or "centos6/". In many cases, these modules are more than 7 years old, based on CentOS 5, and, in the case of MPI-based code, do not work properly with our current environment. The new module system already has several thousand builds of more than 600 applications and so should be able to accommodate any computation required.
We will be consolidating some of the modules during this migration. Particularly for some compiler and MPI-dependent libraries, there are dozens of builds only a handful of which are really used (see NetCDF). We will be using our module load logging data to determine whether applications or libraries are actually used and they will be given priority.
Migration of some legacy applications can be very difficult and time-consuming. RC staff will be available at Office Hours or by appointment to assist where possible.
In addition we will be moving to a new release pattern for modules where updated versions of common modules (such as python, MPI, gcc) will be released on a regular schedule, and older modules will be deprecated. This will help us stay up with the latest software available and reduce the number of redundant builds. More details on this new release pattern will be forthcoming after the upgrade to CentOS 7 is complete.
Check this page for updates
While this project progresses, we will be sending out periodic communications. These communications will be available from this page, so check below for new postings.
Fall greetings from RC!
We are very pleased to unveil the new compute purchased by FAS as part of the Odyssey 3 project (https://www.rc.fas.harvard.edu/odyssey-3-the-next-generation/). This new compute hardware has been benchmarked at 0.5 petaflops, making it the fastest block of compute to date at Harvard. This hardware is now available on...Read More »
Research Computing Odyssey Users,
For those of you returning, welcome back. For those new to Harvard, we hope to provide you with solid services, resources and support. Below is information on some important updates as well as our new training schedule.
Next Maintenance: Due to the Labor Day holiday, the next...Read More »
Greetings Odyssey cluster users.
We'd like to let you know that we are planning several major updates to the Odyssey cluster that will be rolling out over the next few months. This third generation of high performance computing will include increases in raw compute power, a new CentOS 7-based application stack,...Read More »