by Paul Edmon July 23, 2026

Abstract

With the advent of Nvidia deprioritizing 64-bit floating point performance, another metric is needed to fairly weight GPU performance for the sake of fairshare accounting. This blog post describes the rationale behind the new method which is to take the theoretical peak performance in whatever category is the largest and then normalize it to FP64. This new method allows FASRC to account for CPU’s and GPU’s which are not optimized for 64-bit math.

Background

Since the creation of the x86-64 CPU architecture in 1999, 64-bit registers have been the universal standard for chip design. 64-bit registers have a number of benefits that have proven invaluable for scaling out compute:

1. Addressing of more than 4GB of memory: This is due to 32-bit integers (INT32) capping out at an unsigned value of 4,294,967,295 (232-1)
2. Double Precision Floating Point Numbers (FP64 AKA REAL*8 for you Fortran Fans): This allows for 16 digits of precision which is essential for keeping round off error at bay. This is critical for doing math on very large numbers of small values, or values with large dynamic ranges.
3. Lower precision work got a free boost: You can pack two 32-bit values in a 64-bit register, thus getting a free speed up by a factor of two for 32-bit math. This also applies to even lower precision math.

The dominance of 64-bit architecture has led to benchmarks designed specifically for it (e.g. High Performance Linpack (HPL)) and made it the gold standard for comparing machine performance.

The growth of AI workflows has caused reconsideration of accepted hardware design. A plethora of hardware architectures has grown out the quest for performance and power efficiency. One of the more enduring changes has been the reconsideration of FP64 and INT64. Experimentation has proven that the standard AI training methods work at lower precision with minimal loss. This is primarily due to the probabilistic nature of AI, which has higher tolerances for round off error and is more about relative weights than arithmetic accuracy. Many AI workflows are leaning more toward the BF16 standard, which stands for Brain Floating Point 16-bit which has a shorter mantissa than FP16 allowing for a larger exponent.

Responding to this shift, Nvidia is beginning to deprioritize and even remove direct 64-bit support. By doing this they can focus on their chip design and performance on where it really matters for their primary customers, AI hyperscalers. The downside is that the degradation of 64-bit performance will hurt non-AI work. Nvidia is willing to make this trade though as their main users are AI labs and hyperscalers who are constantly on the look out for an edge in computation.

Fairshare Weights

For the first three years of running Slurm (2013-2016), FASRC weighted all labs and all usage equally regardless of what hardware the lab used, or how much hardware that lab contributed to the cluster. In 2016 it was decided to move to a system of weighting fairshare based on hardware type using TRES (Trackable RESources) Billing Weights. This rewarded groups that contributed hardware to the cluster, as well as those who were able to optimize their code for specific hardware types. It also made it possible to differentiate between hardware, which is important because FASRC is a highly heterogeneous cluster.

To do the weighting properly, a universal method of comparing different chipsets was needed. The only standard that was nonarbitrary and noncode dependent was the theoretical peak FLOPS (FLointing point Operations Per Second). Theoretical peak is derivable from first principles for CPUs for most normal chipsets, and for GPU’s it is widely published. It had the advantage of being able to fairly weight CPU’s versus GPU’s, thus allowing GPU users to move compute cycles to CPUs and vice versa. This is good because in Slurm there is only one global fairshare pool, so having a weighting factor that is universal and nonarbitrary is important. At the time 64-bit was the universal standard, that was the number used for the calculations.

Changes in GPU design noted above now necessitate a shift in our methodology. 32-bit performance is not simply a factor of two increment from 64-bit. Chipsets are being designed for 32-bit specifically or even lower precision. 64-bit support is even being removed, meaning that 64-bit math has to occur on 32-bit registers which has enormous performance penalties. In addition, the published performance for GPUs are now being pumped up artificially due to leveraging sparsity (i.e. usng the fact that sparse matrices are full of zeros and thus need no math, boosting the throughput artificially, typically by a factor of two).

To illustrate this lets look at a table of performance metrics for different GPU generations correcting for sparsity along with a CPU standard (FASRC has no B200/300 as of publication, this is rather for illustration):

Performance (GFLOPS)/Architecture	FP64	FP64 Tensor Core	FP32	FP32 Tensor Core	FP16/BF16 Tensor Core	FP8 Tensor Core	FP4 Tensor Core
Intel Xeon Platinum 8268 (Cascade Lake) (per core)	93.25
Nvidia A100	9,700	19,500	19,500	156,000	312,000
Nvidia H200	34,000	67,000	67,000	494,500 (assuming sparsity ½)	989,000 (assuming sparsity ½)	1,978,000 (assuming sparsity ½)
Nvidia B200	37,000	37,000	75,000	1,250,000 (assuming sparsity ½)	2,500,000 (assuming sparsity ½)	5,000,000 (assuming sparsity ½)	10,000,000 (assuming sparsity ½)
Nvidia B300	1,200	1,200	75,000	1,250,000 (assuming sparsity ½)	2,500,000 (assuming sparsity ½)	5,000,000 (assuming sparsity ½)	13,500,000

 

First, you can see the degradation of the FP64 performance in the generation following H200. B200 is only marginally better than H200, and B300 falls off a cliff. Second, you can see that the tensor core numbers are substantially higher than the regular numbers. The tensor cores do the heavy lifting for matrix math and are the heart and soul of the GPU. Third, you will notice that for the most part the FP32 Tensor Core numbers and lower precision are basically powers of 2 of each other. When this happens you know that the lower precision math is leveraging the high precision silicon and the speed up is due to the fact you can do twice the math in the same size register each time you decrease the precision.

To illustrate this let me reproduce the same chart but now normalized based on the precision. Namely FP64: 1, FP32: ½, FP16: ¼, FP8: ⅛, FP4: 1/16:

Performance (GFLOPS)/Architecture	FP64	FP64 Tensor Core	FP32	FP32 Tensor Core	FP16/BF16 Tensor Core	FP8 Tensor Core	FP4 Tensor Core
Intel Xeon Platinum 8268 (Cascade Lake) (per core)	93.25
Nvidia A100	9,700	19,500	9,750	78,000	78,000
Nvidia H200	34,000	67,000	33,500	247,250 (assuming sparsity ½)	247,250 (assuming sparsity ½)	247,250 (assuming sparsity ½)
Nvidia B200	37,000	37,000	37,500	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)
Nvidia B300	1,200	1,200	37,500	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	843,750

With the normalized numbers it is now easy to see what is happening. For the A100 and H200, there are specific FP64 Tensor Cores being used, the FP64 and FP32 use the same registers, and there are specific FP32 Tensor Cores that are being used for all the lower precision work. So really there are just three parts of the GPU, the FP64 block, the FP64 Tensor Core Block, and the FP32 Tensor Core Block. B200’s are the same but the FP64 Tensor Block is now down played. For B300, they sacrificed the FP64 performance and boosted the FP4 and increased the onboard memory (see here for a good comparison), but everything else is similar to B200.

Given all this, what is the most appropriate method for fairly weighting the GPU? Slurm does not have the ability to distinguish in detail the architecture of the GPU. We also need to account for the fact that in the future CPU’s may also have capabilities that enhance lower precision math.

It is beneficial at this point to take a step back and look at the goal of TRES Billing Weights and Fairshare is. The goals are:

1. To ensure all users and labs have fair usage of the cluster regardless of what hardware that they choose to use.
2. That those who purchase hardware for the cluster are given commensurate allocation on the cluster in a way that rewards them for contributing to the cluster.
3. That users are incentivized to select the appropriate hardware and partition for their workflows.
4. That users are incentivized to optimize their code and be efficient in their job design.
5. For the policy to be simple, straightforward, and code/workflow agnostic.

For all of these reasons FASRC has adopted the following approach:

• Normalize all GPU FLOPS to FP64: This allows us to compare apples to apples as FP8 numbers will always look way higher than FP64, but if you divide by 8, the numbers become more comparable. Think of it as looking at total bit throughput instead of FLOPS which can be misleading as lower precision uses less bits and thus is “easier” math.

• Look for the category where the Maximum FLOPs is: One could use fancy math using geometric means, or sum all the FLOPs together. However really your performance limit for a particular code is the part of the GPU that you are bottlenecked on. The different components of a GPU are not additive. As such the maximum theoretical peak of a GPU would be whatever the maximum capability of the GPU is. If it is FP4 (as in the case of the B300) that is where the GPU is optimized to work and an optimal workflow will account for that, remove barriers, and fully leverage the GPU.
• Divide by the baseline CPU: This allows us to normalize the distribution against a standard and have some semblance of comparison between the different processors and GPUs.

With these rules we can reproduce the previous table but now highlight the values we use for the fairshare and give the normalized weights:

Performance (GFLOPS)/Architecture	FP64	FP64 Tensor Core	FP32	FP32 Tensor Core	FP16/BF16 Tensor Core	FP8 Tensor Core	FP4 Tensor Core	TRES Weight
Intel Xeon Platinum 8268 (Cascade Lake) (per core)	93.25							1.0
Nvidia A100	9,700	19,500	9,750	78,000	78,000			836.5
Nvidia H200	34,000	67,000	33,500	247,250 (assuming sparsity ½)	247,250 (assuming sparsity ½)	247,250 (assuming sparsity ½)		2651.5
Nvidia B200	37,000	37,000	37,500	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	6702.4
Nvidia B300	1,200	1,200	37,500	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	625,000 (assuming sparsity ½)	843,750	9048.3

Conclusion

One of the basic tenets of TRES in the above system is that if you move your workflow to a different architecture you should see a speed up equivalent to the increase in cost. For example, say you have a job that runs on the A100 and takes 10 hours. If your code is well optimized to fully utilize the GPU you should see that your job will run 3.16 times faster on a H200 and thus complete in 3.16 hours. The total TRES cost (TRES Billing Weight times time) will be the same.

However that rule will only be true if you are fully utilizing the GPU for its intended design. For instance if you had a FP64 calculation and tried to run it on the B200 series, you would hardly see any speed up between the H200 and B200. That means that you should really use the H200 and leave B200 use for its intended use case. Conversely you may realize that you can actually beat the curve by running FP64 work on the H200 because it is actually 3.5 times faster than the A100 which is better than the ratio between the fairshare factors which are based on the FP32 Tensor Core performance. That, of course, assumes that your code is fully optimized and can fully leverage the GPU in that category.

Note that this also applies to CPUs. There are cases where a code will actually outperform its comparative TRES weight speed up because the code is better optimized for a specific architecture (for instance it uses the cache better or is memory bandwidth dependent). Users need to be cognizant of these factors and benchmark their code rigorously on different architectures. You may be surprised at which chipset has the best TRES efficiency. It may be a previous generation that better matches your code design.

At the end of the day fairshare is really just a method of prioritizing work. We could in theory pick any arbitrary method for assessing how much fairshare each lab gets and what partitions charge what cost. For example, the requeue partitions do not conform to the model above but instead charge a flat rate regardless of architecture. The goal is to be fair and understandable, so you can plan accordingly and select the appropriate hardware to conduct your research.