(Living Doc) Current Test Bench Hardware & List of Methodologies

Objectives

This page will primarily serve as a landing page for test benches that we commonly use. This page is still being built and is under construction. The objective is to provide a single stop to answer questions about what hardware we use.

Our philosophy is to review each component in a vacuum, eliminating as many variables and bottlenecks imposed by other components as possible. This means that GPU testing for GPU performance will use a relatively high-powered CPU and RAM, thus hopefully limiting bottlenecks (to the extent possible) outside of the GPU; however, for that same GPU review, we would use a different test bench for acoustics. That's because the acoustics testing does not require unbound performance to eliminate variables, but instead requires a total (within reason) elimination of noise from non-GPU components instead. In this example, we would use the GPU test bench for thermals and gaming, but the GPU acoustic bench for noise levels and frequency response. Likewise, we use a different bench entirely for GPU power consumption measurements, as the host platform is irrelevant for these and only the power consumption matters (and it allows us to asynchronously test power consumption while our primary benches remain occupied for testing gaming or thermals, which greatly improves logistics to allow more testing in a shorter time period).

A lot of this has to do with a general internal approach of specializing within the team. Typically, the same person will handle GPU power consumption tests for every review, but a different person will handle GPU gaming performance and analysis, and a different person still might handle acoustics. Splitting benches like this not only isolates for variables, but also allows the team's experts in each test method to operate without blocking their teammates, as the benches are not singularly needed for every test in a suite.

Caveats & Limitations

We sometimes have to change-out single components (or revert to older benches) for certain tests. For example, if we are running a suite of CPU tests that requires us to go back to AM4 or Z270 or similar, we obviously won't be using the standardized Intel or AMD platform listed as the current primary, but would instead use one of the prior platforms. We will try to keep the older (retired) platforms listed further down the page for such events, but it will take us some time to comb through the site history and catalog them all in this format.

These test benches might not perfectly represent each individual content piece. Some pieces require a singular variable change to these benches as part of the piece. For example, if we're testing something like Intel APO, we will use a different boot drive (to avoid cross-contamination with the primary bench drive) and will use different driver packages and a potentially different BIOS configuration (to enable the dynamic tuning). Likewise, if we're testing PCIe bandwidth scaling and impact on an NVIDIA *90-class card, we will change the PCIe slot's generation for that test. The DUT (Device Under Test) or feature under test will deviate from the fixed bench as part of the test.

Sometimes we miss listing a detail since there are so many, but we will do our best to explain these caveats in each individual content piece.

Consistency is What Matters

Consistency is all that really matters. For comparative reviews, which is how we conduct our reviews, we just need an unchanging and fixed platform that allows us to reliably see scaling and differences between components. In the above example then, having APO toggling or PCIe generation switches being the new variable on an otherwise unchanging platform allows us the most comparative data against our "baseline" for that platform.

Likewise, this means that components being "old" or back by a couple generations becomes largely irrelevant. It only starts mattering if it's meaningfully limiting either the performance scaling of the product or the technology that can be used for the product. For instance, for purposes of generating heat for a cooler review, the CPU itself doesn't matter as much as its power consumption (and IHS shape, to be fair).

Methodology Pieces

This contains a list of our recently published or current test methodologies. Note that we have not yet gone back to upfit all of the older pieces for this website revision, so some links are to the videos.

This section is under construction. We still have to build this section out fully. It is missing a lot of information.

Recent Game Test Methodologies & Research

• Dragon's Dogma 2 (CPU & GPU Research)
• Starfield: (GPU Research | CPU Research | Graphics Optimization Guide) (Videos)
• Cities Skylines 2: GPU Research
• Baldur's Gate 3: GPU Research

Hardware Test Methodologies & Research

• CPU Cooler Test Methodology (Article) and accompanying video
• GPU Test Methodology (basics - the games have changed, some concepts are the same)

Test Benches

Mini-ITX Case Variable Test Bench (2023-TBD)

GPU Acoustics Test Bench (2023-TBD)

The below test bench details our platform used for GPU acoustics. This is deployed for collecting data such as the below. The GPU Acoustics Bench is not used for performance analysis (other than acoustic performance), and so the core components do not matter. For this bench, the most important aspect is controlling the noise: We use passive cooling, like from the Noctua NH-P1, to ensure no noise interference from other parts of the system. The power supply is used in a 0-RPM mode and validated during testing. The GPU does not perform work, but rather has its fan speed manually adjusted to meet the automatic fan speed we find during load testing (from our GPU FPS & thermal test platform, also on this page). We also adjust the speed in increments of 5% to capture a dBA profile across the RPM range.

Unless otherwise stated, video card coolers are tested at an industry-standard distance of 1m from the center fan (aligned on the horizontal and vertical center of the video card cooler). The video card is situated in a horizontal Thermaltake Core P3 platform with the glass removed, so it is basically just a flat "case."

GPU Test Bench (2022-2023)

The below test bench details our two GPU review platforms that are used for performance evaluation and thermals. The GPU test benches are overclocked, which helps significantly limit the bottlenecks potentially imposed by a CPU. The platforms also run tests at 1080p, 1440p, and 4K, with the higher resolutions further distancing us from CPU scaling limitations. Presently, our platform is primarily GPU-bound, but 1080p is starting to become constrained for high-end cards. This platform will be retired from service at the end of 2023 or early 2024.

Test Applications & Games

We are constantly changing the games that get tested on this platform, and as such, maintaining a list of what we currently show is difficult. Instead, we'll list everything that the platform runs as of November 2, 2023. This list will likely be moved to the GPU methodology piece once we get it into the new website.

Additional tests are run, but are not explicitly listed as they may be more variable (or we just haven't publicly documented them at this time).

CPU Cooler Test Bench (2020-2024)

CPU Test Bench (2024-TBD)

We are still working to fully add our modernized testing methodology for CPU benchmarks and reviews. Often, the review itself will contain verbal explanations of each test and methodological notes. We are slowly compiling those as we continue to build-out the living doc on the website. The below tests are typically run on every CPU; however, we sometimes remove tests for reasons that could relate to time with the product before embargo lift, compatibility, performance being too low (e.g. a CPU that struggles to maintain 30 FPS might just see that test eliminated), game updates that invalidate prior data (and need to be recollected), or if we believe the charts in a review are enough to illustrate the pattern without lengthening the review further.

Our CPU tests include these games and settings:

Production Workloads

Gaming Workloads

CPU Miscellaneous Tests

We will add the test bench hardware information for the next round of reviews, but the vast majority of critical information is listed in the chart subtitles.

Check back soon: We still need to add our case bench, our new cooler bench, and our CPU benches, among other platforms!