Worst-Case Consistency, Thermals, and Power Consumption

The performance of the drives in various real-world access traces as well as synthetic workloads was brought out in the preceding sections. We also looked at the performance consistency for these cases. Power users may also be interested in performance consistency under worst-case conditions, as well as drive power consumption. The latter is also important when used with battery powered devices such as notebooks and smartphones. We analyze each of these in detail below.

Flash-based storage devices tend to slow down in unpredictable ways when subject to a large number of small-sized random writes. Many benchmarks use that scheme to pre-condition devices prior to the actual testing in order to get a worst-case representative number. Fortunately, such workloads are uncommon for direct-attached storage devices, where workloads are largely sequential in nature. Use of SLC caching as well as firmware caps to prevent overheating may cause drop in write speeds when a flash-based DAS device is subject to sustained sequential writes.

Our Sequential Writes Performance Consistency Test configures the device as a raw physical disk (after deleting configured volumes). A fio workload is set up to write sequential data to the raw drive with a block size of 128K and iodepth of 32 to cover 90% of the drive capacity. The internal temperature is recorded at either end of the workload, while the instantaneous write data rate and cumulative total write data amount are recorded at 1-second intervals.

Sequential Write to 90% of Disk Capacity - Performance Consistency

Our primary focus is on the performance in the Haswell testbed. The Extreme PRO v2 starts off around 1800 MBps for up to 30GB of continuous writes before dropping down to around 1500 MBps for the remainder of the workload. There is a 22C rise from 56C to 78C in this process. The P50 also starts off similarly, sustaining around 1800 MBps for 13GB of continuous writes before moving down to 1400 MBps. However, the worrisome part is the drops to sub-50 MBps for 3-5 seconds at a time before recovering for around 10-15 seconds in the latter half of the workload. This symptom is seen after around 610GB of continuous writes. We do not see this problem when the drive is connected via the eGFX enclosure - however, the fill rate is around 1280 MBps consistently for the full duration of the workload in that configuration. With the USB 3.2 Gen 2 port of Alpine Ridge, the P50 is again consistent around 850 MBps. The performance of the other drives in other configurations are along expected lines, as can be observed in the above graphs.

Bus-powered devices can configure themselves to operate within the power delivery constraints of the host port. While Thunderbolt 3 ports are guaranteed to supply up to 15W for client devices, USB 3.0 ports are guaranteed to deliver only 4.5W (900mA @ 5V). In this context, it is interesting to have a fine-grained look at the power consumption profile of the various drives. Using the Plugable USBC-TKEY, the bus power consumption of the drives was tracked while processing the CrystalDiskMark workloads (separated by 30s intervals). The graphs below plot the instantaneous bus power consumption against time, while singling out the maximum and minimum power consumption numbers.

Drive Power Consumption - CrystalDiskMark Workloads

The system lock-up during the RND4K Q32T16 workload component is evident in the above graphs, where the power consumption tracker on a different machine continues to keep tracking the power numbers while the system and the drive itself are frozen for all practical purposes. The most interesting cases for the above set are with the testbed demonstrating maximum performance - the Haswell testbed. The Extreme PRO v2 has a peak of 7.23W, and a minimum of 2.91W. The corresponding numbers for the P50 are 8.13W and 3.17W. Interestingly, we see the peak for the Extreme PRO v2 and the P50 with the eGFX configuration - 7.41W and 8.19W respectively. Using the USB SuperSpeed 10Gbps port, the drives run a bit cooler - Extreme PRO v2 peaks at 6.59W, while the P50 peaks at 7.39W. It must be noted that the P50 comes with a status LED (which glows white when the drive is connected), while the Extreme PRO v2 doesn't.

PCMark 10 Storage Bench - Real-World Access Traces Miscellaneous Aspects and Concluding Remarks
Comments Locked

81 Comments

View All Comments

  • repoman27 - Wednesday, October 7, 2020 - link

    According to the USB4 spec, host-to-host tunneling is possible via inter-domain links even when using a hub, but of course YMMV.

    Thanks for pointing out that Thunderbolt 4 hub, that's awesome! And way cheaper than I was expecting. It looks like a Goshen Ridge hub based on Intel's Thunderbolt 4 Compact Dock reference design to me. However, I'm a little worried that OWC's wording implies that it is not backwards compatible with Thunderbolt 3 hosts. That would seem insane, and it may just be an OS support issue at this juncture, but it sure sounds like it's for Thunderbolt 4 PCs only. Aargh!
  • CaptainChaos - Wednesday, October 7, 2020 - link

    It *might* work, and cost may not be prohibitive & worth a shot! I don't see networking mentioned at all on the product page. Also, aren't thunderbolt 3 cables required to be active to do 40GB speeds?
  • CaptainChaos - Wednesday, October 7, 2020 - link

    ... plus it's described as a hub and not a switch so expectations would need to be tempered accordingly!
  • repoman27 - Wednesday, October 7, 2020 - link

    IP over Thunderbolt is a thing, but you'll never see it advertised. Much like IP over IEEE 1394 (a.k.a. FireWire) was also a thing... that nobody knew about. Where we're going, we don't need Ethernet!

    It is a Thunderbolt 4 / USB4 hub, almost certainly based on the recently launched Intel JHL8440 chip. Per the USB4 spec, it contains a router which "includes a flat point-to-point, configurable switch necessary to create the internal paths between adapters", in addition to a PCIe switch as well as USB 3.2 and USB 2.0 hubs. It is also required to support Thunderbolt 3 interoperability and DisplayPort Alternate mode on all downstream facing ports. It's the real deal.
  • repoman27 - Wednesday, October 7, 2020 - link

    Oh, also, the max length for passive cables at 40 Gbit/s is 0.8 m, and FWIW Apple sells theirs for $39. Full featured active cables are available in lengths up to 2.0 m, but cost $129 (from Apple).

    And throughput for host-to-host communications will be limited by the available bandwidth of the PCIe link between the host CPU and Thunderbolt controller. For Thunderbolt 3 hosts, that tends to be a PCIe Gen3 x4 link which results in real-world throughput of about 22 Gbit/s after accounting for protocol overhead.

    So Thunderbolt isn't exactly going to be a panacea, but if you can live with those constraints, it is still a very fast interface.
  • CaptainChaos - Wednesday, October 7, 2020 - link

    i want one (or 2 - wonder if they can be cascaded?)! Should be interesting to see follow ons from this or other vendors for cost or features. I've been eyeballing the 2.5baseT switch recently released by QNAP but this might be a better option for me.
  • CaptainChaos - Wednesday, October 7, 2020 - link

    ... except that I guess I'd need to use a PC to bridge TB subnet to a 10 GB ethernet network :-(
  • Deprectuod - Saturday, October 10, 2020 - link

    It looks good!
  • xpclient - Tuesday, October 13, 2020 - link

    Hi Anandtech, can you do a performance comparison of 2 PCs networked via Thunderbolt 3 to Thunderbolt 3 standard cables (since it allows running 10Gbps Ethernet) vs networking using USB 20 Gbps (Gen 2x2) to RJ-45 10 Gbps Ethernet adapters between 2 PCs? Which is faster? I bet the Thunderbolt 3 connection will be faster due to the protocol overhead of USB.
  • Howard - Sunday, October 18, 2020 - link

    With the stupidity (or anti-consumerist behavior) so glaringly obvious, It's almost as if USB-IF want people to stick with Lightning for as long as possible.

Log in

Don't have an account? Sign up now