Machine Learning: MLPerf and AI Benchmark 4

Even as a new benchmark in the space, MLPerf has been made available that runs representative workloads on devices and takes advantage of both common ML frameworks such as NNAPI as well as the respective chip libraries for each vendor. Using this benchmark on retail phones to date, Qualcomm has had the lead in almost all the tests, but given that the company is promoting a 4x increase in AI performance, it will be interesting to see if that comes across all of MLPerf’s testing scenarios.

It should be noted that Apple’s CoreML is currently not supported, hence the lack of Apple numbers here.

MLPerf 1.0.1 - Image ClassificationMLPerf 1.0.1 - Object DetectionMLPerf 1.0.1 - Image SegmentationMLPerf 1.0.1 - Image Classification (Offline)

Across the board in these first four tests Qualcomm is making a sizable lead, going above and beyond what the S888 can do. Here we’re seeing up to a 2.2x result, making an average +75% gain. It’s not quite the 4x that Qualcomm promoted in its materials, but there’s a sizable gap with the other high-end silicon we’ve tested to date.

MLPerf 1.0.1 - Language Processing

The only non-lead is with the language processing, where Google’s Tensor SoC is almost 2x what the S8g1 scores. This test is based on a mobileBERT model, and either for software or architecture reasons, it fits a lot better into the Google chip than any other. As smartphones increase their ML capabilities, we might see some vendors optimizing for specific workloads over others, like Google has, or offering different accelerator blocks for different models. The ML space is also fast paced, so perhaps optimizing for one type of model might not be a great strategy long-term. We will see.

AI Benchmark 4 - NNAPI (CPU+GPU+NPU)

In AI Benchmark 4, running in pure NNAPI mode, the Qualcomm S8g1 takes a comfortable lead. Andrei noted in previous reviews with this test that the power consumed during this test can be quite high, up to 14 W, and this is where some chips might be able to pull ahead an efficiency advantage. Unfortunately we didn’t record power at the same time as the test, but it would be good to monitor this in the future.

Testing the Cortex-X2: A New Android Flagship Core System-Wide Testing and Gaming
Comments Locked


View All Comments

  • rtho782 - Friday, December 17, 2021 - link

    Alex Katouzian is like budget Sylvester Stallone :D
  • bogamia - Monday, December 20, 2021 - link

    Apparently, D9000 has been benchmarked by this Chinese YouTuber ( and it outperformed the snapdragon 8 gen 1 in all the areas. The performance power watt gap is massive between these two chips. Even in GPU benchmarks, D9000 outperformed the snapdragon equivalent. This is going to be an interesting generation. I wonder what will be the excuses of OEMs to use Qualcomm in the affordable flagship segments .
  • iphonebestgamephone - Monday, December 20, 2021 - link

    It looks like things like genshin impact is still more optimized for adreno.
  • ChrisGX - Thursday, December 23, 2021 - link

    >>[the D9000] outperformed the snapdragon 8 gen 1 in all the areas<<

    Not, exactly. The vlogger makes clear that the Adreno GPU in the Snapdragon 8 Gen 1 offers slightly higher performance than the Mali G710 in the D9000 but that the small performance increment comes at a significant cost in power consumption terms. So, while the Mali GPU offers slightly lower performance it is definitely the more energy efficient GPU.
  • Kangal - Friday, December 24, 2021 - link

    Interesting video. Here's the recap:

    GeekBench 5 (Single-core)
    D9000: 1287 score, 3.5 Watts, 368 Efficiency
    QC 8g1: 1200 score, 4.2 Watts, 286 Efficiency
    QSD 888: 1135 score, 3.8 Watts, 299 Efficiency

    GeekBench 5 (Multi-thread)
    D9000: 4474 score, 9.8 Watts, 457 Efficiency
    QC 8g1: 3810 score, 11.1 Watts, 343 Efficiency
    QSD 888: 3753 score, 8.9 Watts, 422 Efficiency

    GFXBench Aztec Ruins (High/1440p Offscreen)
    D9000: 43 fps, 8.2 Watts, 5.24 Efficiency
    QC 8g1: 47 fps, 11.2 Watts, 4.20 Efficiency
    QSD 888: 30 fps, 9.0 Watts, 3.34 Efficiency

    Genshin Impact (Quality Mode, Initial Performance)
    D9000: 60 fps, 6.8 Watts, drops 1.1 Watts, 7 minutes then throttles
    QC 8g1: 60 fps, 7.5 Watts, drops 1.2 Watts, 3 min then dips but maintains
    QSD 888: 60 fps, 7.5 Watts, drops 1.6 Watts, 6 minutes then throttles

    Genshin Impact (Quality Mode, after 15min throttling)
    D9000: 48 fps, 5.7 Watts, 8.42 Efficiency, Resolution 1422x640
    QC 8g1: 55 fps, 6.3 Watts, 8.73 Efficiency, Resolution 1422x640
    QSD 888: 48 fps, 5.9 Watts, 8.14 Efficiency, Resolution 1600x720
  • ChrisGX - Tuesday, January 4, 2022 - link

    Xiaobai's Tech Reviews ( ) - a Hong Kong based YouTuber - has come to similar conclusions to the tech vlogger (Geekerwan) that you cite. The benchmark testing conducted by both vloggers seems broader in scope and more careful than the usual fare on YouTube. In both cases the engineering test mules/prototype devices tested were equipped with the latest crop of premium ARM SoCs (that will appear in production Android devices in 2022). The similarity of benchmark results doesn't guarantee the correctness of those results or the completeness of the performance picture drawn, of course, but with independent sources getting very similar results it wouldn't be a shock if production devices follow the evident pattern.

    Golden Reviewer, a Singapore based tech reviewer/vlogger (posting reviews in English) if anything offers an even more critical account of the SD8 Gen 1 and an account of the D9000 that does indeed credit it as the better chip in most areas. Using a pre-production device in testing he singles out overheating as a serious problem for the SD8 Gen 1.
  • maik80 - Tuesday, January 18, 2022 - link

    Fans of android have a mania to use AnTuTu, the owner has already been arrested for fraud in the tests, it has already been confirmed that he received to increase the result. In addition to giving points for the amount of ram memory. it's a joke
  • qnfw3174 - Thursday, March 31, 2022 - link

    Is it finished to test A710/A510? Where can I find the result?
  • yeeeeman - Monday, April 25, 2022 - link

    Can we get a follow up on this?

Log in

Don't have an account? Sign up now