Machine Learning: MLPerf and AI Benchmark 4

Even as a new benchmark in the space, MLPerf has been made available that runs representative workloads on devices and takes advantage of both common ML frameworks such as NNAPI as well as the respective chip libraries for each vendor. Using this benchmark on retail phones to date, Qualcomm has had the lead in almost all the tests, but given that the company is promoting a 4x increase in AI performance, it will be interesting to see if that comes across all of MLPerf’s testing scenarios.

It should be noted that Apple’s CoreML is currently not supported, hence the lack of Apple numbers here.

MLPerf 1.0.1 - Image ClassificationMLPerf 1.0.1 - Object DetectionMLPerf 1.0.1 - Image SegmentationMLPerf 1.0.1 - Image Classification (Offline)

Across the board in these first four tests Qualcomm is making a sizable lead, going above and beyond what the S888 can do. Here we’re seeing up to a 2.2x result, making an average +75% gain. It’s not quite the 4x that Qualcomm promoted in its materials, but there’s a sizable gap with the other high-end silicon we’ve tested to date.

MLPerf 1.0.1 - Language Processing

The only non-lead is with the language processing, where Google’s Tensor SoC is almost 2x what the S8g1 scores. This test is based on a mobileBERT model, and either for software or architecture reasons, it fits a lot better into the Google chip than any other. As smartphones increase their ML capabilities, we might see some vendors optimizing for specific workloads over others, like Google has, or offering different accelerator blocks for different models. The ML space is also fast paced, so perhaps optimizing for one type of model might not be a great strategy long-term. We will see.

AI Benchmark 4 - NNAPI (CPU+GPU+NPU)

In AI Benchmark 4, running in pure NNAPI mode, the Qualcomm S8g1 takes a comfortable lead. Andrei noted in previous reviews with this test that the power consumed during this test can be quite high, up to 14 W, and this is where some chips might be able to pull ahead an efficiency advantage. Unfortunately we didn’t record power at the same time as the test, but it would be good to monitor this in the future.

Testing the Cortex-X2: A New Android Flagship Core System-Wide Testing and Gaming
Comments Locked


View All Comments

  • GeoffreyA - Wednesday, December 15, 2021 - link

    Since about Kit Kat or so, the bytecode of an Android app is compiled to native code at install time. But doubtless C++ would still be faster because Java overdoes object-oriented style and dynamic allocation. C/C++ code can work in Android using the NDK, but it appears to be a nightmare getting it latched in. Sometimes I wonder what they were thinking when they picked Java as their language.
  • Meteor2 - Friday, December 17, 2021 - link

    They were thinking about the number of developers who could code it...
  • TheinsanegamerN - Friday, December 17, 2021 - link

    Developer support, ease of transition, and flexibility.
  • Chaser - Tuesday, December 14, 2021 - link

    Before, I used to care about these SOC improvements/benchmarks, etc. But I don't play games on smartphones. All I care about today is battery life improvements, not performance.
  • Alistair - Tuesday, December 14, 2021 - link

    That's why I switched to Apple. I get battery life and efficiency improvements, and CPU improvements. The only highlight for Snapdragon is GPU. CPU improvements are almost non-existent this year, again. Same for several years, they get farther and farther behind Apple in CPU.
  • flyingpants265 - Thursday, December 16, 2021 - link

    Yeah, but everything else about it sucks.
  • Alistair - Saturday, December 18, 2021 - link

    just browsing the web and using apps that are the same on all platforms, and doing it faster
  • Kangal - Sunday, December 19, 2021 - link

    Apple has a long history, that whenever their next chipset offers more efficiency, they counteract this by using a new (power hungry) feature.

    For instance, newer 5G modem, 120Hz screen, higher resolution, or more processing demand in their next software.

    And at times, they even reduce the battery capacity. This saves weight off the phone, and they counteract this by adding other components like more cameras, or heavier glass construction.

    All in all, the iPhones don't gain more battery life, they follow a general trend, and there are some outliers there (eg iPhone 6S+ and iPhone 13 Max).
  • TheinsanegamerN - Friday, December 17, 2021 - link

    Budget phones like the moto g power annialate apple in real life battery benchmarks for a fraction of the price.
  • Alistair - Tuesday, December 14, 2021 - link

    I'll all in on Mediatek then, this was disappointing. Can't wait for Mediatek comparisons!

Log in

Don't have an account? Sign up now