Machine Learning: MLPerf and AI Benchmark 4

Even as a new benchmark in the space, MLPerf has been made available that runs representative workloads on devices and takes advantage of both common ML frameworks such as NNAPI as well as the respective chip libraries for each vendor. Using this benchmark on retail phones to date, Qualcomm has had the lead in almost all the tests, but given that the company is promoting a 4x increase in AI performance, it will be interesting to see if that comes across all of MLPerf’s testing scenarios.

It should be noted that Apple’s CoreML is currently not supported, hence the lack of Apple numbers here.

MLPerf 1.0.1 - Image ClassificationMLPerf 1.0.1 - Object DetectionMLPerf 1.0.1 - Image SegmentationMLPerf 1.0.1 - Image Classification (Offline)

Across the board in these first four tests Qualcomm is making a sizable lead, going above and beyond what the S888 can do. Here we’re seeing up to a 2.2x result, making an average +75% gain. It’s not quite the 4x that Qualcomm promoted in its materials, but there’s a sizable gap with the other high-end silicon we’ve tested to date.

MLPerf 1.0.1 - Language Processing

The only non-lead is with the language processing, where Google’s Tensor SoC is almost 2x what the S8g1 scores. This test is based on a mobileBERT model, and either for software or architecture reasons, it fits a lot better into the Google chip than any other. As smartphones increase their ML capabilities, we might see some vendors optimizing for specific workloads over others, like Google has, or offering different accelerator blocks for different models. The ML space is also fast paced, so perhaps optimizing for one type of model might not be a great strategy long-term. We will see.

AI Benchmark 4 - NNAPI (CPU+GPU+NPU)

In AI Benchmark 4, running in pure NNAPI mode, the Qualcomm S8g1 takes a comfortable lead. Andrei noted in previous reviews with this test that the power consumed during this test can be quite high, up to 14 W, and this is where some chips might be able to pull ahead an efficiency advantage. Unfortunately we didn’t record power at the same time as the test, but it would be good to monitor this in the future.

Testing the Cortex-X2: A New Android Flagship Core System-Wide Testing and Gaming


View All Comments

  • GeoffreyA - Wednesday, December 15, 2021 - link

    All the best, Andrei. Live long and prosper. Reply
  • OreoCookie - Wednesday, December 15, 2021 - link

    Shame. Whoever takes over your responsibilities has big shoes to fill! Good luck for your future adventures. Your work was much appreciated. Reply
  • Ian Cutress - Wednesday, December 15, 2021 - link

    You're reading that guy's article already 👍 Reply
  • OreoCookie - Thursday, December 16, 2021 - link

    No worries, but I think of you as part of the inventory here at Anandtech! :-)
    (Plus, I had the impression that Andrei was hired because you can't handle the workload all by yourself. This of course isn't meant as a dig towards you, Anandtech has been thoroughly reviewing *a lot* of CPUs and SoCs.)
  • back2future - Wednesday, December 15, 2021 - link

    Thank you and best wishes Reply
  • t.s - Wednesday, December 15, 2021 - link

    Thanks for everything, Andrei. Will miss your articles here. Reply
  • Bik - Thursday, December 16, 2021 - link

    Have a good journey ahead friend. Reply
  • Meteor2 - Friday, December 17, 2021 - link

    Best wishes for your future endeavours, Andrei! You'll be missed here. Reply
  • Trackster11230 - Friday, December 17, 2021 - link

    Best of luck! Reply
  • naagin - Thursday, February 10, 2022 - link

    naagin 6 Today Episode

    Watch naagin 6 Today Live Full Episode, naagin 6 Latest Full Episode Streaming on Mx Player, Hindi Drama naagin All Episode Season 6 Youtube, Tv Drama naagin 6 Today Full Episode Watch Online In High Qualiy Video Youtube Dailymotion, naagin 6 Today Episode Promo, Get all the information about naagin 6 Tv Series

    <a href="">naagin 6 Today Episode</a>

Log in

Don't have an account? Sign up now