Machine Learning: MLPerf and AI Benchmark 4

Even as a new benchmark in the space, MLPerf has been made available that runs representative workloads on devices and takes advantage of both common ML frameworks such as NNAPI as well as the respective chip libraries for each vendor. Using this benchmark on retail phones to date, Qualcomm has had the lead in almost all the tests, but given that the company is promoting a 4x increase in AI performance, it will be interesting to see if that comes across all of MLPerf’s testing scenarios.

It should be noted that Apple’s CoreML is currently not supported, hence the lack of Apple numbers here.

MLPerf 1.0.1 - Image ClassificationMLPerf 1.0.1 - Object DetectionMLPerf 1.0.1 - Image SegmentationMLPerf 1.0.1 - Image Classification (Offline)

Across the board in these first four tests Qualcomm is making a sizable lead, going above and beyond what the S888 can do. Here we’re seeing up to a 2.2x result, making an average +75% gain. It’s not quite the 4x that Qualcomm promoted in its materials, but there’s a sizable gap with the other high-end silicon we’ve tested to date.

MLPerf 1.0.1 - Language Processing

The only non-lead is with the language processing, where Google’s Tensor SoC is almost 2x what the S8g1 scores. This test is based on a mobileBERT model, and either for software or architecture reasons, it fits a lot better into the Google chip than any other. As smartphones increase their ML capabilities, we might see some vendors optimizing for specific workloads over others, like Google has, or offering different accelerator blocks for different models. The ML space is also fast paced, so perhaps optimizing for one type of model might not be a great strategy long-term. We will see.

AI Benchmark 4 - NNAPI (CPU+GPU+NPU)

In AI Benchmark 4, running in pure NNAPI mode, the Qualcomm S8g1 takes a comfortable lead. Andrei noted in previous reviews with this test that the power consumed during this test can be quite high, up to 14 W, and this is where some chips might be able to pull ahead an efficiency advantage. Unfortunately we didn’t record power at the same time as the test, but it would be good to monitor this in the future.

Testing the Cortex-X2: A New Android Flagship Core System-Wide Testing and Gaming
Comments Locked


View All Comments

  • Reflex - Tuesday, December 21, 2021 - link

    Because Lineage's security model is completely broken. It's fun to play with on very old devices but I'd never put it on a device with access to anything I cared about.
  • iphonebestgamephone - Tuesday, December 21, 2021 - link

    What exactly? How is it any different from stock roms?
  • Mujeeb - Tuesday, December 14, 2021 - link

    U have to be kidding right. iPhone se might be costing just $400 in US but surely not outside US. Outside US, it costs $500.
    At $500 one gets a pure andriod flagship here in India. And in sale days, we get them further $100 cheap.
    Let's compare (in India):
    iPhone se $500 vs Xiaomi mi 11x
    A13 (GB score 3400) vs Sd 870 GB score 3400)
    Genshin Impact 53 fps vs 48 fps
    PUBG 90 fps vs 60 fps
    Amoled fhd+ 10 bit 120hz vs ips sd 60hz
    Bigger display 6.6 inch vs 4.7
    Battery 5000 mah vs 1800 mah
    8gb ram 128 GB storage vs 3 gb ram and 64 gb storage
    33w fast charging vs slowing charging
    5g ready vs backward 4g
    Modern minimalistic design vs old boring design
    Punch hole vs old boring design
    Peak brightness 1100 nits vs 480 nits
    Camera 64 mp main vs 12 MP main
    Oh God please don't force me to tarnish this so called se edition here.
    Only ones who will prefer this are losers who think owning an apple mobile makes them richer.
  • 1_rick - Tuesday, December 14, 2021 - link

    "Only ones who will prefer this are losers who think owning an apple mobile makes them richer."

    Don't forget about the tiny hands brigade of twentysomethings with perfect eyes. They like their small screens, too.
  • Fulljack - Tuesday, December 14, 2021 - link

    or people who just prefer small screen, tho.
  • isthisavailable - Wednesday, December 15, 2021 - link

    lol this is such a boomer comment
  • Reflex - Wednesday, December 15, 2021 - link

    I'm not saying Android sucks or trying to start a flame war. But that Android phone you mention comes with between 2-3 years of support before you are on your own, with all the security risks that entails and slow loss of app compat.

    Meanwhile the iPhone 6S, released six years ago, just got iOS 15 which guarantees it at least four more years of support and all apps continue to function on it. A six year old phone will have a more functional lifespan than your brand new Xiaomi.

    For someone trying to get into a phone cheap, have it just work, and who does not have an interest in upgrading (and spending another $400+) every 2-3 years, Apple is dirt cheap comparatively with 3-5x the support lifecycle for only 1.25x more money (using your comparison).

    It's a no-brainer. Again, for those who aren't interested in the upgrade treadmill.
  • flyingpants265 - Thursday, December 16, 2021 - link

    Well, every phone I've used from Android has cost less than $250 so at least it's not as bad as buying a new Galaxy phone every 2 years.
  • flyingpants265 - Thursday, December 16, 2021 - link

    I'm lying because I'm including used phones in this. I bought a lot of used LGs, they drop in value like rocks. There's an LG G7 selling locally right now for $80...

    ...Maybe buying a brand new phone is for suckers.
  • Reflex - Friday, December 17, 2021 - link

    I don't generally recommend a new phone period, but again on Android that lack of support lifecycle is a problem as used cuts significantly into the support timeframe.

Log in

Don't have an account? Sign up now