Concluding Remarks

Futuremark's PCMark 10 is a welcome update to the tried and tested PCMark 8 benchmark. The new version solves many of the pain points that reviewers had to either bear or work-around in PCMark 8. The run-time of the benchmark has been shortened compared to the older version. The workloads are more clearly defined now. Compared to the generic 'Home', 'Creative', and 'Work' scores, we have 'Essentials', 'Productivity', 'Gaming', and 'Digital Content Creation'. Each of the low-level workloads is reported with its own score. This makes it easier to compare systems for specific tasks. Custom workloads are also available. Some of the new low-level workloads include application start-up times, and rendering and visualization. The command line is the interface of choice for professional users and reviewers, and we are happy to report that this aspect has received a much-needed overhaul in PCMark 10. Results can be exported directly in either XML or PDF format, and a lot of confusing options resulting from PCMark 8's 'OpenCL-accelerated' mode are now removed.

The version of PCMark 10 that we evaluated (1.0.1198) does have some scope for improvement. While VP9 and H.264 decode are part of the web browsing benchmarks, HEVC is oddly absent (given that Netflix is using HEVC right now for 4K streaming). Similar to the previous version, PCMark 10 also allows monitoring and recording of various hardware parameters (such as the CPU and GPU clock frequencies, temperature, power consumption etc.) during the processing of the benchmark. This feature is especially useful to identify whether the system throttles under normal consumer workloads. Unfortunately, the recorded information is not truly portable - it can be viewed only by loading the results in the PCMark 10 application.

Yet another aspect that needs improvement is the difference in the level of details between results exported in the XML and PDF formats. While the former includes all low-level workload scores, the PDF only breaks down the scores up to the four distinct workload sets. Another workload / score that we missed in moving from PCMark 8 to PCMark 10 is the storage segment evaluation. While the new application start-up workload does reflect the storage sub-system performance to some extent, PCMark 8's evaluation of that aspect was much more comprehensive.

Futuremark's PCMark 10 is a vendor-neutral benchmark that can help system builders in pitching the right system configuration to high-volume PC buyers (businesses and enterprises). Its main competitor in the space is BAPCo's SYSmark 2014 SE. Both of them use real-world workloads to deliver a measure of the system capabilities. Based on our experience with both benchmarks, we believe that PCMark 10 would do well to gather some aspects of BAPCo's offerings. Ensuring consistent OS settings across all benchmark runs on different systems and adding energy consumption data to get an idea of the power efficiency of the system would be nice additions. On the other hand, the ease of processing PCMark 10 is miles ahead of SYSmark 2014 SE. PCMark 10 also includes rendering and gaming workloads, which is not available in SYSmark 2014 SE. From the viewpoint of our system reviews, we believe presenting results from both benchmarks will provide readers with more insight into the performance of the system.

Benchmark Processing and Sensitivity Analysis
POST A COMMENT

18 Comments

View All Comments

  • surfnaround - Monday, June 5, 2017 - link

    pricing for the commercial version is not mentioned, because it is 1000 dollars. I would assume it is the same price as the previous versions... how could it get anymore expensive? Reply
  • surfnaround - Monday, June 5, 2017 - link

    Ignore the post above... i though i was talking about 3Dmark... D'oh! Reply
  • waltsmith - Monday, June 5, 2017 - link

    Well, sounds like they are headed in the right direction!! Reply
  • TelstarTOS - Monday, June 5, 2017 - link

    Only open souce software is a step back IMO. Is LibreOffice the most used office suite? Doubt that. Same for the lack of acrobat. HEVC is another disappointment. Reply
  • BrokenCrayons - Monday, June 5, 2017 - link

    There's no doubt that LibreOffice is far behind MS Office, however there are signs of growth. In 2015, there were an estimated 100 million users.

    Source: http://news.softpedia.com/news/libreoffice-now-has...
    Reply
  • JoeyJoJo123 - Monday, June 5, 2017 - link

    They'd need some collaboration effort with Microsoft to get their blessing on being able to hook into Office applications at such a native level when doing these benchmarking tasks.

    LibreOffice is a lot easier to do this native testing in due to its open source nature; they can look up the code to see how they can invoke opening a document, generating 1000 pages of "Lorem Ipsum", doing a Ctrl + H find and replace all, etc. from the runtime of their own executable benchmark, and evaluate to a fine level the amount of time it took to complete these tasks to the clock cycle.

    Can you do Word and Excel based productivity benchmarking? Yeah, but it's manual and I hope you're quick and accurate with that stop watch. Why is it manual? Well go ask Microsoft why they continue to build walled garden closed source software where making extensions is difficult or impossible to accomplish.
    Reply
  • ganeshts - Monday, June 5, 2017 - link

    I doubt Word & Excel-based productivity benchmarking is that difficult. SYSmark 2014 SE does it without any issues. Also, you can always use AutoIt scripting to achieve the required 'workload' and 'automate' it similar to the way a real user would interact (PCMark 10 also uses AutoIt). Reply
  • JoeyJoJo123 - Friday, June 9, 2017 - link

    Wasn't actually aware of AutoIt existing. In that case, yes, I do agree that dropping Word/Excel automation, which could otherwise be done via AutoIt is short-sighted. Reply
  • FMJarnis - Thursday, June 22, 2017 - link

    PCMark 8 includes this exact thing as a separate application test. Implementing one is not that difficult. It is likely that a separate test for MS Office is added to PCMark 10 at a later date.

    But requiring everyone who wants to run PCMark 10 to first buy and install Microsoft Office is not really feasible as it is commercial software. For the baseline tests that should be usable without any other prerequisites, what you propose unfortunately doesn't work.
    Reply
  • FMJarnis - Thursday, June 22, 2017 - link

    HVEC is not used because it is not supported on Windows 7 by the OS.

    Also when PCMark 10 development started, HVEC licensing was... complicated. Software implementations on computers still had per-application licensing fees for the codec. Open source solutions were in a legal limbo. Since then HVEC licensing terms have changed, but it was too late for this benchmark.
    Reply

Log in

Don't have an account? Sign up now