Challenging the Xeon

So what caused us to investigate the IBM POWER8 as a viable alternative to the mass market Xeon E5s and not simply the high-end quad (and higher) socket Xeon E7 parts? A lot. IBM sold its x86 server division to Lenovo. So there is only one true server processor left at IBM: the POWER family. But more importantly, the OpenPOWER fondation has a lot of momentum since its birth in 2013. IBM and the OpenPOWER Foundation Partners like Google, NVIDIA, and Mellanox are all committed to innovating around the POWER processor-based systems from the chip level up through the whole platform. The foundation has delivered some tangible results:

  • Open Firmware which includes both the firmware to boot the hardware (similar to the BIOS) ...
  • ... as OPAL (OpenPOWER Abstraction Layer) to boot and launch a hypervisor kernel.
  • OpenBMC
  • Cheaper and available to third parties (!) POWER8 chips
  • CAPI over PCIe, to make it easier to link the POWER8 to GPUs (and other PCIe cards)
  • And much more third party hardware support (Mellanox IB etc.)
  • A much large software ecosystem (see further)

The impact of opening up firmware under the Apache v2 license and BMC (IBM calls it "field processor") code should not be underestimated. The big hyperscale companies - Google, Amazon, Microsoft, Facebook, Rackspace - want as much control over their software stack as they can.

The resuls are that Google is supporting the efforts and Rackspace has even built their own OpenPOWER server called "Barreleye". While Google has been supportive and showing of proof of concepts, Rackspace is going all the way:

... and aim to put Barreleye in our datacenters for OpenStack services early next year.

The end result is that the complete POWER platform, once only available in expensive high end servers, can now be found inside affordable linux based servers, from IBM (S8xxL) and third parties like Tyan. The opinions of usual pundits range from "too little, too late" to "trouble for Intel". Should you check out a POWER8 based server before you order your next Xeon - Linux server? And why? We started with analyzing the available benchmarks carefully.

A Real Alternative? Reading the Benchmarks
Comments Locked

146 Comments

View All Comments

  • jesperfrimann - Monday, November 9, 2015 - link

    Well, I think you should kick Franz Bourlet, for not hooking you up with with a IBM technical Advocate who actually knew the technology. Such a person could have shown you the robes and helped you understand the kit better. Again Franz is a sales guy.

    IMHO selecting Ubuntu as the Linux distro, did not help you. It's new to the POWER platform and does not have the same robustness as for example SLES which have been around for 10+ years on POWER.

    The fact that you are getting better results using gcc generated code rather than xLC, shows me that something is not right.
    And that the IBM JDK isn't working is well also an indicator that something is now right.
    IMHO selecting Ubuntu, did not make Things easier for you Guys.

    And for really optimized code you need to install and use High performance math libraries for POWER (MASS), which is an addon math library.

    And AFAIR having 8 memory modules, only enables half the memory bandwidth of the system.

    So IMHO IBM didn't help you make their system look good.

    But again that is what you get when you get rid of all the clever people :)

    // Jesper
  • nils_ - Wednesday, November 11, 2015 - link

    You can always rent a box at OVH, they offer a huge chunk of an OpenPower System, albeit virtualized through Runlabs.
  • stefstef - Sunday, November 8, 2015 - link

    compared to the pentium 4 the mips r16k with loads of l3 cache was a bzip2 beast, outperforming the pentium 4 which ran at twice the clock speed and more. despite that the usage of zip programs is what these server processors are build.
  • mapesdhs - Tuesday, November 10, 2015 - link

    Just curious, do you know of any comparative results anywhere for bzip2 on old MIPS vs. other CPUs? It's not something I've seen mentioned before, at least not with respect to SGIs, but perhaps I can run som tests the next time I obtain a quad-R16K/1GHz (16MB L2) Tezro. Best I have at is only an R16K/900MHz (8MB L2) single-CPU Fuel and various configs of Tezro and Onyx350 from 4 to 16x 700MHz with 8MB L2. Just a pity SGI never got to employ multi-core MIPS (it was planned, but alas never happened).

    Oddly, back when current, MIPS' real strength was fp. Over time it fell behind badly for general int, though for SGI's core markets that didn't really matter ("It's the bandwidth, stupid!" - famous quote from Mashey IIRC). MIPS could have caught up with MDMX and MIPS V ISA, especially with the initially intended merged Cray vector stuff, but again that all fell away once the design talent moved to Intel in 1996/7.

    Ian.
  • Freen the merciless - Sunday, November 8, 2015 - link

    Heh! Sparc T5 eats Xeon and power for breakfast.
  • kgardas - Monday, November 9, 2015 - link

    I guess you mean T7 with SPARC M7 inside and not T5. If so, then yes, M7 looks quite capable, but unfortunately provides horrible price/performance ratio. POWER8 box starts at ~6.5k $ while T7-1 on ~40k $. So on SPARC front we'll need to see if Oracle is going to change that with Sonoma chip.
  • Michael Bay - Monday, November 9, 2015 - link

    In parallel only.
  • aryonoco - Tuesday, November 10, 2015 - link

    Thank you Johan for this amazingly well written and well researched article.

    I have to agree with a few people here that question your choice of using LE Ubuntu to test. Traditionally people who use Linux on POWER use SUSE, and some use RHEL, but Ubuntu? Nothing against them, and I love apt, but it's just not a mature platform.

    Try with something more representative such as BE SLES and you will find a vastly different types ecosystem maturity.

    But thanks again, and also thanks to AT for caring about such subjects and publishing these tests.
  • JohanAnandtech - Wednesday, November 11, 2015 - link

    Thank you for taking the time to write up some constructive feedback. I have years of experience with ubuntu and linux and I wanted to play it safe. Running benchmarks on "new" hardware with a new ISA (from my perspective) is pretty complex. C-ray and 7-zip are the only exceptions, but most real server apps (NAMD, ElasticSearch, Spark) depends on many layers of software.

    In theory the OS/ distro is more important to get applications working than the ISA. In practice, it might have been better to bet on the distro with the most maturity and adapt our scripts and installation procedures to Suse.

    But as soon as I get the chance, I'll try out BE suse or redhat on a POWER system.
  • mapesdhs - Tuesday, November 10, 2015 - link

    Johan,

    A minor point, please note my home page for C-ray is here:

    http://www.sgidepot.co.uk/c-ray.html

    Blinkenlights is just a mirror, and not the primary mirror either (that would be the vintagecomputers site).

    Btw, it's a pity you didn't use the same image sizes & settings as used on the main c-ray site, because then I could have included the results on my page (ie. 'sphfract' at 800x600, 1024x768 with 8X oversampling, and 7500x3500), or did you just use the same settings that Phoronix employs?

    Also, John Tsiombikas, the guy who wrote C-ray, told me some interesting things about the test and how it works (info included on the page), most especially that it is highly vulnerable to compiler optimisations which can produce results that are even less realistic than real life workloads. I'm glad thought that you did at least use the sphfract test, since at a sensible resolution or with oversampling it easily pushes the test out of just L1 (the 'scene' test is much smaller). But yeah, overall, c-ray was never intended to be used as a benchmark, it's just taken off somehow, perhaps because the scanline method of threading makes it scale very well.

    Hmm, I really must sort out the page formatting one of these days, and move the most complex test tables to the top. Never seem to find the time...

    Thanks!!

    Ian.

    PS. I always obtained the best results by having more threads than the no. of cores/CPUs, or is this something which doesn't work with non-MIPS systems?

Log in

Don't have an account? Sign up now