Sizing Up Servers: Intel's Skylake-SP Xeon versus AMD's EPYC 7000 - The Server CPU Battle of the Decade?
by Johan De Gelas & Ian Cutress on July 11, 2017 12:15 PM EST- Posted in
- CPUs
- AMD
- Intel
- Xeon
- Enterprise
- Skylake
- Zen
- Naples
- Skylake-SP
- EPYC
AMD's EPYC Server CPU
If you have read Ian's articles about Zen and EPYC in detail, you can skip this page. For those of you who need a refresher, let us quickly review what AMD is offering.
The basic building block of EPYC and Ryzen is the CPU Complex (CCX), which consists of 4 vastly improved "Zen" cores, connected to an L3-cache. In a full configuration each core technically has its own 2 MB of L3, but access to the other 6 MB is rather speedy. Within a CCX we measured 13 ns to access the first 2 MB, and 15 to 19 ns for the rest of the 8 MB L3-cache, a difference that's hardly noticeable in the grand scheme of things. The L3-cache acts as a mostly exclusive victim cache.
Two CCXes make up one Zeppelin die. A custom fabric – AMD's Infinity Fabric – ties together two CCXes, the two 8 MB L3-caches, 2 DDR4-channels, and the integrated PCIe lanes. That topology is not without some drawbacks though: it means that there are two separate 8 MB L3 caches instead of one single 16 MB LLC. This has all kinds of consequences. For example the prefetchers of each core make sure that data of the L3 is brought into the L1 when it is needed. Meanwhile each CCX has its own separate (not inside the L3, so no capacity hit) and dedicated SRAM snoop directory (keeping track of 7 possible states). In other words, the local L3-cache communicates very quickly with everything inside the same CCX, but every data exchange between two CCXes comes with a tangible latency penalty.
Moving further up the chain, the complete EPYC chip is a Multi Chip Module(MCM) containing 4 Zeppelin dies.
AMD made sure that each die is only one hop apart from the other, ensuring that the off-die latency is as low as reasonably possible.
Meanwhile scaling things up to their logical conclusion, we have 2P configurations. A dual socket EPYC setup is in fact a "virtual octal socket" NUMA system.
AMD gave this "virtual octal socket" topology ample bandwidth to communicate. The two physical sockets are connected by four bidirectional interconnects, each consisting of 16 PCIe lanes. Each of these interconnect links operates at +/- 38 GB/s (or 19 GB/s in each direction).
So basically, AMD's topology is ideal for applications with many independently working threads such as small VMs, HPC applications, and so on. It is less suited for applications that require a lot of data synchronization such as transactional databases. In the latter case, the extra latency of exchanging data between dies and even CCX is going to have an impact relative to a traditional monolithic design.
Facebook
Twitter
RSS
219 Comments
View All Comments
Panxa - Sunday, July 16, 2017 - link
"Competition has spoiled the naming convention Intels 14 === competetions 7 or 10"The node naming convention used to be the gate length, however that has become irrelevant. Intel 14 nm gate lenghth is about 1.5x and 10 nm about 1.8x. Companies and organizations have developed quite accurate models to asses process density with equations based on process poarameters like CPP and MPP to what they call a "standard node"
"Intel used to maintain 2 year lead now grew that to 3-4year lead"
Don't belive intel propaganda. Intel takes the lead in 2014 with their 14nm process with a standard node value of 12.1. Samsung and then TSMC take the lead in 2017 with their 10nm processes having standard node values of 11.2 and 10.3 respectively. Intel will retake the the lead back when they deliver their 10nm process with a standard node value of 8.3. However it will be a short lived lead, TSMC will retake the lead back with their 7nm with a standard node of 7.9 before GLOBALFOUNDRIES takes the lead in 2018 with their 7nm process with a standard node value of 7.8. The gap is gone !!!
"yet their revenue profits grow year over year"
Wrong. Intel revenue for the last years remained fairly constant
2011 grow
2012 decline
2013 decline
2014 grow
2015 decline
2016 grow
All in all from 2011 to 2016 revenue went from 54 billion to 59 billion. If we take into account inflation $54 billion in the year 2011 is worth $58.70 billion today.
Not to mention that Samsung has overtaken Intel to become the world No.1 semiconductor company, and that a "pure play" foundry like TSMC has surpassed intel in market CAP
johnp_ - Wednesday, July 12, 2017 - link
The Xeon Bronze Table on Page 7 seems to have an error. It lists the 4112 as having 5.50MB L3, but ark says it has 8.25MB, just like the 3104, so it looks like it has an above-average L3/Core:https://ark.intel.com/products/123551
Ian Cutress - Friday, July 14, 2017 - link
I've got Intel documents from our briefings that say it has the regular 1.375MB/core allocation, and others saying it has 8.25MB. I'm double checking.johnp_ - Friday, July 21, 2017 - link
All commercial listings and most reviews I've seen online show the processor with 8.25MB as well.Do you have any further information from Intel?
pepoluan - Wednesday, July 12, 2017 - link
What I'm dying to know: Performance when running as virtualization host. Using Xen, VMware, and Hyper-V.Threska - Saturday, July 22, 2017 - link
Virtualization itself, and more importantly virtualization security.Sparkyman215 - Wednesday, July 12, 2017 - link
Typo here: Intel will seven different versions of the chipset, varying in 10G and QAT support, but also varying in TDP:tmbm50 - Wednesday, July 12, 2017 - link
One thing to consider when considering value is the Microsoft Server 2016 core tax.....assuming your mission critical apps are still tied to MS ;-)Server 2016 now chargers per core with an 8 core socket as the base. The Window license for a 32 core server is NUTS.
I'm surprised AMD and Intel are not pushing Microsoft on this. For datacenters like ourselves its pushing us to 8 core sku's with more 2U nodes.
msroadkill612 - Wednesday, July 12, 2017 - link
Aye, its a fuuny world lad.The way the automobile panned out differently in different countries, was laargely die to fuel tax regimes, rather than technology.
i.e. what is the best way to cheat a bit on the incumbent tax rules of germany/france/uk vs a more laissez faire USA. In UK, u were taxed on horsepower, but u could cheat a bit w/ hi revs & more gears - that sort of thing.
rahvin - Wednesday, July 12, 2017 - link
Who runs any Windows service on bare metal these days? If you haven't virtulalized your windows servers running on KVM you should.