Server Generations

Freedom laid the groundwork for new generations to come. In 2012, Facebook released the specs for its second server design, dubbed "Windmill", which brought updated mainboards supporting Intel's Sandy Bridge-EP platform and AMDs Opteron 6200/6300 CPUs.

Following table details all Facebook-designed OCP servers, starting with Freedom, with highlighted differences.

Facebook-designed OCP Server Generations
  Freedom (Intel) Freedom (AMD) Windmill (Intel) Watermark (AMD) Winterfell Leopard
Platform Westmere-EP Interlagos Sandy Bridge-EP Interlagos Sandy Bridge-EP / Ivy-Bridge EP Haswell-EP
Chipset 5500 SR5650/
SP5100
C602 SR5650/
SR5670/
SR5690
C602 C226
Models X5500/
X5600
Opteron 6200/6300 E5-2600 Opteron 6200/6300 E5-2600 v1 / v2 E5-2600v3
Sockets 2 2 2 2 2 2
Max TDP allowed (in Watt) 95 85 115 85 115 145
RAM per socket 3x DDR3 12x DDR3 8x DDR3 8x DDR3 8x DDR3 8x DDR4
/NVDIMM
~ Node Width (inch) 21 21 8 21 6.5 6.5
Form factor (height in rack U) 1.5 1.5 1.5 1.5 2 2
Fans per node 4 4 2 4 2 2
Fan width (in mm) 60 60 60 60 80 80
Disk bays (3.5'') 6 6 6 6 1 1
Disk interface SATA II SATA II SATA III SATA III SATA III / RAID HBA SATA III
/ M.2
Amount of DIMM slots per socket 9 12 9 12 8 8
DDRX gen 3 3 3 3 3 4
Ethernet connectivity 1 GbE fixed 2 GbE fixed 2 GbE fixed + PCIe mezz 2 GbE fixed 1GbE fixed + 8x PCIe Mezz 8x PCIe
Mezz
Deployed in Prineville, Oregon Prineville Lulea, Sweden Lulea Altoona ?
PSU model PowerOne SPAFCBK- 01G PowerOne SPAFCBK- 01G PowerOne PowerOne N/A N/A
PSU count 1 1 1 1 N/A N/A
PSU capacity (in Watt) 450 450 450 450 N/A N/A
Amount of nodes per sled 1 1 2 2 3 3
BMC No (Intel RMM) No No (Intel RMM) No No (Intel RMM) Yes (Aspeed AST1250
w 1GB
Samsung DDR3 DIMM K4B1G1646G- BCH9 )

Facebook indicated that its expected lifespan for its Freedom nodes is around three years, and is in the process of swapping out deprecated equipment with OpenRack v1 based equipment.

Introduction Integrate: OpenRack
POST A COMMENT

26 Comments

View All Comments

  • Kevin G - Tuesday, April 28, 2015 - link

    Excellent article.

    The efficiency gains are apparent even using suboptimal PSU for benchmarking. (Though there are repeated concurrency values in the benchmarking tables. Is this intentional?)

    I'm looking forward to seeing a more compute node hardware based around Xeon-D, ARM and potentially even POWER8 if we're lucky. Options are never a bad thing.

    Kind of odd to see the Knox mass storage units, I would have thought that OCP storage would have gone the BackBlaze route with vertically mount disks for easier hot swap, density and cooling. All they'd need to develop would have been a proprietary backplane to handle the Kinetic disks from Seagate. Basic switching logic could also be put on the backplane so the only external networking would be high speed uplinks (40 Gbit QSFP+?).

    Speaking of the Kinetic disks, how is redundancy handled with a network facing drive? Does it get replicated by the host generating the data to multiple network disks for a virtual RAID1 redundancy? Is there an aggregator that handles data replication, scrubbing, drive restoration and distribution, sort of like a poor man's SAN controller? Also do the Kinetic drives have two Ethernet interfaces to emulate multi-pathing in the event of a switch failure (quick Googling didn't give me an answer either way)?

    The cold storage racks using Blu-ray discs in cartridges doesn't surprise me for archiving. The issue I'm puzzled with is the process how data gets moved to them. I've been under the impression that there was never enough write throughput to make migration meaningful. For a hypothetical example, by the time 20 TB of data has been written to the discs, over 20 TB has been generated that'd be added to the write queue. Essentially big data was too big to archive to disc or tape. Parallelism here would solve the throughput problem but that get expensive and takes more space in the data center that could be used for hot storage and compute.

    Do the Knox storage and Wedge networking hardware use the same PDU connectivity as the compute units?

    Are the 600 mm wide racks compatible use US Telecom rack width equipment (23" wide)? A few large OEMs offer equipment in that form factor and it'd be nice for a smaller company to mix and match hardware with OCP to suit their needs.
    Reply
  • nils_ - Wednesday, April 29, 2015 - link

    You can use something like Ceph or HDFS for data redundancy which is kind of like RAID over network. Reply
  • davegraham - Tuesday, April 28, 2015 - link

    Also, Juniper Networks has an ONIE-compliant OCP switch called the OCX1100 which is the only Tier1 switch manufacturer (e.g. Cisco, Arista, Brocade) to provide such a device. Reply
  • floobit - Tuesday, April 28, 2015 - link

    This is very nice work. One of the best articles I've seen here all year. I think this points at the future state of server computing, but I really wonder if the more traditional datacenter model (VMware on beefy blades with a proprietary FC-connected SAN) can be integrated with this massively-distributed webapp model. Load-balancing and failovering is presumably done in the app layer, removing the need for hypervisors. As pretty as Oracle's recent marketing materials are, I'm pretty sure they don't have an HR app that can be load-balanced on the app layer in alongside an expense app and an ERP app. Maybe in another 10 years. Then again, I have started to see business suites where they host the whole thing for you, and this could be a model for their underlying infrastructure. Reply
  • ggathagan - Tuesday, April 28, 2015 - link

    In the original article on these servers, it was stated that the PSU's were run on 277v, as opposed to 208v.
    277v involves three phase power wiring, which is common in commercial buildings, but usually restricted to HVAC-related equipment and lighting.
    That article stated that Facebook saved "about 3-4% of energy use, a result of lower power losses in the transmission lines."
    If the OpenRack carries that design over, companies will have to add the cost of bringing power 277v to the rack in order to realize that gain in efficiency.
    Reply
  • sor - Wednesday, April 29, 2015 - link

    208 is 3 phase as well, generally 3x120v phases, with 208 tapping between phases or 120 available to neutral. Its very common for DC equipment. 277 to the rack IS less common, but you seemed to get hung up on the 3 phase part. Reply
  • Casper42 - Monday, May 4, 2015 - link

    3 phase restricted to HVAC?
    Thats ridiculous, I see 3 Phase in DataCenters all the time.
    And Server vendors are now selling 277vAC PSUs for exactly this reason that FB mentions. Instead of converting the 480v main to 220 or 208, you just take a 277 feed right off the 3 phase and use it.
    Reply
  • clehene - Tuesday, April 28, 2015 - link

    You mention a reported $2 Billion in savings, but the article you refer to states $1.2 Billion. Reply
  • FlushedBubblyJock - Tuesday, April 28, 2015 - link

    One is the truth and the other is "NON Generally Accepted Accounting Procedures" aka it's lying equivalent. Reply
  • wannes - Wednesday, April 29, 2015 - link

    Link corrected. Thanks! Reply

Log in

Don't have an account? Sign up now