Testing Overview

In total, we used three different testing locations, an “ideal” setup with the wireless router less than ten feet from the test laptop(s), and two different “distant” locations where the laptop is located several rooms away, with doors closed in between and several sheetrock walls. We also tested with two different routers, a Netgear WNR3500L and a Linksys E4200. At each location, we ran four different networking related tasks. The tests had the server PC connected directly to the Gigabit router via a six-foot CAT6 Ethernet cable, and the test PCs were otherwise idle (no extra processes, services, etc. were running).

The first task consists of copying files from the server to the test laptop. Here we have two different scenarios; the first is a single large file, a 2.2GiB HD movie. We time how long it takes to copy the file and then calculate the throughput in Mbps (1Mb = 1,000,000 bits). The second scenario consists of numerous small files. We use the contents of the Cinebench R10 and Cinebench R11.5 folders, which have a mix of a few larger files with many small files. In total, this test has 8780 files and 511 folders, with the total file size coming to 440MiB. Again we time the copy process and then calculate Mbps. Copying files between PCs is a completely real-world scenario, and with SSDs present in all of the test systems we should be bottlenecked by the networking performance (even a moderate HDD should be able to outpace WiFi speeds, but for Gigabit Ethernet you’ll want SSDs).

The remaining three tests are more theoretical rather than real-world. First up, we have the NTttcp utility. Unlike file copying, this test transfers data from memory between PCs, so the storage subsystem is out of the loop. We run two different scenarios, once with the laptop acting as “client” and the desktop being the “server”, and the second test with the roles reversed. This will test the maximum theoretical throughput of the laptop in both transmit and receive modes; most of the wireless devices do better at receiving vs. sending, but in a few instances the reverse holds. We used the following commands on the client/server:

ntttcpr.exe -m 6,0,[Server IP] -a 4 -l 64000 -n 4000
ntttcps.exe -m 6,0,[Client IP] -a 4 -l 64000 -n 4000

Our third test is very similar to the above, only we use Netperf instead of NTttcp and we conduct TCP as well as UDP testing. We use a precompiled version of Netperf that Bigfoot sent, but a Windows version is available from Chris Wolf that shows essentially the same results. One of the things to be aware of is that Netperf has a variety of options; we tested with the “stock” options, though Bigfoot has a second set of recommended command line parameters. The throughput can be very different depending on what options you specify and what wireless card you’re using. Several of the systems we tested with the Bigfoot parameters performed horribly on the UDP test, so we decided to stick with the stock Netperf command (though we’ll show the performance with the Bigfoot settings on the E4200 Ideal page as a reference point). The default options simply specify the IP address of the host, along with “-t UDP_STREAM” for UDP testing; the Bigfoot recommended commands for TCP and UDP are:

netperf.exe -l 20 -t TCP_STREAM -f m -H [Server IP] -- -m 1460 -M 1460 -s 4000000,4000000 -S 4000000,4000000
netperf.exe -l 20 -t UDP_STREAM -f m -H [Server IP] -- -m 1472 -M 1472 -s 4000000,4000000 -S 4000000,4000000

The key difference between the stock options and the Bigfoot parameters is the selection of a 1472 byte packet. The default UDP packet size is 1000 bytes, so that’s what most manufacturers optimize for, but larger packets are possible. Bigfoot states that video streaming often uses larger packets to improve throughput, so they’ve worked to ensure their drivers perform well with many packet sizes. Skip to the “ideal” testing with the Linksys router for more details of our Netperf UDP testing with the Bigfoot parameters.

Our final test is a measure of latency, and you might be tempted to take the results with a grain of salt as we’re using a utility provided by Bigfoot called GaNE (Gaming Network Efficiency). The purpose of this utility is to measure real latency between a client and a server, down to the microsecond, while simulating a network gaming workload. Most games don’t support any logging of network latency, so GaNE is an easy to use substitute. It sends a 100 byte UDP packet every 200ms, with a timestamp included in the packet. The client receives the packet and sends it back, and by looking at the timestamp the server can determine roundtrip latency. According to Bigfoot, the majority of network games send ~100 byte packets at intervals ranging from every 50ms to a few seconds apart, so they chose a value that would represent a large number of titles. While not all games are the same, there are long-established “best practice” coding standards for network gaming, so a lot of titles should behave similar to GaNE. For the test, two laptops connect to the server at the same time and GaNE reports average latency along with the average latency of the worst 10% of packets—the latter being a better measurement of jitter on your network connection.

Besides GaNE, we conducted some informal testing of latency with other utilities. First up is the Windows ping command; we saw latency spikes that are similar to those in GaNE so the results of both utilities appear valid. Ping doesn’t work like most games, however—it has smaller packets sent less frequently and it uses the ICMP protocol instead of UDP. That brings us to the third latency test. About the only games to include support for latency logging are Source Engine titles, so we tested latency with a local server running Counterstrike: Source using the “net_graph 3” utility. We’re unable to capture raw latency numbers this way, but we could clearly see periods of higher latency on all of the Intel WiFi cards. If anything, the latency blips tended to occur more frequently with CS:S than in GaNE, though interestingly enough the Atheros controller basically tied the Bigfoot in our experience (1ms ping times most of the match, with no discernable spikes). The Realtek card had a higher average latency of around 6ms, but at least there wasn’t a lot of jitter.

In short, latency measurements with GaNE, ping, and CS:S all show similar results, at least when measured over the same local network. Despite concerns about using a Bigfoot provided utility, it doesn’t appear that Bigfoot is doing anything unusual. The bigger concern is what the results mean in the world of Internet gaming. Since we’re running on a local network, latency is already an order of magnitude (or two) lower than what you’d generally experience over the Internet. For online gaming, the latency results we’re reporting end up being more of an additive latency, after the latency of the router to Internet server is factored in. In other words, if you’re playing a game where it takes 60ms for your data packets to get from the server to your router, what GaNE reports is how long it takes those packets to get from the router to your PC. The bigger issue with latency isn’t the 3-5ms average that you’ll get over the local network, but it’s the jitter where 100+ms spikes occur, and as noted GaNE provides an indication of jitter with the “worst 10% average latency” result.

We ran each of the above tests at least five times at each test location on each laptop, and most of the tests were run dozens of times. The reason is that we were trying to measure best-case performance at each location, so we didn’t bother trying to average all of the results. We did notice a distinct lack of consistency across all wireless cards, especially on 2.4GHz connections, but as noted in the introduction, interference from other sources is nearly impossible to avoid. We report the best result for each test, which was never completely out of line with other “good” results. If we averaged the best five runs on each device, we’d be within 3% of our reported result, but we skipped that step in order to keep things manageable.

With the test parameters out of the way, let’s move to our first testing location and see how the various wireless devices perform.

Introducing Bigfoot’s Killer Wireless-N 1102 Netgear 2.4GHz Ideal Performance
Comments Locked

52 Comments

View All Comments

  • Aikouka - Wednesday, August 10, 2011 - link

    Jarred, if you asked me about a year ago... I never thought I'd recommend a Netgear product. My past encounters with their networking devices has been less than stellar. Even with that, I decided to take the plunge, and I purchased a WNDR3700 a few months back. It has treated me rather well so far and that's on stock firmware. It is DD-WRT compatible; so if you prefer their firmware, you're good to flash.

    Compared to the Airport Extreme, it is a bit weaker on the wireless front as it only supports MIMO 2x2, but its newer sibling, the WNDR4000, supports MIMO 3x3.

    The one feature I'd definitely argue for though is simultaneous dual-band. I don't think most people have homes with only 5Ghz devices. So with just a dual-band router, you'd be limited to 2.4Ghz anyway.
  • xand42 - Wednesday, August 10, 2011 - link

    Any idea why the (default parameter) netperf tcp transmit results are so horrible across the board? I just ran netperf on my Advanced-N 6200 notebook and got 190Mbit/s to my Cisco E3000 router, basically the same value as with iperf and netcat and every other benchmark.
  • Rick83 - Wednesday, August 10, 2011 - link

    It would be fairly interesting to see the difference between the Killer NIC, server NICs, laptop NICs and desktop NIC's from different vendors (and in different implementations).
    As I run only a SSD in my desktop, and have my user profile residing on a NAS (Intel i5 powered NAS, though, not limited by the available cycles and RAM) on a RAID 5, with reasonable linear throughput, network performance is quite important for me, as it accelerates such tasks as generating thumbnails of files, listing large directories, local decompression (that's probably the worst offender, and with large archives I do this on the NAS directly) and many tasks that access my user profile that may be latency or throughput limited.

    For this reason I recently specifically got a mainboard with Intel network adapter, hoping for enhanced performance, but only realizing, that I'd have to hack the .inf to get driver support under my OS.

    On the NAS, I am currently using the RTL 8111D chips, one connected to the switch and the other to the modem. If using a decent chip increases my samba/nfs performance, I'd put down the 50 euros for the intel pro chip in a blink.

    Also you could test supported cable length, jumbo frame support, documentation (with some cards the maximum MTU is not properly documented, and it takes ages of non-fragmented pinging to discover the correct MTU) and performance of teaming and fail-over mechanisms.

    Also, 5Ghz was something that I tried in my parent's home, but we couldn't even connect a machine that was one ceiling and wall away, at less than 10 meters distance. Might have been just a spectacularly bad router (Linksys 320N) or maybe 5GHz is really just for line-of-sight applications.
  • pityme - Wednesday, August 10, 2011 - link

    Jarred,

    I think it would be nice to run tests of multiple wireless network interference like the kind observed in apartment/townhouse/condo situations. This is a big problem that no one seems to ever test/talk about.
  • ckryan - Wednesday, August 10, 2011 - link

    True that. I live in a condo building in uptown Charlotte, NC and every unit has a wireless router of one kind or another. To make matters worse, the elevators seem to interfere somehow -- along with every wireless phone and microwave, the acres of glass and the steel structure. To top it off, there's an office building right next door which has many wireless systems on its own.
  • ckryan - Wednesday, August 10, 2011 - link

    Sometimes it seems like someone has cast some manner of evil voodoo on WiFi as a whole.

    I bought a Killer 2100 NIC because I thought it was a good idea -- and it was on sale. It is better than Intel's on-board Ethernet (and better still than Realtek) if only just barely. In reality, I just liked the idea of dedicated hardware for my networking needs. It really did make a difference, and really is some good stuff, but probably not worth the substantial premium over garden variety PCIe NICs (for most people). I'm glad I bought it... but only because it was on sale.

    The Killer product line is in the position of having a product family which does perform mostly as advertised, but becomes a tough sell when you already have onboard ethernet or wifi. It has to be difficult to sell a consumer NIC that costs as much as a 80GB SSD or mid range video card. For $65 or $70 it's great. At $140, not so much. As such, I'm glad to see them trying to put their hardware inside other hardware, like networking cordon bleu. Shoving their "NPU" onto the PCBs of video and sound cards, motherboards, and laptops is a good idea and if nothing else add a little variety to the mix. It would easily be worth an extra $25 -$35 premium for a motherboard in any price range to get their NIC over the bog standard Realtek. It can't possibly cost more than the NF200 or Hydra chips on mid and high end mainboards and is surely worth an extra $20 on top of a laptop as well. Whether they can sell this particular device sans lappy at a reasonable price or even at all remains to be seen.
  • honvl - Wednesday, August 10, 2011 - link

    I'm on a wireless card and want to test my latency. Where can I find this GaNE tool?
  • Flunk - Wednesday, August 10, 2011 - link

    This result really surprised me, I was expecting results like their wired products (no difference). My guess is that the biggest factor is the support for more channels but the performance really does seem markably better. I'm gearing up to by a new Alienware laptop near the end of the year and I may very well check the box for a Killer-N upgrade.
  • StormyParis - Wednesday, August 10, 2011 - link

    Isn't there a wide gap between what tech sites review, and what users use ? I know it's part of the purview of a tech site to go for the 0.001%-of-users, bleeding-edge stuff, but... has anyone ever seen a review of mainstream Wifi adapters/laptop anywhere ? As one of the 99.999%, I'd be interested !
  • JarredWalton - Wednesday, August 10, 2011 - link

    Bigfoot offered, so I thought, "Hey, this could be fun." Well, it wasn't. WiFi testing is a real pain in the butt I've decided. Still, it would be good to look at some other products. We might come back to the subject in the near future; I'm going to see about soliciting a bunch of different WiFi adapters and see what turns up. Obviously, I have at least the 6230 and 6300 from Intel. I want to add the Killer 1103, some Broadcom stuff, etc. Really, I'd want to focus on just dual-band cards, though -- anything that can't do 5GHz becomes less interesting after playing with the Linksys E4200.

Log in

Don't have an account? Sign up now