Our 2009 12-City 3G Data Mega Test: AT&T Won
Posted in: att, feature, new-york, Sprint, TMobile, Today's Chili, top, verizon, Verizon Wireless
Given carrier reputation and our own iPhone call drops, we were pretty surprised to discover, through careful testing in 12 markets, that AT&T’s has pretty consistently the fastest 3G network nationwide, followed closely—in downloads at least—by Verizon Wireless.
Let’s get this straight right away: We didn’t test dropped voice calls, we didn’t test customer service, and we didn’t test map coverage by wandering around in the boonies. We tested the ability of the networks to deliver 3G data in and around cities, including both concrete canyons and picket-fenced ‘burbs. And while every 3G network gave us troubles on occasion, AT&T’s wasn’t measurably more or less reliable than Verizon’s.
It was measurably faster, however, download-wise, in 6 of the 12 markets where we tested, and held a significantly higher national average than the other carriers. Only Verizon came close, winning 4 of the 12 markets. For downloads, AT&T and Verizon came in first or second in nine markets, and in whatever location we tested, both AT&T and Verizon 3G were consistently present. If you’re wondering about upload speeds, AT&T swept the contest, winning 12 for 12.
The Cities
Last year, we did an 8-city coast-to-coast test, and called Sprint the big winner. This year, we have results from 11 cities coast-to-coast, and even got to test (during what was otherwise vacation time) on the Hawaiian island of Maui. Also, unlike last year, we were able to test T-Mobile’s new 3G network, active in all the markets we visited (except, at the time, Maui). For being such a latecomer, T-Mo did well, and the numbers show even more promise from them.
We tried to spread the love around this year, geographically, hitting cities we didn’t get to last year (at the cost of losing a few from ’08). Besides Maui, we hit Atlanta, Chicago, Denver, Las Vegas, Los Angeles, New York, Phoenix, Portland, Seattle, San Francisco/Bay Area and Tampa.
The Methodology
Our testing regimen was based on the same scheme as last year: We picked five locations in each city, including at least one “downtown” location that was considered a suburb. The selections were arbitrary, or fixed but logical—landmarks, residences, etc. (Note: Due to timing constraints, Chicago and Maui only had three test locations.)
Our hardware consisted of two identical stripped-down Acer Timeline laptops running Windows Vista, and four 3G wireless modems requested from the carriers. We allowed them to make the choice of hardware, simply asking for their “best performing” model. Once up and running, here are the tests we ran:
• Bandwidth & Latency: Speedtest.net – Reports upload and download bandwidth in megabits per second, as well as ping latency in milliseconds. We performed this test five times at each location on each modem.
• Pageload: Hubble images at Wikimedia – A 4.42MB web page with 200 4KB thumbnails, it was fully reloaded three times, and timed using the Firefox plug-in YSlow. The three time readings were averaged.
• Download: Wikimedia’s Abell 2667 galaxy cluster photo – This single 7.48MB JPEG is a clear test of how fast you can download stuff from the cloud, and again, we hard refreshed this file three times, and measured time using YSlow for an accurate human-error-free reading.
This was a test of 3G performance. Even though Sprint and its tech partner Clearwire have intrepidly released 4G networks in half of the tested markets—Atlanta, Chicago, Las Vegas, Maui, Portland and Seattle—we only tested Sprint’s 3G network. The reason should be obvious: While we performed the test with laptop cards on PCs, it’s supposed to serve as a test of the network’s ability to deliver service to all devices, including smartphones, dumbphones and laptops. Show us a Palm Pre WiMax edition—better yet, sell 100,000 of them—and then we’ll switch it up. And while you may argue that this 3G test still doesn’t adequately reflect your experience with your iPhone, at least it’s the same network, and may serve to rule out AT&T’s data pipe as the independent cause for all those infamous dropped calls.
(On a side note, when multiple carriers release 4G networks, we’ll definitely conduct a comparative test of them all, using new parameters, and focused around laptop use.)
The Results
Now that you know how we ran the test, here are the top finishers in each market, plus some pretty bar graphs showing you how bandwidth compares.
Though we tested for uploads and downloads, we focused our additional tests on the downstream, as it’s the more important direction, in the minds of most consumers and most carriers. The anomaly there is AT&T, which has dramatically good upload bandwidth, even when its download bandwidth doesn’t keep up. Fast uploads are a priority for AT&T, and will soon be for T-Mobile, which recently turned on faster uploading in NYC, which you can see in our test results. Meanwhile, although Verizon technically came in second in uploads as well as downloads, it doesn’t seem to treat this as a major priority.
When it came to downloads, though, the competition was markedly stiffer:
Atlanta – AT&T, followed by Verizon
Bay Area/San Francisco – AT&T, followed by Verizon
Chicago – AT&T, followed by Verizon then Sprint
Denver – AT&T, followed by Verizon
Las Vegas – Verizon, followed by AT&T
Los Angeles – AT&T, followed by Sprint
Maui – Verizon, followed by AT&T
New York – AT&T, followed by T-Mobile
Phoenix – Verizon, followed by T-Mobile
Portland – T-Mobile, followed by Verizon
Seattle – Verizon, followed by T-Mobile
Tampa – Sprint, followed by AT&T
Is That The End?
No. We’ve compiled the following gallery with all the data from each test location in the 12 markets, so you can see on a neighborhood-by-neighborhood level who won what. This also includes latency, pageload and download numbers, so you can track the performance in several ways. (The data above is bandwidth, though as you’ll see, that was generally representative of the overall performance. If a carrier was tops in bandwidth, it was usually tops in download time.) These tests are all just “snapshots in time,” as the carriers like to say, so feel free to bitch about where your experience doesn’t reflect our results. We stand by them, but acknowledge that network performance is changing all the time, and experiences very regular hiccups.
Regarding latency, you’ll notice it didn’t appear to affect actual user experience—3G isn’t really up for Modern Warfare 2, if that’s what you’re thinking—we will gladly show you latency averages, as well as pageload and file download averages, broken out for every market on the test.
Special thanks to all of the excellent testers we enlisted, Mark Wilson, Chris Mascari, John Herrman, Kyle VanHemert, Dan Nosowitz, Matt Buchanan and Rosa Golijan from our own team, along with Tamara Chadima and the indefatigable Dennis Tarwood. You guys were troopers, and I’m pretty sure FedEx either loves you or hates you. Thanks to John Mahoney for helping develop the initial tests that we’ve continually refined, to Chris Jacob for mapping all the locations, and to Don Nguyen for the mad number crunching—you truly are a spreadsheet pimp.
Note: Some of you may have noticed that San Diego is among the cities highlighted on the top illustration—and that Maui is not. The reason is that while we did testing in three great San Diego locations, one of the locations didn’t get any Sprint or T-Mobile service, and the already fairly thin dataset was rendered too compromised for any kind of usable report. As for Maui’s absence, Maui’s just too far out in the Pacific to make for a pretty map shot.
By federal law, wireless carriers must allow consumers to port their phone number when switching to a different carrier.
