In this Clear Choice test -- Huawei's first public outing in a North American setting -- the world's largest telecom vendor took the humble approach, supplying a pretty basic managed layer-2 switch that is a key building block found in every enterprise wiring closet.
The S5700-52X-PWR-LI-AC (catchy naming isn't a Huawei strong suit) features 48 gigabit Ethernet ports and four 10G Ethernet uplinks. It supports the standard protocols you'd expect in a layer-2 switch, it's excellent on power consumption, and performance is good. Importantly, we found no obvious security vulnerabilities. The list price of the switch, as tested, is $9,000, and we'll assume the street price is much lower.
But given a market already awash in wiring closet switches, we wonder if two extra uplinks and low power usage will be enough of a differentiator. Between that concern and a few rough spots we encountered during testing, the S5700 represents a modest entrance into the North American market. Huawei hit a single here, not a home run.
USABILITY AND MANAGEMENT
Huawei's command-line interface (CLI) is different from the oft-copied Cisco format. There are cheat sheets available that map Cisco and Huawei commands, and some parts of the configuration file do resemble Cisco's format (for example, in interface naming and grouping syntax). Still, this will be a different environment for network engineers familiar with Cisco syntax.
The CLI will look familiar to users of HP's high-end switches. Huawei formerly was in the H3C joint venture with 3Com, which bought out Huawei's share and which in turn was later acquired by HP.
A useful feature in the CLI's configuration mode is the ability to view configuration details in any context with the "display this" command. There are also ways to do this with Cisco and Juniper gear, but those commands involve more typing than just "d th".
In other aspects, device management isn't as polished as some of the competition. The switch lacks a dedicated Ethernet port for out-of-band management, while almost all competing enterprise switches have one. The CLI supports piping of command output like other enterprise switches, but not for all commands.
Overall, though, the switch offers a decent set of device management tools. It supports all the major management methods (over IPv4 only), and has multiple rate-control mechanisms to thwart denial-of-service attacks. It can block against rogue DHCP server and ARP spoofing attacks. Another plus: Huawei offers free lifetime updates for switch software (See Features Spreadsheet below.)
Click here to view the Huawei S5700-52X-PWR-LI-AC performance chart.
POWER CONSUMPTION
Simply put, this is one seriously power-efficient switch. We ran our usual suite of power-consumption tests, measuring watts consumed when the switch was idle and fully loaded, in the latter case with the Spirent TestCenter traffic generator blasting 64-byte frames at line rate to all ports.
In the idle test case, the switch consumed just 59 watts. That's the lowest power usage we've ever seen in a gigabit top-of-rack switch. Fully loaded, power consumption edged up minimally to 61 watts.
Huawei clearly has worked on power efficiency, and it shows. There are cases where power consumption might be higher -- for example, when using an optional redundant power supply or when using the S5700S-28P-HI version of the switch that supplies up to 1,440 watts of power over Ethernet to attached devices. On its own, though, the S5700-52X-PWR-LI-AC is very efficient.
As for the heat generated by the switch, it's vented on the side like most enterprise campus switches. If airflow is an issue, network managers will have to step up to a data center-class switch for true front-to-back (and reversible) cooling.
We also measured the switch's sound levels. It runs at around 70 dB once booted, and at around 80 dB during startup. In a data center or wiring closet, that won't matter at all. In an office setting, it's noticeable but not overpowering.
ABOUT SECURITY
There are many myths about Huawei security, with charges of supposed backdoors and magic kill packets, but no concrete evidence to back up the allegations. This article won't add to the mythology: Our security testing and monitoring found no vulnerabilities.(Network World tester Joel Snyder's analysis of Chinese security fears remains the definitive exploration of real and imagined concerns.)
We assessed Huawei switch security three ways. First, we conducted a "fuzzing" test of the switch's SSH daemon using a Spirent Mu-8010 security analyzer. With SSH fuzzing, the test tool iterates over every field in the SSH protocol headers, generating bogus values in each field (and beyond -- the fuzzing suite also injects excess data in an attempt to create buffer overflows).
We ran additional checks to determine whether SSH fuzzing attacks would interrupt other control- and data-plane operations. Concurrent with the fuzzing test, we generated around 30 SNMP queries per second, and simultaneously offered traffic at 10 percent of line rate to 50 ports (two 10-gigabit Ethernet and 48 gigabit Ethernet).
While the initial test run appeared to find hundreds of issues, none of these represented actual exploits (see "What's a Vulnerability?"). Every fault found by the analyzer simply meant the SSH daemon was too busy processing existing tasks to handle a new request. In no case did any fuzzing attack succeed in a device compromise.
Our other checks validated this: The switch continued to forward data-plane traffic without loss, and it responded to all SNMP queries. Monitoring showed the switch CPU pegged at 90% utilization or higher during the test. In our view, an SSH server is doing the right thing when it doesn't answer new requests because it's busy with existing ones.
In the second security test, we ran the SSH modules from Metasploit, a well-known vulnerability assessment and penetration testing tool maintained by Rapid7. We loaded Metasploit onto a Linux virtual machine running on an iX-2212 storage server supplied by iXsystems.
Here again, SSH testing turned up no vulnerabilities. The tool reported one cracked password, but only because the password was an easily guessable term made up on the spot by Huawei's test engineers. IT professionals who use easy-to-guess passwords in production networks are asking for trouble; we don't consider Metasploit's password discovery a problem.
Our third check of device security involved monitoring all traffic sent to and from the switch, which resided on an isolated network, throughout the entire test project. In more than a month of testing, our logs showed no evidence that the switch sent or received traffic beyond what we'd allowed. Moreover, a packet sniffer running for 24 hours on a switch port showed the switch emitted no frames at all with spanning tree and discovery protocols disabled.
We would never claim any device is secure. The standard weasel words apply here: All we can say is that this switch, running this specific version of software, was not vulnerable to the specific attacks we launched. Actually, we can say one other thing: Our results don't support the notion that a device is automatically less secure because of its country of origin. That's still just a myth.
PERFORMANCE TESTS
At this point, performance testing of any gigabit top-of-rack switch should be pretty cut and dried. After all, switches are usually built around merchant silicon ASICs -- and for more than a decade those chips have run at wire speed with low latency. End of story, right?
In Huawei's case, that's mostly true, but not entirely. On the plus side, the switch introduced relatively low and consistent latency in all unicast and multicast tests, and it delivered wire-speed throughput in the test cases we ended up using (see the figure).
But getting there was a little rocky, as was some of the control-plane testing. The good news is that the suboptimal results we saw in benchmark testing are unlikely to have any impact on production campus enterprise networks.
The switch didn't run at line speed in our first set of tests involving 64-byte frames. That was with four 10G Ethernet uplink ports in one full-mesh pattern and 48 gigabit Ethernet downlink ports in another full mesh. We reran the test with two 10G Ethernet uplinks, similar to the topology we've used with other gigabit top-of-rack switches, and again saw frame loss with 64-byte frames.
When Huawei's engineers changed the switch configuration to hash on IP addresses instead of MAC addresses, all was well: The switched moved all traffic, at any rate and with any frame size, with zero loss. We'd recommend configuring the switch to hash on IP addresses; Huawei told us they were considering changing this to the default.
No switch in an enterprise campus setting will handle only 64-byte frames at wire speed on all ports. Still, the RFC 2544/2889 benchmarks have been a sort of infant mortality test for switches for well more than a decade. It's always surprising to see a top-of-rack switch run at anything less than line rate.
Control-plane testing turned up another surprise, but again it's unlikely to have an impact in campus networks. Huawei's data sheet claims this switch's MAC address table can handle up to 16,000 MAC addresses. And with just two ports and nonrandom MAC addresses involved, that's true. But when we reran the test with all 48 gigabit Ethernet ports, the switch learned at most around 10,000 MAC addresses. Any higher number led to hashing collisions in the silicon.
The switch also learned fewer addresses when we used pseudorandom patterns like those described in RFC 4814. We got to 10,000 addresses only with the use of sequential MAC addresses -- and those aren't found in production settings.
In a data center, where virtualization drives the need for big, flat layer-2 networks, this would be a potential showstopper. In an enterprise campus setting, it's a nonissue.
A campus switch typically handles a few hundred to a few thousand MAC addresses, and this switch does that. The only concern is that the probability an address won't be learned is slightly higher due to the hashing algorithm in use by the switch's ASIC, which seems not to like randomness. (Note to ASIC designers: Switch vendors don't get to choose which MAC addresses their customers will use.)
MULLING OVER MULTICAST
When handling multicast traffic, the switch did fine with 400 IGMPv3 groups. It forwarded traffic at line rate to all groups on all 48 gigabit Ethernet ports, and latency was relatively low.
Another multicast test examines group capacity, and the results were only middling. The switch can handle a maximum of 400 multicast group addresses if there are subscribers to all groups on all ports. That limit is lower than five of seven similar switches in a Clear Choice test from five years ago. Huawei says the switch can support up to 1,000 multicast groups, and that's true -- provided we used only 16 of the 48 gigabit Ethernet ports.
Here again, this is really a matter of specsmanship, and unlikely to be a problem for most campus networks. In a data center, multicast group counts can rise into the thousands; in campus networks, typical numbers are in the hundreds or (more likely) dozens.
NETWORK MANAGEMENT
Although this is a switch test, Huawei also asked us to assess eSight Standard Edition, its $7,000 enterprise network management system (NMS). We ran eSight on a Windows 2008 Server virtual machine, hosted on an iX-2212 storage server supplied by iXsystems.
Using SNMP and ICMP, eSight collects monitoring and performance data from Huawei and non-Huawei devices, including specific templates for Cisco and HP/H3C gear. Commendably, the NMS includes predefined templates for common tasks like tracking CPU and memory utilization, interface traffic, and top talkers.
ESight includes tools for autodiscovery, mapping and reporting. It manages VPNs, terminal access, device configurations, and licenses, and it can import lists of devices from an Excel spreadsheet. It supports a hierarchy of management users. Featurewise, eSight is a reasonably complete NMS.
Functionally, though, eSight's implementation could be better. Although eSight is web-based, it only works with Internet Explorer and Firefox browsers, not Chrome or Safari. The "protocol template" pop-up window wouldn't close with either the cancel or close-window controls. A Javascript control in the NMS menu hung a Firefox browser.
Also, both in eSight and the switch CLI, there are multiple places where it's clear English wasn't the developer's first language. For instance, when exiting eSight, the NMS asks "Are you sure to logout?" While this is seemingly trivial stuff, with no functional impact, appearances count. We're sure Chinese customers could point to multiple cringe-worthy examples of poor translations from U.S.-based vendors. Here, the same is true in the opposite direction. Huawei would improve its prospects in English-speaking markets with better English in its user interfaces.
With the S5700, Huawei makes a solid if unspectacular entrance into North American networking. On features, performance, price, and especially power consumption, it may be attractive to some buyers. On security, it's certainly not the Black Box of Terror it's sometimes made out to be.
On the other hand, there's no true standout feature of the switch, and there were places where we wished the software were more polished. This is a decent, workmanlike effort -- but for Huawei to succeed in North American networking it will need to bring some sizzle along with the steak.
How we tested the Huawei switch
We tested the Huawei switch for performance, features, usability, power consumption, and security.
To evaluate device performance, we used the Spirent TestCenter traffic generator/analyzer to run the industry-standard benchmarks described in RFCs 2544, 2889, and 3918.
For unicast traffic, we assessed the switch in terms of throughput and latency by offering traffic on all ports at line rate. We also determined unicast MAC address capacity.
In the throughput/latency tests, we used separate traffic patterns for gigabit and 10-gigabit Ethernet ports. Initially, we used a four-port fully meshed pattern for the 10-gigabit Ethernet ports, but later scaled that back to a port-pair pattern. We offered fully meshed traffic on all 48 gigabit Ethernet ports.
All tests involved the seven standard Ethernet frame lengths recommended in RFC 2544 -- 64, 128, 256, 512, 1,024, 1,280, and 1,518 bytes -- and the test duration was 60 seconds in all cases. Also as recommended in RFC 2544, we measured latency at the throughput rate.
To measure MAC address capacity, we used the RFC 2889 wizard in Spirent TestCenter. This wizard conducts a binary search to find the largest number of MAC addresses a switch can learn without flooding. In all test iterations, Spirent TestCenter's MAC address aging timer is set to twice that of the switch under test. We ran the RFC 2889 wizard on three ports, and then manually repeated the test on 48 switch ports. We determined MAC address capacity to the nearest 1,000 addresses.
We measured multicast performance with tests of throughput and latency, and of multicast group capacity. For throughput and latency, we followed the aggregated multicast throughput procedure described in RFC 3918. We configured the Spirent TestCenter test tool to act as an IGMP querier, and to offer multicast traffic to one of the switch's 10-gigabit Ethernet ports. We also configured the Spirent tool to join the same 400 multicast groups on all 48 gigabit Ethernet ports, using IGMPv3 join messages. After sending the join messages, Spirent TestCenter offered traffic to all subscribers at line rate. Here again, the test duration was 60 seconds. We measured throughput and latency using the seven standard Ethernet frame lengths from RFC 2544.
To measure multicast group capacity, we used the RFC 3918 wizard in Spirent TestCenter. This wizard joins a fixed number of groups, and then attempts to forward traffic to all groups. The test instrument used a binary search to find the highest number of groups joined successfully. The test passes if the switch forwards traffic to all groups. If the switch fails to forward traffic to one or more of the groups joined, the test fails. We used the most stressful possible condition of having receivers on 48 gigabit Ethernet ports concurrently join all groups. Spirent TestCenter offered multicast traffic to one 10-gigabit Ethernet port.
We measured power consumption using Fluke 335 clamp meters. This test involved three measurements: AC line voltage; AC amperage when idle; and AC amperage when fully loaded. We fully loaded the switch control and data planes by configuring Spirent TestCenter to offer traffic at line rate to all ports. We derived wattage by multiplying voltage and amperage.
To evaluate device security, we used Spirent Mu-8000 security analyzer to run a protocol mutation test on the switch's SSH server daemon. In a mutation test, the Mu analyzer iterates over every field in SSH headers, injecting unexpected and/or illegal values, and then assesses the response of the device under test. Taken individually and together, this test involves nearly 40,000 different mutations of possible SSH requests and responses.
We also used the SSH modules in Rapid7's Metasploit security assessment tool against the switch.
During both the Spirent and Metasploit security tests, we concurrently ran continuous SNMP queries and offered data-plane traffic at 10 percent of line rate to 50 ports (2 10-gigabit Ethernet and 48 gigabit Ethernet). The purpose of this additional traffic was to verify availability of other control- and data-plane functions while the SSH daemon was under attack.
Thanks
Network World gratefully acknowledges the assistance of vendors that supported this project. Thanks to Spirent Communications, which supplied its Spirent TestCenter and Mu-8010 test instruments for this project. Spirent's Liang Kan and Emil Moral also provided engineering support for this project. Thanks also to iXsystems, which supplied an iX-2212 storage server and the FreeNAS storage appliance; to VMware, which supplied its vSphere 5 virtualization software; and to Fluke, which supplied a Fluke 335 clamp meter for power measurement.
Newman is a member of the Network World Lab Alliance and president of Network Test, an independent test lab and engineering services consultancy. He can be reached at dnewman@networktest.com.
Read more about lan and wan in Network World's LAN & WAN section.
