The problem with using broadband to back up branch office MPLS links is 1) you spend all that money on the pipes and most of the time they simply lie fallow, and 2) when MPLS does go down the failover process often takes so long it kills active sessions.
Greenbrier Companies and Hydrite Chemical Co. separately went looking for an answer to this common problem and found Talari Networks, a vendor that sells appliances that can amalgamate the capacity of MPLS and broadband links and share the capacity as necessary.
ANOTHER CASE STUDY: Real estate firm boosts bandwidth, cuts costs with Talari
Steve Eaton, senior network engineer at Greenbrier, says he is using Talari's Mercury appliances to increase reliability and add bandwidth.
Greenbrier is a $1.8 billion supplier of transportation equipment and services to the railroad industry. It makes different types of railroad cars (from tank cars to boxcars, flatbeds, etc.), and has a number of related business, including a rail car repair/maintenance business, and businesses for parts, hydraulics and wheels. It even has a railroad car leasing business.
The company, which employs about 5,000 people, has 45 sites on its MPLS backbone that is used to support a centralized VoIP system and different types of data traffic -- email, access to centralized databases and other back office applications.
Most sites, which have 10 to 20 people with PCs and phones, are connected via a single T-1, meaning if that link goes down operations are severely hampered. The failure that pushed Eaton to look for new answers was a 10-day outage at one of the rail car repair shops that stemmed from a hand-off problem between Verzion, the MPLS provider, and CenturyLink, the local link provider.
"We had SLAs but they just pro-rate the service fees and give you back a few hundred dollars," Eaton says. "The downtime, on the other hand, can cost you thousands upon thousands."
Eaton quickly ruled out dial-up as a suitable backup plan, but considered adding cellular modules to its remote Cisco routers, but determined that that would be too costly.
What's more, the company had another demanding capacity need pending: It wanted to add video surveillance at some remote sites and move away from an expensive video surveillance service. That would make link reliability even more important and drive up bandwidth needs.
Ultimately Eaton became aware of Talari, which sells appliances that would let him add DSL or cable modem links to each site for redundancy, while also using the secondary links in a load-balancing scenario instead of just reserving that bandwidth for backup. The appliances constantly assess the quality of both links and route traffic accordingly.
"Business-grade broadband connections are good because the service profile -- usually something like 3Mbps down and 1Mbps up -- matches the typically traffic to our remote sites, which tend to pull more stuff down versus post it up," Eaton says.
Today Eaton has Talari appliances installed at four locations with the short-term plan to add 10 more.
At the remote offices the appliances sit between the router and the remote office LAN. The T-1 for the MPLS remains attached to the router and the new broadband link -- which is running Eaton anywhere from $60-$90 per month. -- attaches directly to the Talari box. No firewall is necessary because the link tunnels back to HQ.
In use, most traffic still goes over the MPLS links. Eaton says the system is configured to defer voice traffic to MPLS, but the appliance samples for congestion and jitter on both links and can and does route traffic over the broadband links as necessary. Generally speaking, he is seeing broadband picking up about a quarter of all branch office traffic.
Eaton says he justified the purchase on the reliability factor and the support for video and is happy with the returns. Installation was straightforward and, other than a few minor routing issues, went without a hitch.
Hydrite
Like Greenbrier, Hydrite Chemical turned to Talari for much the same reasons: adding more bandwidth to its MPLS network and making the network more reliable.
Hydrite, a private company, manufactures and distributes chemicals used in food processing, industry, paper and other industries. It has 3,000-plus products and 10 offices spread across Wisconsin, Illinois, Indiana, Iowa and California.
All locations are supported by a data-only MPLS network with T-1 access links.
Most offices have 15-90 people, but two locations account for 80% of the branch office network traffic.
While MPLS typically provides 99% uptime, multiplied across 10 branches, even short outages add up, says Bob Fourness, computer operations manager at Hydrite. He wrestles with two scenarios: sporadic outages, where a branch goes down two to three times per year; and chronic problems, where there is an outage every week for two to three months.
While the company has added DSL backup on some of its remote routers, the failover processes takes two to five minutes, disconnecting the company's IBM 5250 terminal sessions, meaning users have to log back in when the connection is re-established and, oftentimes, have to call IT.
"We have a number of plants that work three shifts, so if that happened at night it would involve a call to one our people outside of normal hours to reset printers and that sort of thing," Fourness says.
Besides the terminal traffic to the company's CRM app on an IBM iSeries AS400, the network supports PC backups, Internet access (backhauled to HQ), Lotus Notes replication, terminal emulation and file sharing.
So reliability was a driver, but so was the need for additional bandwidth, in particular at those two main branches.
The first attempt at capacity planning for those branches saw the company install Riverbed WAN optimization tools. "While that made the pipes wider, they didn't address uptime," Fourness says.
Adding a second MPLS link to each office for redundancy was going to cost $1,000/location/month and add management complexity, which was enough of an incentive for Fourness to look for alternatives. Hydrite found that in Talari.
After piloting Talari's appliances in a few locations, the company decided to roll them out to all branches so they could use the additional broadband capacity and benefit from the increased reliability.
At one of the two larger locations, the one housing the company's newly consolidated customer service function, Fourness found a cost-effective fiber alternative to broadband.
"We're paying $800 month for T-1/MPLS to that location and we added a symmetric 5Mbps fiber link for $450/month," Fourness says. "Talari lets me use that direct-to-Internet access line to get back to headquarters, and I get the reliability of fiber, symmetric bandwidth (which I wasn't getting with DSL), and it is significantly less expensive than if I had doubled up on MPLS. Instead of 1.5Mbps, I now have 6.5Mbps and only pay 50% more."
While he can't find fiber deals like that for every branch, he says he was able to get a 10 meg link for another location for a little more than $600/month. Otherwise he is using mostly DSL.
And those DSL links are pulling their weight, he says: "I thought I would get half of the rated DLS throughput, but I'm seeing almost the same throughput on the DSL as I'm seeing through the MPLS T1 links at a fraction of the cost; $80 vs. several hundred."
Then there is the reliability factor. With two independent links to each site and the Talari box dynamically shifting between them as necessary, IT no longer gets calls in the middle of the night. "If an MPLS link goes down at 2 p.m., no intervention on our part is needed," he says. "The link might seem slower at the branch, but there is no interruption to the terminal session or printing."
Another side benefit: "Talari gives us much better monitoring, second by second, of our lines, and the DSLs have impressed me with up time and line quality. Previously I only had binary status information -- the link was either up or down. Now I can check the status of the circuit and assess the quality of the line."
That has proved to be much more useful than he anticipated. "I reviewed the Talari installation with our MPLS supplier and, on their own, they came back with a number of options on how they could increase bandwidth too us at a lower cost. The point being, my MPLS providers understand where this is going. They came back to us with a lower rate on higher bandwidth. They see the writing on the wall."
Fourness says he cost justified the move to Talari on avoiding the need to add more MPLS links, but is seeing additional payback in the form of more up time, less management overhead, and the fact that he doesn't have people working outside of normal business hours.
"I hadn't heard about Talari a year ago. But once I put it in, I wonder how I got along without it," Fourness says.
Read more about lan and wan in Network World's LAN & WAN section.
