Providers address capacity, supply-chain challenges brought on by COVID-19

Service providers are adapting to changing traffic demands driven by COVID-19 quarantines by building up more capacity, expanding the locations of their points of presence, and even shifting their supply chains, according to several that discussed the measures during a virtual conference.

Network leaders from Dropbox, Equinix, Netflix and Zoom talked about their experiences during an event hosted by network-visibility and performance-management vendor Kentik on March 25. For the most part, they reported that their services and the underlying networks they rely on are holding up well, but they have also had to make adjustments.

Scaling capacity quickly, strategically

The popularity of Zoom’s conferencing service has surged along with the coronavirus, as coworkers, families and friends try to stay connected while maintaining physical distance. Alex Guerrero, senior manager of SaaS operations at Zoom, says the company was in a good position to deliver before the outbreak. “Luckily, we were kind of ahead of the curve, as we had probably 50% more capacity than we needed at our maximum usage to begin with. So we try to keep that [buffer] in place,” Guerrero said. “The last couple of weeks, we’ve been all-hands-on-deck to keep that standard.”

The general philosophy has been to procure as much bandwidth as needed. At the same time the company is trying to think strategically about how it wants to scale over the long term. For example, Zoom uses Equinix’s network-interconnection platform to establish private connections to its technology partners and service-provider customers.

Guerrero is looking at ways to improve its proximity to end users by augmenting Zoom’s peering relationships, ordering more transit, and increasing bandwidth on existing interconnections. “Our product can handle a lot of latency, but the closer you are to the eyeballs, the better performance you’re going to get across the board. So that’s mainly what I’m looking at,” Guerrero said.

Zoom uses 19 data centers globally, where it’s connected to the biggest exchange points in each market. “But now we’re looking at the second-biggest, and maybe the third-biggest” to move even closer to more end users, he said.

Zoom hasn’t experienced problems scaling its public cloud resources, and neither has Netflix, which relies on AWS cloud services for its internal infrastructure. “We’ve invested an awful lot of time and energy and money into making that … quite scalable, in such a way that that we don’t feel like we’re stressing our cloud infrastructure by the current events,” said Dave Temkin, vice president of networks at Netflix.

Dropbox operates its own data centers and relies on AWS to burst capacity. But one challenge for Dropbox has been a shift in the way people access its platform. Before coronavirus, people tended to access Dropbox from corporate and university networks with highly concentrated user populations. Now users are dispersed, working from home and accessing Dropbox from many different locations. It’s driving the company to review its last-mile connectivity practices and consider investing further in peering relationships, said Dzmitry Markovith, senior director of engineering at Dropbox.

On the content delivery side, the internet seems to be scaling pretty well, Temkin said. Netflix, which operates its own content delivery network, has seen some degradation, but often it’s minor enough that services are still usable, he said. “We’re also seeing that generally nothing is absolutely melting down. It is scaling quite well, both our system and other people’s systems.”

Netflix announced plans to reduce the quality of its streaming service in Europe, which is expected to cut traffic on its European networks by 25%. “We’ve made some changes, as we’ve talked about publicly, to try to ease some of that pressure. But by and large, we’re seeing that things are holding up quite well,” Temkin said.

More broadly, data center giant Equinix spans 55 metro areas and 26 countries through its colocation and interconnection services, which gives it a unique view into the global impact of COVID-19 on network traffic. Around the world, Equinix has seen traffic increases ranging from 10% to more than 40% as the coronavirus began to spread. “The good news is that all that core infrastructure is actually scaling really well,” said Bill Long, senior vice president of product management at Equinix.

The surge in traffic came “at a sort of fortuitous time in [our] technology upgrade cycle,” Long said. Equinix is a few years into upgrading its networks from 10 Gigabit to 100 Gigabit links, so “luckily, there was a lot of extra headroom left and core capacity.” A lot of that headroom is filling up, but Equinix isn’t seeing many internet exchange ports hitting peak capacity.

He said Equinix customers are accelerating their long-range network plans by scaling up much faster than they intended. They are exploring Equinix’s IP-transit, peering, and virtual-network services, for example, to be able to accommodate some of these new and unexpected traffic flows, Long said. “What we were expecting to take two years is now taking two weeks or two months.”

Coronavirus impact on supply chain

In terms of physical infrastructure, Netflix had to overcome some supply-chain obstacles. “We have had multiple fires at this point with our supply chain,” Temkin said. For example, the primary server manufacturer for Netflix is located in Santa Clara County, Calif., where residents have been ordered to shelter in place. “We had 24 hours to figure out how to get as many of the boxes out of there as we possibly could,” he said.

Netflix has resolved those supply issues, for the most part, by sourcing elsewhere. “By and large, we’ve been able to use most of the infrastructure we have deployed. Partners like Equinix have been great about getting cross-connects provisioned quickly where we need them in order to get interconnects beefed up in certain markets,” Temkin said.

On the content-production side, there’s not a lot happening – at Netflix or anywhere else – as studios halt film and TV production to avoid further fueling the outbreak. “One of the big challenges we are trying to figure out is: what parts of it can we restart?” Temkin said.

Post-processing tasks for animation or visual effects, for example, are not traditionally done from home because they require significant compute power and bandwidth. “We’re undergoing efforts to figure out how we can get that operating out of people’s houses. And we’ve seen some early success in that space,” Temkin said.

In the big picture, Temkin has worked to avoid hoarding servers or network capacity at the expense of more critical services like healthcare, e-learning and collaboration. “I’ve asked my team to pursue [resources] wherever they possibly can and to pull things in. But we also recognize there’s a lot of other things going on that are more important than Netflix,” he said.

Temkin is also trying to help his team maintain balance and stay healthy. To that end, “it’s important that we don’t scramble everyone to get everything done right now,” Temkin said. “Everyone’s just trying to figure out what the new normal is.”

Learn more about tech response to COVID-19

• The impact of COVID-19 on public networks and security
• Tips for cleaning data-center gear in response to coronavirus
• Coronavirus challenges remote networking
• Coronavirus prep could prompt better disaster recovery
• VIDEO: Coronavirus, working from home and your home Wi-Fi network
• Coronavirus challenges capacity, but core networks are holding up