Artillery: Finding open source success between dev and ops

For Artillery’s creator, the key to the popularity of the open source load testing tool is a focus on the needs of both developers and operations. Early bets on Node.js and YAML didn’t hurt.

Contributor, InfoWorld |

Artillery: Finding open source success between dev and ops — bigstock

Given enough cash and luck, you can manufacture a pop phenomenon like One Direction. But if you want to foster a popular open source project, no amount of cash is going to buy success.

Instead, as Artillery.io founder Hassy Veldstra related in an interview, often the best way to create the next big open source project is to pay attention to how enterprises work, and to look for rough edges that can be smoothed out. In the case of Artillery, an open source load testing tool, that meant focusing on the cross-functional interplay between developers and operations as it relates to software performance.

While there was no guarantee that Artillery would become a big success, Veldstra and the Artillery community made some early technology bets—on JavaScript and YAML, for example—that have proved to be prescient. It suggests they will continue to make smart investments that help improve cross-functional collaboration within the enterprise.

The key to Veldstra’s early bets, however, is something anyone can emulate: Get in the trenches, get hands-on. Here’s how that approach has paid off for the Artillery community.

Scratching an itch

Today Artillery sees several hundred thousand downloads each month, growing 15 percent month over month. What Veldstra started as a solo project in 2016 now attracts dozens of contributors and users from pretty much every industry you can think of—even a forestry service, Veldstra notes.

All of this stemmed from a conference talk Veldstra gave five years ago. He was talking about the chat app he was building but also mentioned the load testing tool he’d created to help with it. Pretty much all of the audience questions focused on the load tester, which he soon pushed to GitHub on a whim to see if there would be interest. There was. “I put it up in GitHub and before I knew it, I was getting PR requests, comments, and requests for new features,” Veldstra recalled. “It was great.”

Great—but also surprising, perhaps, because there didn’t seem to be a need for another open source load testing tool.

If you pay attention to the performance testing market, you’re familiar with Apache JMeter. Veldstra was, too. But while working on a chat application, he needed a different approach than JMeter or other options provided.

One problem with JMeter was its graphical user interface, which might be perfect for many users, but not for Veldstra. “As a developer, I wanted something that would play well with source control; something that would allow me to just write code,” he says. “I didn’t want to click around to assemble my script.”

JMeter’s use of “a really verbose XML format” made it even more unfriendly for use in source control operations, Veldstra adds. And there were other issues:

JMeter was very difficult to containerize because it’s written in Java. It’s quite a bit easier today, but back then it was a nightmare. And that was important because it was really difficult to run tests written with JMeter as part of your continuous integration pipeline. That was probably the number one requirement [for users] who want to run those performance tests regularly as part of the release process.

And it was very difficult to extend JMeter to test anything that wasn’t just HTTP. That was a very important requirement for us because we were writing this real-time system, which was speaking something other than HTTP, and at the time, I actually tried extending JMeter to add support for this protocol, and it was just too difficult.

And then the final reason was that it was very difficult to plug JMeter into third-party monitoring systems.

But Veldstra wasn’t simply thinking about his needs as a developer. He was focused on cross-functional collaboration with the operations team. In fact, in 2016 performance testing was considered “QA’s job,” something that developers didn’t really think about. This has changed. Today, there’s a “shift-left movement” to instantiate testing early in the development process.

Because Veldstra was in the trenches, he had a first-hand view of this shift, and just how much else was moving across organizational boundaries, and what these changes might mean.

Seeing around corners

Veldstra’s involvement with operations teams helped him to see, for example, that a load testing tool really had to support third-party monitoring systems, otherwise it wouldn’t be useful for operations folks. At the same time, the test scripts had to be really easy to write and read, so that non-technical QA people or product managers (who might not write code) could understand what was going on. This led him to adopt YAML.

“If you look at Artillery’s scripts, you don’t really need to know a lot about the underlying stacks that they’re testing,” Veldstra says. “They’re very close to English, really.” YAML has become de rigueur for the Kubernetes crowd, but back in 2016 it wasn’t necessarily an obvious bet.

As with YAML, Veldstra sensed the momentum building behind Docker and the cloud early on, when they were still relatively nascent. “You could see that’s the way things were going,” he recalls. “So it made sense to build something that would plug into those new workflows and support the direction in which the world was moving.”

Similarly, Veldstra’s big bet on JavaScript and Node.js seem obvious today, but Node.js was just a few years old in 2016. “It just made sense that there was going to be loads of libraries built for Node.js,” he remembers. If this proved to be true, it would make it really easy to extend Artillery’s load testing scripts.

The example of Artillery shows that open source innovation (or, really, any innovation) is most likely to happen when the developer, or operations person, is deeply entrenched in the problem. This gives them a better vantage point to see rough edges that need smoothing. “It really helps to be hands-on,” Veldstra advises. “I don’t know how to do it without being hands-on. As a practitioner, as someone who works with the tools every day and focused on developers, you get a sense for where things are going.”

In other words, technology innovation isn’t something a venture capitalist is going to be able to spot while crunching spreadsheets over her morning latte. It’s not something that will be manufactured in top-down fashion. It happens from the bottom up, and it starts in the trenches.

Scratching an itch

Seeing around corners

Read more about open source: