The State of Simulation Testing - 2015 Results

  • sim-testing
  • clojure

Well, it’s been about a month since I initiated the State of Simulation Testing - 2015 survey, and the results are in. In total, 29 people responded, and the results were quite informative about the state of simulation testing today.

Since the number of respondents was on the low-side, I think the most appropriate summary would be a qualitative summary. For raw data, I encourage you to peruse the raw responses (names and companies redacted).

General Demographics

In the survey, I asked a number of demographic questions. There weren’t many surprises, but it won’t hurt to go over the responses:

In what capacity do you work?

Capacity Count Percent
At a company 19 66%
At a consulting company 5 17%
Self-employed 5 17%

What industry do you/your clients work in?

Industry Count Percent
E-Commerce 7 23%
Financial Services 6 20%
Healthcare 3 10%
Professional Services 2 7%
Apartment Rentals 1 3%
Automotive 1 3%
Charity 1 3%
Education 1 3%
Business-process Outsourcing 1 3%
IT / Data analytics 1 3%
Media applications 1 3%
Mobile 1 3%
Music 1 3%
Online entertainment 1 3%
Software Development 1 3%
Technology 1 3%

A lot of E-Commerce and Financial applications, as I had expected, but I was surprised by the myriad of other places people are applying the technique.

How big is your company?

Size Count Percent
1-5 employees 10 34%
6-25 employees 7 24%
25-100 employees 4 14%
100-500 employees 2 7%
500+ employees 6 21%

Results here were a surprise to me. I had expected predominately large organizations, but in fact the most predominant response was “1-5 employees”.

How would you categorize your company?

Category Count Percent
Startup 13 45%
Small-to-medium Sized Business 11 38%
Large Enterprise 5 17%

A little unexpected. I had thought large enterprise might be the biggest category, but in fact companies of all sorts are using simulation testing.

What is at risk if your systems fail to operate correctly?

At risk Count Percent
We stand to lose customers 14 48%
We stand to lose revenue 9 31%
We could face legal action 1 3%
We stand to lose developer/business time 3 10%
There would be no substantial impact 1 3%
All of the above 1 3%

As I expected, losing customers/revenue occupies the majority of the responses (nearly 80%). Elaborations ranged from direct revenue loss, to less direct, intangible loss. Some interesting quotes:

  • “We are a self-coaching tool […] if we’re down or buggy [then] folks [will] lose confidence in our tool and associate that with our approach and concept.”
  • “If there was any more than a 5 minute outage, we could lose a large contract and subsequently a large portion of our business.”
  • “Our customers' inventory flow would be incorrect, either too much or not enough inventory moving between trading partners.”

Other General Facts

  • 72% of all systems under test were Clojure systems.
  • Most respondents learned about simulation testing via Cognitect’s efforts (not surprising).
  • Most companies were introduced to simulation testing by a developer or team lead.
  • Nearly every company considering simulation testing performs at least unit testing, if not integration, and other internal QA. Not a big surprise, given simulation testing is somewhat of a more rigorous form of testing. I wouldn’t think a company not testing at all would considering going with simulation testing.

Who’s using it?

Of those the 29 responses, 21 have considered simulation testing (positively), 5 have implemented one, and 1 considered, but rejected. 2 respondents had not yet considered simulation testing internally.

Why yes?

Of the 5 respondents who already have simulation tests, these were some of the most serious customers. In fact, of one respondent I happen to know personally, I can almost certainly guarantee they’d make national (possibly international) news if they had a serious failure.

These respondents are also different in that they’re generally larger (25-500 employees), and generally classified their risks as either loss of customers or revenue. Interestingly, only one of these customers uses Simulant (and the one that does, uses a fork of it). I can only presume this is because a) “simulation testing” as a concept is not entirely novel, and b) because these established businesses needed to mitigate risks within their system before the advent of Simulant.

Why no?

The single respondent whose company decided against simulation testing, did so primarily because it “seemed like a lot of investment to try the approach.” This foreshadows “What Could Be Better”, but I can only presume this was because of the lack of maturity around tooling and lack of success-stories.

Why maybe?

The most common thread through respondents positively considering simulation testing was the existence of mission critical systems that were complicated and hard to test. Less common, but still prevalent was a general curiosity about the technique.

The most interesting response to me, by far, was a respondent who reported their system was an event-sourced system, that they could (relatively) trivially re-feed into a simulation test. I want to work on that system!

Not Considered

Only 2 respondents had yet to consider simulation testing. These two companies were both on the small side (< 25), and noted a lack of familiarity with the technique/benefits.

What could be better?

Rounding out the survey, I asked two general questions:

  • What could make simulation testing easier to implement or use, and
  • Do you have any general comments regarding simulation testing.

The responses to these questions are probably the most informative. Through nearly every response ran one common theme: we need more resources. Being as obscure a technique as it is, simulation testing lacks a lot of things techniques like TDD have going for them. This is supported by responses to “How would you rate your team’s skills…”, with 75% of respondents familiar with only the basics of simulation testing, or not at all.

respondents identified a lack of:

  • A body of knowledge – think blog articles, tutorials, videos, books, etc.
  • Helpful tools – think libraries, templates, and well-known best practices.
  • Public success stories – A tool in some ways. Let me explain: a few respondents noted the difficulty of making a business-case for simulation testing, primarily based on the lack of public success stories. It’s one thing to sell a fancy new technique on its own merits, it’s another thing entirely to do it without any big companies putting their name behind it.

Summary

In general, I’m really happy with how the survey turned out. I had been prepared for maybe 5-10 results, but 30, boy, that’s actually a workable number of responses. I’m also really happy with the depth of the responses I received–nearly everyone elaborated on why they made certain choices and how simulation testing could be better.

Going forward this year, simulation testing is going to be my primary focus. In fact, I’m building Homegrown Labs (my company) around it. If you want to learn more about simulation testing, I encourage you to sign up for my mailing list below. In the near future, I’ll be blogging about techniques, open-sourcing useful tools, and talking about how you can convince your boss to let you make take the plunge into simulation testing.

Like this post? Subscribe to my newsletter.

Get fresh content on Clojure, Architecture and Software Development, each and every week.

comments powered by Disqus