Simulation allows a rigorous, scalable, and reproducible approach to testing:

Statistical modeling

Simulation begins with statistical models of the use of your system. This model includes facts such as “we have identified four customer profiles, each with different browsing and purchasing patterns” or “the analytics query for the management report must run every Wednesday afternoon.” Models are versioned and kept in a database.

Activity streams

The statistical models are used to create activity streams. Each agent in the system represents a human user or external process interacting with the system, and has its own timestamped stream of interactions. With a large number of agents, simulations can produce the highly concurrent activity expected in a large production system.

Distributed execution

Agents are scaled across as many machines as are necessary to both handle the simulation load, and give access to the system under test. The simulator coordinates time, playing through the activity streams for all the agents.

Capture results

Every step of the simulation process, including modeling, activity stream generation, execution, and the code itself, is captured and stored in a database for further analysis. You will typically also capture whatever logs and metrics your system produces.

Validation

Since all phases of a simulation are kept in a database, validation can be performed at any time. This differs markedly from many approaches to testing, which require in-the-moment validation against the live system.

Separation of concerns

The separation of concerns above, and the use of a versioned, time-aware database, gives simulation great power. Imagine that you get a bug report from the field, and you realize that the bug corresponds to a corner case that you failed to consider. With a simulation-based approach, you can write a new validation for the corner case, and run that validation against your past simulation results, without ever running your actual system.

This talk will introduce simulation testing, walking through a complete example using an open-source simulation library.