Model Verification

Ensuring model accuracy and validity is a critical task and there are a number of procedures that may be carried out in order to help obtain confidence in your model results. These confidence-building activities include:

·         Historical Tests: Comparing the results of the model to known historical values.

·         Extreme Value Tests: Testing how the model behaves when extreme values are inputted into it. Does the model behave as one would expect or does it exhibit wild behavior?

·         Submodel Tests: For large models it is sometimes possible to isolate one portion of the model and compare it to some other more detailed model designed for analyzing that specific feature.

In addition to these methods, Simgua includes a rich suite of features designed for documenting the assumptions that your model is based on and ensuring that these assumptions are not violated during model simulation.

For instance, imagine you are designing a model simulating an ecosystem. As part of this complex model, you may have a stock representing the quantity of water in a lake. When designing your model, you know that the value of the lake stock should never become negative. If it does, there is clearly something wrong with your results and you should analyze the model to see what is causing this error. Unfortunately, though this constraint is fresh in your mind when you are focusing on the lake, it is very easy to forget about it when you focus on a different item. When you move on to another part of your model you might inadvertently make a change to your model that violates the assumption about the lake. Possibly you add another flow that results in the lake stock becoming negative and, since your focus distracted, you do not realize this mistake until much later.

In order to prevent mistake like these, Simgua includes a set of tools for documenting and enforcing your model’s assumptions as you develop them. These tools include: Unit Checking, Constraint Checking, and Live Checking.  All of these tools can be accessed by clicking the model verification button that appears in the configuration pane when you select a stock, flow, converter, or parameter within your model. Once you have defined verification checks for a primitive, a checkmark will appear on the primitive’s verification button letting you know that primitive will be validated during the simulation.

Units are at the heart of the simulation method Simgua uses. The flows into a stock should have the same units as the contents of that stock. Similarly, you should be able to specify the units for the different parameters and converters in your model. This is an underlying assumption of System Dynamics models, and it is one that you should adhere to as you construct your models. If you do construct models with disregard to units, it is still possible to develop accurate simulations, however it is much more likely that you will create errors or your model will fail to reflect the underlying processes driving reality.

Simgua allows you to enforce units using its unit mechanisms. As you construct your model, you may define the units for the different parts of your model. For instance, you may add a stock representing a lake with units of Cubic Meters. You may have a river flowing into the lake with units of Cubic Meters per Second. When you run a simulation, Simgua will check to make sure that the units you have defined are consistent. If they are not consistent (for instance you try to subtract a number with units of Meters from one with units of Liters) Simgua will display an error and terminate the simulation.

When you define a number in Simgua, it is automatically defined as Unitless. When the value of a valued primitive (stock, flow, parameter, or converter) whose units are defined is referenced elsewhere in your model, Simgua will evaluate its equation. If the result of this evaluation is a Unitless number, Simgua will then assign the units of that primitive to the value. If, however, the value is not Unitless and the units conflict with the primitive’s units, then an error will be displayed. You may add and multiply Unitless numbers by and to numbers with defined units and the results will take on the units of the number for which units were defined. You may also define a number with specified units in your equations using a special, curly-bracket notation like follows:

The above equation defines a value of 2 meters. Simgua also contains the ability to carry out automatic unit conversion between certain sets of units. Thus if you subtract a value of 10 centimeters from a value of 2 meters the result will be 190 centimeters or 1.9 meters.  The following table illustrates some examples of unit operations within Simgua:

Simgua comes with a large number of units built into it. However, it also contains the ability for you to define custom units using the material primitive. For instance in a Predator-Prey model that simulated the interactions between wolves and rabbits, you could define a unit “Wolves” and another unit “Rabbits”. Then, for a parameter containing the number of rabbits eaten by a wolf per time period you could create a third custom unit called “Rabbits per Wolf” with units of “Rabbits/Wolves”. Custom units are created by adding Simgua material primitives to your model. Materials are very powerful and we will not discuss all of their features now.

When you create a new material primitive, a custom unit is defined with the name of the material primitive. Thus, if you create a material primitive called “Prey”, a new custom unit will be available also called “Prey”. You may then define the relationship of the new custom unit to already existing units. For instance, if you were defining a new unit type representing an Imperial Quart, you would create a new material can name it “Quarts”. Then you would add a new synonym with units of Gallons with an exponent of 1. Then you would enter a scaling factor for the unit of 0.25 (there are four quarts in a gallon) Alternatively, you could add a synonym of Meters with an exponent of 3 (Cubic Meters). Then you would specify of scaling factor of 0.000946. When specifying scaling factors and synonyms it is important to always direct your unit conversions in one direction. This means that if you have two units which can convert from one to the other, you should only enter a synonym and conversion factor for one of the units, not for both.

One additional point should be made. Sometimes you will have model equations where there is an intentional unit inconsistency. This can often occur for empirically derived equations. In this case, you may specify the units for the result of the equation and select “Override units” to cause the calculated units to be replaced with the specified units without displaying an error.

Simgua constraints allow you to very simply set the maximum and minimum allowable values for a primitive. Thus, for our lake example, we would set a minimum value of 0. If the constraints are ever violated during the simulation, an error message will be displayed and the simulation will be terminated.

Live Checks are a powerful method of creating arbitrary validations of your model. Each Live Check is a Simgua equation that should evaluate to True or False. Simgua will evaluate each of the Live Checks over the course of your simulation. If any Live Check evaluates to False, Simgua will display an error and terminate the simulation. Live Checks can also be multiline blocks of Visual Basic code and you should see the Advanced Simgua portion of the manual for more information about writing complex Simgua equations.

The following are two examples of possible Live Checks.  The first ensures that the primitive My Quantity always has at least twice the value of the primitive Their Quantity. The second Live Check ensures that if there are no animals in the Animals primitive, then the Consumption primitive is also 0.