Learn how the new engine helps with previous limitations and what are the trade-offs.

Engine Update Timeline

We will be releasing the new version of the Scenarios generation engine by the end of July. Upon release, both engines will be available side-by-side to give you time to transition. This project should have no impact on how the current engine works.

We will be retiring the old engine by the end of August.

How to activate the new engine

Overall, you will be performing the same steps you normally would to generate the scenarios table.

You will now see 2 options under the Scenarios section: “Original” represents the old engine, “Standard” – the new one. As mentioned, you can use both initially, but we will eventually transition to only supporting the “Standard” version.

What are the advantages of the new engine

Improved Speed; larger models with a lot of constraints will show larger improvements.
- In larger models and/or at higher coverage strengths, you will see scenario rows being “streamed” in real time (instead of waiting for the entire table to finish before rendering)
- In some instances, you will also seem improved value variety across rows (even if the incremental coverage % stays the same)
Full rendering; there will never be “any valid values” present in the generated scenarios.
Improved Constraint Handling; “No possible value” as the table element will never appear with the new engine.There were 2 types of NPVs:
- user error in constraints;
- complex logic (especially involving Skip constraints) that actually had viable values available in the scenario slot, but the old engine couldn’t figure it out.
The new engine solves the second issue automatically. For the first one, there will be a warning that certain values cannot be used in the test suite and the Scenarios table won’t generate.

What are the trade-offs of the new engine

Constraints and Forced Interactions alignment

As the consequence of improved constraint handling, it is no longer possible to have constraint vs forced interaction contradictions (previously doable when accepting the warning). That means certain implementations of one-off scenarios or n-way constraints are not supported by the new engine.

We are working on a dedicated option for n-way constraints and evaluating other approaches for one-off tests.

(IMPORTANT) Non-determinism

As you will notice, clicking the “Scenarios”->”Standard” link multiple times throughout the modeling process can result in different content and counts of scenarios even when it seems counterintuitive. For example,

Copying a model can lead to a lower or higher count;
Adding a value to the largest parameter can lead to a lower scenario count;
Adding a value to the smallest parameter can lead to a higher scenario count;
Removing a value can lead to a higher scenario count;
Replacing a value can lead to a lower or higher count.

That is because the new algorithm searches for the approximate optimum and uses some non-deterministic variation in the process. In other words, the engine “knows” that the optimal number is in the ballpark of X, but doesn’t prioritize always returning exactly X (unlike the old engine) – hence the trade-off between accuracy/consistency and speed/other benefits mentioned above.

Therefore, the scenarios chosen for each iteration will be slightly unique – e.g., if you run the uncached (see below) generation 10 times, you can have 10 different outcomes – even if the model shape remains more or less the same. The count difference in these counterintuitive instances is usually within 5.

The only (guaranteed) unchanging scenarios are those in the cache, and any change to the model shape will break the cache. For example, the following will force a “fresh” generation:

Adding/Removing parameters
Adding/Removing values
Adding/Removing/Editing constraints
Adding/Removing/Editing forced interactions
Copying the model/reverting to a different version of the model

The examples of model actions that do not break cache:

Editing value expansions
Renaming values
Navigating between tabs
Adding notes
Editing scripts

Importantly, just switching the computation strength within the same engine (e.g. from 2-way to 3-way) does not remove the 2-way cache.

Equally importantly (for the transition period), switching between engines will wipe out the cache (so, going Standard -> Original -> Standard can have new results/count).

A couple of aspects to keep in mind:

No redundancy – despite the count differences, there will be no duplicate scenarios unless you intentionally include them via forced interactions. The delta will just tap into higher coverage strengths.

Freezing – once you are happy with the set of scenarios, if you want to maintain consistency for execution, we recommend using the “Freeze these Scenarios” option in the top-right of the Scenarios screen. That will make sure that any changes due to model updates and non-determinism are happening at the bottom of the table, to newly added scenarios, where they can be more easily incorporated or ignored.

Keep in mind that Freezing does not affect the functions built on top of values (e.g. value expansions, ranges). Those elements may show different choices when a frozen set of scenarios goes through regeneration. The more value expansions there are and/or the wider ranges are, the more likely you will see discrepancies.

Analysis, Scripting, and Export – the Coverage Matrix/Graph and scripting/export options will adjust to the latest scenarios count & content for the selected strength.

July 2024

A new “Optimized” compute option for the engine that was released last year is now live.

Key points:

Selective Activation:
- The "optimized" compute option will not affect existing computations automatically.
- Users can select the "optimized" option from a dropdown menu on the Scenarios screen.
- To activate, users must not only select “optimized”, but also fill in a text field with the word “optimize” and confirm it. This intentional extra step ensures that the optimized compute is used sensibly.
Enhanced Computation Process:
- The optimized scenario option runs the computation on the same engine 50 times with a single click.
- It selects the outcome with the least number of tests, often resulting in slightly reduced scenario counts compared to the standard option.
- The non-deterministic variation is crucial for achieving this optimization.
Cache Management:
- Switching between standard and optimized options does reset the cache. For example, if you calculate using the standard option, switch to optimized, and then back to standard, a new data table will be generated, therefore, creating a new set of scenarios.
- Staying with the same compute option does not reset the cache. For instance, if you calculate using the optimized option, switch to another screen without making any cache-resetting edits, and then return to optimized, the same table will be retrieved.
Recommended Workflow:
- Utilize the standard scenario computation while the model is undergoing edits.
- Utilize the optimized scenario computation when you are ready to export the final results to include in test executions.