Advanced test generation concepts are known to require more processing power or time to resolve in particular as model complexity increases during model development. Delaying the use of such concepts, simplifies in particular troubleshooting in cases where test generation does not reach expected coverage.
To make most effective use of time, users are advised to delay covering of large data sets or use of advanced test generation options up to the point where in principle the model is complete and coverage with basic test generation options is 100%, i.e., the basic correctness of the model has been validated. Note that use of large data sets including combinatorial data generally does not increase your functional coverage but purely increases data coverage.
Recommendation: Validate all functionality to be tested first and make sure it is covered in tests before using modeling constructs like large external spreadsheets or combinatorial data, and advanced test generation options like “Enable Only Finalized Runs” or “Enable Data Distribution”. |
Following the approach to validate models without using advanced test generation concepts faithfully, significantly simplifies troubleshooting for possible causes for lack of coverage due to
a) A default test target setting can actually never be covered due to model construction, which can be resolved by setting the target to “DON’T_CARE” in the Test Targets view in the Conformiq Test Design
b) Model construction or complexity, which can be resolved by either model restructuring or increasing the test generation time limit
When using advanced test generation concepts users should expect longer test generation times as compared to basic test generation options, i.e., use of advanced test generations options generally never reduce test generation complexity and time.
Recommendation: Ensure you have suitable hardware and use the 64-bit Creator installation when using advanced test generation concepts. |
With the “Enable Finalized Runs” option Conformiq Creator only ends test generation if after covering every target that test can also be ended in an explicit or implicit return node, i.e., even if a test target can be covered but a return node cannot be reached then the test is disregarded and target is reported as not covered.
Recommendation: Always first ensure 100% coverage is achievable with a model before generating tests with the “Enable Only Finalized Runs” option. Be aware that test generation may take significantly longer and increase your time limit settings. For model validation or coverage analysis disable in general the option to reduce test generation time. |
Surprisingly, in often it is not data conditions or constraints that lead to this increase in test generation time but instead it is the number of control explicit and implicit merge nodes encountered in flows before reaching a return node – the more merge nodes are encountered the longer the required test generation. The best solution to reduce test generation time is to restructure the model either by reducing control flow merges in activity diagrams or even by splitting modelled functionality across two Conformiq projects instead of keeping all in one.
Recommendation: In case that test generation time with “Enable Only Finalized Runs” becomes too large, restructure your model refactoring common control flows into sub activity diagrams and reducing the amount of control flow merges. In case of high model complexity split the model into multiple models. |
Ultimately there is always the possibility to accept the optimized test set produced instead of using “Enable Only Finalized Runs”.
The “Enable Data Distribution” option has been introduced to generate tests with data distribution for alternative values specified in input action constraints which are not used in any flow branching or other decision making in the model, i.e., as soon as for example a decision node compares an input action result to an alternative value then this value can no longer be freely distributed. Use of this advanced option can also impact significantly test generation time for larger models and should therefore only be used in final stages of modeling.
Recommendation: Use the “Enable Distributed Data” option only for test generation with validated models and when there is alternative values that can actually be distributed. |