...
Now how do we know when have we gone too far and modelled too much detail? One way to get an understanding about that is by measuring model complexity. Measurement of model complexity does not provide all the answers but it also helps to create models that can be maintained by multiple people – a model that more people understand than just the author. Complex models usually lead to longer test generation times and longer turnaround times to analyze and validate your model based on test generation results. Therefore managing model complexity saves your valuable time.
5.1 Measuring Model Complexity
There is no single simple way how to define or measure model complexity. Clearly there is a point in the lifetime of a model where the model becomes “too big” to understand even by its author, i.e., where analysis such as “why is target not covered” becomes too time consuming. An easy rule of thumb is that: if your peer can not easily review and understand how generated tests map back to your model then it is very likely to be too complex.
Note that increased test generation times generally go hand in hand with higher model complexity but model complexity ironically is often not the actual root cause for longer test generation times. Commonly improper use of (in particular) advanced model language constructs are the main cause for longer test generation times.
Four simple ways for measuring model complexity are:
a. Number of testable requirements referenced in a model
Benefits: Requirements (and their complexity) are generally well understood among all stakeholders. The review of poorly worded or open ended requirements with other stakeholders leads to the benefit that if requirements get improved then all stakeholders benefit from that – not just the tester. Requirements and their coverage are easy to monitor and trace through entire testing process.
Challenges: Complexity and quality of requirements often differ depending on their source, i.e., one requirement can actually mean ten or more requirements from testing point of view (like “the application should work well”). Another challenge is that many testers lack to this day availability (and access to) requirements in a format that can be easily automatically processed, e.g., ALM or Excel. This measure focuses on overall model complexity but cannot help to identify diagram complexity or unused definitions.
...
Recommendation: Before assessing number of requirements or even using them in a model normalize them to a common granularity, i.e., break complex requirement up into smaller actually testable requirements. Use a spreadsheet to keep track of testable requirements if you do not have access to a requirement management system.
b. Number of tests generated from a model
Benefits: Extremely easy to measure and understand by all stakeholders.
Challenges:Thismeasureiseasily“polluted”byuseofexternalspreadsheetand/orcombinatorial test data. It focuses on overall model complexity but cannot help to identify diagram complexity or unused definitions.
c. Diagram structure, i.e., number/nesting depth of activity diagrams, number of nodes/data objects/variables per diagram, control flow fan in/out per diagram, number of advanced modelling constructs, number of widgets per screen, etc
Benefits: Easy to measure – note that some parts of diagram complexity are already measured and reported today in activity diagram editor (but usually ignored). Helps to catch and reduce modelling flaws introduced by modelling complexity.
Challenges: Requires understanding of model to understand metric – meaningful to the user but not really to other stakeholders like managers. It is also difficult to completely avoid complex diagrams if it is complex functionality which is to be tested.
d. Test Generation Time, ie., time taken for test generation to provide 100% test target coverage and produce an optimized test suite.
Benefits: Easy to measure. Test generation time is a factor of various parameters and includes the complete model construct, including the diagram structure, combinatorial data, number of requirements etc,. Since Creator treats different models with its own internal logic while generating tests and is the only constant across different models, the total time taken serves as a good comparative parameter.
Challenges: Test generation time is not known until model is complete and test generation process is invoked. Also, test generation time can easily vary depending on Conformiq options. Test generation includes those AD which are referenced by Main AD (directly or indirectly); however, Creator includes “all” files in “model” folder in various levels of processing.
...
Recommendation: User should measure model complexity based on a mix of criteria which allows all stakeholders to understand the complexity of each model.
...
Recommendation: Generate tests frequently to review test generation time, and not until all the model is complete. Also, set identical Conformiq options across models while doing complexity comparison across models.
5.2 Limit Model Complexity per Creator Project
After selecting a way to measure model complexity users should define limits to what they consider reasonable limits. Consider using different limits for junior vs senior modelers.
...
Recommendation: Limit main complexity measures per Creator project as follows:
Do not exceed and try to cover more than a maximum of 100 testable requirement references
Do not exceed a maximum of 100 tests to cover model logic, i.e., not counting tests whose only difference or value is to cover combinatorial or spreadsheet data
Do not exceed 20 activity diagrams project and do not nest them deeper than 5 levels.
Less than 20% of all activity diagrams should be of high diagram complexity.
Keep diagram complexity in line with your knowledge level
Note that the limits of 100 requirements and 100 tests are not coincidental – we should target to have at least one test per requirement. In other words we can use the number of tests as a basis to reason about requirements even if they are not explicitly referenced in a model. If you want to increase the number of tests per requirement, then rather reduce the maximum number of requirements than increasing the maximum number of tests.
5.3 Restructure and Distribute Complex Models across Multiple Creator Projects
What to do when my model complexity exceeds recommended limits? In the case of high diagram complexity the cause is frequently the use of advanced modelling constructs like combinatorial actions, data tables, (high) variables. Ask your peers of similar or more advanced skill level (e.g., your CoE team) for a model review and second opinion to ensure you are using modelling constructs as intended. Check if the model can be simplified by restructuring it.
...
Recommendation: When encountering high diagram complexity and you are not able to resolve coverage issues ask your peer or even CoE team for a review and try to restructure and simplify your diagrams. Do not only contact experts in times of crises, but schedule regular model reviews with them.
In many cases, models start to exceed complexity limits because they simply “grow too big”. In such cases, split off functionality (generally activity diagram structuring already tells a lot about how to split functionality. Remember to use structure diagram libraries (see “section Use Structure Diagram (SD) Libraries to Reuse Interfaces across Multiple Projects) to have a single point for maintaining changes or extensions to interfaces available for testing.
...
Recommendation: When exceeding model complexity in general, split the modelled functionality, use structure diagram libraries to use the same structure diagrams in multiple Creator projects, and move activity diagrams to a new Creator project.
If your manager sets your objective, e.g., to generate 1000 tests, then plan in advance and think about how you could break up functionality to be tested into 10 projects. Maybe at the end it will be more or less projects but that is not the point. Split based on specification related information, e.g., your task is to test an application then create one project per module or feature file or product requirement definition or alike (or any other meaningful minimum number of such).
...
Recommendation: For new projects ideally plan as early as possible how to distribute functionality to be tested across different projects, i.e., do not wait for model complexity warnings to come up. Then implement project by project and this way learn from project to project more about good and bad ideas to construct your model. After the first three projects you will be a lot smarter!
5.4 Make Model Complexity Transparent to Help Everyone
Model complexity should not only concern users but also management. It helps all parties involved to understand why modelling or test generation is taking longer for person/model A than for person/model B. Model complexity (especially diagram complexity) is not generally “bad” and may even be required to solve a complex problem in some cases. However, in many cases it has been proven in the past the transparency has marked a turning point in the overall discussions and planning where the effort required to work on a model starts to become unreasonable.
Use and tracking of model complexity also helps in project planning, e.g., how much effort will (still) be required to model and generate tests for this functionality. It can be used to see how closely users follow best practices or to identify a good time for a model review or more training.
...
| Page Tree | ||||||||
|---|---|---|---|---|---|---|---|---|
|