Evaluating the generation of an ILP using RACR
Correctness Tests
Setup
The correctness test is split into two parts without invalidating the results, because of technical problems. Normally one would generate the ILP from the model, solve it afterwards and finally check the solution. Because solving is currently done with the command-line interface of GLPK, glpsol, either glpsol has to be invoked from within Scheme, or the generation and checking has to be seperated.
Currently, the tests are driven by a Python script. The following steps are executed:
- Get the valid test-case ids by executing ilp_test.scm with
ranges. - Create a test-case for each id, in which
- the ILP is generated and saved to disk by invoking ilp_test.scm with
run - the ILP is solved by invoking glpsol, creating a solution file
- the solution is check by invoking ilp_test.scm with
check
- the ILP is generated and saved to disk by invoking ilp_test.scm with
With this, the Python script does not have to know about the concrete test-cases, but merely invoking the same steps for each test-case.
Categories of test-cases
The test-cases are grouped by categories, each testing a different aspect of the ILP generation. In the following, the categories ids are listed along with a short description. The description r x (c * i * m) describes a system with r resources and c software components with i implementations having m modes each.
| ID | System | Description |
|---|---|---|
| 1 - 16 | 2x(1*1*2) | Basic selection within modes. |
| 30 - 36 | 2x(1*2*1) | Basic selection within modes. |
| 100 - 129 | 3x(1*2*2) | Basic selection within implementation and modes. |
| 200 - 207, 300 - 307 | 3x(2*2*2) | Component requiring another component. For the 3xx category, only the first implementation has a constraint for the required component. |
| 400 - 407 | 3x(1*2*2) | Resources with different resource types. |
| 500 - 503 | [2,3]x(1*1*2) | New resources entering the hardware system. |
| 600 - 605 | 2x([1,2]*[1,2]*[2,3]) | New components, implementations and modes entering the software system. |
| 700 - 702 | 30x(1*1*2) | Tree-like hardware structure. |
| 900 - 907 | varies | Unsolveable problems, i.e. negative test-cases. |
Profiling Tests
Setup
- Create the initial system state, i.e. the example ast
ea - Evaluate
(=to-ilp ea), i.e. generate the ILP and measure the time for generation - Change some hardware resources, i.e. do a rewrite of terminales representing the values inside the provision clauses of the respective resources
- Repeat steps 2 and 3 for different values.
Instead of changing the hardware, changing the software is also feasible, e.g. to simulate new components or contract runtime feedback. The latter example means, that a contract is re-calculated after the (monitored) execution of a component and its values are updated.