|
# Evaluating the generation of an ILP using RACR
|
|
# Evaluating the generation of an ILP using RACR
|
|
|
|
|
|
## Setup
|
|
## Correctness Tests
|
|
|
|
|
|
- the initial system state is a generated system using the following parameters
|
|
### Setup
|
|
- `num-pe` – total number of hardware resources
|
|
|
|
- `num-pe-subs` – number of subresources per resource (i.e. breadth of the HW-tree)
|
|
The correctness test is split into two parts without invalidating the results, because of technical problems. Normally one would
|
|
- `num-comp` – number of SW components
|
|
generate the ILP from the model, solve it afterwards and finally check the solution. Because solving is currently done with the
|
|
- `impl-per-comp` – number of implementations per component
|
|
command-line interface of [GLPK](http://www.gnu.org/software/glpk/), *glpsol*, either glpsol has to be invoked from within
|
|
- `mode-per-impl` – number of modes per implementation
|
|
Scheme, or the generation and checking has to be seperated.
|
|
- possible other, not implemented possibilities to describe the system include
|
|
|
|
- randomness for number of Impls and Modes
|
|
Currently, the tests are driven by a Python script. The following steps are executed:
|
|
- minima and maxima for properties
|
|
|
|
|
|
1. Get the valid test-case ids by executing *ilp_test.scm* with `ranges`.
|
|
## Test
|
|
2. Create a test-case for each id, in which
|
|
|
|
- the ILP is generated and saved to disk by invoking *ilp_test.scm* with `run`
|
|
1. create the initial system state, i.e. the example ast `ea`
|
|
- the ILP is solved by invoking *glpsol*, creating a solution file
|
|
2. `(att-value 'to-ilp ea)` – generate the ILP, and measure the time for generation
|
|
- the solution is check by invoking *ilp_test.scm* with `check`
|
|
3. optional: solve the ILP and measure the time for solving
|
|
|
|
4. change some hardware resources, i.e. do a rewrite of some terminales representing the values inside
|
|
With this, the Python script does not have to know about the concrete test-cases, but merely invoking the same steps for each
|
|
the provision clauses of the respective resources
|
|
test-case.
|
|
5. GOTO 2.
|
|
|
|
|
|
### Categories of test-cases
|
|
Instead of changing the hardware, changing the software is also feasible, e.g. to simulate
|
|
|
|
new components or contract runtime feedback. The latter example means, that a contract is
|
|
The test-cases are grouped by categories, each testing a different aspect of the ILP generation. In the following, the categories
|
|
re-calculated after the (monitored) execution of a component and its values are updated. |
|
ids are listed along with a short description. The description $r \times c \ast i \ast m$ describes a $r$$c$
|
|
|
|
software components with $i$ implementations having $m$ modes each.
|
|
|
|
|
|
|
|
-----------------------------------------------------------------------------------------------------------------------
|
|
|
|
ID System Description
|
|
|
|
----------- ------------------------------- ---------------------------------------------------------------------------
|
|
|
|
1 - 16 $2 \times 1 \ast 1 \ast 2$ Basic selection within modes.
|
|
|
|
|
|
|
|
30 - 36 $2 \times 1 \ast 2 \ast 1$ Basic selection within modes.
|
|
|
|
|
|
|
|
100 - 129 $3 \times 1 \ast 2 \ast 2$ Basic selection within implementation and modes.
|
|
|
|
|
|
|
|
200 - 207 $3 \times 2 \ast 2 \ast 2$ Component requiring another component. For the 3xx category, only the
|
|
|
|
300 - 307 first implementation has a constraint for the required component.
|
|
|
|
|
|
|
|
400 - 407 $3 \times 1 \ast 2 \ast 2$ Resources with different resource types.
|
|
|
|
|
|
|
|
500 - 503 $2/3 \times 1 \ast 1 \ast 2$ New resources entering the hardware system.
|
|
|
|
|
|
|
|
600 - 605 $2 \times 1 \ast 1 \ast 2$ New components, implementations and modes entering the software system.
|
|
|
|
|
|
|
|
700 - 702 $30 \times 1 \ast 1 \ast 2$ Tree-like hardware structure.
|
|
|
|
|
|
|
|
900 - 907 *varies* Unsolveable problems, i.e. negative test-cases.
|
|
|
|
-----------------------------------------------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
## Profiling Tests
|
|
|
|
|
|
|
|
### Setup
|
|
|
|
|
|
|
|
1. Create the initial system state, i.e. the example ast `ea`
|
|
|
|
2. Evaluate `(=to-ilp ea)`, i.e. generate the ILP and measure the time for generation
|
|
|
|
3. Change some hardware resources, i.e. do a rewrite of terminales representing the values inside the provision clauses of the
|
|
|
|
respective resources
|
|
|
|
4. Repeat steps 2 and 3 for different values.
|
|
|
|
|
|
|
|
Instead of changing the hardware, changing the software is also feasible, e.g. to simulate new components or contract runtime
|
|
|
|
feedback. The latter example means, that a contract is re-calculated after the (monitored) execution of a component and its
|
|
|
|
values are updated. |