Test-scenario.md

 # Evaluating the generation of an ILP using RACR
 
-## Setup
-
-- the initial system state is a generated system using the following parameters
-	- `num-pe` – total number of hardware resources
-	- `num-pe-subs` – number of subresources per resource (i.e. breadth of the HW-tree)
-	- `num-comp` – number of SW components
-	- `impl-per-comp` – number of implementations per component
-	- `mode-per-impl` – number of modes per implementation
-- possible other, not implemented possibilities to describe the system include
-	- randomness for number of Impls and Modes
-	- minima and maxima for properties
-
-## Test
-
-1. create the initial system state, i.e. the example ast `ea`
-2. `(att-value 'to-ilp ea)` – generate the ILP, and measure the time for generation
-3. optional: solve the ILP and measure the time for solving
-4. change some hardware resources, i.e. do a rewrite of some terminales representing the values inside
-   the provision clauses of the respective resources
-5. GOTO 2.
-
-Instead of changing the hardware, changing the software is also feasible, e.g. to simulate
-new components or contract runtime feedback. The latter example means, that a contract is
-re-calculated after the (monitored) execution of a component and its values are updated.
+## Correctness Tests
+
+### Setup
+
+The correctness test is split into two parts without invalidating the results, because of technical problems. Normally one would 
+generate the ILP from the model, solve it afterwards and finally check the solution. Because solving is currently done with the 
+command-line interface of [GLPK](http://www.gnu.org/software/glpk/), *glpsol*, either glpsol has to be invoked from within 
+Scheme, or the generation and checking has to be seperated.
+
+Currently, the tests are driven by a Python script. The following steps are executed:
+
+1. Get the valid test-case ids by executing *ilp_test.scm* with `ranges`.
+2. Create a test-case for each id, in which
+	- the ILP is generated and saved to disk by invoking *ilp_test.scm* with `run`
+	- the ILP is solved by invoking *glpsol*, creating a solution file
+	- the solution is check by invoking *ilp_test.scm* with `check`
+
+With this, the Python script does not have to know about the concrete test-cases, but merely invoking the same steps for each 
+test-case.
+
+### Categories of test-cases
+
+The test-cases are grouped by categories, each testing a different aspect of the ILP generation. In the following, the categories 
+ids are listed along with a short description. The description $r \times c \ast i \ast m$ describes a $r$$c$ 
+software components with $i$ implementations having $m$ modes each.
+
+-----------------------------------------------------------------------------------------------------------------------
+	ID				System					Description
+----------- ------------------------------- ---------------------------------------------------------------------------
+  1 -  16	$2 \times 1 \ast 1 \ast 2$		Basic selection within modes.
+
+ 30 -  36	$2 \times 1 \ast 2 \ast 1$		Basic selection within modes.
+
+100 - 129	$3 \times 1 \ast 2 \ast 2$		Basic selection within implementation and modes.
+
+200 - 207	$3 \times 2 \ast 2 \ast 2$		Component requiring another component. For the 3xx category, only the
+300 - 307									first implementation has a constraint for the required component.
+
+400 - 407	$3 \times 1 \ast 2 \ast 2$		Resources with different resource types.
+
+500 - 503	$2/3 \times 1 \ast 1 \ast 2$	New resources entering the hardware system.
+
+600 - 605	$2 \times 1 \ast 1 \ast 2$		New components, implementations and modes entering the software system.
+
+700 - 702	$30 \times 1 \ast 1 \ast 2$		Tree-like hardware structure.
+
+900 - 907	*varies*						Unsolveable problems, i.e. negative test-cases.
+-----------------------------------------------------------------------------------------------------------------------
+
+## Profiling Tests
+
+### Setup
+
+1. Create the initial system state, i.e. the example ast `ea`
+2. Evaluate `(=to-ilp ea)`, i.e. generate the ILP and measure the time for generation
+3. Change some hardware resources, i.e. do a rewrite of terminales representing the values inside the provision clauses of the 
+   respective resources
+4. Repeat steps 2 and 3 for different values.
+
+Instead of changing the hardware, changing the software is also feasible, e.g. to simulate new components or contract runtime 
+feedback. The latter example means, that a contract is re-calculated after the (monitored) execution of a component and its 
+values are updated.