Compare revisions

.gitlab-ci

.gitlab-ci

Dockerfile

Dockerfile

.dockerignore

.dockerignore

trainbenchmark-docker.tar

models

models

.git

.git

root = true

[*.java]

charset = utf-8

end_of_line = lf

indent_size = 4

indent_style = tab

insert_final_newline = true

max_line_length = 140

trim_trailing_whitespace = true

[*.jadd]

charset = utf-8

end_of_line = lf

indent_size = 4

indent_style = tab

insert_final_newline = true

max_line_length = 140

trim_trailing_whitespace = true

[*.jrag]

charset = utf-8

end_of_line = lf

indent_size = 4

indent_style = tab

insert_final_newline = true

max_line_length = 140

trim_trailing_whitespace = true

# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)

# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)

!gradle-wrapper.jar

!gradle-wrapper.jar

!relational-rags*.zip

# output directory

ModelValidationWithRAGs/

# Cache of project

# Cache of project

.gradletasknamecache

.gradletasknamecache

~*.*

~*.*

*.~*

*.~*

jastadd-gen/

xtend-gen/

xtend-gen/

neo4j-server/

neo4j-server/

out/

out/

Görel Hedin

Görel Hedin

gorel.hedin@cs.lth.se

gorel.hedin@cs.lth.se

Emma Söderberg}

Emma Söderberg

emma.soderberg@cs.lth.se

emma.soderberg@cs.lth.se

Thomas Kühn}

Thomas Kühn

thomas.kuehn3@tu-dresden.de

thomas.kuehn3@tu-dresden.de

Niklas Fors

Niklas Fors

Jesper Öqvist

Jesper Öqvist

jesper.oqvist@cs.lth.se

jesper.oqvist@cs.lth.se

Uwe Aßmann

Uwe Aßmann

uwe.assmann@tu-dresden.de

uwe.assmann@tu-dresden.de

WORKDIR /trainbenchmark

WORKDIR /trainbenchmark

COPY trainbenchmark/ /trainbenchmark/

COPY trainbenchmark/ /trainbenchmark/

RUN chown -R user:user /trainbenchmark

COPY docker /trainbenchmark/docker

COPY docker /trainbenchmark/docker

RUN chown -R user:user /trainbenchmark/docker

COPY docker/bashrc /home/user/.bashrc

COPY docker/bashrc /home/user/.bashrc

RUN chown -R user:user /trainbenchmark

RUN chown -R user:user /home/user/.bashrc

USER user

RUN ./gradlew --no-daemon preprocess \

	&& ./gradlew --no-daemon build shadowJar -x test \

	&& ./scripts/configure.sh 1 1 900 10 \

	&& ./gradlew --no-daemon generate \

	&& ./scripts/configure.sh 1 1024 900 10

# USER root

# RUN apt-get update -y && apt-get install -y \

# 	vim \

# 	--no-install-recommends \

# 	&& apt-get clean \

# 	&& rm -rf /var/lib/apt/lists/*

# USER user

RUN ./gradlew --no-daemon preprocess

RUN ./gradlew --no-daemon build shadowJar -x test

RUN ./gradlew --no-daemon initScripts

# when benchmarking run:

# when benchmarking run:

# RUN ./gradlew --no-daemon generate

# RUN ./gradlew --no-daemon generate

Copyright (c) 2018, The Authors

All rights reserved.

Redistribution and use in source and binary forms, with or without

modification, are permitted provided that the following conditions are met:

    * Redistributions of source code must retain the above copyright notice,

      this list of conditions and the following disclaimer.

    * Redistributions in binary form must reproduce the above copyright notice,

      this list of conditions and the following disclaimer in the documentation

      and/or other materials provided with the distribution.

    * Neither the name of the Lund University nor the names of its contributors

      may be used to endorse or promote products derived from this software

      without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND

ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE

DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR

ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES

(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON

ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3666664.svg)](https://doi.org/10.5281/zenodo.3666664)

# Artifacts for "Continuous Model Validation Using Reference Attribute Grammars"

# Artifacts for "Continuous Model Validation Using Reference Attribute Grammars"

### Authors

- Johannes Mey <johannes.mey@tu-dresden.de>

- Carl Mai <carl.mai@tu-dresden.de>

- René Schöne <rene.schoene@tu-dresden.de>

- Görel Hedin <gorel.hedin@cs.lth.se>

- Emma Söderberg <emma.soderberg@cs.lth.se>

- Thomas Kühn <thomas.kuehn3@tu-dresden.de>

- Niklas Fors <niklas.fors@cs.lth.se>

- Jesper Öqvist <jesper.oqvist@cs.lth.se>

- Uwe Aßmann <uwe.assmann@tu-dresden.de>

### Introduction

### Introduction

The paper discusses the utilization of reference attribute grammars (RAGs) for model validation and presents two specific contributions.

The paper discusses the utilization of reference attribute grammars (RAGs) for model validation and presents two specific contributions.

First, the differences between models and trees specified by reference attribute grammars, specifically non-containment references, are discussed and a manual, yet optimised method to efficiently overcome these differences is presented.

First, the differences between models and trees specified by reference attribute grammars, specifically non-containment references, are discussed and a manual, yet optimised method to efficiently overcome these differences is presented.

Secondly, an extension of RAG grammar specifications is proposed to model noncontainment references automatically.

Secondly, an extension of RAG grammar specifications is proposed to model non-containment references automatically.

The proposed modelling techniques are compared to state-of-the-art modelling tools utilizing a benchmarking framwork for continuous model validation, the *Train Benchmark*.

The proposed modelling techniques are compared to state-of-the-art modelling tools utilizing a benchmarking framework for continuous model validation, the *Train Benchmark*.

### Structure of the Supplementary Artifacts

### Structure of the Supplementary Artifacts

The artifacts are structured in three parts:

The artifacts are structured in four parts:

- Full collection of all measurement data and diagrams mentioned in the paper

- A standalone example of the non-containment references preprocessor (relational-rags-0.2.3.zip)

- Benchmark code to reproduce the measurements, including all relevant source codes

- Benchmark code to reproduce the measurements, including all relevant source codes

- A standalone example of non-containment references preprocessor

    - as a zip file (ModelValidationWithRAGs.zip)

    - as a docker container (trainbenchmark-docker.tar)

- Full collection of all measurement data and diagrams mentioned in the paper (paper-results.zip)

### General Remarks on the presented Listings and Measurements

### General Remarks on the presented Listings and Measurements

For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resuting data.

For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resulting data.

Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams.

Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams.

The following table shows the relation of the terminology used in the paper and in the code.

The following table shows the relation of the terminology used in the paper and in the code.

+-----------------------+-----------------------------+----------------------------+

| Name used in paper and result data | Name used in source code   |

| Name used in Paper    | Name used in result data    | Name used in source code   |

|------------------------------------|----------------------------|

+=======================+=============================+============================+

| Name Lookup                        | jastadd-namelookup         |     

| Name Lookup           | Jastadd (Name Lookup)       | jastadd-namelookup         |     

| Intrinsic References               | jastadd-intrinsic          |

+-----------------------+-----------------------------+----------------------------+

| Grammar Extension                  | jastadd-relast             |

| Intrinsic References  | Jastadd (Optimized)         | jastadd-optimized          |

+-----------------------+-----------------------------+----------------------------+

| Grammar Extension     | Jastadd (Specialized)       | jastadd-specialized        |

+-----------------------+-----------------------------+----------------------------+

## The Grammar Extension Preprocessor `RelAst`

## The Grammar Extension Preprocessor *RelAst*

To transform the grammar extension we provide a preprocessor for JastAdd

To transform the grammar extension we provide a preprocessor for JastAdd.

This preprocessor including its source code is provided in the `preprocessor` subdirectory.

This preprocessor including its source code is provided in the `preprocessor` subdirectory.

Its usage is:

Its usage is:

- Running the preprocessor `java -jar relast-compiler.jar`

Run preprocessor on train benchmark (output written to standard output):

- Build the preprocessor

    - `./gradlew build jar`

    - copy the jar `cp build/libs/relational-rags-0.2.3.jar relast-compiler.jar`

- Run preprocessor on train benchmark (output written to standard output):  

    - `cat examples/TrainBenchmark.relast`

    - `cat examples/TrainBenchmark.relast`

    - `java -jar relast-compiler.jar examples/TrainBenchmark.relast`

    - `java -jar relast-compiler.jar examples/TrainBenchmark.relast`

Run preprocessor and write output to files:

- Run preprocessor and write output to files:

    - `java -jar relast-compiler.jar examples/TrainBenchmark.relast --file`

	`java -jar relast-compiler.jar examples/TrainBenchmark.relast --file`

    - `cat examples/TrainBenchmarkGen.ast`

	`cat examples/TrainBenchmarkGen.ast`

    - `cat examples/TrainBenchmarkGen.jadd`

	`cat examples/TrainBenchmarkGen.jadd`

Run preprocessor and write output to files (with a different list class):

	`java -jar relast-compiler.jar examples/TrainBenchmark.relast --listClass=MyListClass --file`

	`cat examples/TrainBenchmarkGen.ast`

	`cat examples/TrainBenchmarkGen.jadd`

## The Train Benchmark

## The Train Benchmark

2. **Queries:** The queries are used to find the aforementioned faults. For each fault, there are two queries: *repair*, to find the fault, and *inject*, to find places where a fault can be injected.

2. **Queries:** The queries are used to find the aforementioned faults. For each fault, there are two queries: *repair*, to find the fault, and *inject*, to find places where a fault can be injected.

3. **Transformations:** The transformations performed by the benchmark are, again, two sets: *inject* and *repair* transformations.

3. **Transformations:** The transformations performed by the benchmark are, again, two sets: *inject* and *repair* transformations.

4. **Transformation Strategies:** The benchmark does not perform the operation on all matches.

4. **Transformation Strategies:** The benchmark does not perform the operation on all matches.

   The strategy *fixed* performs the transformation on a given number of matches, while the *proportional* strategy performes them on a given percentage of all matches.

   The strategy *fixed* performs the transformation on a given number of matches, while the *proportional* strategy performs them on a given percentage of all matches.

These settings are defined in a *benchmark scenario*, which can be edited before running the benchmark.

These settings are defined in a *benchmark scenario*, which can be edited before running the benchmark.

### Measurement Data

### Measurement Data

The result data are stored in the directory [paper-results/](paper-results/).

The result data is stored in the directory [paper-results/](paper-results/).

This directory contains two subdirectories

This directory contains two subdirectories:

- [measurements](paper-results/measurements) contains two directories.

- [measurements](paper-results/measurements) contains two directories.

  The [inject](paper-results/measurements/inject) subdirectory contains the measurements for the *inject* scenario, which is also included in [inject/BenchmarkScript.groovy](paper-results/measurements/inject/BenchmarkScript.groovy).

  The [individual](paper-results/measurements/individual) subdirectory contains the measurements for individual queries for both the *inject* and *repair* scenario.

  The [repair](paper-results/measurements/repair) subdirectory contains the same data for the *repair* scenario in [repair/BenchmarkScript.groovy](paper-results/measurements/repair/BenchmarkScript.groovy).

  The [all-queries](paper-results/measurements/all-queries) subdirectory contains the same data for the a run including all queries in sequence.

  Both directories contain files with time measurement data (starting with `times`) and the numbers of matches (starting with `matches`).

  Both directories contain files with time measurement data (starting with `times`) and the numbers of matches (starting with `matches`).

  Each file name contains information on the tool used, the query, and the size of the model.

  Each file name contains information on the tool used, the query, and the size of the model.

- [diagrams](paper-results/diagrams) contains the same subdirectories, both containing diagrams with the respective measurements.

- [diagrams](paper-results/diagrams) contains the same subdirectories, containing diagrams with the respective measurements.

  The diagrams are generated from the same data as in the paper, but enlarged for better readability.

  The diagrams are generated from the same data as in the paper, but enlarged for better readability.

  In particular, the six diagrams presented in the paper are

    - [Fig. 7a. Read and Check for RouteSensor (repair)](paper-results/diagrams/repair/Read-and-Check-RouteSensor.pdf)

    - [Fig. 7b. Read and Check for ConnectedSegments (repair)](paper-results/diagrams/repair/Read-and-Check-ConnectedSegments.pdf)

    - [Fig. 7c. Transformation and Recheck for RouteSensor (inject)](paper-results/diagrams/inject/Transformation-and-Recheck-RouteSensor.pdf)

    - [Fig. 7d. Transformation and Recheck for ConnectedSegments (inject)](paper-results/diagrams/inject/Transformation-and-Recheck-ConnectedSegments.pdf)

    - [Fig. 7e. Transformation and Recheck for RouteSensor (repair)](paper-results/diagrams/repair/Transformation-and-Recheck-RouteSensor.pdf)

    - [Fig. 7f. Transformation and Recheck for ConnectedSegments (repair)](paper-results/diagrams/repair/Transformation-and-Recheck-ConnectedSegments.pdf)

**Please Note:** The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes.

**Please Note:** The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes.

For this publication, we tried to modify the source code of the benchmark itself as little as possible.

For this publication, we tried to modify the source code of the benchmark itself as little as possible.

Therefore, unfortunately, the code base is rather large and confusing. The following section tries to point to the parts relevant for this paper.

Therefore, unfortunately, the code base is rather large and confusing. The following section tries to point to the parts relevant for this paper.

The benchmark is structures in modules, some of which form the code of the benchmark, some are provided by the contesting tools, and some are related to required model serializations.

The benchmark is structured in modules, some of which form the code of the benchmark, some are provided by the contesting tools, and some are related to required model serializations.

There are some naming conventions:

There are some naming conventions:

- Tool-related modules are in directories starting with `trainbenchmark-tool`.

- Tool-related modules are in directories starting with `trainbenchmark-tool`.

- Model serialization-related modules start with `trainbenchmark-generator`.

- Model serialization-related modules start with `trainbenchmark-generator`.

- All other modules are core modules of the bechmark.

- All other modules are core modules of the benchmark.

Since the JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant, it is even more compolicated.

The JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant.

Each JastAdd configuraration must be presented to the benchmark as a separate tool. Thus there are two directories for each variant, one for the bacht processing mode and one for the incremental mode.

Each JastAdd configuration must be presented to the benchmark as a separate tool. Thus, there are two directories for each variant, one for the batch processing mode and one for the incremental mode.

Because these two modes share almost all the source code, a third directory is used to store this shared code.

Because these two modes share almost all the source code, a third directory is used to store this shared code.

Finally, there is a directory for code shared between all JastAdd variants.

Finally, there is a directory for code shared between all JastAdd variants.

These are the important directories:

These are the important directories:

    - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/train.ast)

    - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/train.ast)

    - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries)

    - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries)

    - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

    - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

- [JastAdd with Intrinsic References](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base)

- [JastAdd with Intrinsic References](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base)

    - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/train.ast)

    - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/train.ast)

    - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/queries)

    - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/queries)

    - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

    - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

- [JastAdd with Grammar Extension](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base)

- [JastAdd with Grammar Extension](trainbenchmark/trainbenchmark-tool-jastadd-relast-base)

    - [(Extended) Grammar](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/Train.relast)

    - [(Extended) Grammar](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/Train.relast)

    - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/queries)

    - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/queries)

    - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

    - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

- [Common JastAdd Code](trainbenchmark/trainbenchmark-tool-jastadd-base)

- [Common JastAdd Code](trainbenchmark/trainbenchmark-tool-jastadd-base)

### Reproducing the Measurements

### Reproducing the Measurements

**Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested to run the benchmark with a smaller maximum problem size, less repetitions, and a shorter timeout.** Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes.

**<span style="color:red">Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested running the benchmark with a smaller maximum problem size, fewer repetitions, and a shorter timeout.</span>** Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes. Note that running the benchmark requires a significant amount of disk space (up to 10GB when running the full benchmark).

To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly.

To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly.

Alternatively, it is, on course, also possible to simply run the provided gradle build scripts.

Alternatively, it is, of course, also possible to simply run the provided gradle build scripts.

However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We stronly suggest running the Docker variant.

However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We strongly suggest running the Docker variant.

#### Running the Benchmark with Docker

#### Running the Benchmark with Docker

##### Loading the Docker Image

##### Loading the Docker Image

- Variant 1 (recommended): Load the provided docker image

- Variant 1 (*recommended*): Load the provided docker image

    - Prerequisites: An installation of Docker in the `PATH`

    - Prerequisites: An installation of Docker in the `PATH`

    - Steps:

    - Steps:

        - Unpack the provided archive and open a terminal in the extracted directory

        - Unpack the provided archive and open a terminal in the extracted directory

    - Steps:

    - Steps:

        - Unpack the provided archive and open a terminal in the extracted directory

        - Unpack the provided archive and open a terminal in the extracted directory

        - `docker build -t trainbenchmark .`

        - `docker build -t trainbenchmark .`

        - `docker run -it trainbenchmark`

##### Running the Docker Image

##### Running the Docker Image

- `docker run -it trainbenchmark`

- `docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark`

  To make the results available outside the container, run

- This makes the results and diagrams available outside the container in the directories `docker-results` and `docker-diagrams` respectively

  `docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark`

- Once running, a command prompt is opened and some information is displayed

- A command prompt is opened and some information is displayed

- Follow the instructions below

- Follow the instructions below

#### Running the Benchmark directly

#### Running the Benchmark directly

- For running a standard run,

- For running a standard run, use one of the following commands:

    - `./run_small`

    - `./run_medium`

| Name   | Command        | Minimum size | Maximum size | Timeout | Runs |

    - `./run_full`

| ------ | -------------- | ------------ | ------------ | ------- | ---- |

| Small  | `./run_small`  | 1            | 32           | 60s     | 1    |

| Medium | `./run_medium` | 1            | 64           | 10min   | 5    |

| Full   | `./run_full`   | 1            | 512          | 15min   | 10   |

- For running a custom run,

- For running a custom run,

    - run `./gradlew preprocess` to generate the 

    - run `./gradlew preprocess` to generate the grammar from the extended grammar specification

    - run `./gradlew build shadowJar -x test`

    - run `./gradlew build shadowJar -x test`

    - run `./gradlew initScripts`

    - configure the scripts by running `./scripts/configure.sh 1 <MAXSIZE> <TIMEOUT in s> <REPETITIONS>`

    - configure the scripts either by running `./scripts/configure.sh 1 <MAXSIZE> <TIMEOUT in s> <REPETITIONS>`

        - Where MAXSIZE is one of 2, 4, 8, 16, 32, 64, 128, 256, 512, or, 1024. The larger sizes use **a lot of** disk space!

      Where MAXSIZE is one of 2,4,8,16,32,64,128,256,512,1024. The larger sizes use **a lot of** disk space!

    - *Alternatively*, run ``

    - run `./gradlew initScripts`

    - run `./gradlew generate`

    - run `./gradlew generate`

    - run the benchmark

    - run the benchmark

        - run `./gradlew individualInjectBenchmark` for the *inject* scenarios

        - run `./gradlew individualInjectBenchmark` for the *inject* scenarios

        - run `./gradlew individualRepairBenchmark` for the *repair* scenarios

        - run `./gradlew individualRepairBenchmark` for the *repair* scenarios

    - Plot the diagrams for the current run: `./gradlew plotIndividual`

    - Plot the diagrams for the current run: `./gradlew plotIndividual`

- The resulting data are in the `results` and the `diagrams` folder

- The resulting data and diagrams is placed in the `results` and the `diagrams` folder

    - When running with docker, the data are also in `docker-results` and `docker-diagrams` on the host machine.

    - When running with docker, the data is also in `docker-results` and `docker-diagrams` on the host machine.

Artifacts for "Continuous Model Validation Using Reference Attribute Grammars"

==============================================================================

### Authors

-   Johannes Mey <johannes.mey@tu-dresden.de>

-   Carl Mai <carl.mai@tu-dresden.de>

-   René Schöne <rene.schoene@tu-dresden.de>

-   Görel Hedin <gorel.hedin@cs.lth.se>

-   Emma Söderberg <emma.soderberg@cs.lth.se>

-   Thomas Kühn <thomas.kuehn3@tu-dresden.de>

-   Niklas Fors <niklas.fors@cs.lth.se>

-   Jesper Öqvist <jesper.oqvist@cs.lth.se>

-   Uwe Aßmann <uwe.assmann@tu-dresden.de>

### Introduction

The paper discusses the utilization of reference attribute grammars

(RAGs) for model validation and presents two specific contributions.

First, the differences between models and trees specified by reference

attribute grammars, specifically non-containment references, are

discussed and a manual, yet optimised method to efficiently overcome

these differences is presented. Secondly, an extension of RAG grammar

specifications is proposed to model non-containment references

automatically. The proposed modelling techniques are compared to

state-of-the-art modelling tools utilizing a benchmarking framework for

continuous model validation, the *Train Benchmark*.

### Structure of the Supplementary Artifacts

The artifacts are structured in four parts:

-   A standalone example of the non-containment references preprocessor

    (relational-rags-0.2.3.zip)

-   Benchmark code to reproduce the measurements, including all relevant

    source codes

    -   as a zip file (ModelValidationWithRAGs.zip)

    -   as a docker container (trainbenchmark-docker.tar)

-   Full collection of all measurement data and diagrams mentioned in

    the paper (paper-results.zip)

### General Remarks on the presented Listings and Measurements

For reasons of readability and simplicity, there are some minor

differences in naming in the source codes and the measured resulting

data. Most importantly, the names of the three presented JastAdd

implementation variants are different in the code and the diagrams.

The following table shows the relation of the terminology used in the

paper and in the code.

  Name used in paper and result data   Name used in source code

  ------------------------------------ --------------------------

  Name Lookup                          jastadd-namelookup

  Intrinsic References                 jastadd-intrinsic

  Grammar Extension                    jastadd-relast

The Grammar Extension Preprocessor *RelAst*

-------------------------------------------

To transform the grammar extension we provide a preprocessor for

JastAdd. This preprocessor including its source code is provided in the

`preprocessor` subdirectory.

Its usage is:

-   Build the preprocessor

    -   `./gradlew build jar`

    -   copy the jar

        `cp build/libs/relational-rags-0.2.3.jar relast-compiler.jar`

-   Run preprocessor on train benchmark (output written to standard

    output):

    -   `cat examples/TrainBenchmark.relast`

    -   `java -jar relast-compiler.jar examples/TrainBenchmark.relast`

-   Run preprocessor and write output to files:

    -   `java -jar relast-compiler.jar examples/TrainBenchmark.relast --file`

    -   `cat examples/TrainBenchmarkGen.ast`

    -   `cat examples/TrainBenchmarkGen.jadd`

The Train Benchmark

-------------------

### Structure of the Train Benchmark

The benchmark is able to measure different scenarios specified by

configurations with several kinds of parameters:

1.  **Input Data:** There are two types of input data used in the

    benchmark, the `inject` and the `repair` data set. The former

    contains *valid* models, i.e., models, which do not contain any of

    the faults that are supposed to be found by the presented queries.

    The latter, `repair`, contains models already containing faults.

2.  **Queries:** The queries are used to find the aforementioned faults.

    For each fault, there are two queries: *repair*, to find the fault,

    and *inject*, to find places where a fault can be injected.

3.  **Transformations:** The transformations performed by the benchmark

    are, again, two sets: *inject* and *repair* transformations.

4.  **Transformation Strategies:** The benchmark does not perform the

    operation on all matches. The strategy *fixed* performs the

    transformation on a given number of matches, while the

    *proportional* strategy performs them on a given percentage of all

    matches.

These settings are defined in a *benchmark scenario*, which can be

edited before running the benchmark.

### Measurement Data

The result data is stored in the directory

[paper-results/](paper-results/). This directory contains two

subdirectories:

-   [measurements](paper-results/measurements) contains two directories.

    The [individual](paper-results/measurements/individual) subdirectory

    contains the measurements for individual queries for both the

    *inject* and *repair* scenario. The

    [all-queries](paper-results/measurements/all-queries) subdirectory

    contains the same data for the a run including all queries in

    sequence. Both directories contain files with time measurement data

    (starting with `times`) and the numbers of matches (starting with

    `matches`). Each file name contains information on the tool used,

    the query, and the size of the model.

-   [diagrams](paper-results/diagrams) contains the same subdirectories,

    containing diagrams with the respective measurements. The diagrams

    are generated from the same data as in the paper, but enlarged for

    better readability.

**Please Note:** The measurements were conducted using a timeout for the

whole run. If a run was not completed, no individual times of the steps

appear in the measurements and diagrams. Thus, some tools do not have

measurements for all problem sizes.

### The Source Code

For this publication, we tried to modify the source code of the

benchmark itself as little as possible. Therefore, unfortunately, the

code base is rather large and confusing. The following section tries to

point to the parts relevant for this paper.

The benchmark is structured in modules, some of which form the code of

the benchmark, some are provided by the contesting tools, and some are

related to required model serializations. There are some naming

conventions: - Tool-related modules are in directories starting with

`trainbenchmark-tool`. - Model serialization-related modules start with

`trainbenchmark-generator`. - All other modules are core modules of the

benchmark.

The JastAdd-based solutions use a preprocessor to generate Java files,

for the presented variant. Each JastAdd configuration must be presented

to the benchmark as a separate tool. Thus, there are two directories for

each variant, one for the batch processing mode and one for the

incremental mode. Because these two modes share almost all the source

code, a third directory is used to store this shared code. Finally,

there is a directory for code shared between all JastAdd variants. These

are the important directories:

-   [JastAdd with Name

    Lookup](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base)

    -   [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/train.ast)

    -   [Queries](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries)

    -   [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

-   [JastAdd with Intrinsic

    References](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base)

    -   [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/train.ast)

    -   [Queries](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/queries)

    -   [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

-   [JastAdd with Grammar

    Extension](trainbenchmark/trainbenchmark-tool-jastadd-relast-base)

    -   [(Extended)

        Grammar](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/Train.relast)

    -   [Queries](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/queries)

    -   [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations)

-   [Common JastAdd

    Code](trainbenchmark/trainbenchmark-tool-jastadd-base)

### Reproducing the Measurements

**[Please Note: Reproducing the graphs as presented in the paper and

supplied here takes a very long time depending on the utilized hardware.

It is strongly suggested running the benchmark with a smaller maximum

problem size, fewer repetitions, and a shorter

timeout.]{style="color:red"}** Most results of the benchmark are

observable with more restricted setup as well. In the following, we will

provide a suggested way to run the benchmark in different sizes. Note

that running the benchmark requires a significant amount of disk space

(up to 10GB when running the full benchmark).