diff --git a/.gitignore b/.gitignore index 6804809b49e243976cc8f3c62cbfff8eef54d6cb..412d6823d3069fefed5ba2632f0f3c0f324324fe 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,10 @@ gradle-app.setting # Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored) !gradle-wrapper.jar +!relational-rags*.zip + +# output directory +ModelValidationWithRAGs/ # Cache of project .gradletasknamecache diff --git a/README.html b/README.html index 805d15c5c77af670d9eb970465bfcb3593e1fc8a..69ab7c12e4050a63e49ee1860b0618b07884bb36 100644 --- a/README.html +++ b/README.html @@ -1,68 +1,28 @@ -<!DOCTYPE html> -<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang=""> -<head> - <meta charset="utf-8" /> - <meta name="generator" content="pandoc" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" /> - <title>README</title> - <style type="text/css"> - code{white-space: pre-wrap;} - span.smallcaps{font-variant: small-caps;} - span.underline{text-decoration: underline;} - div.column{display: inline-block; vertical-align: top; width: 50%;} - </style> - <!--[if lt IE 9]> - <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script> - <![endif]--> - <style type="text/css"> - table { - border-spacing: 0; - border-bottom: 2px solid black; - border-top: 2px solid black; - } - table th { - padding: 3px 10px; - background-color: white; - border-top: none; - border-left: none; - border-right: none; - border-bottom: 1px solid black; - } - table td { - padding: 3px 10px; - border-top: none; - border-left: none; - border-bottom: none; - border-right: none; - } - code { - background-color: lightgrey; - } - </style> -</head> -<body> <h1 id="artifacts-for-continuous-model-validation-using-reference-attribute-grammars">Artifacts for “Continuous Model Validation Using Reference Attribute Grammars”</h1> -<p><em>Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We encourage using the one including the image, since building the image takes a long time.</em></p> <h3 id="authors">Authors</h3> <ul> -<li>Johannes Mey <a href="mailto:johannes.mey@tu-dresden.de">johannes.mey@tu-dresden.de</a></li> -<li>Carl Mai <a href="mailto:carl.mai@tu-dresden.de">carl.mai@tu-dresden.de</a></li> -<li>René Schöne <a href="mailto:rene.schoene@tu-dresden.de">rene.schoene@tu-dresden.de</a></li> -<li>Görel Hedin <a href="mailto:gorel.hedin@cs.lth.se">gorel.hedin@cs.lth.se</a></li> -<li>Emma Söderberg <a href="mailto:emma.soderberg@cs.lth.se">emma.soderberg@cs.lth.se</a></li> -<li>Thomas Kühn <a href="mailto:thomas.kuehn3@tu-dresden.de">thomas.kuehn3@tu-dresden.de</a></li> -<li>Niklas Fors <a href="mailto:niklas.fors@cs.lth.se">niklas.fors@cs.lth.se</a></li> -<li>Jesper Öqvist <a href="mailto:jesper.oqvist@cs.lth.se">jesper.oqvist@cs.lth.se</a></li> -<li>Uwe Aßmann <a href="mailto:uwe.assmann@tu-dresden.de">uwe.assmann@tu-dresden.de</a></li> +<li>Johannes Mey <a href="mailto:johannes.mey@tu-dresden.de" class="email">johannes.mey@tu-dresden.de</a></li> +<li>Carl Mai <a href="mailto:carl.mai@tu-dresden.de" class="email">carl.mai@tu-dresden.de</a></li> +<li>René Schöne <a href="mailto:rene.schoene@tu-dresden.de" class="email">rene.schoene@tu-dresden.de</a></li> +<li>Görel Hedin <a href="mailto:gorel.hedin@cs.lth.se" class="email">gorel.hedin@cs.lth.se</a></li> +<li>Emma Söderberg <a href="mailto:emma.soderberg@cs.lth.se" class="email">emma.soderberg@cs.lth.se</a></li> +<li>Thomas Kühn <a href="mailto:thomas.kuehn3@tu-dresden.de" class="email">thomas.kuehn3@tu-dresden.de</a></li> +<li>Niklas Fors <a href="mailto:niklas.fors@cs.lth.se" class="email">niklas.fors@cs.lth.se</a></li> +<li>Jesper Öqvist <a href="mailto:jesper.oqvist@cs.lth.se" class="email">jesper.oqvist@cs.lth.se</a></li> +<li>Uwe Aßmann <a href="mailto:uwe.assmann@tu-dresden.de" class="email">uwe.assmann@tu-dresden.de</a></li> </ul> <h3 id="introduction">Introduction</h3> <p>The paper discusses the utilization of reference attribute grammars (RAGs) for model validation and presents two specific contributions. First, the differences between models and trees specified by reference attribute grammars, specifically non-containment references, are discussed and a manual, yet optimised method to efficiently overcome these differences is presented. Secondly, an extension of RAG grammar specifications is proposed to model non-containment references automatically. The proposed modelling techniques are compared to state-of-the-art modelling tools utilizing a benchmarking framework for continuous model validation, the <em>Train Benchmark</em>.</p> <h3 id="structure-of-the-supplementary-artifacts">Structure of the Supplementary Artifacts</h3> -<p>The artifacts are structured in three parts:</p> +<p>The artifacts are structured in four parts:</p> <ul> -<li>A standalone example of the non-containment references preprocessor</li> -<li>Benchmark code to reproduce the measurements, including all relevant source codes</li> -<li>Full collection of all measurement data and diagrams mentioned in the paper</li> +<li>A standalone example of the non-containment references preprocessor (relational-rags-0.2.3.zip)</li> +<li>Benchmark code to reproduce the measurements, including all relevant source codes +<ul> +<li>as a zip file (ModelValidationWithRAGs.zip)</li> +<li>as a docker container (trainbenchmark-docker.tar)</li> +</ul></li> +<li>Full collection of all measurement data and diagrams mentioned in the paper (paper-results.zip)</li> </ul> <h3 id="general-remarks-on-the-presented-listings-and-measurements">General Remarks on the presented Listings and Measurements</h3> <p>For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resulting data. Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams.</p> @@ -70,26 +30,22 @@ <table> <thead> <tr class="header"> -<th>Name used in Paper</th> -<th>Name used in result data</th> -<th>Name used in source code</th> +<th style="text-align: left;">Name used in paper and result data</th> +<th style="text-align: left;">Name used in source code</th> </tr> </thead> <tbody> <tr class="odd"> -<td>Name Lookup</td> -<td>Jastadd (Name Lookup)</td> -<td>jastadd-namelookup</td> +<td style="text-align: left;">Name Lookup</td> +<td style="text-align: left;">jastadd-namelookup</td> </tr> <tr class="even"> -<td>Intrinsic References</td> -<td>Jastadd (Optimized)</td> -<td>jastadd-optimized</td> +<td style="text-align: left;">Intrinsic References</td> +<td style="text-align: left;">jastadd-intrinsic</td> </tr> <tr class="odd"> -<td>Grammar Extension</td> -<td>Jastadd (Specialized)</td> -<td>jastadd-specialized</td> +<td style="text-align: left;">Grammar Extension</td> +<td style="text-align: left;">jastadd-relast</td> </tr> </tbody> </table> @@ -97,6 +53,11 @@ <p>To transform the grammar extension we provide a preprocessor for JastAdd. This preprocessor including its source code is provided in the <code>preprocessor</code> subdirectory.</p> <p>Its usage is:</p> <ul> +<li>Build the preprocessor +<ul> +<li><code>./gradlew build jar</code></li> +<li>copy the jar <code>cp build/libs/relational-rags-0.2.3.jar relast-compiler.jar</code></li> +</ul></li> <li>Run preprocessor on train benchmark (output written to standard output): <ul> <li><code>cat examples/TrainBenchmark.relast</code></li> @@ -108,12 +69,6 @@ <li><code>cat examples/TrainBenchmarkGen.ast</code></li> <li><code>cat examples/TrainBenchmarkGen.jadd</code></li> </ul></li> -<li>Run preprocessor and write output to files (with a different list class): -<ul> -<li><code>java -jar relast-compiler.jar examples/TrainBenchmark.relast --listClass=MyListClass --file</code></li> -<li><code>cat examples/TrainBenchmarkGen.ast</code></li> -<li><code>cat examples/TrainBenchmarkGen.jadd</code></li> -</ul></li> </ul> <h2 id="the-train-benchmark">The Train Benchmark</h2> <h3 id="structure-of-the-train-benchmark">Structure of the Train Benchmark</h3> @@ -126,18 +81,10 @@ </ol> <p>These settings are defined in a <em>benchmark scenario</em>, which can be edited before running the benchmark.</p> <h3 id="measurement-data">Measurement Data</h3> -<p>The result data is stored in the directory <a href="paper-results/" class="uri">paper-results/</a>. This directory contains two subdirectories:</p> +<p>The result data is stored in the directory <a href="paper-results/">paper-results/</a>. This directory contains two subdirectories:</p> <ul> -<li><a href="paper-results/measurements">measurements</a> contains two directories. The <a href="paper-results/measurements/inject">inject</a> subdirectory contains the measurements for the <em>inject</em> scenario, which is also included in <a href="paper-results/measurements/inject/BenchmarkScript.groovy">inject/BenchmarkScript.groovy</a>. The <a href="paper-results/measurements/repair">repair</a> subdirectory contains the same data for the <em>repair</em> scenario in <a href="paper-results/measurements/repair/BenchmarkScript.groovy">repair/BenchmarkScript.groovy</a>. Both directories contain files with time measurement data (starting with <code>times</code>) and the numbers of matches (starting with <code>matches</code>). Each file name contains information on the tool used, the query, and the size of the model.</li> -<li><a href="paper-results/diagrams">diagrams</a> contains the same subdirectories, both containing diagrams with the respective measurements. The diagrams are generated from the same data as in the paper, but enlarged for better readability. In particular, the six diagrams presented in the paper are -<ul> -<li><a href="paper-results/diagrams/repair/Read-and-Check-RouteSensor.pdf">Fig. 7a. Read and Check for RouteSensor (repair)</a></li> -<li><a href="paper-results/diagrams/repair/Read-and-Check-ConnectedSegments.pdf">Fig. 7b. Read and Check for ConnectedSegments (repair)</a></li> -<li><a href="paper-results/diagrams/inject/Transformation-and-Recheck-RouteSensor.pdf">Fig. 7c. Transformation and Recheck for RouteSensor (inject)</a></li> -<li><a href="paper-results/diagrams/inject/Transformation-and-Recheck-ConnectedSegments.pdf">Fig. 7d. Transformation and Recheck for ConnectedSegments (inject)</a></li> -<li><a href="paper-results/diagrams/repair/Transformation-and-Recheck-RouteSensor.pdf">Fig. 7e. Transformation and Recheck for RouteSensor (repair)</a></li> -<li><a href="paper-results/diagrams/repair/Transformation-and-Recheck-ConnectedSegments.pdf">Fig. 7f. Transformation and Recheck for ConnectedSegments (repair)</a></li> -</ul></li> +<li><a href="paper-results/measurements">measurements</a> contains two directories. The <a href="paper-results/measurements/individual">individual</a> subdirectory contains the measurements for individual queries for both the <em>inject</em> and <em>repair</em> scenario. The <a href="paper-results/measurements/all-queries">all-queries</a> subdirectory contains the same data for the a run including all queries in sequence. Both directories contain files with time measurement data (starting with <code>times</code>) and the numbers of matches (starting with <code>matches</code>). Each file name contains information on the tool used, the query, and the size of the model.</li> +<li><a href="paper-results/diagrams">diagrams</a> contains the same subdirectories, containing diagrams with the respective measurements. The diagrams are generated from the same data as in the paper, but enlarged for better readability.</li> </ul> <p><strong>Please Note:</strong> The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes.</p> <h3 id="the-source-code">The Source Code</h3> @@ -151,17 +98,17 @@ <li><a href="trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries">Queries</a></li> <li><a href="trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations">Transformations</a></li> </ul></li> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-optimized-base">JastAdd with Intrinsic References</a> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base">JastAdd with Intrinsic References</a> <ul> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/train.ast">Grammar</a></li> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/queries">Queries</a></li> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations">Transformations</a></li> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/train.ast">Grammar</a></li> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/queries">Queries</a></li> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations">Transformations</a></li> </ul></li> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-specialized-base">JastAdd with Grammar Extension</a> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-relast-base">JastAdd with Grammar Extension</a> <ul> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/Train.relast">(Extended) Grammar</a></li> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/queries">Queries</a></li> -<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations">Transformations</a></li> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/Train.relast">(Extended) Grammar</a></li> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/queries">Queries</a></li> +<li><a href="trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations">Transformations</a></li> </ul></li> <li><a href="trainbenchmark/trainbenchmark-tool-jastadd-base">Common JastAdd Code</a></li> </ul> @@ -192,7 +139,7 @@ </ul> <h5 id="running-the-docker-image">Running the Docker Image</h5> <ul> -<li><code>docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark</code></li> +<li><code>docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark</code></li> <li>This makes the results and diagrams available outside the container in the directories <code>docker-results</code> and <code>docker-diagrams</code> respectively</li> <li>Once running, a command prompt is opened and some information is displayed</li> <li>Follow the instructions below</li> @@ -204,38 +151,38 @@ <table> <thead> <tr class="header"> -<th>Name</th> -<th>Command</th> -<th>Minimum size</th> -<th>Maximum size</th> -<th>Timeout</th> -<th>Runs</th> +<th style="text-align: left;">Name</th> +<th style="text-align: left;">Command</th> +<th style="text-align: left;">Minimum size</th> +<th style="text-align: left;">Maximum size</th> +<th style="text-align: left;">Timeout</th> +<th style="text-align: left;">Runs</th> </tr> </thead> <tbody> <tr class="odd"> -<td>Small</td> -<td><code>./run_small</code></td> -<td>1</td> -<td>32</td> -<td>60s</td> -<td>1</td> +<td style="text-align: left;">Small</td> +<td style="text-align: left;"><code>./run_small</code></td> +<td style="text-align: left;">1</td> +<td style="text-align: left;">32</td> +<td style="text-align: left;">60s</td> +<td style="text-align: left;">1</td> </tr> <tr class="even"> -<td>Medium</td> -<td><code>./run_medium</code></td> -<td>1</td> -<td>64</td> -<td>10min</td> -<td>5</td> +<td style="text-align: left;">Medium</td> +<td style="text-align: left;"><code>./run_medium</code></td> +<td style="text-align: left;">1</td> +<td style="text-align: left;">64</td> +<td style="text-align: left;">10min</td> +<td style="text-align: left;">5</td> </tr> <tr class="odd"> -<td>Full</td> -<td><code>./run_full</code></td> -<td>1</td> -<td>512</td> -<td>15min</td> -<td>10</td> +<td style="text-align: left;">Full</td> +<td style="text-align: left;"><code>./run_full</code></td> +<td style="text-align: left;">1</td> +<td style="text-align: left;">512</td> +<td style="text-align: left;">15min</td> +<td style="text-align: left;">10</td> </tr> </tbody> </table> @@ -261,5 +208,3 @@ <li>When running with docker, the data is also in <code>docker-results</code> and <code>docker-diagrams</code> on the host machine.</li> </ul></li> </ul> -</body> -</html> diff --git a/README.md b/README.md index c9e10dc0a03514c17553b824b9ff8146d2527713..77cc279d635e5a4bf65dd48d244e3077beb7d07d 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +[](https://doi.org/10.5281/zenodo.3666664) + # Artifacts for "Continuous Model Validation Using Reference Attribute Grammars" ### Authors @@ -22,11 +24,13 @@ The proposed modelling techniques are compared to state-of-the-art modelling too ### Structure of the Supplementary Artifacts -The artifacts are structured in three parts: +The artifacts are structured in four parts: -- A standalone example of the non-containment references preprocessor +- A standalone example of the non-containment references preprocessor (relational-rags-0.2.3.zip) - Benchmark code to reproduce the measurements, including all relevant source codes -- Full collection of all measurement data and diagrams mentioned in the paper + - as a zip file (ModelValidationWithRAGs.zip) + - as a docker container (trainbenchmark-docker.tar) +- Full collection of all measurement data and diagrams mentioned in the paper (paper-results.zip) ### General Remarks on the presented Listings and Measurements @@ -38,8 +42,8 @@ The following table shows the relation of the terminology used in the paper and | Name used in paper and result data | Name used in source code | |------------------------------------|----------------------------| | Name Lookup | jastadd-namelookup | -| Intrinsic References | jastadd-optimized | -| Grammar Extension | jastadd-specialized | +| Intrinsic References | jastadd-intrinsic | +| Grammar Extension | jastadd-relast | ## The Grammar Extension Preprocessor *RelAst* @@ -48,6 +52,10 @@ This preprocessor including its source code is provided in the `preprocessor` su Its usage is: +- Build the preprocessor + - `./gradlew build jar` + - copy the jar `cp build/libs/relational-rags-0.2.3.jar relast-compiler.jar` + - Run preprocessor on train benchmark (output written to standard output): - `cat examples/TrainBenchmark.relast` - `java -jar relast-compiler.jar examples/TrainBenchmark.relast` @@ -57,11 +65,6 @@ Its usage is: - `cat examples/TrainBenchmarkGen.ast` - `cat examples/TrainBenchmarkGen.jadd` -- Run preprocessor and write output to files (with a different list class): - - `java -jar relast-compiler.jar examples/TrainBenchmark.relast --listClass=MyListClass --file` - - `cat examples/TrainBenchmarkGen.ast` - - `cat examples/TrainBenchmarkGen.jadd` - ## The Train Benchmark @@ -117,14 +120,14 @@ These are the important directories: - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/train.ast) - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries) - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) -- [JastAdd with Intrinsic References](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base) - - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/train.ast) - - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/queries) - - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) -- [JastAdd with Grammar Extension](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base) - - [(Extended) Grammar](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/Train.relast) - - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/queries) - - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) +- [JastAdd with Intrinsic References](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base) + - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/train.ast) + - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/queries) + - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) +- [JastAdd with Grammar Extension](trainbenchmark/trainbenchmark-tool-jastadd-relast-base) + - [(Extended) Grammar](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/Train.relast) + - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/queries) + - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) - [Common JastAdd Code](trainbenchmark/trainbenchmark-tool-jastadd-base) ### Reproducing the Measurements diff --git a/README.txt b/README.txt index 52503e99458e57f6ffe40e2045ee81378604c07b..593510fde2031781d3908382013867a308932f87 100644 --- a/README.txt +++ b/README.txt @@ -1,193 +1,251 @@ -# Artifacts for "Continuous Model Validation Using Reference Attribute Grammars" - -*Note: please use the HTML version of this README.* - -*Also Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We encourage using the one including the image, since building the image takes a long time.* +Artifacts for "Continuous Model Validation Using Reference Attribute Grammars" +============================================================================== ### Authors -- Johannes Mey <johannes.mey@tu-dresden.de> -- Carl Mai <carl.mai@tu-dresden.de> -- René Schöne <rene.schoene@tu-dresden.de> -- Görel Hedin <gorel.hedin@cs.lth.se> -- Emma Söderberg <emma.soderberg@cs.lth.se> -- Thomas Kühn <thomas.kuehn3@tu-dresden.de> -- Niklas Fors <niklas.fors@cs.lth.se> -- Jesper Öqvist <jesper.oqvist@cs.lth.se> -- Uwe Aßmann <uwe.assmann@tu-dresden.de> - +- Johannes Mey <johannes.mey@tu-dresden.de> +- Carl Mai <carl.mai@tu-dresden.de> +- René Schöne <rene.schoene@tu-dresden.de> +- Görel Hedin <gorel.hedin@cs.lth.se> +- Emma Söderberg <emma.soderberg@cs.lth.se> +- Thomas Kühn <thomas.kuehn3@tu-dresden.de> +- Niklas Fors <niklas.fors@cs.lth.se> +- Jesper Öqvist <jesper.oqvist@cs.lth.se> +- Uwe Aßmann <uwe.assmann@tu-dresden.de> ### Introduction -The paper discusses the utilization of reference attribute grammars (RAGs) for model validation and presents two specific contributions. -First, the differences between models and trees specified by reference attribute grammars, specifically non-containment references, are discussed and a manual, yet optimised method to efficiently overcome these differences is presented. -Secondly, an extension of RAG grammar specifications is proposed to model non-containment references automatically. -The proposed modelling techniques are compared to state-of-the-art modelling tools utilizing a benchmarking framework for continuous model validation, the *Train Benchmark*. +The paper discusses the utilization of reference attribute grammars +(RAGs) for model validation and presents two specific contributions. +First, the differences between models and trees specified by reference +attribute grammars, specifically non-containment references, are +discussed and a manual, yet optimised method to efficiently overcome +these differences is presented. Secondly, an extension of RAG grammar +specifications is proposed to model non-containment references +automatically. The proposed modelling techniques are compared to +state-of-the-art modelling tools utilizing a benchmarking framework for +continuous model validation, the *Train Benchmark*. ### Structure of the Supplementary Artifacts -The artifacts are structured in three parts: +The artifacts are structured in four parts: -- A standalone example of the non-containment references preprocessor -- Benchmark code to reproduce the measurements, including all relevant source codes -- Full collection of all measurement data and diagrams mentioned in the paper +- A standalone example of the non-containment references preprocessor + (relational-rags-0.2.3.zip) +- Benchmark code to reproduce the measurements, including all relevant + source codes + - as a zip file (ModelValidationWithRAGs.zip) + - as a docker container (trainbenchmark-docker.tar) +- Full collection of all measurement data and diagrams mentioned in + the paper (paper-results.zip) ### General Remarks on the presented Listings and Measurements -For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resulting data. -Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams. +For reasons of readability and simplicity, there are some minor +differences in naming in the source codes and the measured resulting +data. Most importantly, the names of the three presented JastAdd +implementation variants are different in the code and the diagrams. -The following table shows the relation of the terminology used in the paper and in the code. +The following table shows the relation of the terminology used in the +paper and in the code. -| Name used in Paper | Name used in result data | Name used in source code | -|-----------------------|-----------------------------|----------------------------| -| Name Lookup | Jastadd (Name Lookup) | jastadd-namelookup | -| Intrinsic References | Jastadd (Optimized) | jastadd-optimized | -| Grammar Extension | Jastadd (Specialized) | jastadd-specialized | + Name used in paper and result data Name used in source code + ------------------------------------ -------------------------- + Name Lookup jastadd-namelookup + Intrinsic References jastadd-intrinsic + Grammar Extension jastadd-relast -## The Grammar Extension Preprocessor *RelAst* +The Grammar Extension Preprocessor *RelAst* +------------------------------------------- -To transform the grammar extension we provide a preprocessor for JastAdd. -This preprocessor including its source code is provided in the `preprocessor` subdirectory. +To transform the grammar extension we provide a preprocessor for +JastAdd. This preprocessor including its source code is provided in the +`preprocessor` subdirectory. Its usage is: -- Run preprocessor on train benchmark (output written to standard output): - - `cat examples/TrainBenchmark.relast` - - `java -jar relast-compiler.jar examples/TrainBenchmark.relast` - -- Run preprocessor and write output to files: - - `java -jar relast-compiler.jar examples/TrainBenchmark.relast --file` - - `cat examples/TrainBenchmarkGen.ast` - - `cat examples/TrainBenchmarkGen.jadd` - -- Run preprocessor and write output to files (with a different list class): - - `java -jar relast-compiler.jar examples/TrainBenchmark.relast --listClass=MyListClass --file` - - `cat examples/TrainBenchmarkGen.ast` - - `cat examples/TrainBenchmarkGen.jadd` - - -## The Train Benchmark +- Build the preprocessor + - `./gradlew build jar` + - copy the jar + `cp build/libs/relational-rags-0.2.3.jar relast-compiler.jar` +- Run preprocessor on train benchmark (output written to standard + output): + - `cat examples/TrainBenchmark.relast` + - `java -jar relast-compiler.jar examples/TrainBenchmark.relast` +- Run preprocessor and write output to files: + - `java -jar relast-compiler.jar examples/TrainBenchmark.relast --file` + - `cat examples/TrainBenchmarkGen.ast` + - `cat examples/TrainBenchmarkGen.jadd` + +The Train Benchmark +------------------- ### Structure of the Train Benchmark -The benchmark is able to measure different scenarios specified by configurations with several kinds of parameters: - -1. **Input Data:** There are two types of input data used in the benchmark, the ``inject`` and the ``repair`` data set. - The former contains *valid* models, i.e., models, which do not contain any of the faults that are supposed to be found by the presented queries. - The latter, `repair`, contains models already containing faults. -2. **Queries:** The queries are used to find the aforementioned faults. For each fault, there are two queries: *repair*, to find the fault, and *inject*, to find places where a fault can be injected. -3. **Transformations:** The transformations performed by the benchmark are, again, two sets: *inject* and *repair* transformations. -4. **Transformation Strategies:** The benchmark does not perform the operation on all matches. - The strategy *fixed* performs the transformation on a given number of matches, while the *proportional* strategy performs them on a given percentage of all matches. - -These settings are defined in a *benchmark scenario*, which can be edited before running the benchmark. - +The benchmark is able to measure different scenarios specified by +configurations with several kinds of parameters: + +1. **Input Data:** There are two types of input data used in the + benchmark, the `inject` and the `repair` data set. The former + contains *valid* models, i.e., models, which do not contain any of + the faults that are supposed to be found by the presented queries. + The latter, `repair`, contains models already containing faults. +2. **Queries:** The queries are used to find the aforementioned faults. + For each fault, there are two queries: *repair*, to find the fault, + and *inject*, to find places where a fault can be injected. +3. **Transformations:** The transformations performed by the benchmark + are, again, two sets: *inject* and *repair* transformations. +4. **Transformation Strategies:** The benchmark does not perform the + operation on all matches. The strategy *fixed* performs the + transformation on a given number of matches, while the + *proportional* strategy performs them on a given percentage of all + matches. + +These settings are defined in a *benchmark scenario*, which can be +edited before running the benchmark. ### Measurement Data -The result data is stored in the directory [paper-results/](paper-results/). -This directory contains two subdirectories: - -- [measurements](paper-results/measurements) contains two directories. - The [inject](paper-results/measurements/inject) subdirectory contains the measurements for the *inject* scenario, which is also included in [inject/BenchmarkScript.groovy](paper-results/measurements/inject/BenchmarkScript.groovy). - The [repair](paper-results/measurements/repair) subdirectory contains the same data for the *repair* scenario in [repair/BenchmarkScript.groovy](paper-results/measurements/repair/BenchmarkScript.groovy). - Both directories contain files with time measurement data (starting with `times`) and the numbers of matches (starting with `matches`). - Each file name contains information on the tool used, the query, and the size of the model. -- [diagrams](paper-results/diagrams) contains the same subdirectories, both containing diagrams with the respective measurements. - The diagrams are generated from the same data as in the paper, but enlarged for better readability. - In particular, the six diagrams presented in the paper are - - [Fig. 7a. Read and Check for RouteSensor (repair)](paper-results/diagrams/repair/Read-and-Check-RouteSensor.pdf) - - [Fig. 7b. Read and Check for ConnectedSegments (repair)](paper-results/diagrams/repair/Read-and-Check-ConnectedSegments.pdf) - - [Fig. 7c. Transformation and Recheck for RouteSensor (inject)](paper-results/diagrams/inject/Transformation-and-Recheck-RouteSensor.pdf) - - [Fig. 7d. Transformation and Recheck for ConnectedSegments (inject)](paper-results/diagrams/inject/Transformation-and-Recheck-ConnectedSegments.pdf) - - [Fig. 7e. Transformation and Recheck for RouteSensor (repair)](paper-results/diagrams/repair/Transformation-and-Recheck-RouteSensor.pdf) - - [Fig. 7f. Transformation and Recheck for ConnectedSegments (repair)](paper-results/diagrams/repair/Transformation-and-Recheck-ConnectedSegments.pdf) - -**Please Note:** The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes. - +The result data is stored in the directory +[paper-results/](paper-results/). This directory contains two +subdirectories: + +- [measurements](paper-results/measurements) contains two directories. + The [individual](paper-results/measurements/individual) subdirectory + contains the measurements for individual queries for both the + *inject* and *repair* scenario. The + [all-queries](paper-results/measurements/all-queries) subdirectory + contains the same data for the a run including all queries in + sequence. Both directories contain files with time measurement data + (starting with `times`) and the numbers of matches (starting with + `matches`). Each file name contains information on the tool used, + the query, and the size of the model. +- [diagrams](paper-results/diagrams) contains the same subdirectories, + containing diagrams with the respective measurements. The diagrams + are generated from the same data as in the paper, but enlarged for + better readability. + +**Please Note:** The measurements were conducted using a timeout for the +whole run. If a run was not completed, no individual times of the steps +appear in the measurements and diagrams. Thus, some tools do not have +measurements for all problem sizes. ### The Source Code -For this publication, we tried to modify the source code of the benchmark itself as little as possible. -Therefore, unfortunately, the code base is rather large and confusing. The following section tries to point to the parts relevant for this paper. - -The benchmark is structured in modules, some of which form the code of the benchmark, some are provided by the contesting tools, and some are related to required model serializations. -There are some naming conventions: -- Tool-related modules are in directories starting with `trainbenchmark-tool`. -- Model serialization-related modules start with `trainbenchmark-generator`. -- All other modules are core modules of the benchmark. - -The JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant. -Each JastAdd configuration must be presented to the benchmark as a separate tool. Thus, there are two directories for each variant, one for the batch processing mode and one for the incremental mode. -Because these two modes share almost all the source code, a third directory is used to store this shared code. -Finally, there is a directory for code shared between all JastAdd variants. -These are the important directories: - -- [JastAdd with Name Lookup](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base) - - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/train.ast) - - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries) - - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) -- [JastAdd with Intrinsic References](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base) - - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/train.ast) - - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/jastadd/queries) - - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-optimized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) -- [JastAdd with Grammar Extension](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base) - - [(Extended) Grammar](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/Train.relast) - - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/jastadd/queries) - - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-specialized-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) -- [Common JastAdd Code](trainbenchmark/trainbenchmark-tool-jastadd-base) +For this publication, we tried to modify the source code of the +benchmark itself as little as possible. Therefore, unfortunately, the +code base is rather large and confusing. The following section tries to +point to the parts relevant for this paper. + +The benchmark is structured in modules, some of which form the code of +the benchmark, some are provided by the contesting tools, and some are +related to required model serializations. There are some naming +conventions: - Tool-related modules are in directories starting with +`trainbenchmark-tool`. - Model serialization-related modules start with +`trainbenchmark-generator`. - All other modules are core modules of the +benchmark. + +The JastAdd-based solutions use a preprocessor to generate Java files, +for the presented variant. Each JastAdd configuration must be presented +to the benchmark as a separate tool. Thus, there are two directories for +each variant, one for the batch processing mode and one for the +incremental mode. Because these two modes share almost all the source +code, a third directory is used to store this shared code. Finally, +there is a directory for code shared between all JastAdd variants. These +are the important directories: + +- [JastAdd with Name + Lookup](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base) + - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/train.ast) + - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/jastadd/queries) + - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) +- [JastAdd with Intrinsic + References](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base) + - [Grammar](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/train.ast) + - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/jastadd/queries) + - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-intrinsic-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) +- [JastAdd with Grammar + Extension](trainbenchmark/trainbenchmark-tool-jastadd-relast-base) + - [(Extended) + Grammar](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/Train.relast) + - [Queries](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/jastadd/queries) + - [Transformations](trainbenchmark/trainbenchmark-tool-jastadd-relast-base/src/main/java/de/tudresden/inf/st/train/jastadd/transformations) +- [Common JastAdd + Code](trainbenchmark/trainbenchmark-tool-jastadd-base) ### Reproducing the Measurements -**<span style="color:red">Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested running the benchmark with a smaller maximum problem size, fewer repetitions, and a shorter timeout.</span>** Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes. Note that running the benchmark requires a significant amount of disk space (up to 10GB when running the full benchmark). - -To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly. -Alternatively, it is, of course, also possible to simply run the provided gradle build scripts. -However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We strongly suggest running the Docker variant. +**[Please Note: Reproducing the graphs as presented in the paper and +supplied here takes a very long time depending on the utilized hardware. +It is strongly suggested running the benchmark with a smaller maximum +problem size, fewer repetitions, and a shorter +timeout.]{style="color:red"}** Most results of the benchmark are +observable with more restricted setup as well. In the following, we will +provide a suggested way to run the benchmark in different sizes. Note +that running the benchmark requires a significant amount of disk space +(up to 10GB when running the full benchmark). + +To reproduce the measurements, there are several options. We provide a +prepared Docker image that can be run directly. Alternatively, it is, of +course, also possible to simply run the provided gradle build scripts. +However, since there are some software requirements imposed by the +benchmark, particularly for creating the diagrams using R. We strongly +suggest running the Docker variant. #### Running the Benchmark with Docker ##### Loading the Docker Image -- Variant 1 (*recommended*): Load the provided docker image - - Prerequisites: An installation of Docker in the `PATH` - - Steps: - - Unpack the provided archive and open a terminal in the extracted directory - - `docker load --input trainbenchmark-docker.tar` -- Variant 2: Build the docker image from the provided Dockerfile - - Prerequisites: An installation of Docker in the `PATH` - - Steps: - - Unpack the provided archive and open a terminal in the extracted directory - - `docker build -t trainbenchmark .` +- Variant 1 (*recommended*): Load the provided docker image + - Prerequisites: An installation of Docker in the `PATH` + - Steps: + - Unpack the provided archive and open a terminal in the + extracted directory + - `docker load --input trainbenchmark-docker.tar` +- Variant 2: Build the docker image from the provided Dockerfile + - Prerequisites: An installation of Docker in the `PATH` + - Steps: + - Unpack the provided archive and open a terminal in the + extracted directory + - `docker build -t trainbenchmark .` ##### Running the Docker Image -- `docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark` -- This makes the results and diagrams available outside the container in the directories `docker-results` and `docker-diagrams` respectively -- Once running, a command prompt is opened and some information is displayed -- Follow the instructions below +- `docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark` +- This makes the results and diagrams available outside the container + in the directories `docker-results` and `docker-diagrams` + respectively +- Once running, a command prompt is opened and some information is + displayed +- Follow the instructions below #### Running the Benchmark directly -- For running a standard run, use one of the following commands: - -| Name | Command | Minimum size | Maximum size | Timeout | Runs | -| ------ | -------------- | ------------ | ------------ | ------- | ---- | -| Small | `./run_small` | 1 | 32 | 60s | 1 | -| Medium | `./run_medium` | 1 | 64 | 10min | 5 | -| Full | `./run_full` | 1 | 512 | 15min | 10 | - -- For running a custom run, - - run `./gradlew preprocess` to generate the grammar from the extended grammar specification - - run `./gradlew build shadowJar -x test` - - configure the scripts by running `./scripts/configure.sh 1 <MAXSIZE> <TIMEOUT in s> <REPETITIONS>` - - Where MAXSIZE is one of 2, 4, 8, 16, 32, 64, 128, 256, 512, or, 1024. The larger sizes use **a lot of** disk space! - - run `./gradlew generate` - - run the benchmark - - run `./gradlew individualInjectBenchmark` for the *inject* scenarios - - run `./gradlew individualRepairBenchmark` for the *repair* scenarios - - Plot the diagrams for the current run: `./gradlew plotIndividual` - -- The resulting data and diagrams is placed in the `results` and the `diagrams` folder - - When running with docker, the data is also in `docker-results` and `docker-diagrams` on the host machine. +- For running a standard run, use one of the following commands: + + Name Command Minimum size Maximum size Timeout Runs + -------- ---------------- -------------- -------------- --------- ------ + Small `./run_small` 1 32 60s 1 + Medium `./run_medium` 1 64 10min 5 + Full `./run_full` 1 512 15min 10 + +- For running a custom run, + - run `./gradlew preprocess` to generate the grammar from the + extended grammar specification + - run `./gradlew build shadowJar -x test` + - configure the scripts by running + `./scripts/configure.sh 1 <MAXSIZE> <TIMEOUT in s> <REPETITIONS>` + - Where MAXSIZE is one of 2, 4, 8, 16, 32, 64, 128, 256, 512, + or, 1024. The larger sizes use **a lot of** disk space! + - run `./gradlew generate` + - run the benchmark + - run `./gradlew individualInjectBenchmark` for the *inject* + scenarios + - run `./gradlew individualRepairBenchmark` for the *repair* + scenarios + - Plot the diagrams for the current run: + `./gradlew plotIndividual` +- The resulting data and diagrams is placed in the `results` and the + `diagrams` folder + - When running with docker, the data is also in `docker-results` + and `docker-diagrams` on the host machine. diff --git a/create_release.sh b/create_release.sh index b2940fb8b306318d1e55369493f8cdfebd667e6b..6e8e49ada176774095e675c6b9aa4629847904fd 100755 --- a/create_release.sh +++ b/create_release.sh @@ -8,8 +8,6 @@ cp -a \ .dockerignore \ README.{md,html,txt} AUTHORS.txt \ Dockerfile docker/ run_docker.sh \ - paper-results/ \ - preprocessor/ \ trainbenchmark/ \ ModelValidationWithRAGs/ # the target directory