<h1id="artifacts-for-continuous-model-validation-using-reference-attribute-grammars">Artifacts for “Continuous Model Validation Using Reference Attribute Grammars”</h1>
<p><em>Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We encourage using the one including the image, since building the image takes a long time.</em></p>
<p>The paper discusses the utilization of reference attribute grammars (RAGs) for model validation and presents two specific contributions. First, the differences between models and trees specified by reference attribute grammars, specifically non-containment references, are discussed and a manual, yet optimised method to efficiently overcome these differences is presented. Secondly, an extension of RAG grammar specifications is proposed to model non-containment references automatically. The proposed modelling techniques are compared to state-of-the-art modelling tools utilizing a benchmarking framework for continuous model validation, the <em>Train Benchmark</em>.</p>
<h3id="structure-of-the-supplementary-artifacts">Structure of the Supplementary Artifacts</h3>
<p>The artifacts are structured in three parts:</p>
<p>The artifacts are structured in four parts:</p>
<ul>
<li>A standalone example of the non-containment references preprocessor</li>
<li>Benchmark code to reproduce the measurements, including all relevant source codes</li>
<li>Full collection of all measurement data and diagrams mentioned in the paper</li>
<li>A standalone example of the non-containment references preprocessor (relational-rags-0.2.3.zip)</li>
<li>Benchmark code to reproduce the measurements, including all relevant source codes
<ul>
<li>as a zip file (ModelValidationWithRAGs.zip)</li>
<li>as a docker container (trainbenchmark-docker.tar)</li>
</ul></li>
<li>Full collection of all measurement data and diagrams mentioned in the paper (paper-results.zip)</li>
</ul>
<h3id="general-remarks-on-the-presented-listings-and-measurements">General Remarks on the presented Listings and Measurements</h3>
<p>For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resulting data. Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams.</p>
...
...
@@ -70,26 +30,22 @@
<table>
<thead>
<trclass="header">
<th>Name used in Paper</th>
<th>Name used in result data</th>
<th>Name used in source code</th>
<thstyle="text-align: left;">Name used in paper and result data</th>
<thstyle="text-align: left;">Name used in source code</th>
<p>To transform the grammar extension we provide a preprocessor for JastAdd. This preprocessor including its source code is provided in the <code>preprocessor</code> subdirectory.</p>
<p>Its usage is:</p>
<ul>
<li>Build the preprocessor
<ul>
<li><code>./gradlew build jar</code></li>
<li>copy the jar <code>cp build/libs/relational-rags-0.2.3.jar relast-compiler.jar</code></li>
</ul></li>
<li>Run preprocessor on train benchmark (output written to standard output):
<h3id="structure-of-the-train-benchmark">Structure of the Train Benchmark</h3>
...
...
@@ -126,18 +81,10 @@
</ol>
<p>These settings are defined in a <em>benchmark scenario</em>, which can be edited before running the benchmark.</p>
<h3id="measurement-data">Measurement Data</h3>
<p>The result data is stored in the directory <ahref="paper-results/"class="uri">paper-results/</a>. This directory contains two subdirectories:</p>
<p>The result data is stored in the directory <ahref="paper-results/">paper-results/</a>. This directory contains two subdirectories:</p>
<ul>
<li><ahref="paper-results/measurements">measurements</a> contains two directories. The <ahref="paper-results/measurements/inject">inject</a> subdirectory contains the measurements for the <em>inject</em> scenario, which is also included in <ahref="paper-results/measurements/inject/BenchmarkScript.groovy">inject/BenchmarkScript.groovy</a>. The <ahref="paper-results/measurements/repair">repair</a> subdirectory contains the same data for the <em>repair</em> scenario in <ahref="paper-results/measurements/repair/BenchmarkScript.groovy">repair/BenchmarkScript.groovy</a>. Both directories contain files with time measurement data (starting with <code>times</code>) and the numbers of matches (starting with <code>matches</code>). Each file name contains information on the tool used, the query, and the size of the model.</li>
<li><ahref="paper-results/diagrams">diagrams</a> contains the same subdirectories, both containing diagrams with the respective measurements. The diagrams are generated from the same data as in the paper, but enlarged for better readability. In particular, the six diagrams presented in the paper are
<ul>
<li><ahref="paper-results/diagrams/repair/Read-and-Check-RouteSensor.pdf">Fig. 7a. Read and Check for RouteSensor (repair)</a></li>
<li><ahref="paper-results/diagrams/repair/Read-and-Check-ConnectedSegments.pdf">Fig. 7b. Read and Check for ConnectedSegments (repair)</a></li>
<li><ahref="paper-results/diagrams/inject/Transformation-and-Recheck-RouteSensor.pdf">Fig. 7c. Transformation and Recheck for RouteSensor (inject)</a></li>
<li><ahref="paper-results/diagrams/inject/Transformation-and-Recheck-ConnectedSegments.pdf">Fig. 7d. Transformation and Recheck for ConnectedSegments (inject)</a></li>
<li><ahref="paper-results/diagrams/repair/Transformation-and-Recheck-RouteSensor.pdf">Fig. 7e. Transformation and Recheck for RouteSensor (repair)</a></li>
<li><ahref="paper-results/diagrams/repair/Transformation-and-Recheck-ConnectedSegments.pdf">Fig. 7f. Transformation and Recheck for ConnectedSegments (repair)</a></li>
</ul></li>
<li><ahref="paper-results/measurements">measurements</a> contains two directories. The <ahref="paper-results/measurements/individual">individual</a> subdirectory contains the measurements for individual queries for both the <em>inject</em> and <em>repair</em> scenario. The <ahref="paper-results/measurements/all-queries">all-queries</a> subdirectory contains the same data for the a run including all queries in sequence. Both directories contain files with time measurement data (starting with <code>times</code>) and the numbers of matches (starting with <code>matches</code>). Each file name contains information on the tool used, the query, and the size of the model.</li>
<li><ahref="paper-results/diagrams">diagrams</a> contains the same subdirectories, containing diagrams with the respective measurements. The diagrams are generated from the same data as in the paper, but enlarged for better readability.</li>
</ul>
<p><strong>Please Note:</strong> The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes.</p>
<h5id="running-the-docker-image">Running the Docker Image</h5>
<ul>
<li><code>docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark</code></li>
<li><code>docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark</code></li>
<li>This makes the results and diagrams available outside the container in the directories <code>docker-results</code> and <code>docker-diagrams</code> respectively</li>
<li>Once running, a command prompt is opened and some information is displayed</li>
