<h1id="artifacts-for-continuous-model-validation-using-reference-attribute-grammars">Artifacts for "Continuous Model Validation Using Reference Attribute Grammars"</h1>
<h1id="artifacts-for-continuous-model-validation-using-reference-attribute-grammars">Artifacts for "Continuous Model Validation Using Reference Attribute Grammars"</h1>
<p><em>Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We ecourage using the one including the image, since building the image takes a long time.</em></p>
<p><em>Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We encourage using the one including the image, since building the image takes a long time.</em></p>
<p>For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resulting data. Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams.</p>
<p>For reasons of readability and simplicity, there are some minor differences in naming in the source codes and the measured resulting data. Most importantly, the names of the three presented JastAdd implementation variants are different in the code and the diagrams.</p>
<p>The following table shows the relation of the terminology used in the paper and in the code.</p>
<p>The following table shows the relation of the terminology used in the paper and in the code.</p>
<table>
<table>
<colgroup>
<colwidth="28%"/>
<colwidth="35%"/>
<colwidth="35%"/>
</colgroup>
<thead>
<thead>
<trclass="header">
<trclass="header">
<th>Name used in Paper</th>
<th>Name used in Paper</th>
...
@@ -74,19 +69,19 @@
...
@@ -74,19 +69,19 @@
</thead>
</thead>
<tbody>
<tbody>
<trclass="odd">
<trclass="odd">
<td><p>Name Lookup</p></td>
<td>Name Lookup</td>
<td><p>Jastadd (Name Lookup)</p></td>
<td>Jastadd (Name Lookup)</td>
<td><p>jastadd-namelookup</p></td>
<td>jastadd-namelookup</td>
</tr>
</tr>
<trclass="even">
<trclass="even">
<td><p>Intrinsic References</p></td>
<td>Intrinsic References</td>
<td><p>Jastadd (Optimized)</p></td>
<td>Jastadd (Optimized)</td>
<td><p>jastadd-optimized</p></td>
<td>jastadd-optimized</td>
</tr>
</tr>
<trclass="odd">
<trclass="odd">
<td><p>Grammar Extension</p></td>
<td>Grammar Extension</td>
<td><p>Jastadd (Specialized)</p></td>
<td>Jastadd (Specialized)</td>
<td><p>jastadd-specialized</p></td>
<td>jastadd-specialized</td>
</tr>
</tr>
</tbody>
</tbody>
</table>
</table>
...
@@ -139,8 +134,8 @@
...
@@ -139,8 +134,8 @@
<p><strong>Please Note:</strong> The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes.</p>
<p><strong>Please Note:</strong> The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes.</p>
<h3id="the-source-code">The Source Code</h3>
<h3id="the-source-code">The Source Code</h3>
<p>For this publication, we tried to modify the source code of the benchmark itself as little as possible. Therefore, unfortunately, the code base is rather large and confusing. The following section tries to point to the parts relevant for this paper.</p>
<p>For this publication, we tried to modify the source code of the benchmark itself as little as possible. Therefore, unfortunately, the code base is rather large and confusing. The following section tries to point to the parts relevant for this paper.</p>
<p>The benchmark is structured in modules, some of which form the code of the benchmark, some are provided by the contesting tools, and some are related to required model serializations. There are some naming conventions: - Tool-related modules are in directories starting with <code>trainbenchmark-tool</code>. - Model serialization-related modules start with <code>trainbenchmark-generator</code>. - All other modules are core modules of the bechmark.</p>
<p>The benchmark is structured in modules, some of which form the code of the benchmark, some are provided by the contesting tools, and some are related to required model serializations. There are some naming conventions: - Tool-related modules are in directories starting with <code>trainbenchmark-tool</code>. - Model serialization-related modules start with <code>trainbenchmark-generator</code>. - All other modules are core modules of the benchmark.</p>
<p>The JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant. Each JastAdd configuraration must be presented to the benchmark as a separate tool. Thus there are two directories for each variant, one for the batch processing mode and one for the incremental mode. Because these two modes share almost all the source code, a third directory is used to store this shared code. Finally, there is a directory for code shared between all JastAdd variants. These are the important directories:</p>
<p>The JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant. Each JastAdd configuration must be presented to the benchmark as a separate tool. Thus, there are two directories for each variant, one for the batch processing mode and one for the incremental mode. Because these two modes share almost all the source code, a third directory is used to store this shared code. Finally, there is a directory for code shared between all JastAdd variants. These are the important directories:</p>
<ul>
<ul>
<li><ahref="trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base">JastAdd with Name Lookup</a>
<li><ahref="trainbenchmark/trainbenchmark-tool-jastadd-namelookup-base">JastAdd with Name Lookup</a>
<h3id="reproducing-the-measurements">Reproducing the Measurements</h3>
<h3id="reproducing-the-measurements">Reproducing the Measurements</h3>
<p><strong><spanstyle="color:red">Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested to run the benchmark with a smaller maximum problem size, less repetitions, and a shorter timeout.</span></strong> Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes. Note that running the benchmark requires a significant amount of disk space (up to 10GB when running the full benchmark).</p>
<p><strong><spanstyle="color:red">Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested running the benchmark with a smaller maximum problem size, fewer repetitions, and a shorter timeout.</span></strong> Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes. Note that running the benchmark requires a significant amount of disk space (up to 10GB when running the full benchmark).</p>
<p>To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly. Alternatively, it is, on course, also possible to simply run the provided gradle build scripts. However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We stronly suggest running the Docker variant.</p>
<p>To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly. Alternatively, it is, on course, also possible to simply run the provided gradle build scripts. However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We strongly suggest running the Docker variant.</p>
<h4id="running-the-benchmark-with-docker">Running the Benchmark with Docker</h4>
<h4id="running-the-benchmark-with-docker">Running the Benchmark with Docker</h4>
<h5id="loading-the-docker-image">Loading the Docker Image</h5>
<h5id="loading-the-docker-image">Loading the Docker Image</h5>
<ul>
<ul>
...
@@ -189,7 +184,7 @@
...
@@ -189,7 +184,7 @@
</ul>
</ul>
<h5id="running-the-docker-image">Running the Docker Image</h5>
<h5id="running-the-docker-image">Running the Docker Image</h5>
<ul>
<ul>
<li><code>docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark</code></li>
<li><code>docker run -it -v "$PWD"/:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark</code></li>
<li>This makes the results and diagrams available outside the container in the directories <code>docker-results</code> and <code>docker-diagrams</code> respectively</li>
<li>This makes the results and diagrams available outside the container in the directories <code>docker-results</code> and <code>docker-diagrams</code> respectively</li>
<li>Once running, a command prompt is opened and some information is displayed</li>
<li>Once running, a command prompt is opened and some information is displayed</li>
*Note: please use the HTML version of this README.*
*Note: please use the HTML version of this README.*
*Also Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We ecourage using the one including the image, since building the image takes a long time.*
*Also Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We encourage using the one including the image, since building the image takes a long time.*
### Authors
### Authors
...
@@ -39,15 +39,11 @@ Most importantly, the names of the three presented JastAdd implementation varian
...
@@ -39,15 +39,11 @@ Most importantly, the names of the three presented JastAdd implementation varian
The following table shows the relation of the terminology used in the paper and in the code.
The following table shows the relation of the terminology used in the paper and in the code.
@@ -120,10 +116,10 @@ The benchmark is structured in modules, some of which form the code of the bench
...
@@ -120,10 +116,10 @@ The benchmark is structured in modules, some of which form the code of the bench
There are some naming conventions:
There are some naming conventions:
- Tool-related modules are in directories starting with `trainbenchmark-tool`.
- Tool-related modules are in directories starting with `trainbenchmark-tool`.
- Model serialization-related modules start with `trainbenchmark-generator`.
- Model serialization-related modules start with `trainbenchmark-generator`.
- All other modules are core modules of the bechmark.
- All other modules are core modules of the benchmark.
The JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant.
The JastAdd-based solutions use a preprocessor to generate Java files, for the presented variant.
Each JastAdd configuraration must be presented to the benchmark as a separate tool. Thus there are two directories for each variant, one for the batch processing mode and one for the incremental mode.
Each JastAdd configuration must be presented to the benchmark as a separate tool. Thus, there are two directories for each variant, one for the batch processing mode and one for the incremental mode.
Because these two modes share almost all the source code, a third directory is used to store this shared code.
Because these two modes share almost all the source code, a third directory is used to store this shared code.
Finally, there is a directory for code shared between all JastAdd variants.
Finally, there is a directory for code shared between all JastAdd variants.
These are the important directories:
These are the important directories:
...
@@ -144,11 +140,11 @@ These are the important directories:
...
@@ -144,11 +140,11 @@ These are the important directories:
### Reproducing the Measurements
### Reproducing the Measurements
**<span style="color:red">Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested to run the benchmark with a smaller maximum problem size, less repetitions, and a shorter timeout.</span>** Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes. Note that running the benchmark requires a significant amount of disk space (up to 10GB when running the full benchmark).
**<span style="color:red">Please Note: Reproducing the graphs as presented in the paper and supplied here takes a very long time depending on the utilized hardware. It is strongly suggested running the benchmark with a smaller maximum problem size, fewer repetitions, and a shorter timeout.</span>** Most results of the benchmark are observable with more restricted setup as well. In the following, we will provide a suggested way to run the benchmark in different sizes. Note that running the benchmark requires a significant amount of disk space (up to 10GB when running the full benchmark).
To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly.
To reproduce the measurements, there are several options. We provide a prepared Docker image that can be run directly.
Alternatively, it is, on course, also possible to simply run the provided gradle build scripts.
Alternatively, it is, on course, also possible to simply run the provided gradle build scripts.
However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We stronly suggest running the Docker variant.
However, since there are some software requirements imposed by the benchmark, particularly for creating the diagrams using R. We strongly suggest running the Docker variant.
#### Running the Benchmark with Docker
#### Running the Benchmark with Docker
...
@@ -167,7 +163,7 @@ However, since there are some software requirements imposed by the benchmark, pa
...
@@ -167,7 +163,7 @@ However, since there are some software requirements imposed by the benchmark, pa
##### Running the Docker Image
##### Running the Docker Image
- `docker run -it -v "$PWD"/docker-results:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark`
- `docker run -it -v "$PWD"/:/trainbenchmark/results:Z -v "$PWD"/docker-diagrams:/trainbenchmark/diagrams:Z trainbenchmark`
- This makes the results and diagrams available outside the container in the directories `docker-results` and `docker-diagrams` respectively
- This makes the results and diagrams available outside the container in the directories `docker-results` and `docker-diagrams` respectively
- Once running, a command prompt is opened and some information is displayed
- Once running, a command prompt is opened and some information is displayed