diff --git a/README.md b/README.md index 52503e99458e57f6ffe40e2045ee81378604c07b..c9e10dc0a03514c17553b824b9ff8146d2527713 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,5 @@ # Artifacts for "Continuous Model Validation Using Reference Attribute Grammars" -*Note: please use the HTML version of this README.* - -*Also Note: There is a variant of this submission including a docker image (provided as a link) and one without it (uploaded in HotCRP). We encourage using the one including the image, since building the image takes a long time.* - ### Authors - Johannes Mey <johannes.mey@tu-dresden.de> @@ -39,11 +35,11 @@ Most importantly, the names of the three presented JastAdd implementation varian The following table shows the relation of the terminology used in the paper and in the code. -| Name used in Paper | Name used in result data | Name used in source code | -|-----------------------|-----------------------------|----------------------------| -| Name Lookup | Jastadd (Name Lookup) | jastadd-namelookup | -| Intrinsic References | Jastadd (Optimized) | jastadd-optimized | -| Grammar Extension | Jastadd (Specialized) | jastadd-specialized | +| Name used in paper and result data | Name used in source code | +|------------------------------------|----------------------------| +| Name Lookup | jastadd-namelookup | +| Intrinsic References | jastadd-optimized | +| Grammar Extension | jastadd-specialized | ## The Grammar Extension Preprocessor *RelAst* @@ -90,20 +86,13 @@ The result data is stored in the directory [paper-results/](paper-results/). This directory contains two subdirectories: - [measurements](paper-results/measurements) contains two directories. - The [inject](paper-results/measurements/inject) subdirectory contains the measurements for the *inject* scenario, which is also included in [inject/BenchmarkScript.groovy](paper-results/measurements/inject/BenchmarkScript.groovy). - The [repair](paper-results/measurements/repair) subdirectory contains the same data for the *repair* scenario in [repair/BenchmarkScript.groovy](paper-results/measurements/repair/BenchmarkScript.groovy). + The [individual](paper-results/measurements/individual) subdirectory contains the measurements for individual queries for both the *inject* and *repair* scenario. + The [all-queries](paper-results/measurements/all-queries) subdirectory contains the same data for the a run including all queries in sequence. Both directories contain files with time measurement data (starting with `times`) and the numbers of matches (starting with `matches`). Each file name contains information on the tool used, the query, and the size of the model. -- [diagrams](paper-results/diagrams) contains the same subdirectories, both containing diagrams with the respective measurements. +- [diagrams](paper-results/diagrams) contains the same subdirectories, containing diagrams with the respective measurements. The diagrams are generated from the same data as in the paper, but enlarged for better readability. - In particular, the six diagrams presented in the paper are - - [Fig. 7a. Read and Check for RouteSensor (repair)](paper-results/diagrams/repair/Read-and-Check-RouteSensor.pdf) - - [Fig. 7b. Read and Check for ConnectedSegments (repair)](paper-results/diagrams/repair/Read-and-Check-ConnectedSegments.pdf) - - [Fig. 7c. Transformation and Recheck for RouteSensor (inject)](paper-results/diagrams/inject/Transformation-and-Recheck-RouteSensor.pdf) - - [Fig. 7d. Transformation and Recheck for ConnectedSegments (inject)](paper-results/diagrams/inject/Transformation-and-Recheck-ConnectedSegments.pdf) - - [Fig. 7e. Transformation and Recheck for RouteSensor (repair)](paper-results/diagrams/repair/Transformation-and-Recheck-RouteSensor.pdf) - - [Fig. 7f. Transformation and Recheck for ConnectedSegments (repair)](paper-results/diagrams/repair/Transformation-and-Recheck-ConnectedSegments.pdf) - + **Please Note:** The measurements were conducted using a timeout for the whole run. If a run was not completed, no individual times of the steps appear in the measurements and diagrams. Thus, some tools do not have measurements for all problem sizes. diff --git a/trainbenchmark/docs/advanced b/trainbenchmark/docs/advanced index 9a26f62e25ecca2e893d16d2aeff392d4c2f37ca..1d7f2de1edb36e3650113ec336275ab223800c4e 100644 --- a/trainbenchmark/docs/advanced +++ b/trainbenchmark/docs/advanced @@ -12,8 +12,8 @@ echo -e "* ./gradlew generate" echo -e "" echo -e "\033[1;91mBenchmarking" echo -e "\033[0mCan be configured inside:" -echo -e "* trainbenchmark-scripts/src-template/IndividualBenchmarkInjectScript.groovy" -echo -e "* trainbenchmark-scripts/src-template/IndividualBenchmarkRepairScript.groovy" +echo -e "* trainbenchmark-scripts/src/main/resources/basic-settings.json" echo -e "When you are finished, run:" echo -e "* ./gradlew individualInjectBenchmark" echo -e "* ./gradlew individualRepairBenchmark" +echo -e "* ./gradlew combinedBenchmark" diff --git a/trainbenchmark/docs/basic b/trainbenchmark/docs/basic index 4daa18a5ccd3c79cd9cd04945251569557697746..1c97f54d5f7a2a28aa927eb654d1f59a77db76b3 100644 --- a/trainbenchmark/docs/basic +++ b/trainbenchmark/docs/basic @@ -4,8 +4,9 @@ echo -e "\033[0m* configure_small" echo -e "* ./gradlew generate # will generate the test-models in default settings takes ~40gb" echo -e "* ./gradlew individualInjectBenchmark" echo -e "* ./gradlew individualRepairBenchmark" +echo -e "* ./gradlew combinedBenchmark" echo -e "" echo -e "\033[1;96mReading the Benchmark results" echo -e "\033[0mResults of individualInjectBenchmark and individualRepairBenchmark are stored inside:" -echo -e "* XXXX for graphs" -echo -e "* XXXX for csv - use command: csv_viewer data.csv" +echo -e "* diagrams for graphs" +echo -e "* results for csv - use command: csv_viewer data.csv" diff --git a/trainbenchmark/trainbenchmark-scripts/build.gradle b/trainbenchmark/trainbenchmark-scripts/build.gradle index f2b403dc1093cbdcc91dd155eea139e885436864..1aed89a683929aa659f0e979ff518a804cf7c8fe 100644 --- a/trainbenchmark/trainbenchmark-scripts/build.gradle +++ b/trainbenchmark/trainbenchmark-scripts/build.gradle @@ -68,14 +68,3 @@ task generate(dependsOn: 'classes', type: JavaExec) { classpath = sourceSets.main.runtimeClasspath } -//task memory(dependsOn: 'classes', type: JavaExec) { -// group = 'Benchmark' -// main = 'MemoryScript' -// classpath = sourceSets.main.runtimeClasspath -//} -// -//task qpt(dependsOn: 'classes', type: JavaExec) { -// group = 'Benchmark' -// main = 'QueryPlanTester' -// classpath = sourceSets.main.runtimeClasspath -//}