Merge branch '20-extend-documentation-for-subtree-update-and-incremental-eval' into 'dev'

Resolve "Extend documentation for subtree update and incremental eval"

See merge request !15

pages/docs/adding.md

@@ -90,7 +90,7 @@ task ragConnect(type: JavaExec) {
 
 You might need to add another task for [compiling relast specifications](#compiling-relast-specifications).
 
-## Compiling RelAst specifications
+# Compiling RelAst specifications
 
 The task to compile `RagConnect` specifications is typically accompanied by a task to invoke the [RelAst compiler](http://relational-rags.eu/) and the [JastAdd gradle plugin](https://plugins.gradle.org/plugin/org.jastadd).
 Currently, the parameter `--useJastAddNames` is **required**, and it may cause incompatibilities if not set.
@@ -124,13 +124,3 @@ One also has to specify the dependencies to get correct ordering of tasks.
 generateAst.dependsOn relastToJastAdd
 relastToJastAdd.dependsOn ragConnect
 ```
-
-## Introduced dependencies
-
-RagConnect itself does not introduce dependencies.
-However, depending on the selected protocols (see [compiler options](/using#compiler-options)), additional dependencies are required.
-
-| Protocol | Dependency (Gradle format) | Remarks |
-|---|---|---|
-| `mqtt` | `group: 'org.fusesource.mqtt-client', name: 'mqtt-client', version: '1.15'` | Mqtt is selected by default, so this dependency therefore is required "by default". Might work with other versions as well. |
-| `rest` | `group: 'com.sparkjava', name: 'spark-core', version: '2.9.3'` | Might work with newer versions as well. For debugging, it is beneficial to include an implementation for [SLF4J](http://www.slf4j.org/). |

pages/docs/compiler.md

+# Compiler options
+
+The compiler is JastAdd-compliant, i.e., it accepts all flags available for JastAdd, though there is no process how to chain pre-processors _yet_.
+Additional options are as follows.
+
+| Name | Required (Default) | Description |
+|-|-|-|
+| `--rootNode` | Yes | Root node in the base grammar. |
+| `--protocols` | No (`mqtt`) | Protocols to enable, currently available: `mqtt, rest`. |
+| `--printYaml` | No (false) | Print out YAML instead of generating files. |
+| `--verbose` | No (false) | Print more messages while compiling. |
+| `--logReads` | No (false) | Enable logging for every received message. |
+| `--logWrites` | No (false) | Enable logging for every sent message. |
+| `--logIncremental` | No (false) | Enable logging for observer in incremental dependency tracking. |
+| `--experimental-jastadd-329` | No (false) | Use trace events `INC_FLUSH_START` and `INC_FLUSH_END` ([JastAdd issue #329][jastadd-issue-329]), see [section about automatic dependency tracking](/using#dependency-tracking-automatically-derived). |
+| `--incremental` | No (false) | Enables incremental dependency tracking (if `trace` is also set appropriately). |
+| `--trace[=flush]` | No (false) | Enables incremental dependency tracking (if `incremental` is also set appropriately). |
+| `--version` | No (false) | Print version info and exit (reused JastAdd option) |
+| `--o` | No (`.`) | Output directory (reused JastAdd option) |
+
+All files to be process have to be passed as arguments.
+Their type is decided by the file extension (`ast` and `relast` for input grammars, `connect` and `ragconnect` for RagConnect definitions file).
+
+# Additional software dependencies
+
+Using RagConnect itself does not introduce dependencies.
+However, depending on the selected protocols and/or used features, additional dependencies are required when using the generated code.
+
+## Communication protocol characteristics
+
+### MQTT
+
+- Protocol identifier: `mqtt`
+- URI scheme: `mqtt://<broker-host>[:port]/<topic>`
+- Default port: 1883
+- Type for mapping definitions: `byte[]`
+- Required runtime dependencies:
+    - `group: 'org.fusesource.mqtt-client', name: 'mqtt-client', version: '1.15'`
+- Additional remarks:
+    - First leading slash not included in topic.
+    - Mqtt is selected by default, so this dependency therefore is required "by default".
+    - Might work with other versions of `org.fusesource.mqtt-client.mqtt.client` as well.
+
+### REST
+
+- Protocol identifier: `rest`
+- URI scheme: `rest://localhost[:port]/<path>`
+- Default port: 4567
+- Type for mapping definitions: `String`
+- Required runtime dependencies:
+    - `group: 'com.sparkjava', name: 'spark-core', version: '2.9.3'`
+- Additional remarks:
+    - Host is always `localhost`.
+    - Might work with newer versions of `com.sparkjava.spark-core` as well.
+    - For debugging, it is beneficial to include an implementation for [SLF4J](http://www.slf4j.org/).
+
+## Used features
+
+### Automatic dependency tracking
+
+- Condition: When passing `--incremental` and `--trace=flush` to RagConnect
+- Required runtime dependencies: _none_
+- Required options for RelAST compiler: _none_
+- Required options for JastAdd:
+    - `--incremental`
+    - `--trace=flush`
+- Remarks:
+    - Other (additional) values passed to those two options must be equal (e.g., `--incremental=param` passed to RagConnect must be also passed to JastAdd)
+    - Other values besides `flush` can be added to `--trace`
+    - [Feature description](/using#dependency-tracking-automatically-derived)
+
+### (Safer) Automatic dependency tracking
+
+- Condition: When passing `--experimental-jastadd-329` to RagConnect
+- Required runtime dependencies: _none_
+- Required options for RelAST compiler: _none_
+- Required options for JastAdd: _none_
+- Remarks:
+    - JastAdd version has to support `INC_FLUSH_START` and `INC_FLUSH_END` (i.e., has [issue #329][jastadd-issue-329] resolved)
+    - [Feature description](/using#dependency-tracking-automatically-derived)
+
+### Tree/List Endpoints
+
+- Condition: When using `tree` or `list` endpoints along with default mappings
+- Required runtime dependencies:
+    - `group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.12.1'`
+    - `group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.12.1'`
+- Required options for RelAST compiler:
+    - `--serializer=jackson`
+- Required options for JastAdd: _none_
+- Remarks:
+    - [Feature description](/using#an-advanced-example)
+
+
+[jastadd-issue-329]: https://bitbucket.org/jastadd/jastadd2/issues/329/add-event-for-completion-of-flush

pages/docs/using.md

@@ -2,14 +2,14 @@
 
 The full example is available at <https://git-st.inf.tu-dresden.de/jastadd/ragconnect-minimal>.
 
-## Preparation
+## Preparation and Specification
 
 The following examples are inspired by the real test case [read1write2](https://git-st.inf.tu-dresden.de/jastadd/ragconnect-tests/-/tree/master/ragconnect.tests/src/test/01-input/read1write2)
 The idea is to have two nonterminals, where input information is received on one of them, and - after transformation - is sent out by both.
 
 Let the following grammar be used:
 
-```bnf
+```
 A ::= <Input:String> /<OutputOnA:String>/ B* ;
 B ::= /<OutputOnB:String>/ ;
 ```
@@ -46,8 +46,11 @@ eq A.getB().input() = getInput();
 ```
 
 In other words, `OutputOnA` depends on `Input` of the same node, and `OutputOnB` depends on `Input` of its parent node.
-Currently, those dependencies have to be explicitly written down. It is expected, that in future version, this won't be necessary anymore.
-This happens also in the DSL (dependencies have to be named to uniquely identify them):
+Currently, those dependencies can be explicitly written down, or incremental evaluation can be used.
+
+### Dependency tracking: Manually specified
+
+This specification happens also in the DSL (dependencies have to be named to uniquely identify them):
 
 ```java
 // dependency definitions
@@ -55,6 +58,16 @@ A.OutputOnA canDependOn A.Input as dependencyA ;
 B.OutputOnB canDependOn A.Input as dependencyB ;
 ```
 
+### Dependency tracking: Automatically derived
+
+To automatically track dependencies, the two additional parameters `--incremental` and `--trace=flush` have to be provided to both RagConnect and (in the later stage) JastAdd.
+This will generate a different implementation of RagConnect relying on enabled incremental evaluation of JastAdd.
+The value for `incremental` has only been tested for `incremental=param`.
+The value for `trace` can include other values besides `flush`.
+
+An experimental, optimized version can be selected using `--experimental-jastadd-329` reducing the risk of conflicts between concurrent attribute evaluations.
+However, this requires a version of JastAdd that resolved the [issue 329](https://bitbucket.org/jastadd/jastadd2/issues/329/add-event-for-completion-of-flush).
+
 ## Using generated code
 
 After specifying everything, code will be generated if [setup properly](/adding).
@@ -70,7 +83,7 @@ a.addB(b1);
 a.addB(b2);
 ```
 
-Now, we have to set the dependencies as described earlier.
+If necessary, we have to set the dependencies as [described earlier](#dependency-tracking-manually-specified).
 
 ```java
 // a.OutputOnA -> a.Input
@@ -82,7 +95,7 @@ b2.addDependencyB(a);
 ```
 
 Finally, we can actually _connect_ the tokens.
-Depending on the enabled protocols, [different URI schemes are allowed](#communication-protocol-characteristics).
+Depending on the enabled protocols, [different URI schemes are allowed](/compiler#communication-protocol-characteristics).
 In this example, we use the default protocol: MQTT.
 
 ```java
@@ -98,39 +111,14 @@ Please note, that the first leading slash (`/`) is removed for MQTT topics, e.g.
 
 For sending endpoints, there is a second boolean parameter to specify whether the current value shall be sent immediately after connecting.
 
-## Communication protocol characteristics
-
-| Protocol | URI scheme | Default port | Type for mapping definitions | Remarks |
-|-|-|-|-|-|
-| `mqtt` | `mqtt://<broker-host>[:port]/<topic>` | 1883 | `byte[]` | First leading slash not included in topic. |
-| `rest` | `rest://localhost[:port]/<path>` | 4567 | `String` | Host is always `localhost`. |
-
-## Compiler options
-
-The compiler is JastAdd-compliant, i.e., it accepts all flags available for JastAdd, though there is no process how to chain pre-processors _yet_.
-Additional options are as follows.
-
-| Name | Required (Default) | Description |
-|-|-|-|
-| `--rootNode` | Yes | Root node in the base grammar. |
-| `--protocols` | No (`mqtt`) | Protocols to enable, currently available: `mqtt, rest`. |
-| `--printYaml` | No (false) | Print out YAML instead of generating files. |
-| `--verbose` | No (false) | Print more messages while compiling. |
-| `--logReads` | No (false) | Enable logging for every received message. |
-| `--logWrites` | No (false) | Enable logging for every sent message. |
-| `--version` | No (false) | Print version info and exit (reused JastAdd option) |
-| `--o` | No (`.`) | Output directory (reused JastAdd option) |
-
-All files to be process have to be passed as arguments. Their type is decided by the file extension (`ast` and `relast` for input grammars, `connect` and `ragconnect` for RagConnect definitions file).
-
-## Remarks
+## Remarks for using manual dependency tracking
 
 When constructing the AST and connecting it, one should always set dependencies before connecting, especially if updates already arriving for receiving endpoints.
 Otherwise, updates might not be propagated after setting dependencies, if values are equal after applying transformations of mapping definitions.
 
 As an example, when using the following grammar and definitions for RagConnect ...
 
-```bnf
+```
 A ::= <Input:int> /<Output:String>/ ;
 ```
 
@@ -146,3 +134,121 @@ Round maps float f to int {:
 ```
 
 ... connecting first could mean to store the first rounded value and not propagating this update, since no dependencies are set, and not propagating further updates leading to the same rounded value even after setting the dependencies.
+
+# An advanced example
+
+Non-terminal children can also be selected as endpoints (not only tokens).
+
+## Normal Non-Terminal Children
+
+Receiving normal non-terminal children and optionals means to replace them with a new node deserialized from the received message.
+Sending them involves serializing a node, and sending this representation in a message.
+
+Suppose, the following (shortened) grammar is used (inspired from the testcase [tree](https://git-st.inf.tu-dresden.de/jastadd/ragconnect-tests/-/tree/master/ragconnect.tests/src/test/01-input/tree))
+
+```
+Root ::= SenderRoot ReceiverRoot ;
+SenderRoot ::= <Input:int> /Alfa/ ;
+ReceiverRoot ::= Alfa ;
+Alfa ::= // some content ...
+```
+
+Now, the complete node of type `Alfa` can be sent, and received again using the following connect specification:
+
+```
+send tree SenderRoot.Alfa ;
+receive tree ReceiverRoot.Alfa ;
+```
+
+Currently, receiving and sending trees requires the explicit demarcation from tokens using the keyword `tree`.
+
+To process non-terminals, default mappings are provided for every non-terminal type of the used grammar.
+They use the JSON serialization offered by the RelAST compiler, i.e., interpret the message as a `String`, deserialize the content reading the message as JSON, or vice versa.
+Additional dependencies are required to use this feature, as detailed in [the compiler section](/compiler#treelist-endpoints).
+
+## Receiving List Children
+
+When receiving list children, there are a few more options to match the connection to given requirements.
+
+Suppose we use a similar grammar as above, i.e.:
+
+```
+SenderRoot ::= /AlfaList:Alfa*/ /SingleAlfa:Alfa/;
+ReceiverRoot ::= Alfa* ;
+```
+
+Several options are possible:
+
+### list
+
+A message for a list endpoint can be interpreted as a complete list (a sequence of nodes of type `Alfa`) by using the `list` keyword instead of `tree`:
+
+```
+receive list ReceiverRoot.Alfa ;
+```
+
+### list + with add
+
+Upon receiving the message, the deserialized list can also be appended to the existing list instead of replace the latter.
+This can be achieved using the keyword `with add` in addition to the keyword `list`:
+
+```
+receive list with add ReceiverRoot.Alfa ;
+```
+
+### tree (indexed)
+
+A message for a list endpoint can also be interpreted as an element of this list.
+
+```
+receive tree ReceiverRoot.Alfa ;
+```
+
+Upon connection, the index of the deserialized element to set, has to be passed (`1` in the example below).
+The list must have enough elements once a message is received.
+
+```java
+receiverRoot.connectAlfa("<some-url>", 1);
+```
+
+### tree (wildcard)
+
+Similar to the `tree (indexed)` case above, messages are interpreted as an element of the list, but the connection can also be made using a "wildcard topic" and without an index.
+Then, once a message is received from a new concrete topic, the deserialized element will be appended to the list and this topic is associated with the index of the newly added element.
+Any further message from that topic will replace the element at the associated index.
+In the short example below, MQTT is used to with a wildcard topic, as `#` matches every sub-topic.
+
+```java
+receiverRoot.connectAlfa("mqtt://<broker>/some/topic/#");
+
+// list is initially empty
+assertEquals(receiverRoot.getAlfaList(), list());
+// after receiving "1" on new topic "some/topic/one" (index 0)
+assertEquals(receiverRoot.getAlfaList(), list("1"));
+// after receiving "other" on new topic "some/topic/two" (index 1)
+assertEquals(receiverRoot.getAlfaList(), list("1", "other"));
+// after receiving "new" on existing topic "some/topic/one" (index 0)
+assertEquals(receiverRoot.getAlfaList(), list("new", "other"));
+```
+
+### tree (indexed/wildcard) + with add
+
+Combining `tree` and `with add` results in a connection, where messages are interpreted as elements of the list, and new elements are appended to the existing list.
+In that case, wildcard and non-wildcard connections behave in the same way, as no index has to be passed, and the element is always append at the end.
+Reusing the example from above, the following observations can be made.
+
+```java
+receiverRoot.connectAlfa("mqtt://<broker>/some/topic/#");
+// or
+receiverRoot.connectAlfa("mqtt://<broker>/some/topic/one");
+receiverRoot.connectAlfa("mqtt://<broker>/some/topic/two");
+
+// list is initially empty
+assertEquals(receiverRoot.getAlfaList(), list());
+// after receiving "1" on topic "some/topic/one"
+assertEquals(receiverRoot.getAlfaList(), list("1"));
+// after receiving "other" on topic "some/topic/two"
+assertEquals(receiverRoot.getAlfaList(), list("1", "other"));
+// after receiving "new" on topic "some/topic/one"
+assertEquals(receiverRoot.getAlfaList(), list("1", "other", "new"));
+```

pages/mkdocs.yml

@@ -6,6 +6,7 @@ nav:
   - "RagConnect by Example": using.md
   - "Use Cases": use_cases.md
   - "Adding RagConnect to your project": adding.md
+  - "Compiler options": compiler.md
   - "Inner workings": inner-workings.md
   - "Extending RagConnect": extending.md
   - "Changelog": changelog.md