Update

sections/ODGAppr.tex

@@ -4,7 +4,7 @@ We saw the constraint-based approach tests with parameter dependencies. There is
 What RESTTESTGEN does differently is it analyzes operation dependencies of an API and computes an operation dependency graph.
 
 %Process
-This graph is directed, which means that bidirectional relations are not always expected. If there are two operations derived as nodes and one edge between them, the edge is labeled with a data. This data is an output data of one operation and an input data of the another operation (e.g. an operations getUsers and getUserById could have a data dependency with the data userId). 
+This graph is a directed graph. If there are two operations derived as nodes and one edge between them, the edge is labeled with a data. This data is an output data of one operation and an input data of the another operation (e.g. an operations getUsers and getUserById could have a data dependency with the data userId). 
 After this analysis is done, RESTTESTGEN is ready to generate valid and also invalid test cases. \\
 Firstly, test cases are automatically generated with the module, Nominal Tester. Inputs of this module are an OpenAPI specification and its analyzed operation dependency graph. Test cases created in this module comform the specification and its constraints. \\
 Subsequently, Error Handling Tester takes generated valid test cases as input and constructs several invalid test cases based on the constructions of inputs. Sending invalid test cases provokes the data validation of the target API and may create unexpected accepted responses.

sections/background.tex

 \chapter{Background}\label{ch:background}
-In this section, basic technologies related to this work, (i.e. Software Testing, REST API, OpenAPI Specification, Reference Attribute Grammar) are presented.
+In this Chapter, concepts and technologies related and used as the basis of this work are presented.
 
 \input{sections/testing}
 \input{sections/restapi}

sections/experimentresults.tex

 \chapter{Suitability of RAGs for OpenAPI Testing}\label{ch:suitability}
 
+To evaluate how constructive and suitable RAGs are for OpenAPI testing, we review research questions in following sections. In \Cref{sec:feasibility}, results for \textbf{RQ1} and \textbf{RQ2} are summarized. In \Cref{sec:benefits}, RAG's features that were helpful at implementing and features that could be extended more in RAGO API are discussed.
+
 \input{sections/feasibility}
 \input{sections/benefits}
\ No newline at end of file
-
-\begin{comment}
-> To evaulate how well-suited RAGs are for API testing, we revisit the research questions...
-
-### 6.1 Feasibility
-
-RQ1 RQ2
-
-- current testing: mostly fuzzing
-- different additions:
-  - inference
-  - test model
-- this can be solved with the RAG tool presented in \cref{chap:RAGO}
-- in \cref{sec:inf} we show how inference can be done
-- \cite{ttc-rag-von-ceur} for test model/model connections
-  - http://ceur-ws.org/Vol-2310/ A JastAdd- and ILP-based Solution to the Software-Selection and Hardware-Mapping-Problem at the TTC 2018
-
-### 6.2 Benefits
-- schöne beschreibung mit attributen (verweis auf beispiele aus 5)
-- erweiterbarkeit von RAGs
-  - https://programming-journal.org/2020/4/15/
-  - https://scholar.google.de/scholar?hl=de&as_sdt=0%2C5&q=exstensible+java+compiler+hedin&btnG=
-\end{comment}
\ No newline at end of file

sections/feasibility.tex

 \section{Feasibility} \label{sec:feasibility}
 
+\begin{description}
+\item{\textbf{RQ1 : Which approaches and techniques for automated tests of OpenAPI specifications are researched and developed so far?}}
+\end{description}
+
+The results of current existing approaches are shown in Table 3.1. %\cite{tab:appr-sum}
+As we can see, most of current approaches to test OpenAPI are in black-box and use Fuzzing method to generate test cases. There are differences between approaches at test generation phase. Several of them inference parameters or operations or use test model generation. They also have differences at bug types. They all consider 500 status codes as bugs, but, three of them consider 200 status codes in test cases expected errors and 4xx status codes in test cases expected successful responses as bugs. Exact bug reports of approaches which experimented with industrial APIs excluded Property-based Approach formulate that they can be interesting research targets combined with RAG.
+
+\begin{description}
+\item{\textbf{RQ2 : Are suggested testing approaches from the literatures also available in RAG?}}
+\end{description}
+
+To determine an answer of this question, we have constructed a data structure for OpenAPI specification in \Cref{ch:rago} to parse OpenAPI documents and validated the parser with 98 commercial APIs, i.e. the parser taken an OpenAPI document returns the same elements except elements with empty values and sibling elements of references in an OpenAPI document. The input and output documents still semantically are same. In \Cref{ch:stm}, we have also implemented two Fuzzing approaches (Random Testing, Parameter Inference). Parameter Inference is motivated by Specification-based Approach\cite{ed2018automatic} and RESTTESTGEN\cite{viglianisi2020resttestgen}, where seperate properties of responses are collected and used as inputs in a parameter with the same schema of a response. \\
+
+
 \begin{comment}
 > To evaulate how well-suited RAGs are for API testing, we revisit the research questions...
 
@@ -16,4 +30,39 @@ RQ1 RQ2
 - \cite{ttc-rag-von-ceur} for test model/model connections
   - http://ceur-ws.org/Vol-2310/ A JastAdd- and ILP-based Solution to the Software-Selection and Hardware-Mapping-Problem at the TTC 2018
 
+The results of Nominal Tester module on the set of case
+studies are shown in Table I. As we can see, all the 87 case
+studies (first line) have been subject to automated test case
+generation and for 62 of them (second line) at least one test
+for a nominal execution scenario (status code 2xx) could be
+automatically generated. For 20 case studies, the test case
+automatically generated by RESTTESTGEN exposed errors
+that were not properly handled by the REST API (status
+code 5xx). On 66 case studies, invalid response messages
+were observed, they are responses inconsistent with their
+schema defined in the Swagger.
+Table II shows a more detailed perspective, focusing on
+the case study operations. Among the total 2,612 operations, for 2,560 of them test cases could be generated by
+RESTTESTGEN. The untested operations are due to some
+failure of the tool, such as unsupported input parameter
+generation (e.g., files to be uploaded).
+In particular, automatically generated test cases found a
+nominal execution for 625 of them (status code 2xx) and an
+ungraceful error for 151 (status code 5xx).
+For 435 test cases the status code was 4xx, but they are not
+shown in the table, because they were hard to classify with
+a black-box access. In fact, they might be graceful errors
+due to programming defects, or just rejected requests due to
+failures by RESTTESTGEN in generating appropriate inputs.
+Test cases with validation errors are still a majority: in
+1,733 tests the response did not match the declared schema.
+Considering these results, we can formulate the following
+answer to RQN :
+The Nominal Tester module of RESTTESTGEN is effective
+in automatically generating test cases with black box access, because it was able to test 2,560 operations
+out of 2,612 operations on real world REST APIs. These
+tests exposed 151 faults in the form of not correctly
+handled internal errors and 1,733 inconsistent response
+messages.
+
 \end{comment}
\ No newline at end of file

sections/rag.tex

@@ -2,12 +2,6 @@
 One interesting formal system which is applicable as an approach for automated testing with OpenAPI specifications is RAGs.\\
 RAG is an extension of Attribute Grammars (AG) while AG is an extension to context-free grammars again.
 
-\begin{figure}
-\centering
-\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{ast}
-\caption{Example AST, motivated by \cite{hedin2009introductory}}\label{fig:ast}
-\end{figure}
-
 %Attribute Grammar
 The concept of AG was introduced by Knuth \cite{knuth1968semantics} as a solution of problems with context-free grammars.
 Context-free grammars (e.g. EBNF, BNF) can only specify syntax with terminal, non-terminal symbols and production rules, but in computer programs it is necessary to define the meaning of semantic rules, data types and values while AGs allow this possible. \\
@@ -27,8 +21,8 @@ Such features of RAG represent advantages over AG, largely in efficiency. It is
 
 \begin{figure}
 \centering
-\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{jastadd}
-\caption{Architecture of JastAdd}\label{fig:jastadd}
+\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{ast}
+\caption{Example AST, motivated by \cite{hedin2009introductory}}\label{fig:ast}
 \end{figure}
 
 %JastAdd
@@ -51,6 +45,8 @@ DivExp:BinExp ;
 MinusExp:BinExp ;
 \end{lstlisting}
 
+\newpage
+
 \begin{lstlisting}[language=JRAG,label={lst:example-attributes},caption={Example of synthesized attributes}]
 aspect Printing{
   syn String ASTNode.print();
@@ -60,6 +56,12 @@ aspect Printing{
 }
 \end{lstlisting}
 
+\begin{figure}
+\centering
+\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{jastadd}
+\caption{Architecture of JastAdd}\label{fig:jastadd}
+\end{figure}
+
 %Research questions
 %With the RAG-based tool generator system, JastAdd, there are two questions this work tries to clarify about automated testing of OpenAPI interfaces. These are following: \\ (i) How can OpenAPI specifications be described in RAG? \\ (ii) Which and how much efficiency can RAGs produce by using them while automated testing of OpenAPI interfaces?
 

sections/ragoGram.tex

@@ -5,11 +5,11 @@ As mentioned in section 2.3, OpenAPI specifications are written in structured JS
 % AST
 To use OpenAPI in RAG, it is firstly necessary to rewrite the OpenAPI structure in an AST. We have constructed this AST in 95 AST-nodes to define 30 objects. The version of OpenAPI considered in this framework is 3.0.1. To have a better overview, the definition of Parameter Object in our AST is shown in Listing 4.1 and can be compared with the definition in the OpenAPI official GitHub$^{1}$. 
  
-\begin{lstlisting}[language=AST,label={lst:parameter-ast},caption={Parameter Object}]
+\begin{lstlisting}[language=AST,label={lst:rago-ast},caption={Parameter Object}]
 abstract ParameterOb;
 ParameterReference : ParameterOb ::= <Ref>;
 ParameterObject : ParameterOb ::= <Name> <In> <Description> <Required:Boolean> <DeprecatedBoolean:Boolean> <AllowEmptyValue:Boolean> <Style> <Explode:Boolean> <AllowReserved:Boolean> [SchemaOb] <Example:Object> ExampleTuple* ContentTuple* Extension*;
-\end{lstlisting}
+\end{lstlisting} \label{lst:rago-one}
 
 During transferring the structure from OpenAPI to RAG, several properties of JastAdd could be clearly noticed:
 \begin{enumerate}

sections/restapi.tex

@@ -6,7 +6,7 @@ The concept of REST has been proposed by Roy Fielding in 2000 \cite{fielding2000
 %Constraints
 It offers constraints to increase simplicity, performance, visibility, modifiability, portability and reliability of APIs. \\
 Constraints defined in REST are following: 
-\begin{itemize}
+\begin{description}
 \item Client-Server: API must be a connection in a client-server architecture.
 \item Stateless: Every request must be stateless (i.e. independent of other requests).
 \item Cache: Data in API must be cacheable.
@@ -19,30 +19,30 @@ Constraints defined in REST are following:
 \item Each message includes enough information to describe how to process the message.
 \item HATEOAS (Hypermedia as the Engine of Application State) is available.
 \end{itemize}
-\end{itemize}
+\end{description}
 
 %Operation
 With this set of constraints system resources are characterized by URIs, sent as requests and modified with CRUD operations (Create, Read, Update, Delete), that are mapped to the HTTP methods (POST, GET, PUT, DELETE). Additionally, URIs, header and body objects are modified by parameters (Query, Path, Body, Header, Form). To change URIs and call an operation with variables, parameters in Query and Path are used. \\
 An example for a web service with REST API might be a web service of a pet store. Example operations for this REST API could be: \\
-\begin{itemize}
+\begin{description}
 \item GET /pets (returns all pet information)
 \item GET /pets/findByStatus?k1=v1\&k2=v2 (returns information of users with Query parameters k1 = v1 and k2 = v2 )
 \item GET /users/\{id\} (returns information of an user with the given Path parameter, id)
 \item POST /users (creates a new user)
 \item PUT /users/\{id\} (updates information of an user with the given Path parameter, id)
 \item DELETE /users/\{id\} (removes an user with the given Path parameter, id)
-\end{itemize}
+\end{description}
 
 %Process
 When a client request is made via a REST API, it transmits a representation of the resource status to the client or endpoint. \\ This representation or information is provided in mostly JSON or XML. JSON is the most popular programming language because it is language-agnostic despite its name and can be read by both humans and machines. \\
 After a client sends a HTTP request, a corresponding server is going to send back a HTTP response with headers and optionally a payload (data pack for a GET request). Additionally, a numeric status code will be a part of this response as well. Staus codes are specified in one of five categories:
-\begin{itemize}
+\begin{description}
 \item 1xx : Informational provisional responses
 \item 2xx : Request is successfully processed
 \item 3xx : Request requires more information to complete the request
 \item 4xx : Client error (invalid request, non-existent resource, client not authenticated or authorized)
 \item 5xx : Server error (Server can not supply a valid response)
-\end{itemize}
+\end{description}
 
 HTTP status code is an important component to this work, because it provides to test REST APIs and find out errors (e.g. requests expected as valid return 4xx status code or requests expected as invalid return 2xx status code).
 

sections/rq2.tex

-\section{Expression of approaches in RAG}
-
-
-
-RQ1 : Can we express approaches from related work? \\
-
-To answer this question, we 
-
-- test data structure and mapping \\
-- request generation \\
-- parameter inference \\
-- for future work: more approaches like analysis in RESTler, Constructing Operation Dependency Graph in RESTest, precise inference with stemming algor\\
-- \\
-- \\
\ No newline at end of file

sections/rq3.tex

-\section{Adavantages of RAG in REST API Testing}
-
-- RQ2 : What makes it easier to use RAGs than normal code, what makes it harder?\\
-- attributes for navigation/name resolution \\
-- implementing relative bigger and structured codes with small definitions\\
-- effort to be used to JastAdd\\
-
-% Helpful attributes (Collection, inherited, easier return with =)
-During the implementation phase of parameter inference, RAG was beneficial at writing codes. At writing codes for parser and back-transformation, attributes could not be applied, because the functional parts change the structure of AST, which conflicts to the definition of attribute and also in JastAdd. So, it could not be realized that attributes can be practical. But, following attributes and features of JastAdd were helpful to construct the generators and implement them:
-
-\begin{itemize}
-\item As long as a returning value can be called or storable in primitive way, the syntax in RAG is reduced with an equals sign and simpler than in common programming language. Therefore, the source codes are compact. To compare how RAG could be implemented effectively, see the implementation of an attribute in this work in Listing 5.3 and how this attribute is defined in common Java description in Listing 5.4.
-\item To iterate all paths available in OpenAPI, an inherited attribute facilitated the writing effort. Paths Objects are child nodes of OpenAPI Object, so, all that should have been done was only two lines of code instead of writing an extra for loop. See Listing 5.5.
-\item As its name, Reference Attribute Grammar, every AST node can be referenced by an attribute. For instance, with a collection attribute. A reference defined in Listing 4.1 contains only a String variable named Ref, it still could provide its Object like in Listing 5.6.
-\end{itemize}
-
-\begin{lstlisting}[language=JRAG,label={lst:attribute-infParam},caption={Attribute InferredParameter.value()}]
-syn String InferredParameter.value() = getParameter().substring( getParameter().indexOf("?") + 1 );
-\end{lstlisting} 
-
-\begin{lstlisting}[language=JRAG,label={lst:java-infParam},caption={InferredParameter.value() in Java}]
-public String InferredParameter.value() {
-  return getParameter().substring( getParameter().indexOf("?") + 1 );
-}
-\end{lstlisting} 
-
-\begin{lstlisting}[language=JRAG,label={lst:inherited},caption={Inherited Attribute in Paths Object}]
-inh boolean PathsObject.inferUrl();
-eq OpenAPIObject.getPathsObject(int i).inferUrl(){
-  ...
-}
-\end{lstlisting} 
-
-\begin{lstlisting}[language=JRAG,label={lst:coll-param},caption={Collection Attribute in Parameter Object}]
-coll List<ParameterTuple> OpenAPIObject.parameterTuples() [new ArrayList<>()] root OpenAPIObject;
-  ParameterTuple contributes this
-  to OpenAPIObject.parameterTuples();
-...
-syn ParameterObject ParameterOb.parameterObject();
-eq ParameterObject.parameterObject() = this;
-eq ParameterReference.parameterObject() {
-  for( ParameterTuple t : root().parameterTuples() ){
-  if( t.getKey().equals(getRef().substring(getRef().lastIndexOf("/")+1, getRef().length())) )
-    return t.getParameterOb().parameterObject();
-  }
-  return new ParameterObject();
-}
-\end{lstlisting} 
\ No newline at end of file

sections/stm.tex

 \chapter{Test Methods}\label{ch:stm}
 
-As we have discussed in Section 3, most black-box REST API tests principally use Fuzzing, sending unexpected, random data or data providing errors into input interfaces. For this purpose, the newly restructured OpenAPI can be extended, while test cases are generated and generated requests are sent into API. The implementation of this work contains two basic approaches related to Fuzzing. Following sections present Random Testing in Section 5.1 and Parameter Inference in Section 5.2.
+As we have discussed in \Cref{ch:curr-appr}, most black-box REST API tests principally use Fuzzing, sending unexpected, random data or data providing errors into input interfaces. For this purpose, the newly restructured OpenAPI can be extended, while test cases are generated and generated requests are sent into API. The implementation of this work contains two basic approaches related to Fuzzing. Following sections present Random Testing in \Cref{sec:stm1} and Parameter Inference in \Cref{sec:stm2}.
 
 \input{sections/stm1}
 \input{sections/stm2}
\ No newline at end of file

sections/stm1.tex

-\section{Random Testing}
+\section{Random Testing} \label{sec:stm1}
 
 OpenAPI defines a parameter in an operation in four types, Query, Path, Header and Cookie\footnote{\url{https://swagger.io/docs/specification/describing-parameters}}. In this work, only Query and Path parameters are considered to research the functionality of the Fuzzing prototype in RAG. These parameters are clearly describable in String values and also comfortably testable, because they target only variable URIs. \\
 For the experiments, the OpenAPI document of a Pet Store is mainly used\footnote{\url{https://petstore.swagger.io}}. Additionally, only GET and POST operations are tested to research basic functionality firstly.

sections/stm2.tex

-\section{Parameter Inference}
-Random testing is a one of easiest way to test API and can be useful in some situations. However, it is not effective in REST API testing, because the coverage of the tested API would be particularly low and random values are unusually valid. During the observation in Section 5.1, it was clear to see that random testing mostly produces only requests that receive only 4xx HTTP stauts codes from commercial APIs. 
+\section{Parameter Inference} \label{sec:stm2}
+Random testing is a one of easiest way to test API and can be useful in some situations. However, it is not effective in REST API testing, because the coverage of the tested API would be particularly low and random values are unusually valid. During the observation in \Cref{sec:stm1}, it was clear to see that random testing mostly produces only requests that receive only 4xx HTTP stauts codes from commercial APIs. 
 
 % General
-To solve this problem, most of REST API testing approaches use a stateful process, because it enables to analyze properties of APIs and infer inputs which are more appropriate than random inputs. There are several suggestions in Section 3, this framework investigates a inference of parameters with operation dependency motivated by RESTTESTGEN [5]. Generally, there is an operation dependency, if a returned value in a response is applicable as an input in a request. This dependency is inferred by three strategies.
+To solve this problem, most of REST API testing approaches use a stateful process, because it enables to analyze properties of APIs and infer inputs which are more appropriate than random inputs. There are several suggestions in \Cref{ch:curr-appr}, this framework investigates a inference of parameters with operation dependency motivated by Specification-based Approach \cite{ed2018automatic} and RESTTESTGEN \cite{viglianisi2020resttestgen}. Generally, it collects all responses and inferences parameters contributing the same schema of a succesful response. \\
 
-\begin{itemize}
+
+If there is a schema set in a request and a response, parameters of them are inferred by three strategies.
+
+\begin{description}
 \item Case insensitive
 \item Id completion in a field name (e.g. if a property is named with "id", it gets an additional field name available in the specification)
 \item Stemming algorithm (e.g. pet and pets are considered as a same value.)
-\end{itemize}
+\end{description}
 
 % Process
 In the implementation of this work, case insensitive comparison and id completion are utilized to create the basic functionality. Stemming algorithm can be also extended in the future. \\
-The follwing code in Listing 5.2 shows how the parameter inference is compiled with predefined attributes:
+The follwing code in \Cref{lst:parameter-inference} shows how the parameter inference is compiled with predefined attributes:
 
 \begin{lstlisting}[language=JRAG,label={lst:parameter-inference},caption={Parameter Inference}]
 generateRequests();
@@ -48,9 +51,9 @@ As results, the test case generator with parameter inference implemented in this
 During the implementation phase of parameter inference, RAG was beneficial at writing codes. At writing codes for parser and back-transformation, attributes could not be applied, because the functional parts change the structure of AST, which conflicts to the definition of attribute and also in JastAdd. So, it could not be realized that attributes can be practical. But, following attributes and features of JastAdd were helpful to construct the generators and implement them:
 
 \begin{itemize}
-\item As long as a returning value can be called or storable by an AST node, the syntax in RAG is reduced with an equals sign and simpler than in common programming language. Therefore, the source codes are compact. To compare how RAG could be implemented effectively, see the implementation of an attribute in this work in Listing 5.3 and how this attribute is defined in common Java description in Listing 5.4.
-\item To iterate all paths available in OpenAPI, an inherited attribute facilitated the writing effort. Paths Objects are child nodes of OpenAPI Object, so, all that should have been done was only two lines of code instead of writing an extra for loop. See Listing 5.5.
-\item As its name, Reference Attribute Grammar, every AST node can be referenced by an attribute. For instance, with a collection attribute. A reference defined in Listing 4.1 contains only a String variable named Ref, it still could provide its Object like in Listing 5.6.
+\item As long as a returning value can be called or storable by an AST node, the syntax in RAG is reduced with an equals sign and simpler than in common programming language. Therefore, the source codes are compact. To compare how RAG could be implemented effectively, see the implementation of an attribute in this work in \Cref{lst:attribute-infParam} and how this attribute is defined in common Java description in \Cref{lst:java-infParam}.
+\item To iterate all paths available in OpenAPI, an inherited attribute facilitated the writing effort. Paths Objects are child nodes of OpenAPI Object, so, all that should have been done was only two lines of code instead of writing an extra for loop. See \Cref{lst:inherited}.
+\item As its name, Reference Attribute Grammar, every AST node can be referenced by an attribute. For instance, with a collection attribute. A reference defined in \Cref{lst:rago-ast} contains only a String variable named Ref, it still could provide its Object like in \Cref{lst:coll-param}.
 \end{itemize}
 
 \begin{lstlisting}[language=JRAG,label={lst:attribute-infParam},caption={Attribute InferredParameter.value()}]
@@ -58,11 +61,15 @@ syn String InferredParameter.value() = getParameter().substring( getParameter().
 \end{lstlisting} 
 
 \begin{lstlisting}[language=JRAG,label={lst:java-infParam},caption={InferredParameter.value() in Java}]
-public String InferredParameter.value() {
+Class InferredParameter {
+...
+constructor;
+...
+  public String value() {
     return getParameter().substring( getParameter().indexOf("?") + 1 );
   }
+}
 \end{lstlisting} 
-
 \begin{lstlisting}[language=JRAG,label={lst:inherited},caption={Inherited Attribute in Paths Object}]
 inh boolean PathsObject.inferUrl();
 eq OpenAPIObject.getPathsObject(int i).inferUrl(){