\chapter{Suitability of RAGs for OpenAPI Testing}\label{ch:suitability}
% Helpful attributes (Collection, inherited, easier return with =)
During the implementation phase of parameter inference, RAG was beneficial at writing codes. At writing codes for parser and back-transformation, attributes could not be applied, because the functional parts change the structure of AST, which conflicts to the definition of attribute and also in JastAdd. So, it could not be realized that attributes can be practical. But, following attributes and features of JastAdd were helpful to construct the generators and implement them:
\begin{itemize}
\item As long as a returning value can be called or storable in primitive way, the syntax in RAG is reduced with an equals sign and simpler than in common programming language. Therefore, the source codes are compact. To compare how RAG could be implemented effectively, see the implementation of an attribute in this work in Listing 5.3 and how this attribute is defined in common Java description in Listing 5.4.
\item To iterate all paths available in OpenAPI, an inherited attribute facilitated the writing effort. Paths Objects are child nodes of OpenAPI Object, so, all that should have been done was only two lines of code instead of writing an extra for loop. See Listing 5.5.
\item As its name, Reference Attribute Grammar, every AST node can be referenced by an attribute. For instance, with a collection attribute. A reference defined in Listing 4.1 contains only a String variable named Ref, it still could provide its Object like in Listing 5.6.
RQ1 : Can we express approaches from related work? \\
To answer this question, we
- test data structure and mapping \\
- request generation \\
- parameter inference \\
- for future work: more approaches like analysis in RESTler, Constructing Operation Dependency Graph in RESTest, precise inference with stemming algor -> conclusion\\
- \\
- \\
- RQ2 : What makes it easier to use RAGs than normal code, what makes it harder?\\
- attributes for navigation/name resolution \\
- implementing relative bigger and structured codes with small definitions\\
- effort to be used to JastAdd\\
> To evaulate how well-suited RAGs are for API testing, we revisit the research questions...
### 6.1 Feasibility
RQ1 RQ2
- current testing: mostly fuzzing
- different additions:
- inference
- test model
- this can be solved with the RAG tool presented in \cref{chap:RAGO}
- in \cref{sec:inf} we show how inference can be done
- \cite{ttc-rag-von-ceur} for test model/model connections
- http://ceur-ws.org/Vol-2310/ A JastAdd- and ILP-based Solution to the Software-Selection and Hardware-Mapping-Problem at the TTC 2018
### 6.2 Benefits
- schöne beschreibung mit attributen (verweis auf beispiele aus 5)
@@ -18,7 +18,7 @@ To sum up, following three questions are formulated as research questions:
\item{\textbf{RQ3 : Which advantages can RAG provide with its features at expressing testing approaches?}}
\end{description}
\textbf{RQ1} is intended to investigate which approaches might be able to implement in RAG. Corresponding literatures are introduced in Section 3. \\\textbf{RQ2} focuses on the implementation the data structure and testing methods in RAG and is answered in Section 4 and Section 5. \\ In Section 6, Results for \textbf{RQ3} shows concretely which features (attributes) of RAG are used and which benefits exist at this point.
\textbf{RQ1} is intended to investigate which approaches might be able to implement in RAG. Corresponding literatures are introduced in \Cref{ch:curr-appr}. \\\textbf{RQ2} focuses on the implementation the data structure and testing methods in RAG and is answered in \Cref{ch:rago} and \Cref{ch:stm}. \\ In \Cref{ch:suitability}, Results for \textbf{RQ3} shows concretely which features of RAG are usable and which benefits exist at this point.
% [1] https://swagger.io/specification/
% [2] Basu, Anirban (2015). Software Quality Assurance, Testing and Metrics. PHI Learning. ISBN 978-81-203-5068-7.
There are already many approaches suggesting to test REST APIs with OpenAPI specifications automatically. Mostly, they are black-box approaches where an access to a source code is irrelevant and use Fuzzing [2] - [6]. Fuzzing is sending unexpected, random data or data providing errors into input interfaces and observing if interfaces are stable and provide expected responses. There is also a white box approach [11] which is not the main research target of this paper, but still an interesting way. \\ This section introduces which approaches have been released so far, how they work and which did they achieve.
@@ -44,8 +44,47 @@ Before it starts with the parameter inference, the random testing generator of S
% Result
As results, the test case generator with parameter inference implemented in this framework could create maximum over 300 acceptable URIs for the parameter petId in the selected API, pet store, at the operation getPetById. It generated also numerous requests denied by the server. After the observation of several execution iterations, it can be assumed that the API with this operation sends 200 or 404 status codes randomly or according to some rules, because a same URI provided status codes in that way.
% Future work
In the future, this parameter inference can be more precise with more constraints of the schema. In this novel version of Fuzzing tool in RAG, it only has schema constraints with types array, string, integer and consideration of a parameter enumeration. Apart from Schema Object, it has not been enough extended to support authorization. Several approaches do not support analysis in terms of authorization, either. It is an open research topic.
% Helpful attributes (Collection, inherited, easier return with =)
During the implementation phase of parameter inference, RAG was beneficial at writing codes. At writing codes for parser and back-transformation, attributes could not be applied, because the functional parts change the structure of AST, which conflicts to the definition of attribute and also in JastAdd. So, it could not be realized that attributes can be practical. But, following attributes and features of JastAdd were helpful to construct the generators and implement them:
\begin{itemize}
\item As long as a returning value can be called or storable in primitive way, the syntax in RAG is reduced with an equals sign and simpler than in common programming language. Therefore, the source codes are compact. To compare how RAG could be implemented effectively, see the implementation of an attribute in this work in Listing 5.3 and how this attribute is defined in common Java description in Listing 5.4.
\item To iterate all paths available in OpenAPI, an inherited attribute facilitated the writing effort. Paths Objects are child nodes of OpenAPI Object, so, all that should have been done was only two lines of code instead of writing an extra for loop. See Listing 5.5.
\item As its name, Reference Attribute Grammar, every AST node can be referenced by an attribute. For instance, with a collection attribute. A reference defined in Listing 4.1 contains only a String variable named Ref, it still could provide its Object like in Listing 5.6.
