[WIP] feat: new assertion engine (kubeshop#1378)

* feat: new expression engine (kubeshop#1350) * fix: opensearch example by pinning docker image versions (kubeshop#1358) pin docker image versions * add parsing rules * add filters to the parser * add more test cases for filters * add basic comparisons * add attributes support * remove dead code * support duration comparisons * fix string interpolation * add more tests for interpolation * add support for filters and add json_path filter * support meta attributes * fix meta attributes and duration precision bug * update assertion engine to use new expression engine * update cli to use new assertion format * fix examples * patch tracetesting tests * fix duration conversion * fix regex to detect id field in definition file * remove dead code from assertions * improve filters and add regex filters * add new test to assert spaces are allowed in filter args * improve filter implementation to use typed values * use const instead of string Co-authored-by: Sebastian Choren <[email protected]> * rename types and use iota * remove github actions branch filter for pr pipelines Co-authored-by: Sebastian Choren <[email protected]> * disable otel demo on beta and demo envs (kubeshop#1356) * 1353 fixing test card styles (kubeshop#1355) * fix deployments (kubeshop#1360) * [transactions] Implement transactions details page (kubeshop#1351) * docs: new expressions (kubeshop#1361) * docs: new expressions * apply Julianne changes Co-authored-by: Julianne Fermi <[email protected]> * Update docs/expressions.md Co-authored-by: Julianne Fermi <[email protected]> * Apply suggestions from code review Co-authored-by: Julianne Fermi <[email protected]> Co-authored-by: Julianne Fermi <[email protected]> * feat: add count filter (kubeshop#1377) * feat: add count filter * update docs * feat: allow 'last' as argument for get_index filter (kubeshop#1380) * feat: allow 'last' as arg for get_index filter * update documentation * fix CLI rebase * feat: support variables in expressions (kubeshop#1388) * feat: support variables in expressions * docs: add documentation about variables in expressions * docs: fix variables title * add benchmark * Update docs/expressions.md Co-authored-by: Julianne Fermi <[email protected]> Co-authored-by: Julianne Fermi <[email protected]> * feat: new expression engine (kubeshop#1350) * fix: opensearch example by pinning docker image versions (kubeshop#1358) pin docker image versions * add parsing rules * add filters to the parser * add more test cases for filters * add basic comparisons * add attributes support * remove dead code * support duration comparisons * fix string interpolation * add more tests for interpolation * add support for filters and add json_path filter * support meta attributes * fix meta attributes and duration precision bug * update assertion engine to use new expression engine * update cli to use new assertion format * fix examples * patch tracetesting tests * fix duration conversion * fix regex to detect id field in definition file * remove dead code from assertions * improve filters and add regex filters * add new test to assert spaces are allowed in filter args * improve filter implementation to use typed values * use const instead of string Co-authored-by: Sebastian Choren <[email protected]> * rename types and use iota * remove github actions branch filter for pr pipelines Co-authored-by: Sebastian Choren <[email protected]> * 1353 fixing test card styles (kubeshop#1355) * [transactions] Implement transactions details page (kubeshop#1351) * docs: new expressions (kubeshop#1361) * docs: new expressions * apply Julianne changes Co-authored-by: Julianne Fermi <[email protected]> * Update docs/expressions.md Co-authored-by: Julianne Fermi <[email protected]> * Apply suggestions from code review Co-authored-by: Julianne Fermi <[email protected]> Co-authored-by: Julianne Fermi <[email protected]> * feat: add count filter (kubeshop#1377) * feat: add count filter * update docs * feat: allow 'last' as argument for get_index filter (kubeshop#1380) * feat: allow 'last' as arg for get_index filter * update documentation * fix CLI rebase * feat: support variables in expressions (kubeshop#1388) * feat: support variables in expressions * docs: add documentation about variables in expressions * docs: fix variables title * add benchmark * Update docs/expressions.md Co-authored-by: Julianne Fermi <[email protected]> Co-authored-by: Julianne Fermi <[email protected]> * adding the base editor * adding the base editor * Test outputs (kubeshop#1368) * basic editors integration * fixing FE unit tests * feat: support generator functions in expressions (kubeshop#1393) * remove old replacer implementation * add function calls to expression engine parser * add full support for function calls in expresions * remove old functions package and fix tests * docs: update generator functions docs * Apply suggestions from code review Co-authored-by: Julianne Fermi <[email protected]> * Apply suggestions from code review Co-authored-by: Sebastian Choren <[email protected]> * merge count and length filters into a single length filter Co-authored-by: Julianne Fermi <[email protected]> Co-authored-by: Sebastian Choren <[email protected]> * feat: add "type" filter and support arrays in expressions (kubeshop#1397) * feat: array support and type filter * update docs * add test case * extract value to its own package * make expression engine use value.Value instead of string * add Type function to value.Value * overload contains operator to work with arrays * add example of array contains in docs * add chaining type filter example * Encoding refactor (kubeshop#1399) * improving expressions input integration * improving expressions input integration * updates and fixes * Handle file execution in backend (kubeshop#1404) Co-authored-by: Sebastian Choren <[email protected]> Co-authored-by: Oscar Reyes <[email protected]> Co-authored-by: Jorge Padilla <[email protected]> Co-authored-by: Julianne Fermi <[email protected]>
olange · Oct 21, 2022 · 2cf036c · 2cf036c
1 parent cbeaeb4
commit 2cf036c
Show file tree

Hide file tree

Showing 235 changed files with 7,017 additions and 5,624 deletions.
diff --git a/.github/workflows/pull-request.yaml b/.github/workflows/pull-request.yaml
@@ -5,8 +5,6 @@ on:
       - opened
       - reopened
       - synchronize
-    branches:
-      - main
     paths:
       - .github/workflows/pull-request.yaml
       - api/**

diff --git a/api/definition.yaml b/api/definition.yaml
@@ -0,0 +1,27 @@
+openapi: 3.0.0
+components:
+  schemas:
+
+    ExecuteDefinitionResponse:
+      type: object
+      properties:
+        id:
+          type: string
+          description: resource ID
+        runId:
+          type: string
+          description: run ID
+        type:
+          type: string
+          description: resource type
+        url:
+          type: string
+          description: resource web UI url
+
+    TextDefinition:
+      type: object
+      properties:
+        runInformation:
+          $ref: "./tests.yaml#/components/schemas/TestRunInformation"
+        content:
+          type: string
diff --git a/api/openapi.yaml b/api/openapi.yaml
@@ -6,6 +6,34 @@ info:
 servers:
   - url: /api
 paths:
+  /definition.yaml:
+    post:
+      tags:
+        - api
+      summary: "Execute a definition"
+      description: "Execute a definition"
+      operationId: executeDefinition
+      requestBody:
+        content:
+          text/json:
+            schema:
+              $ref: "./definition.yaml#/components/schemas/TextDefinition"
+      responses:
+        201:
+          description: Definition created
+          content:
+            application/json:
+              schema:
+                $ref: "./definition.yaml#/components/schemas/ExecuteDefinitionResponse"
+
+        200:
+          description: Definition updated
+          content:
+            application/json:
+              schema:
+                $ref: "./definition.yaml#/components/schemas/ExecuteDefinitionResponse"
+        422:
+          description: "trying to create a test with an already existing ID"
   /tests:
     get:
       tags:
@@ -80,54 +108,6 @@ paths:
         400:
           description: "trying to create a test with an already existing ID"
 
-  /tests/definition.yaml:
-    post:
-      tags:
-        - api
-      summary: "Create new test using the yaml definition"
-      description: "Create new test using the yaml definition"
-      operationId: createTestFromDefinition
-      requestBody:
-        content:
-          text/json:
-            schema:
-              $ref: "./tests.yaml#/components/schemas/TextDefinition"
-      responses:
-        200:
-          description: successful operation
-          content:
-            application/json:
-              schema:
-                $ref: "./tests.yaml#/components/schemas/Test"
-        400:
-          description: "trying to create a test with an already existing ID"
-
-  /tests/{testId}/definition.yaml:
-    put:
-      tags:
-        - api
-      parameters:
-        - in: path
-          name: testId
-          schema:
-            type: string
-          required: true
-      summary: "update test from definition file"
-      description: "update test action from definition file"
-      operationId: updateTestFromDefinition
-      requestBody:
-        content:
-          text/json:
-            schema:
-              $ref: "./tests.yaml#/components/schemas/TextDefinition"
-      responses:
-        200:
-          description: successful operation
-          content:
-            application/json:
-              schema:
-                $ref: "./tests.yaml#/components/schemas/Test"
-
   /tests/{testId}:
     get:
       tags:

diff --git a/api/tests.yaml b/api/tests.yaml
@@ -19,11 +19,33 @@ components:
         specs:
           $ref: "#/components/schemas/TestSpecs"
           description: specification of assertions that are going to be made
+        outputs:
+          description: define test outputs, in a key/value format. The value is processed as an expression
+          type: array
+          items:
+            $ref: "#/components/schemas/TestOutput"
+          example:
+            - name: USER_ID
+              selector: span[name="user_signup" result="success"]
+              value: "attr:myapp.signup.user_id"
+            - name: PROFILE_URL
+              selector: span[tracetest.span.type="general" name="Tracetest trigger"]
+              value: "${attr:tracetest.response.body | json_path '.user.profileUrl' }"
         summary:
           $ref: "#/components/schemas/TestSummary"
           description: summary of test data
           readOnly: true
 
+    TestOutput:
+      type: object
+      properties:
+        name:
+          type: string
+        selector:
+          $ref: "#/components/schemas/Selector"
+        value:
+          type: string
+
     TestSummary:
       type: object
       properties:
@@ -60,7 +82,7 @@ components:
               assertions:
                 type: array
                 items:
-                  $ref: "#/components/schemas/Assertion"
+                  type: string
       example:
         specs:
           - selector:
@@ -78,16 +100,6 @@ components:
               comparator: "="
               expected: "200"
 
-    Assertion:
-      type: object
-      properties:
-        attribute:
-          type: string
-        comparator:
-          type: string
-        expected:
-          type: string
-
     TestRun:
       type: object
       properties:
@@ -134,6 +146,16 @@ components:
           $ref: "./trace.yaml#/components/schemas/Trace"
         result:
           $ref: "#/components/schemas/AssertionResults"
+        outputs:
+          type: array
+          items:
+            type: object
+            properties:
+              name:
+                type: string
+              value:
+                type: string
+
         metadata:
           type: object
           additionalProperties:
@@ -198,7 +220,7 @@ components:
       type: object
       properties:
         assertion:
-          $ref: "#/components/schemas/Assertion"
+          type: string
         allPassed:
           type: boolean
         spanResults:
@@ -284,9 +306,3 @@ components:
           $ref: "#/components/schemas/Test"
         run:
           $ref: "#/components/schemas/TestRun"
-
-    TextDefinition:
-      type: object
-      properties:
-        content:
-          type: string
diff --git a/api/transactions.yaml b/api/transactions.yaml
@@ -0,0 +1,21 @@
+openapi: 3.0.0
+components:
+  schemas:
+
+    Transaction:
+      type: object
+      properties:
+        id:
+          type: string
+          readOnly: true
+        name:
+          type: string
+        description:
+          type: string
+        version:
+          type: integer
+          description: version number of the test
+        steps:
+          type: array
+          items:
+            type: string
diff --git a/cli/actions/export_test_action.go b/cli/actions/export_test_action.go
@@ -47,7 +47,12 @@ func (a exportTestAction) Run(ctx context.Context, args ExportTestConfig) error
 		return fmt.Errorf("could not get definition from server: %w", err)
 	}
 
-	err = file.SaveDefinition(args.OutputFile, definition)
+	f, err := file.New(args.OutputFile, []byte(definition))
+	if err != nil {
+		return fmt.Errorf("could not process definition from server: %w", err)
+	}
+
+	_, err = f.Write()
 	if err != nil {
 		return fmt.Errorf("could not save exported definition into file: %w", err)
 	}