Add JSON Schema for validating testsuite files

[armingerten]

Overview

This pull request is about adding a JSON Schema file that can be used to validate test-suite files.

Most popular IDEs (IntelliJ, Visual Studio Code, etc.) support applying schemas to YAML files using a JSON Schema. This provides comprehensive documentation as well as code completion while editing the test-suite file:

In addition, test-suite files can be validated while editing so wrongfully added additional properties or incorrect data types can be detected while editing:

Based on the current documentation, I created such a schema file containing (hopefully) all possible configuration options (test suites, test jobs, assertions, ...).

How to use

Visual Studio Code

When developing with VSCode, the very popular YAML plug-in (created by RedHat) allows adding references to schemas by adding a comment line on top of the file:

# yaml-language-server: $schema=https://raw.githubusercontent.com/armingerten/helm-unittest/feature/testsuite-schema-validation/schema/helm-testsuite.json
suite: http-service.configmap_test.yaml
templates: [configmap.yaml]
release:
  name: test-release
  namespace: TEST_NAMESPACE

Alternatively, you can add the schema globally to the IDE, using a well defined pattern:

"yaml.schemas": {
  "https://raw.githubusercontent.com/armingerten/helm-unittest/feature/testsuite-schema-validation/schema/helm-testsuite.json": ["charts/*/tests/*_test.yaml"]
}

Once this pull request is merged, the above mentioned URLs should be changed to the URL within this repo: https://raw.githubusercontent.com/quintush/helm-unittest/master/schema/helm-testsuite.json

IntelliJ

Similar to VSCode, IntelliJ allows mapping file patterns to schemas via preferences: Languages & Frameworks -> Schemas and DTDs -> JSON Schema Mappings

Future outlook

Afore mentioned IDEs (and also others) support automatic download of JSON schemas from schemastore.org. If the schema was added to schemastore.org, no additional configuration in the IDE would be necessary. New users to helm-unittest would automatically get schema validation out of the box.

I'd like to follow through with that idea and provide such a pull request for the schema store, but of course only with your consent @quintush . WDYT?

[rquinio1A]

This is awesome !
Sadly with latest IJ 2021.3.2 I'm getting an error "Schema validation: Validates to more than one variant" on each assert element, even though auto-completion works fine.
Seems confused by the oneOf, could be https://youtrack.jetbrains.com/issue/IDEA-236928, I see other tickets about similar error have been fixed though.

[armingerten]

Oh, that is too bad :/ I tested mostly on VSCode, so I did not catch that IntelliJ is not properly visualizing the oneOf keyword. I checked the draft-07 again and the keyword seems to exist in that version ...

However, thanks for testing that and providing screenshots of the look-and-feel in IntelliJ! :-) I hope this gets fixed soonish ... It would also be possible to change the schema and avoid usage of oneOf, but that would have a negative impact on the code completion (since different kind of assertions are mutually exclusive).

[rohit-gohri]

This would be really great! I recently added json schema for our charts, and it would be great to have it for the tests too. Makes it really easy to onboard new people to the tool