fixed by https://github.com/networknt/json-schema-validator/pull/697
i propose moving the specification files from JSON Schema Test Suite to . This would provide the following benefits:
src/test/suite
- Makes merging updates from that project into this one easier as it reduces the chance of removing or overwriting this project's files
- Makes it easier to see exactly what changed in the specification thus making it easier to see what functional changes are required to conform (e.g., moved from to and was removed)
content.jsondraft2019-09/optionaldraft2019-09draft2019-09/optional/zeroTerminatedFloats.json- Enables the opportunity to populate by linking to that project thus eliminating the need to manually merge its changes into this project (it is also possible for Maven to download and unpack a specific version of the source code)
src/test/suite- Enables more complicated processing of specification files. JSON Schema Test Suite has informed validators to expect specification files to appear at the root of the folder. The validators are now expected to extract from the specification file rather than using the folder name (e.g.,
tests$schematests/draft2019-09)This change depends on #690 being accepted.
JSON Schema Test Suite has several tests for each test-case and there is one or more test-cases in each specification file. Currently, this project treats each specification file as a single unit-test. As a result, a single test failure causes the unit-test to fail and no subsequent tests are performed. If the first test in a specification file fails, for example, no other tests are executed.
Several specification files are currently disabled simply because a single test fails. For example, content.json has four test-cases with a total of 18 tests. The unit-test is disabled so none of the tests are executed.
I propose refactoring the unit-tests to use JUnit Jupiter's dynamic test facility to create a separate unit-test for each test in a test-case (and each test-case in a specification file). This would provided several benefits:
- Better test coverage since more tests are actually executed
- Better reporting since each test is a single line-item in the report
- Better reporting of the issue found (e.g., each test can report more detailed information than "expected true but was false")
- Easier to investigate the cause of some obscure bugs such as JSON Schema Syntax Validator #285, V2019-09 not validating field type under properties #313 and V2020-12 not validating items: { "type" } correctly #587
- More fine-grained control over which tests are excluded (i.e., instead of excluding a whole specification file, support excluding a specific test-case or individual test)
The specification files also need refreshing but that is best left to a separate ticket.
This is an example of how this would appear when executed from the command line.
<span style="color:#1f2328"><span style="background-color:#ffffff"><span style="color:var(--fgColor-default, var(--color-fg-default))"><span style="background-color:var(--bgColor-muted, var(--color-canvas-subtle))"><code>[INFO] +--JSON Schema Test Suite - 1.063s
[INFO] | +-- [OK] validation of duration strings (src/test/resources/draft2019-09/optional/format/duration.json) Draft 2019-09 no elements present - 0.001s
[INFO] | +-- [XX] validation of duration strings (src/test/resources/draft2019-09/optional/format/duration.json) Draft 2019-09 no time elements present - 0s
[INFO] | +-- [OK] validation of duration strings (src/test/resources/draft2019-09/optional/format/duration.json) Draft 2019-09 no date or time elements present - 0s
</code></span></span></span></span>This is how it appears in Eclipse (IntelliJ has similar views).
