Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File based tests for the normalizers. supply a raw file and an expected serialized output file and this will create a new ad hoc test #2545

Open
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

underdarknl
Copy link
Contributor

Changes

This adds a tester which looks for specific input and output files in the normaliser folders.
It then runs the input agains the normalizer and checks if the output matches the output file.

Issue link

#2534

Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
  • I have written unit tests for the changes or fixes I made.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have made corresponding changes to the documentation, if necessary.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.


Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

This test looks for specific test-files in the boefjes/normalizer folders (named test-input and test-output) and creates pytest testcases on the fly for all of them.
@underdarknl underdarknl self-assigned this Feb 21, 2024
@underdarknl underdarknl requested a review from a team as a code owner February 21, 2024 17:08
@underdarknl underdarknl marked this pull request as draft February 22, 2024 10:18
@underdarknl
Copy link
Contributor Author

underdarknl commented Feb 22, 2024

Todo's:

  • Design a input/output filename scheme.
  • Add documentation for creating and running these tests.
  • Validate each yielded object one by one against the expected objects
  • Optionally provide a way to only check certain fields on the yielded objects, instead of all
  • Use JSON content for output files
  • Optionally allow the input__ooi to be specified

@underdarknl
Copy link
Contributor Author

underdarknl commented Mar 5, 2024

As discussed, I will add Python file support next to Json file support for output files.
The discussion around filenames is still open, and projecting these changes onto testing of reports, and testing of individual plugins is also still open.

@underdarknl underdarknl marked this pull request as ready for review May 21, 2024 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

1 participant