Refactoring of automated cucumber tests #1113
Conversation
Reduced the content in any given line to just the essentials. Use of quotes in cucumber is odd but workable. Allowing some columns to be blank was not intuitive but is also workable. The line is much shorter with little to know repetitive information. The checks are more complete and StepDefs is completely cleaned up. Weill, probably not completely but good enoough for next step.
|
@jordanpadams @nutjob4life @tloubrieu-jpl Look at the output of cucumber (follow github action links) to make sure the output is sufficient. As you can see by the code changes a lot has changed - mostly deleted. My next step (maybe tomorrow but probably next week) is to write a script that converts the old tables to the new ones. I am keeping the multiple files for no good reason. I do not like having to grep the feature file to find the test that is failing. I like having the smaller files. Usually when I am in the feature file, I am actively working the bug so I know which file to find it in. Hence no good reason to go any direction other than to leave it as is. |
|
@al-niessner I can't comment if the output looks okay but I can celebrate code deletion and also this: So woot woot! 🎉 |
There was a problem hiding this comment.
Visual inspection: ✓
Maven tests: ✓
[INFO] Results:
[INFO]
[WARNING] Tests run: 25, Failures: 0, Errors: 0, Skipped: 23
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
| Then produced output from validate command should be similar to reference <refOutputValue> or no error reported. | ||
|
|
||
| @v3.6.x | ||
| Feature: 3.6.x |
There was a problem hiding this comment.
@al-niessner looks great! a couple comments:
- will need a
testIdcolumn in the feature file. this ties to our I&T system and ensure we have a unique identifier for each test that traces back to ticket. @tloubrieu-jpl can maybe provide some more details about what is expected there, but I am pretty sure atestIdcolumn is needed. - I would prefer we keep the tags in the files, if possible, so you can run
mvn testfor a specific subset of tests, e.g.-Dcucumber.filter.tags='@pre.v3.6.x' - @tloubrieu-jpl, do we need the a specific feature name associated with these tests for TestRail, or does it not matter?
There was a problem hiding this comment.
- testId still exists but comes in two parts: column 1 and column 2 if first index is 1. The scenerio outline contains all of the boilerplate and joins those two columns. In eclipse it looks like:
Scenario Outline: NASA-PDS/validate#1028 1 �[90m# src/test/resources/features/3.6.x.feature:9�[0m
�[32mGiven �[0m�[32man �[0m�[32m�[1m1028�[0m�[32m, �[0m�[32m�[1m1�[0m�[32m, and �[0m�[32m�[1m"github1028"�[0m �[90m# cucumber.StepDefs.an_and(java.lang.Integer,java.lang.Integer,java.lang.String)�[0m
�[32mWhen �[0m�[32mexecute validate with �[0m�[32m�[1m"--skip-context-validation -t {datasrc}/xa.s16..shz.1976.070.0.xml --schema {datasrc}/PDS4_PDS_1N00.xsd --schematron {datasrc}/PDS4_PDS_1N00.sch"�[0m �[90m# cucumber.StepDefs.execute_validate(java.lang.String)�[0m
�[32mThen �[0m�[32mcompare to the .�[0m �[90m# cucumber.StepDefs.compare_to_the()�[0m
Scenario Outline: NASA-PDS/validate#1028 2 �[90m# src/test/resources/features/pre.3.6.x.feature:9�[0m
�[32mGiven �[0m�[32man �[0m�[32m�[1m1028�[0m�[32m, �[0m�[32m�[1m2�[0m�[32m, and �[0m�[32m�[1m"github1028"�[0m �[90m# cucumber.StepDefs.an_and(java.lang.Integer,java.lang.Integer,java.lang.String)�[0m
�[32mWhen �[0m�[32mexecute validate with �[0m�[32m�[1m"--skip-context-validation -t {datasrc}/xa.s16..shz.1976.070.0.xml --schema {datasrc}/PDS4_PDS_1N00.xsd --schematron {datasrc}/PDS4_PDS_1N00.sch"�[0m �[90m# cucumber.StepDefs.execute_validate(java.lang.String)�[0m
�[32mThen �[0m�[32mcompare to the .�[0m �[90m# cucumber.StepDefs.compare_to_the()�[0m
If you look at the output from github actions you will see the same thing:
Scenario Outline: NASA-PDS/validate#1028 1 # src/test/resources/features/3.6.x.feature:9
Given an 1028, 1, and "github1028" # cucumber.StepDefs.an_and(java.lang.Integer,java.lang.Integer,java.lang.String)
When execute validate with "--skip-context-validation -t {datasrc}/xa.s16..shz.1976.070.0.xml --schema {datasrc}/PDS4_PDS_1N00.xsd --schematron {datasrc}/PDS4_PDS_1N00.sch" # cucumber.StepDefs.execute_validate(java.lang.String)
Then compare to the . # cucumber.StepDefs.compare_to_the()
Scenario Outline: NASA-PDS/validate#1028 1 # src/test/resources/features/pre.3.6.x.feature:9
Given an 1028, 1, and "github1028" # cucumber.StepDefs.an_and(java.lang.Integer,java.lang.Integer,java.lang.String)
When execute validate with "--skip-context-validation -t {datasrc}/xa.s16..shz.1976.070.0.xml --schema {datasrc}/PDS4_PDS_1N00.xsd --schematron {datasrc}/PDS4_PDS_1N00.sch" # cucumber.StepDefs.execute_validate(java.lang.String)
Then compare to the . # cucumber.StepDefs.compare_to_the()
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.888 s -- in cucumber.CucumberTest
Yeah, I updated my pre.3.6.x to be subtest 2 while both are 1 in the commited files. This problem exists no matter how you do it and can use a python script to check the files that the same ticket number and subtest do not exist.
There was a problem hiding this comment.
- I did not know there was a purpose to that redundant information. Add it back.
Bulk of the work is done. Made the report.json match (give or take) for the tests from when run on main as well. If they are wrong, they are as wrong as they ere anyway. Added a new test function for running single scenerios as though typed in on the command line. cut-n-paste is no longer your friend when translating scenerios to command lines. However, what it should look like is printed out so you can just grab it from there now.
|
Please review when you can. Once this is merged, I will rebase then integrate all of the outstanding update PRs on validate. There are 2 new tools for developing in validate. cucumber.SingleScenerio lets you run any one scenerio to check that it will pass later. The other is RunFeatureScenerioAsValidateFromCLI. It translates any one scenerio into a set of command line args then sends it off to validate as though it was called from the command line. I did both of these because cut-n-paste does not work so well in eclipse anymore. Lastly, if we can figure out how, then we can run these in parallel. Order no longer matters - YAY!!! - and all tests are independent. I started looking at doing them in parallel then gave up quickly because all tests were not working yet. They are now, so lets schedule moving to parallel processing. |
|
@al-niessner not sure if this is intermittent or something else, but I am getting a failure on a few test cases. I made some updates to this branch, which I thought broke it, but I rolled those back and it still fails on my laptop (MacOS): |
|
I am seeing the same errors. Something external to PDS must have changed. Looking into it... |
|
Seems we were not able to load some resources last week or whenever this was developed. All of the errors were related to not be able to download schema and schematron resources. Shows that the testing still is dependent on the outside world which is a serious weakness. One that I think we will have to live with. I noticed that you did not like the blank subtest number. I changed the _ to a - to fit the rest of the nonsense. |
Three of the tests were trying to download or access network resources that were not available during the recreation of the feature files. These tests had some erroneous expectations because of the external lack of resources. While the fault is clear it does demonstrate a weakness of the tests that they can fail for reasons beyond the control of the validate repo.
|
|
Got to love the internet... tests pass 🎉 |
jordanpadams
changed the title
🗒️ Summary
Complete rewrite of cucumber testing.
⚙️ Test Data and/or Report
All unit tests should pass and there should be 251 cucumber tests.
♻️ Related Issues
Closes #1102