Update webdriver tests README.md

webdriver/README.md

@@ -1,17 +1,130 @@
 # wpt.fyi Webdriver tests
 
-This directory covers Webdriver tests for the `webapp`, written in a 3rd-party
-Golang Webdriver client, [tebeka/selenium](https://github.com/tebeka/selenium).
+This directory containers integration tests for webapp/. These tests bring up
+the full webserver then use a Golang Webdriver client,
+[tebeka/selenium](https://github.com/tebeka/selenium), to load pages in a
+browser (Chrome or Firefox) and assert that the webapp behaves as expected.
 
 To run the tests, from the root `wpt.fyi` directory, run:
 
-    make go_webdriver_test
+    make go_large_test
 
-If you want to actually see the tests in action, disable the frame buffer.
+You can run just one of Chrome or Firefox via:
 
-    make webdriver_deps
-    go test --frame_buffer=false -tags=large ./webdriver
+    make go_chrome_test
+    make go_firefox_test
 
-If you want to use a custom installed location of selenium / browser / driver
-binaries, the required flags are shown in [the Makefile](../Makefile)
-`go_webdriver_test' rule.
+Note that when running these tests outside of docker (see [Running in
+docker](#running-in-docker)), they will use your locally installed browser and
+webdriver clients, so it is worth making sure they are the versions you expect.
+
+## Debugging
+
+Debugging the webdriver/ tests can be difficult as they are integration tests
+and the problem can occur anywhere from controlling the browser, to the webapp
+frontend, to the backend - and other weird bugs inbetween! This section
+contains some tips on how to effectively debug them.
+
+After running one of the above `make` commands at least once, one can directly
+run the golang tests via:
+
+```
+# You can copy GECKODRIVER_PATH out of the make output; it should be installed
+# locally under webapp/node_modules/selenium-standalone/...
+go test -v -timeout=15m -tags=large ./webdriver -args \
+    -firefox_path=/usr/bin/firefox \
+    -geckodriver_path=$GECKODRIVER_PATH \
+    -chrome_path=/usr/bin/google-chrome \
+    -chromedriver_path=/usr/bin/chromedriver \
+    -frame_buffer=true \
+    -staging=false \
+    -browser=chrome  # Or firefox
+```
+
+It is worth comparing this command-line against the Makefile, in case this
+documentation becomes out of date.
+
+### Running only one test
+
+If you only need to run one test, you can use the golang test [`-run`
+parameter](https://golang.org/pkg/testing/#hdr-Subtests_and_Sub_benchmarks)).
+For example:
+
+```
+go test -v -timeout=15m -tags=large ./webdriver \
+    -run TestProductParam_Order/Order \
+    -args \
+    -firefox_path=/usr/bin/firefox \
+    -geckodriver_path=$GECKODRIVER_PATH \
+    -chrome_path=/usr/bin/google-chrome \
+    -chromedriver_path=/usr/bin/chromedriver \
+    -frame_buffer=true \
+    -staging=false \
+    -browser=chrome  # Or firefox
+```
+
+### Visual Output
+
+Many of the tests run some javascript (or click on an element, etc) and expect
+to find some resulting change on the page. When that doesn't occur, they
+usually just timeout and it can be difficult to know why. One very useful trick
+is to enable visual output, so that you can actually see the webpage as the
+test runs.
+
+To do this, set the `frame_buffer` argument to `false`, e.g.:
+
+```
+go test -v -timeout=15m -tags=large ./webdriver -args \
+    -firefox_path=/usr/bin/firefox \
+    -geckodriver_path=$GECKODRIVER_PATH \
+    -chrome_path=/usr/bin/google-chrome \
+    -chromedriver_path=/usr/bin/chromedriver \
+    -frame_buffer=false \  # <--- this is set to false
+    -staging=false \
+    -browser=chrome  # Or firefox
+```
+
+### Verbose webdriver output
+
+By default, webdriver output is hidden as it is very noisy. You can re-enable
+it by passing `-debug=true` to the tests, e.g.:
+
+```
+go test -v -timeout=15m -tags=large ./webdriver -args \
+    -firefox_path=/usr/bin/firefox \
+    -geckodriver_path=$GECKODRIVER_PATH \
+    -chrome_path=/usr/bin/google-chrome \
+    -chromedriver_path=/usr/bin/chromedriver \
+    -frame_buffer=true \
+    -staging=false \
+    -browser=chrome \
+    -debug=true
+```
+
+Redirecting stderr to stdout and saving it to a log-file is recommended due to
+the verbosity of webdriver logs (append `2>&1 | tee my-log.txt` to the above
+command).
+
+### Running in docker
+
+Sometimes bugs only occur in a docker-like environment. This can be difficult
+to reproduce, but a first step is to run the tests inside of docker. To do
+this, first start the docker container in one terminal tab:
+
+```
+./util/docker-dev/run.sh
+```
+
+Then, in another tab, we need to get the instance id of the container, exec
+'bash' inside of it, and run our test:
+
+```
+docker container ls
+[ note the CONTAINER ID ]
+docker exec -it $CONTAINER_ID bash
+user@abce84dd426d:~/wpt.fyi$
+[now you can run 'make go_chrome_test', or 'go test ...' directly, etc]
+```
+
+Note that this maps the host machine's wpt.fyi checkout into docker, so any
+code edits you make on the host are reflected in the container and vice-versa.