Deployment (for testing) failure

[alanruttenberg]

On OSX Yosemite:

brew cask install Caskroom/cask/vagrant
brew cask install Caskroom/cask/virtualbox
brew install ansible
vagrant up
vagrant ssh

so far so good.

Then, (in box)

cd /var/www/purl.obolibrary.org
make

which results in:

rm -rf temp tests
kwalify -f tools/config.schema.yml config/*.yml \
    | awk '{print} /INVALID/ {status=1} END {exit status}'
bash: kwalify: command not found
mkdir backup
mkdir -p temp/obo/aeo
tools/translate-entries.py config/aeo.yml temp/obo/aeo/.htaccess
Traceback (most recent call last):
  File "tools/translate-entries.py", line 59, in <module>
import argparse, sys, yaml, re
ImportError: No module named 'yaml'
make: *** [temp/obo/aeo/.htaccess] Error 1

[alanruttenberg]

In case it matters, the first time I ran vagrant up I got a message saying I needed a new version of virtualbox. Installed that. Second time it complained ansible wasn't installed. Third time "vagrant up" succeeded, with message:

==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

[alanruttenberg]

Even though it said box was provisioned, I tried running vagrant provision (I'm at monkey now, though)

Output:

vagrant provision
/opt/vagrant/embedded/gems/gems/vagrant-1.9.0/lib/vagrant/util/platform.rb:24: warning: Insecure world writable dir /opt/local/sbin in PATH, mode 040777
==> default: Running provisioner: ansible...
default: Running ansible-playbook...
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and 
make sure become_method is 'sudo' (default).
This feature will be removed in a 
future release. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [default]

TASK [Update apt-get cache] ****************************************************
changed: [default]

TASK [Install system-wide packages (not Travis)] *******************************
changed: [default] => (item=[u'ntp', u'git', u'ruby', u'ruby-dev', u'python3'])

TASK [Install system-wide packages (All)] **************************************
changed: [default] => (item=[u'apache2', u'python3-pip'])

TASK [Install Travis command-line] *********************************************
fatal: [default]: FAILED! => {"changed": true, "cmd": ["gem", "install", "travis", "-v", "1.8.0", "--no-rdoc", "--no-ri"], "delta": "0:01:02.071028", "end": "2017-01-10 20:48:15.689382", "failed": true, "rc": 1, "start": "2017-01-10 20:47:13.618354", "stderr": "ERROR:  Error installing travis:\n\tjson requires Ruby version ~> 2.0.", "stdout": "Building native extensions.  This could take a while...", "stdout_lines": ["Building native extensions.  This could take a while..."], "warnings": []}
    to retry, use: --limit @/Users/alanr/repos/purl.obolibrary.org/tools/site.retry

PLAY RECAP *********************************************************************
default                    : ok=4    changed=3    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

[cmungall]

I don't really know vagrant The tools/site.yml file has an ansible configuration, that has the relevant "gem install kwalify". Guess vagrant doesn't use this? Why don't we use docker here? (ducks)

[jamesaoverton]

Thanks for trying this Alan. I don't think anybody else has tried the Vagrant stuff, and I haven't run it in a while either, so there's bound to be some problems.

The travis gem is only used to see if the build is green before updating. You don't need it for a local test, so you can remove those lines from the site.yml and see if the rest works. That gem has given me trouble in the past, and it would be nice to avoid using it entirely.

I kind of like Ansible, and I used it to deploy to the curent PURL server (Debian on EC2). It's nice to use the same deployment tool aimed at the local Vagrant machine, and I made it work in Travis after jumping through some hoops. But I've had some trouble with Vagran file sharing in the past, and using it for Travis tests is slow.

I've had nothing but grief from Docker, and I won't be investing any more time in it until it is much more mature. For something as simple as this, it might work fine, but I leave that task to another brave soul.

[alanruttenberg]

Removing the travis install lets vagrant provision complete. Will try the rest now. Thanks! (should I update the instructions after success or do you want to?)

[cmungall]

On 10 Jan 2017, at 13:27, James A. Overton wrote: Thanks for trying this Alan. I don't think anybody else has tried the Vagrant stuff, and I haven't run it in a while either, so there's bound to be some problems. The travis gem is only used to see if the build is green before updating. You don't need it for a local test, so you can remove those lines from the `site.yml` and see if the rest works. That gem has given me trouble in the past, and it would be nice to avoid using it entirely.

I don't love it but we should have some form of checking the yaml tags (beyond yaml syntax checking) in travis? There is a python implementation that accepts the same schema files. I'm not wedded to kwalify as the schema language at all, but it's what we have at the moment...

I kind of like Ansible, and I used it to deploy to the curent PURL server (Debian on EC2). It's nice to use the same deployment tool aimed at the local Vagrant machine, and I made it work in Travis after jumping through some hoops. But I've had some trouble with Vagran file sharing in the past, and using it for Travis tests is slow. I've had nothing but grief from Docker, and I won't be investing any more time in it until it is *much* more mature. For something as simple as this, it might work fine, but I leave that task to another brave soul.

ok!

[alanruttenberg]

make test

In the below, is the first warning/stacktrace something I should worry about?
I think the validate error at the bottom is iao.owl (next alphabetically) but it doesn't say so. "would be nice".

rm -rf temp tests
kwalify -f tools/config.schema.yml config/*.yml \
    | awk '{print} /INVALID/ {status=1} END {exit status}'
/usr/lib/ruby/1.9.1/psych.rb:206:in `parse': (<unknown>): mapping values are not allowed in this context at line 103 column 14 (Psych::SyntaxError)
    from /usr/lib/ruby/1.9.1/psych.rb:206:in `parse_stream'
    from /usr/lib/ruby/1.9.1/psych.rb:289:in `load_stream'
    from /usr/lib/ruby/1.9.1/psych/deprecated.rb:28:in `load_documents'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:299:in `block in _validate_files'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:267:in `each'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:267:in `_validate_files'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:215:in `validate_files'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:111:in `execute'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:124:in `main'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/bin/kwalify:13:in `<top (required)>'
    from /usr/local/bin/kwalify:23:in `load'
    from /usr/local/bin/kwalify:23:in `<main>'
config/aeo.yml#0: valid.
config/aero.yml#0: valid.
config/agro.yml#0: valid.
...
config/hp.yml#0: valid.
config/hsapdv.yml#0: valid.
make: *** [validate] Error 1

[jamesaoverton]

Yes. kwalify is checking the YAML for each file against a schema, and the check is failing for one of the files. I agree that it's not friendly output. You're usually pretty sure that it's the file you're working is the one that failed validation.

[jamesaoverton]

@cmungall I'm not sure what you mean. The Travis tests run kwalify, but they also install Apache (using Ansible), run Apache with the PURL configuration, and check a large portion of the redirects against their targets.

The Travis gem uses their REST API to tell Make whether master is green, so Make can pull and update.

If we trust that master is always green, we don't need the Travis gem. But last week one of our maintainers commited directly to master, so I'm not sure it's safe.

[alanruttenberg]

Ok, It's working for me. Thanks!

Doc suggestions (while fresh in my mind)

• Document ansible dependency
• Mention vagrant provision
• Suggest that on OSX brew be used to install vagrant, ansible, virtualbox (commands above)
• Give command line example to validate one yml file (faster debugging)
• Mention setting up /etc/hosts to locally test (the raw IP is ok for testing a single file, but fails when testing against ontologies that import other ontologies).

Other suggestions:

• Set up so that inside the machine there's a git remote which is the outside machine's cloned repo (the one you're running vagrant from). git push/pull local my-branch can then be used to sync.
• That needs an ssh key from host to be copied to box. Since I don't know vagrant well, I was fumbling around how to transfer a file in :)
• Document the travis issue, or remove it from the default Site.yml

Thanks again. I found and fixed bugs it my iao.yml, which I was rearranging because it was a mess.

[alanruttenberg]

just noticed shared folder. Nice. so nix: - Set up so that inside the machine there's a git remote which is the outside machine's cloned repo (the one you're running vagrant from). git push/pull local my-branch can then be used to sync.

[alanruttenberg]

BTW, I committed changes to iao.yml, but they haven't shown up in the last day. Is there something that needs to be poked to refresh the server?

[jamesaoverton]

Thanks for pointing this out!

Updates are automatic and usually take <20 minutes. But the automatic updates were broken by the problem you reported in #287. I fixed the immediale problem in #288, then created an issue for the general problem in #290.

[jamesaoverton]

Hi @alanruttenberg. PRs #289, #291, and #292 should address all of your suggestions except the one about an SSH key for Vagrant. This shouldn't be necessary because the files on the host machines are synced into the VM. The new docs make this more clear.

Please take a look at #292.