Maintaining lobid API

configure

There is a web/conf/resources.conf_template which is to be copied to web/conf/resources.conf and configured properly.

webhook: updating, creating and switching an index

The webhook configuration is part of web/conf/resources.conf.

webhook = {
    alma = {
            update = {
                    filename = "/data/other/datenportal/export/alma/prod/update.xml.bgzf"
                    indexname = "resources-alma-fix-staging"
            }
            basedump = {
                    filename = "/data/other/datenportal/export/alma/prod/baseline.xml.bgzf"
                    indexname = "resources-alma-fix"
                    switch = {
                            automatically = "false"
                            minDocs = "83000000"
                            minSize = "49613349376"
                    }
            }
            token = "secret"
    }
    email = "change@me"
}
index = {
  name = "resources-alma-fix-staging"

update an index

If you want to update an index, call the webhook listener like this:

sol@quaoar1:$ curl http://localhost:7507/resources/webhook/update-alma?token=secret

The paramter token must have the value configured in webhook.alma.token. This update call will ETL the file configured in webhook.alma.update.filename into the index named as described in webhook.alma.update.indexname.

create an index

If you want to create an index, call the webhook listener like this:

sol@quaoar3:$ curl http://localhost:7507/resources/webhook/basedump-alma?token=secret

It's deliberately not possible to create an index not suffixed with a timestamp. It's also deliberately not possible to assign an alias not suffixed with -staging. Both (suffixing timestamp to the index-name and -staging to the index-alias) is done automatically. The index-alias is created by the value found in webhook.alma.basedump.indexname automatically suffixing -staging.

more details

Principally you can call the webhook from wherever you want, but it's not that safe (or: transparent) as it's not guaranteed which webhook is configured or how its configuration looks like. You would call it like this:

$ curl http://test.lobid.org/resources/webhook/update-alma?token=secret

This domain is ("normally") wired to the play app currently running at quaoar3 at port 7502. Note also that you should only assign lower cased index-names, otherwise there may be problems.

switching index aliases

We always have at least two indices and two aliases, one of them with the -staging suffix. This way we can always switch quickly back to an older index if there is a problem with the new one. Also, we have a stage index for testing purpose (if we don't switch the index automatically, that is).

automatically

The switching of the aliases is done automatically if webhook.alma.basedump.switch.automatically is set to true and the values defined there (minDocs and minSize) are fullfilled in the newly created index.

manually

If you want to switch an alias from e.g. resources-alma-fix-staging to resources-alma-fix, making the index available at https://alma.lobid.org/resources/ , you can switch the alias manually in two ways:

using the elasticearch plugin "cerebro"

(see (only hbz-internally): https://dienst-wiki.hbz-nrw.de/display/SEM/Elasticsearch+Indexe?src=contextnavpagetreemode)

invoking the switch-alias webhook

$ curl -Lvvv http://localhost:7507/resources/webhook/switchalias?token=secret

The index alias, which is to be switched, is configured at: webhook.alma.basedump.indexname . This alias is switched with the alias consisting of this index name plus suffixed -staging.

deployment

note: the deployment is automatized via cron. If the deployment at Friday is enough for you, you don't need to do anything here. From the cron:

00 20 * * Fri ssh sol@quaoar1 "cd /home/sol/git/lobid-resources-alma/ && git pull && mvn install -DskipTests=true && cd web/ && bash restart.sh lobid-resources-alma"
00 20 * * Tue ssh sol@quaoar3 "cd /home/sol/git/lobid-resources-alma/ && git pull && mvn install -DskipTests=true && cd web/ && bash restart.sh lobid-resources-alma"

If you want to deploy a new FIX or new programs: It is crucial to know that it is not sufficient to reside in the git repo (sol@quaoar3:[~/git/lobid-resources-test/] ) and do your normal git pull and mvn clean install -DskipTests=true - you have to also restart the play app so that the new maven build and morphs are taken into account. Note that changes to the webhook parameters in web/conf/resources.conf are loaded dynamically whenever a webhook is called, so you don't need to restart anything when changing these configs.

Important: as the web app is deployed in a high available setting (i.e. mirrored at two different servers) you have to deploy at both servers: quaoar1 and quaoar3 to have them both in sync. See following "server" section how to do this.

Have a look at the logs in web/logs/*.log to see whats going on: application.log is the log file of the web app, ETL.log is the log file of the invoked ETL processes (that's the library you've built by doing mvn clean install at the root of the repo).

production

To invoke a "restart" of an API it's sufficient to kill the running web app - it's restarted automatically (see section "server.production" for details). You kill it by first getting the PID ("Process ID") with lsof -i:7507 and kill it by kill $PID. Wait a minute and see if the web app has started automatically by checking with lsof -i:7507 again.

test

Test runs at quaoar3:7502. It's also monitored, thus Killing the PID invokes a restart (which may take up to 5 minutes). You also could start it manually: first # monit stop lobid-resources-test and then: rm /home/sol/git/lobid-resources-test/web/target/universal/stage/RUNNING_PID; sbt "start 7502"

server

We have git-cloned repos at three locations:

production

The production web app serves "https://alma.lobid.org/resources/". It's deployed at two servers, one of them is the fallback server. If the production web app is failing, apache (using the HA proxy directive) automatically switches to the fallback server at quaoar2:7507 when quaoar1:7507 is down, e.g. when quaoar1:7507 is restarted when deploying. Thus we always have a seemingly working API. Apache switches back to quaoar1:7507 if that one is up again. Both apps are monitored by monit.

The web app (aka "the API") uses the index configured by index.name.

quaoar1, port 7507

At sol@quaoar1:~/git/lobid-resources-alma the production API is deployed.

quaoar3, port 7507

At sol@quaoar3:~/git/lobid-resources-alma the production API fallback is deployed.

test

This serves "http://test.lobid.org/resources/".

quaoar3, port 7502

At sol@quaoar3:~/git/lobid-resources-test the test API is deployed.