-
Notifications
You must be signed in to change notification settings - Fork 6
Weblate server
This page describes how the Weblate service running at translate.rx.studio was configured, and how it is maintained.
The service runs on a single t3a.large node with 20GB storage (was extended from 10GB due to Docker pulls) in the us-east-1 region of AWS, with termination protection enabled. AWS Backup is configured to take weekly snapshots of the node with a retention window of 1 year.
Weblate is running using Docker and Docker compose:
version: '3'
services:
weblate:
image: weblate/weblate
volumes:
- weblate-data:/app/data
env_file:
- ./environment
restart: always
depends_on:
- database
- cache
environment:
WEBLATE_ENABLE_HTTPS: 1
WEBLATE_IP_PROXY_HEADER: HTTP_X_FORWARDED_FOR
database:
image: postgres:14-alpine
env_file:
- ./environment
volumes:
- postgres-data:/var/lib/postgresql/data
restart: always
cache:
image: redis:6-alpine
restart: always
command: [redis-server, --save, '60', '1']
volumes:
- redis-data:/data
https-portal:
image: steveltn/https-portal:1
ports:
- 80:80
- 443:443
restart: always
environment:
STAGE: production
PROXY_READ_TIMEOUT: 3600
CLIENT_MAX_BODY_SIZE: 100M
volumes:
- ssl-certs:/var/lib/https-portal
volumes:
weblate-data: {}
postgres-data: {}
redis-data: {}
ssl-certs: {}The environment file has no modifications, as the env var overrides are defined in the docker-compose-https.override.yml file:
version: '3'
services:
weblate:
environment:
WEBLATE_EMAIL_HOST: email-smtp.us-east-1.amazonaws.com
WEBLATE_EMAIL_PORT: 587
WEBLATE_EMAIL_USE_SSL: 0
WEBLATE_EMAIL_USE_TLS: 1
WEBLATE_EMAIL_HOST_USER: *******
WEBLATE_EMAIL_HOST_PASSWORD: *******
WEBLATE_SERVER_EMAIL: [email protected]
WEBLATE_DEFAULT_FROM_EMAIL: [email protected]
WEBLATE_SITE_DOMAIN: translate.rx.studio
WEBLATE_ADMIN_EMAIL: *******
WEBLATE_ADMIN_PASSWORD: *******
# TODO migrate to Google Analytics v4
WEBLATE_GOOGLE_ANALYTICS_ID: UA-*******-*
# enable daily pulls from R core subversion
WEBLATE_AUTO_UPDATE: true
https-portal:
environment:
DOMAINS: 'translate.rx.studio -> http://weblate:8080'The current image ids:
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
weblate/weblate latest 2ea7d086a8c4 4 days ago 1.1GB
redis 6-alpine 3616f0c0705d 7 days ago 27.1MB
postgres 14-alpine 9d94e6318ef2 7 days ago 242MB
steveltn/https-portal 1 1f41166c2e81 2 weeks ago 351MB
weblate/weblate <none> 02c6327da166 9 months ago 804MB
postgres <none> 07c710d28b91 9 months ago 216MB
redis <none> 57c580553a4d 10 months ago 25.5MB
postgres <none> ea498678e2bd 10 months ago 189MB
weblate/weblate <none> 475045de4358 10 months ago 847MB
steveltn/https-portal <none> 26e5bea459df 12 months ago 274MBCurrent weblate etc versions listed at https://translate.rx.studio/about/
Commands to start the service:
docker-compose -f docker-compose-https.yml -f docker-compose-https.override.yml build
docker-compose -f docker-compose-https.yml -f docker-compose-https.override.yml upTo update, follow the official docs, but in short:
docker-compose -f docker-compose-https.yml -f docker-compose-https.override.yml pull
docker-compose -f docker-compose-https.yml -f docker-compose-https.override.yml down
docker-compose -f docker-compose-https.yml -f docker-compose-https.override.yml up -dTo generate a report on the translation updates in a time period:
- Visit https://translate.rx.studio/projects/r-project/#reports
- Select time period and generate report in rST format
- Convert ~markdown to HTML and share in the R Contributors slack group's #core-translation channel
To submit a patch file on the translations found in Weblate but not in the trunk of the main R subversion repo:
-
Update the weblate repo from Subversion at https://translate.rx.studio/projects/r-project/#repository
-
Clone the weblate git repo from https://translate.rx.studio/git/r-project/base-r-gui/
-
Drop empty translation files to reduce noise, e.g. something like:
library(data.table) library(logger) last_commit <- '...' po_files <- \() list.files(pattern = '\\.po$', recursive = TRUE, full.names = TRUE) set_branch <- \(branch) system2('git', c('checkout', branch)) new_files <- po_files() set_branch(last_commit) old_files <- po_files() set_branch('main') added_files <- setdiff(new_files, old_files) for (f in added_files) { if (fread(cmd = paste('pocount --csv', f), sep = ',', fill = TRUE)$`Translated Messages` == 0) { log_info('dropping empty {f}') unlink(f) } else { log_info('keeping {f}') } }
-
Clone the subversion R repo or its git clone from [email protected]:wch/r-source.git
-
Make sure that both repos are up-to-date!
-
Copy over
src/libraryfrom the weblate repo to R/trunk. -
Generate a patch file from the diff, going back to the most recent commit with translations merged, e.g.
git diff --no-prefix 366f45a4599e04e00df59d063c67ccfadf27ae96
-
Share the patch file on the R Contributors Slack group's #core-translation channel and kindly ping @MichaelLawrence for his assistance on getting the patch file applied on the trunk of R dev to get it merged. We should do this ~once per quarter.
Components might become locked due to "not being able to push back to the upstream repo" (expected) or when upstream is not updated for a long time. In such case, components can be unlocked on the UI, but this needs to happen at the component level, so better to use the CLI instead:
weblate unlock_translation r-projectWhen there's a conflict between SVN and the Weblate git repo, and Weblate cannot pull from the upstream repo, you will need to resolve the conflicts manually after seeing an error like the below:
Rebasing (1/210)
Rebasing (2/210)
...
Rebasing (50/210)
error: could not apply 93fd9017... Added translation using Weblate (Spanish)
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 93fd9017... Added translation using Weblate (Spanish)
CONFLICT (add/add): Merge conflict in src/library/stats/po/es.po
Auto-merging src/library/stats/po/es.po
rebase refs/remotes/origin/trunk: command returned error: 1Weblate will provide a suggestion on how to fix, but in short:
-
SSH to the Weblate server
-
Attach the Docker container:
sudo docker exec -ti weblate-docker_weblate_1 bash -
Enter the git folder:
cd app/data/vcs/r-project/base-r-gui -
Pull from SVN and start the rebase process to see the actual errors:
git svn fetch git svn rebase
-
Resolve the conflict via a text editor, then continue. Let's say
src/library/base/po/es.pois affected:mcedit src/library/base/po/es.po git add src/library/base/po/es.po git commit -m "resolve conflict" git rebase --continue
To see the current progress of conflict resolution, run something like:
( RMD="$( git rev-parse --git-path 'rebase-merge/' )" && N=$( cat "${RMD}msgnum" ) && L=$( cat "${RMD}end" ) && echo "${N} / ${L}" ; )And make sure to use rerere (reuse recorded resolution) to speed up resolving the individual conflicts:
git config --global rerere.enabled trueCurrently @daroczig and @MichaelChirico have admin access to Weblate, both in the web UI and via SSH.
In case of any questions, reach out to them on the R-devel Slack's #core-translation channel, pinging both their usernames.
To add a new admin in the Weblate app:
- Go to the Users page
- Select the user
- Go to "Groups"
- Add to the "The R Project for Statistical Computing / Administration" group
To grant SSH access:
- Share your public key with one of the admins, who will add that to
~/.ssh/authorized_keys - Ask for the SSH config from one of the admins
There are language teams defined in each project with the below extra permissions:
- Manage glossary
- Review strings
- Manage translation memory
See the general Weblate FAQ on how to manage members.