Skip to content

Latest commit

 

History

History
277 lines (194 loc) · 17.4 KB

README.md

File metadata and controls

277 lines (194 loc) · 17.4 KB
Raw

Materia and Docker

Materia is deployed in docker containers orchestrated through docker compose in both development and production environments.

Overview

Container Architecture

  1. webserver (Nginx) runs the NGINX web server (proxies to phpfpm for app and serves static files directly).
  2. app (PHP-FPM) runs and manages the PHP processes for the application.
  3. mysql for storing relational application data. We recommend an external database source in production.
  4. memcached for caching data and sessions.
  5. fakeS3 mocks AWS S3 behavior for asset uploading. This should not be used in production.

Docker Compose files

The default docker-compose.yml file contains some base-level configurations for Materia:

  • The app service definition and its associated image.
  • The webserver service definition and its associated image.
  • The frontend and backend docker networks.

The docker-compose.development.yml file serves as a baseline template for the creation of a docker-compose.override.yml file via the first-run scripts and is not referenced directly by default. The docker-compose.override.test.yml is utilized for the test suite to separate test conditions from the running application.

The docker-compose.override.yml file is not tracked and generated by either the run_first_for_dev.sh or run_first_for_nondev.sh scripts. By default, docker compose up will initialize with both the docker-compose.yml and docker-compose.override.yml (if present) configurations. Docker compose can always be run with a different set of compose files via the -f flag:

docker compose -f docker-compose.yml -f docker-compose.my-override.yml up

Volumes

Docker volumes are used to mount files from the host machine into one or more containers, and to share file contents between containers. Depending on whether you are pursuing a development or more prod-like instance, volume definitions in your compose file(s) will be different.

  • Development: Your entire project directory is mounted to the app container. Static assets and widget assets are cross-mounted to the webserver container.
  • Production: Static (compiled) assets are virtually mounted on the app container and cross-mounted on the webserver container. Widget engine files and user media files (if using the file asset storage driver) are volume mounted from the host machine.

In both situations, the webserver has additional volume-mounted files: the NGINX configuration and typically (though dependent on your production setup), the key and certificate pair for SSL.

Data Persistence

Because docker containers are relatively ephemeral, certain resources and configurations should be sourced from the host machine or an external service and volume mounted to the containers through compose. This is especially important to facilitate future updates to the system. Let's review each:

The database

We highly recommend using an external database service like Amazon RDS, Azure Database for MySQL, or Cloud SQL. Alternatively, the database file can be volume-mounted from the host to enable persistence. The default development environment persists the mysql database in a virtual volume, which is not recommended for production. Regardless of where your database is located, you should perform regular backups.

Configuration files, like env variables

These configuration files should be defined on the host machine or an external service like AWS Secrets Manager. The docker compose file can define individual environment variables or import a local .env or .env.local file. The .env file in the root project directory serves as a template for production use; in docker/, the .env, .env.local, and .env.local.mysql files are used for the default development and nondevelopment instance.

Installed widget files

Widget files are installed to /var/www/html/public/widget on the application container, which is volume mounted from the host machine (in the Materia project directory: public/widget) and cross-mounted to the webserver container. These should be synced with an external service like AWS S3 or an external filesystem on a semi-regular interval.

User uploaded media

User media storage will depend on your ASSET_STORAGE_DRIVER configuration. If the file storage driver is in use, user media is saved to /var/www/html/fuel/app/media on the application container by default, custom configuration changes notwithstanding. Like installed widget files, this directory is volume mounted from the home machine (in the Materia project directory: fuel/app/media) and should be synced on a frequent interval with an external service like AWS S3.

Setup Details

Setup: Development Environment

Use ./run_first_for_dev.sh in the docker/ directory to initialize the application for development. Environment configurations specific to development include:

  1. Use of memcached for cache and session storage.
  2. Use of fakes3 for asset storage.
  3. Use of a local mysql database on a dedicated mysql container.
  4. Volume mounting the entire project directory into the application container, so that modifications of server files on the host machine are reflected in the running application.
  5. Configuration of an additional port (8008) to simulate serving static assets from a second domain.

Since your entire project directory is volume mounted, local changes to compiled JS and CSS assets will be reflected on the running application. Use yarn dev to run the webpack dev server and facilitate live reloading of assets. This requires a local installation of node and yarn.

Setup: Non-Development Environment

The non-dev configuration is ideal for those looking to play around with Materia locally without doing any local development. Additionally, it serves as a reasonable starting point for standing up a production instance of Materia.

The nondev startup script walks you through a series of configuration decisions to dynamically write the docker-compose.override.yml file. These include:

  1. The IP you use to access docker on your host machine (usually localhost).
  2. The use of a local database in a dedicated mysql container or an external one.
  3. The cache driver configuration (memcached or file).
  4. The session driver configuration (memcached, file, or db).
  5. The asset storage driver configuration (file or db).

Additionally, the script will explicitly ask whether or not you want to install the default set of widgets. While this is required for fresh instances, if using or migrating from an existing database, this may not be desired.

Environment Configuration

The docker/ directory contains a default .env file, plus additional files based on the nondev script:

  1. .env contains default dev-specific default values. This file is tracked, and changes to it are generally not recommended.
  2. .env.local is created by either of the run_first scripts. It contains environment configuration overrides and should be considered the authoritative source for environment variables.
  3. .env.local.mysql is created by the run_first_for_nondev.sh script to contain credential values for the mysql service if configured.

If a different .env file is desired, make sure to update your docker-compose.override.yml file with changes to the env_file parameter in the associated service definition (typically app, potentially mysql).

The Materia application will defer to environment variables to populate many configuration options. Storing these in environment variables that are loaded via docker compose ensures configurations will persist across container instances.

For a full breakdown of the environment variables available, refer to the Server Variables page in the documentation.

Image Versioning

Both the dev and nondev scripts use the -dev versions of the app and webserver images. These are automatically built and deployed to the GitHub package registry for any version tag, including alpha and rc tags. For convenience, a number of different image tags are provided: -dev, -stable, -<version>, and -<commit hash>.

-stable images are only created for stable releases (v10.3.0 as opposed to v10.3.0-alpha.1).

In a production instance, we recommend locking images to a specific version in your docker-compose.override.yml:

services:
  app:
	image: ghcr.io/ucfopen/materia:app-v10.3.0

By default, the base docker-compose.yml file will contain the image definitions for the app and webserver services. These can be overridden in the override compose file or applied to a standalone compose file depending on your preferred compose configuration.

Database Migrations

Updating to a new version of Materia may require migrations be performed on the database. These are handled by the built-in migration utility in FuelPHP. Database migration state is stored in the database in the migration table.

Important

As with all database modifications, you should ensure a backup is performed prior to running a migration.

In docker/:

./run.sh php oil r migrate

Note that running oil commands in production may require shell access to the app container if the Materia repository (and docker/ directory with its utility scripts) are not present (see Upgrading Materia in Production below).

Using Materia in Production

While the nondev startup script can assist with creating a compose file and configuring certain environment variables, a true production instance of Materia does not need the repository present on the host machine or instance.

At minimum, the host machine will require the following:

  1. Environment variable configurations written to a .env file.
  2. A docker-compose.yml file or combination docker-compose.yml and docker-compose.override.yml files.
  3. An NGINX configuration *see note below.
  4. A valid key and cert to provide to NGINX *see note below.
  5. A media directory with proper write permissions.
  6. A widget directory with proper write permissions.

Note

Two important considerations with NGINX:

  1. The webserver container comes pre-supplied with the docker/nginx-production.conf file copied to /etc/nginx/nginx.conf. You can use volume mounts to override this, as seen in docker-compose.development.yml or the override compose file, if a different configuration is needed.
  2. Whether the webserver includes a volume-mounted key and cert depends on where SSL terminates relative to the host machine. For example, if your Materia instance is located behind a load balancer, it may be sufficient to only listen for traffic on port 80. If your webserver is listening on ports 80 and 443, the webserver will require an NGINX configuration that includes references to a key and cert that are volume mounted from the host machine.

Based on the above, additional modifications to the docker compose file(s) should include:

  1. Importing the correct environment variables by ensuring the correct file is selected in a env_file: directive or variables are individually imported via a environment: directive. You can use the root .env as a template: just be sure to update the env_file: path for services accordingly.
  2. Ensuring the local paths for volume mounts for the widget and media directories are updated and correct.
  3. Ensuring the local paths for volume mounts for the NGINX configuration and key/cert pairs in the webserver service definition are updated and correct (if included).
  4. Selecting the preferred versions of the app and webserver images. For production, we recommend version-specific tags (e.g., app-v10.3.0 and webserver-v10.3.0) or the app-stable and webserver-stable tags.
  5. Any additional configurations for the webserver service definition as far as port assignments or considerations for network traffic reaching the host machine.

Important

Several environment variable configurations must be set or updated in a production instance. These include:

  • FUEL_ENV=production
  • LTI_KEY, LTI_SECRET, LTI_GUID, and LTI_TOOL_ID must be set if you intend to use Materia with an LMS.
  • AUTH_SALT, AUTH_SIMPLEAUTH_SALT, and CIPHER_KEY: see the commands section below for generating a unique salt hash for these values.
  • CRYPTO_IV, CRYPTO_HMAC, and CRYPTO_KEY are legacy encryption keys. They are not required for new copies of Materia, but can be set using unique salt hashes in the same way as CIPHER_KEY.

For easy reference, use the .env in the root project directory as a template for your production configurations.

Once configured, starting Materia on the host machine is as simple as:

docker compose pull
docker compose up -d

The -d flag for compose runs the containers in a detached state, so a terminal session for the compose process does not need to persist.

Upgrading Materia in Production

Upgrading to newer versions of Materia is a straightforward process under most circumstances, though specific upgrades may require configuration changes or database migrations. These requirements are specified in each release. This upgrade process makes several basic assumptions about your production setup:

  • No direct changes have been made to files on the container fileystems (app and webserver), such as modifications to PHP files. These changes are inherently ephemeral and will be lost as soon as the container is stopped or restarted.
  • The database is not hosted directly within the mysql container on a virtual volume. One exception is if the database itself (not the mysql process) is located on the host machine and volume mounted into the mysql container, in which case the database will persist across containers.
  • Environment variables, configs and credentials (such as NGINX), widget engine files, and user media files (if applicable) are volume mounted from the host machine.

The upgrade process involves the following:

  1. Stop the running containers (ctrl + c the running process or docker compose stop app if running in detached mode). At minimum, app and webserver should be stopped.
  2. Remove the stopped containers via docker compose rm app.
  3. Modify your docker-compose.yml, docker-compose.override.yml, or whatever other compose file you have configured that includes image: definitions for the app and webserver containers. If using the -stable tag, this will not be necessary. If using version tags, specify the new version:
services:
  app:
	image: ghcr.io/ucfopen/materia:app-v10.3.0 # use the semver value for the new version if using version tags

	...additional app definitions...

  webserver:
	image: ghcr.io/ucfopen/materia:webserver-v10.3.0 # use the semver value for the new version if using version tags

	...additional webserver definitions...
  1. Run docker compose pull.
  2. Run docker compose up or docker compose up -d to run the containers in detached mode.
  3. If a database migration is required, perform the migration by first getting shell access to the app container and then running the migrate command:
<host machine shell>  $ docker exec -it <app container id> sh
<app container shell> $ php oil r migrate

Configuring Authentication

Refer to the Authentication section of the base Materia README for an explanation of the authentication options available.

Commands and Utilities

Common Dev Commands

  • Run commands on the app container (like php, composer, or fuelphp oil commands)

     ./run.sh php -i
 ./run.sh php oil r admin:help
 ./run.sh composer run --list

  • Stop containers (db data is retained)

     docker compose stop

  • Stop and destroy the containers (deletes database data!, first_run.sh required after)

     docker compose down

  • Install composer libraries on the app container

     ./run.sh composer install

  • Install all Widgets in fuel/app/tmp/widget_packages/*.wigt

     ./run_widgets_install.sh '*.wigt'

  • Run Tests for development

    ./run_tests.sh

  • Run Tests with code coverage

    ./run_tests_coverage.sh

  • Create a user based on your docker host machine's current user

    $ max_power@ucf: ./run_create_me.sh
User Created: max_power password: kogneato
max_power now in role: super_user
max_power now in role: basic_author

  • Create a user manually

     ./run.sh php oil r admin:new_user username firstname mi lastname email password

  • Installing widgets: Copy the widget file you want to install into app/fuel/app/tmp/widget_packages/ and then run install_widget.sh passing the name of the widget file to install. Example:

     cp my_widget.wigt ~/my_projects/materia_docker/app/fuel/app/tmp
 cd ~/my_projects/materia_docker
 ./run_widgets_install.sh my_widget.wigt

  • Generate a unique salt hash for auth, cipher, and crypto configuration values:

     docker compose run --rm app php -r "echo(sodium_bin2hex(random_bytes(SODIUM_CRYPTO_STREAM_KEYBYTES)));"

Default User Accounts

If you wish to log into Materia, there are 3 default accounts created for you based on the config. If you're on MacOS or Linux, you'll also get a user based on the username you use on the host machine.