Merge branch 'develop'

user_documentation/rst/dashboard.rst

@@ -3,7 +3,11 @@
 Dashboard
 *********
 
-MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[port]``.
+MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[PORT]``, where 
+
+* [IP] is the ip address of MiCADO master, the virtual machine you have launched in Step 4 of :ref:`deployment`
+
+* [PORT] is the port number configured during Step 4 of :ref:`deployment`, its value is held by the ``web_listening_port`` variable specified in the ``micado-master.yml`` ansible file. 
 
 The following webpages are currently exposed:
 

user_documentation/rst/deployment.rst

@@ -1,3 +1,5 @@
+.. _deployment:
+
 Deployment
 **********
 
@@ -11,6 +13,13 @@ We recommend to perform the installation remotely as all your configuration file
 Prerequisites
 =============
 
+For cloud interfaces supported by MiCADO:
+
+* EC2 (tested on Amazon and OpenNebula)
+* Nova (tested on OpenStack)
+* CloudSigma
+* CloudBroker
+
 For the MiCADO master:
 
 * Ubuntu 16.04
@@ -36,7 +45,7 @@ To install Ansible on Ubuntu 16.04, use these commands:
    sudo apt-get update
    sudo apt-get install ansible
 
-To install Ansible on other operation system follow the `official installation guide <https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html>`__.
+To install Ansible on other operation systems follow the `official installation guide <https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html>`__.
 
 Git
 ---
@@ -47,7 +56,7 @@ To install Git on Ubuntu, use this command:
 
    sudo apt-get install git-all
 
-To install Git on other operating system follow the `official installation guide <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`__.
+To install Git on other operating systems follow the `official installation guide <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`__.
 
 Installation
 ============
@@ -61,42 +70,42 @@ Step 1: Download the ansible playbook.
 
    git clone https://github.com/micado-scale/ansible-micado.git ansible-micado
    cd ansible-micado
-   git checkout v0.6.0
+   git checkout v0.6.1
 
-Step 2: Specify credential for instantiating MiCADO workers.
-------------------------------------------------------------
+Step 2: Specify cloud credential for instantiating MiCADO workers.
+------------------------------------------------------------------
 
-MiCADO master will use this credential to start/stop VM instances (MiCADO workers) to host the application and to realize scaling. Credentials here should belong to the same cloud as where MiCADO master is running. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the credential in a file, called credentials.yml. Please, do not modify the structure of the template!
+MiCADO master will use this credential to start/stop VM instances (MiCADO workers) to host the application and to realize scaling. Credentials here should belong to the same cloud as where MiCADO master is running. We recommend making a copy of our predefined template and edit it. MiCADO expects the credential in a file, called credentials-cloud-api.yml before deployment. Please, do not modify the structure of the template!
 
 ::
 
-   cp sample-credentials.yml credentials.yml
-   vi credentials.yml
+   cp sample-credentials-cloud-api.yml credentials-cloud-api.yml
+   edit credentials-cloud-api.yml
 
-Edit credentials.yml to add cloud credentials. You will find predefined sections in the template for each cloud interface type MiCADO supports. Fill only the section belonging to your target cloud.
+Edit credentials-cloud-api.yml to add cloud credentials. You will find predefined sections in the template for each cloud interface type MiCADO supports. Fill only the section belonging to your target cloud.
 
 Optionally you can use the `Ansible Vault <https://docs.ansible.com/ansible/2.4/vault.html>`_ mechanism to keep the credential data in an encrypted format. To achieve this, create the above file using Vault with the command
 
 ::
 
-    ansible-vault create credentials.yml
+    ansible-vault create credentials-cloud-api.yml
 
 
-This will launch *vi* or the editor defined in the ``$EDITOR`` environment variable to make changes to the file. If you wish to make any changes to the previously encrypted file, you can use the command
+This will launch the editor defined in the ``$EDITOR`` environment variable to make changes to the file. If you wish to make any changes to the previously encrypted file, you can use the command
 
 ::
 
-    ansible-vault edit credentials.yml
+    ansible-vault edit credentials-cloud-api.yml
 
-Step 3a: (Optional) Specify security settings and credentials.
---------------------------------------------------------------
+Step 3a: Specify security settings and credentials to access MiCADO.
+--------------------------------------------------------------------
 
-MiCADO master will use these security-related settings and credentials during provisioning.
+MiCADO master will use these security-related settings and credentials to authenticate its users for accessing the REST API and Dashboard.
 
 ::
 
-   cp sample-security-cred.yml security-cred.yml
-   vi security-cred.yml
+   cp sample-credentials-micado.yml credentials-micado.yml
+   edit credentials-micado.yml
 
 Specify the provisioning method for the x509 keypair used for TLS encryption of the management interface in the ``tls`` subtree:
 
@@ -108,34 +117,40 @@ Specify the default username and password for the administrative we user in the
 Optionally you may use the Ansible Vault mechanism as described in Step 2 to protect the confidentiality and integrity of this file as well.
 
 
-Step 3b: (Optional) Specify details of your private Docker repository.
-----------------------------------------------------------------------
+Step 3b: (Optional) Specify credentials to use private Docker registries.
+-------------------------------------------------------------------------
 
-Set the Docker login credentials of your private Docker registries in which your personal containers are stored. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the docker registry details in a file, called docker-cred.yml. Please, do not modify the structure of the template!
+Set the Docker login credentials of your private Docker registry in which your private containers are stored. We recommend making a copy of our predefined template and edit it. MiCADO expects the docker registry credentials in a file, called credentials-docker-registry.yml. Please, do not modify the structure of the template!
 
 ::
 
-   cp sample-docker-cred.yml docker-cred.yml
-   vi docker-cred.yml
+   cp sample-credentials-docker-registry.yml credentials-docker-registry.yml
+   edit credentials-docker-registry.yml
 
-Edit docker-cred.yml and add username, password, and repository url. To login to the default docker_hub, leave DOCKER_REPO as is (a blank string).
+Edit credentials-docker-registry.yml and add username, password, and registry url. To login to the default docker_hub, leave DOCKER_REPO as is (a blank string).
 
 Optionally you may use the Ansible Vault mechanism as described in Step 2 to protect the confidentiality and integrity of this file as well.
 
 Step 4: Launch an empty cloud VM instance for MiCADO master.
 ------------------------------------------------------------
 
-This new VM will host the MiCADO master core services. Use any of aws, ec2, nova, etc command-line tools or web interface of your target cloud to launch a new VM. We recommend a VM with 2 cores, 4GB RAM, 20GB disk. Make sure you can ssh to it (password-free i.e. ssh public key is deployed) and your user is able to sudo (to install MiCADO as root). Store its IP address which will be referred as ``IP`` in the following steps. The following ports should be open on the virtual machine:
+This new VM will host the MiCADO core services. 
+
+**a)** Default port number for MiCADO service is ``443``. Optionally, you can modify the port number stored by the variable called ``web_listening_port`` defined in the ansible playbook file called ``micado-master.yml``. 
+
+**b)** Configure a cloud firewall settings which opens the following ports on the MiCADO master virtual machine:
 
 ::
 
    TCP: 22,2377,7946,8300,8301,8302,8500,8600,[web_listening_port]
    UDP: 4789,7946,8301,8302,8600
 
-**NOTE:** ``web_listening_port`` is defined in the ansible inventory file called ``hosts`` and defaults to ``443``
+**NOTE:** replace ``[web_listening_port]`` with the actual value specified in Step 4a.
 
 **NOTE:** MiCADO master has built-in firewall, therefore you can leave all ports open at cloud level.
 
+**c)** Finally, launch the virtual machine with the proper settings (capacity, ssh keys, firewall): use any of aws, ec2, nova, etc command-line tools or web interface of your target cloud to launch a new VM. We recommend a VM with 2 cores, 4GB RAM, 20GB disk. Make sure you can ssh to it (password-free i.e. ssh public key is deployed) and your user is able to sudo (to install MiCADO as root). Store its IP address which will be referred as ``IP`` in the following steps.
+
 Step 5: Customize the inventory file for the MiCADO master.
 -----------------------------------------------------------
 
@@ -144,7 +159,7 @@ We recommend making a copy of our predefined template and edit it. Use the templ
 ::
 
    cp sample-hosts hosts
-   vi hosts
+   edit hosts
 
 Edit the ``hosts`` file to set ansible variables for MiCADO master machine. Update the following parameters:
 
@@ -154,8 +169,6 @@ Edit the ``hosts`` file to set ansible variables for MiCADO master machine. Upda
 * **ansible_become**: specifies if account change is needed to become root, defaults to "True"
 * **ansible_become_method**: specifies which command to use to become superuser, defaults to "sudo"
 * **ansible_python_interpreter**: specifies the interpreter to be used for ansible on the target host, defaults to "/usr/bin/python3"
-* **docker_cred_path**: sets the path of file storing the credentials for private docker registries, defaults to "./docker-cred.yml"
-* **web_listening_port**: specifies the listening port of the management interface including the MiCADO dashboard and the REST interface, defaults to the default HTTPS port (443/TCP)
 
 Please, revise all the parameters, however in most cases the default values are correct.
 
@@ -186,8 +199,8 @@ Check the logs
 
 You can SSH into MiCADO master and check the logs at any point after MiCADO is succesfully deployed. All logs are kept under ``/var/log/micado`` and are organised by components. Scaling decisions, for example, can be inspected under ``/var/log/micado/policykeeper``
 
-Accessing user service
-======================
+Accessing user-defined service
+==============================
 
 In case your application contains container exposing a service, you have two alternatives to access its endpoint:
 

user_documentation/rst/release_notes.rst

@@ -1,6 +1,21 @@
 Release Notes
 *************
 
+**v0.6.1 (15 Oct 2018)**
+
+- enable VM-only deployments
+- add support for special characters in SSL credentials
+- fix missing vm instance number reset at undeployment
+- add option to disable auto-updates on worker nodes
+- modify default launch-order of TOSCA adaptors
+- add cloud-specific TOSCA templates and improve helper scripts for stressng
+- flatten CPU scaling policies
+- improve virtual machine build time
+- fix Zorp starting dependency
+- fix Docker login timing issue
+- remove unnecessary port from docker compose file
+- enable Prometheus DB export
+
 **v0.6.0 (10 Sept 2018)**
 
 - introduce documentation repository and host its content at http://micado-scale.readthedocs.io
@@ -13,7 +28,7 @@ Release Notes
 - fix issue with worker node deployment in EC2 clouds
 - fix issue with user-defined Docker networks in OpenStack clouds
 - make Submitter response message structure uniform
-- add 'nodes' and 'services' query methods to REST API 
+- add 'nodes' and 'services' query methods to REST API
 - improve 'stressng' and 'cqueue' test helper scripts
 - add more compose properties to custom TOSCA definition
 - fix floating ip issues in the Dashboard component

user_documentation/rst/tutorials.rst

@@ -3,7 +3,7 @@
 Tutorials
 *********
 
-You can find test application(s) under the subdirectories of the ‘testing’ directory. The current tests are configured for CloudSigma.
+You can find test application(s) under the subdirectories of the ‘testing’ directory. The current **stressng** test can be configured for use with CloudSigma, AWS EC2, OpenStack Nova and deployments via CloudBroker. The current **cqueue** test is configured for CloudSigma.
 
 stressng
 ========
@@ -12,15 +12,17 @@ This application contains a single service, performing a constant CPU load. The
 
 **Note:** make sure you have the ``jq`` tool installed required by the helper scripts.
 
-*  Step1: add your ``public_key_id`` to both the ``stressng.yaml`` and ``stressng-update.yaml`` files. Without this CloudSigma does not execute the contextualisation on the MiCADO worker nodes. The ID must point to your public ssh key under your account in CloudSigma. You can find it on the CloudSigma Web UI under the “Access & Security/Keys Management” menu as **Uuid**.
-* Step2: add a proper ``firewall_policy`` to both the ``stressng.yaml`` and ``stressng-update.yaml`` files. Without this MiCADO master will not reach MiCADO worker nodes. Firewall policy ID can be retrieved from a rule defined under the “Networking/Policies” menu. The following ports must be opened for MiCADO workers: all inbound connections from MiCADO master
-*  Step3: Update the parameter file, called ``_settings``. You need the ip address for the MiCADO master and should name the application by setting the APP_ID  ***the application ID can not contain any underscores ( _ )**
-*  Step4: run ``1-submit-tosca-stressng.sh`` to create the minimum number of MiCADO worker nodes and to deploy the docker stack including the stressng service defined in the ``stressng.yaml`` TOSCA description. A few minutes after successful deployment, the system should respond by scaling up virtual machines and containers to the maximum specified.
+*  Step1: make a copy of the TOSCA file which is appropriate for your cloud - ``stressng_<your_cloud>.yaml`` - and name it ``stressng.yaml`` (ie. by issuing the command ``cp stressng_cloudsigma.yaml stressng.yaml``)
+*  Step2: fill in the requested fields beginning with ``ADD_YOUR_...`` . These will differ depending on which cloud you are using. 
+ * In CloudSigma, for example, the ``libdrive_id`` , ``public_key_id`` and ``firewall_policy`` fields must be completed. Without these, CloudSigma does not have enough information to launch your worker nodes. All information is found on the CloudSigma Web UI. ``libdrive_id`` is the long alphanumeric string in the URL when a drive is selected under “Storage/Library”. ``public_key_id`` is under the “Access & Security/Keys Management” menu as **Uuid**. ``firewall_policy`` can be found when selecting a rule defined under the “Networking/Policies” menu. The following ports must be opened for MiCADO workers: *all inbound connections from MiCADO master*
+*  Step3: Update the parameter file, called ``_settings``. You need the ip address for the MiCADO master and should name the application by setting the APP_ID  ***the application ID can not contain any underscores ( _ )** You should also change the SSL user/password/port information if they are different from the default.
+*  Step4: run ``1-submit-tosca-stressng.sh`` to create the minimum number of MiCADO worker nodes and to deploy the docker stack including the stressng service defined in the ``stressng.yaml`` TOSCA description.
 *  Step4a: run ``2-list-apps.sh`` to see currently running applications and their IDs
 *  Step4b: run ``query-services.sh`` to see the details of docker services of your application
 *  Step4c: run ``query-nodes.sh`` to see the details of docker nodes hosting your application
-*  Step5: run ``3-update-tosca-stressng.sh`` to update the service and reduce the CPU load. After a few moments the system should respond by scaling down virtual machines and containers to the minimum specified.
-*  Step6: run ``4-undeploy-stressng.sh`` to remove the stressng stack and all the MiCADO worker nodes
+*  Step5: run ``3-stress-cpu-stressng.sh 85`` to stress the service and increase the CPU load. After a few minutes, you will see the system respond by scaling up virtual machines and containers to the maximum specified.
+*  Step6: run ``3-stress-cpu-stressng.sh 10`` to update the service and decrease the CPU load. After a few moments the system should respond by scaling down virtual machines and containers to the minimum specified.
+*  Step7: run ``4-undeploy-stressng.sh`` to remove the stressng stack and all the MiCADO worker nodes
 
 cqueue
 ======