From 3214af80650a6d882726988c73510682b9018f14 Mon Sep 17 00:00:00 2001 From: jaydesl <35102795+jaydesl@users.noreply.github.com> Date: Thu, 27 Sep 2018 14:16:57 +0100 Subject: [PATCH 1/7] Update dashboard.rst --- user_documentation/rst/dashboard.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/user_documentation/rst/dashboard.rst b/user_documentation/rst/dashboard.rst index e869637..387c3d6 100644 --- a/user_documentation/rst/dashboard.rst +++ b/user_documentation/rst/dashboard.rst @@ -3,7 +3,7 @@ Dashboard ********* -MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[port]``. +MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[port]``. This port number is configured during initial setup of the MiCADO master *(Docs > Deployment > Installation > Step 5 > web_listening_port)* and defaults to port 443. The following webpages are currently exposed: From 39038eabac47425d0c27b3881c14aef5730451d8 Mon Sep 17 00:00:00 2001 From: jaydesl <35102795+jaydesl@users.noreply.github.com> Date: Thu, 27 Sep 2018 14:21:12 +0100 Subject: [PATCH 2/7] Update deployment.rst --- user_documentation/rst/deployment.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/user_documentation/rst/deployment.rst b/user_documentation/rst/deployment.rst index 13e7c4a..5191dd3 100644 --- a/user_documentation/rst/deployment.rst +++ b/user_documentation/rst/deployment.rst @@ -15,6 +15,10 @@ For the MiCADO master: * Ubuntu 16.04 +For the MiCADO worker nodes: + +* A MiCADO-supported cloud: OpenStack, OpenNebula, AmazonEC2 or CloudSigma + For the host where the Ansible playbook is executed (differs depending on local or remote): * Ansible 2.4 or greater From 87d27072f89c20b18b5f2f8b41ad6c1dc62309d8 Mon Sep 17 00:00:00 2001 From: jaydesl <35102795+jaydesl@users.noreply.github.com> Date: Fri, 28 Sep 2018 10:50:29 +0100 Subject: [PATCH 3/7] Update deployment.rst Step 3a is not optional ```cp sample-security-cred.yml security-cred.yml``` is a necessary step --- user_documentation/rst/deployment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/user_documentation/rst/deployment.rst b/user_documentation/rst/deployment.rst index 5191dd3..4b487b9 100644 --- a/user_documentation/rst/deployment.rst +++ b/user_documentation/rst/deployment.rst @@ -92,7 +92,7 @@ This will launch *vi* or the editor defined in the ``$EDITOR`` environment varia ansible-vault edit credentials.yml -Step 3a: (Optional) Specify security settings and credentials. +Step 3a: Specify security settings and credentials. -------------------------------------------------------------- MiCADO master will use these security-related settings and credentials during provisioning. From cea6f7cc8bc6860a87f80c735310297715dcee24 Mon Sep 17 00:00:00 2001 From: jaydesl <35102795+jaydesl@users.noreply.github.com> Date: Thu, 11 Oct 2018 13:37:16 +0100 Subject: [PATCH 4/7] Update stressng tutorial --- user_documentation/rst/tutorials.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/user_documentation/rst/tutorials.rst b/user_documentation/rst/tutorials.rst index 74a4ed8..7c57462 100644 --- a/user_documentation/rst/tutorials.rst +++ b/user_documentation/rst/tutorials.rst @@ -3,7 +3,7 @@ Tutorials ********* -You can find test application(s) under the subdirectories of the ‘testing’ directory. The current tests are configured for CloudSigma. +You can find test application(s) under the subdirectories of the ‘testing’ directory. The current **stressng** test can be configured for use with CloudSigma, AWS EC2, OpenStack Nova and deployments via CloudBroker. The current **cqueue** test is configured for CloudSigma. stressng ======== @@ -12,15 +12,17 @@ This application contains a single service, performing a constant CPU load. The **Note:** make sure you have the ``jq`` tool installed required by the helper scripts. -* Step1: add your ``public_key_id`` to both the ``stressng.yaml`` and ``stressng-update.yaml`` files. Without this CloudSigma does not execute the contextualisation on the MiCADO worker nodes. The ID must point to your public ssh key under your account in CloudSigma. You can find it on the CloudSigma Web UI under the “Access & Security/Keys Management” menu as **Uuid**. -* Step2: add a proper ``firewall_policy`` to both the ``stressng.yaml`` and ``stressng-update.yaml`` files. Without this MiCADO master will not reach MiCADO worker nodes. Firewall policy ID can be retrieved from a rule defined under the “Networking/Policies” menu. The following ports must be opened for MiCADO workers: all inbound connections from MiCADO master -* Step3: Update the parameter file, called ``_settings``. You need the ip address for the MiCADO master and should name the application by setting the APP_ID ***the application ID can not contain any underscores ( _ )** -* Step4: run ``1-submit-tosca-stressng.sh`` to create the minimum number of MiCADO worker nodes and to deploy the docker stack including the stressng service defined in the ``stressng.yaml`` TOSCA description. A few minutes after successful deployment, the system should respond by scaling up virtual machines and containers to the maximum specified. +* Step1: make a copy of the TOSCA file which is appropriate for your cloud - ``stressng_.yaml`` - and name it ``stressng.yaml`` (ie. by issuing the command ``cp stressng_cloudsigma.yaml stressng.yaml``) +* Step2: fill in the requested fields beginning with ``ADD_YOUR_...`` . These will differ depending on which cloud you are using. + * In CloudSigma, for example, the ``libdrive_id`` , ``public_key_id`` and ``firewall_policy`` fields must be completed. Without these, CloudSigma does not have enough information to launch your worker nodes. All information is found on the CloudSigma Web UI. ``libdrive_id`` is the long alphanumeric string in the URL when a drive is selected under “Storage/Library”. ``public_key_id`` is under the “Access & Security/Keys Management” menu as **Uuid**. ``firewall_policy`` can be found when selecting a rule defined under the “Networking/Policies” menu. The following ports must be opened for MiCADO workers: *all inbound connections from MiCADO master* +* Step3: Update the parameter file, called ``_settings``. You need the ip address for the MiCADO master and should name the application by setting the APP_ID ***the application ID can not contain any underscores ( _ )** You should also change the SSL user/password/port information if they are different from the default. +* Step4: run ``1-submit-tosca-stressng.sh`` to create the minimum number of MiCADO worker nodes and to deploy the docker stack including the stressng service defined in the ``stressng.yaml`` TOSCA description. * Step4a: run ``2-list-apps.sh`` to see currently running applications and their IDs * Step4b: run ``query-services.sh`` to see the details of docker services of your application * Step4c: run ``query-nodes.sh`` to see the details of docker nodes hosting your application -* Step5: run ``3-update-tosca-stressng.sh`` to update the service and reduce the CPU load. After a few moments the system should respond by scaling down virtual machines and containers to the minimum specified. -* Step6: run ``4-undeploy-stressng.sh`` to remove the stressng stack and all the MiCADO worker nodes +* Step5: run ``3-stress-cpu-stressng.sh 85`` to stress the service and increase the CPU load. After a few minutes, you will see the system respond by scaling up virtual machines and containers to the maximum specified. +* Step6: run ``3-stress-cpu-stressng.sh 10`` to update the service and decrease the CPU load. After a few moments the system should respond by scaling down virtual machines and containers to the minimum specified. +* Step7: run ``4-undeploy-stressng.sh`` to remove the stressng stack and all the MiCADO worker nodes cqueue ====== From 9d9c8899fcdce02af727580c61de90c2ceafdf8a Mon Sep 17 00:00:00 2001 From: Jozsef Kovacs Date: Fri, 12 Oct 2018 10:37:14 +0000 Subject: [PATCH 5/7] add supported cloud interfaces, rename credential files, rearrange step4 (port, firewall, vm) --- user_documentation/rst/deployment.rst | 75 +++++++++++++++------------ 1 file changed, 42 insertions(+), 33 deletions(-) diff --git a/user_documentation/rst/deployment.rst b/user_documentation/rst/deployment.rst index 4b487b9..f6d8456 100644 --- a/user_documentation/rst/deployment.rst +++ b/user_documentation/rst/deployment.rst @@ -1,3 +1,5 @@ +.. _deployment: + Deployment ********** @@ -11,13 +13,16 @@ We recommend to perform the installation remotely as all your configuration file Prerequisites ============= -For the MiCADO master: +For cloud interfaces supported by MiCADO: -* Ubuntu 16.04 +* EC2 (tested on Amazon and OpenNebula) +* Nova (tested on OpenStack) +* CloudSigma +* CloudBroker -For the MiCADO worker nodes: +For the MiCADO master: -* A MiCADO-supported cloud: OpenStack, OpenNebula, AmazonEC2 or CloudSigma +* Ubuntu 16.04 For the host where the Ansible playbook is executed (differs depending on local or remote): @@ -40,7 +45,7 @@ To install Ansible on Ubuntu 16.04, use these commands: sudo apt-get update sudo apt-get install ansible -To install Ansible on other operation system follow the `official installation guide `__. +To install Ansible on other operation systems follow the `official installation guide `__. Git --- @@ -51,7 +56,7 @@ To install Git on Ubuntu, use this command: sudo apt-get install git-all -To install Git on other operating system follow the `official installation guide `__. +To install Git on other operating systems follow the `official installation guide `__. Installation ============ @@ -67,40 +72,40 @@ Step 1: Download the ansible playbook. cd ansible-micado git checkout v0.6.0 -Step 2: Specify credential for instantiating MiCADO workers. ------------------------------------------------------------- +Step 2: Specify cloud credential for instantiating MiCADO workers. +------------------------------------------------------------------ -MiCADO master will use this credential to start/stop VM instances (MiCADO workers) to host the application and to realize scaling. Credentials here should belong to the same cloud as where MiCADO master is running. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the credential in a file, called credentials.yml. Please, do not modify the structure of the template! +MiCADO master will use this credential to start/stop VM instances (MiCADO workers) to host the application and to realize scaling. Credentials here should belong to the same cloud as where MiCADO master is running. We recommend making a copy of our predefined template and edit it. MiCADO expects the credential in a file, called credentials-cloud-api.yml before deployment. Please, do not modify the structure of the template! :: - cp sample-credentials.yml credentials.yml - vi credentials.yml + cp sample-credentials-cloud-api.yml credentials-cloud-api.yml + edit credentials-cloud-api.yml -Edit credentials.yml to add cloud credentials. You will find predefined sections in the template for each cloud interface type MiCADO supports. Fill only the section belonging to your target cloud. +Edit credentials-cloud-api.yml to add cloud credentials. You will find predefined sections in the template for each cloud interface type MiCADO supports. Fill only the section belonging to your target cloud. Optionally you can use the `Ansible Vault `_ mechanism to keep the credential data in an encrypted format. To achieve this, create the above file using Vault with the command :: - ansible-vault create credentials.yml + ansible-vault create credentials-cloud-api.yml -This will launch *vi* or the editor defined in the ``$EDITOR`` environment variable to make changes to the file. If you wish to make any changes to the previously encrypted file, you can use the command +This will launch the editor defined in the ``$EDITOR`` environment variable to make changes to the file. If you wish to make any changes to the previously encrypted file, you can use the command :: - ansible-vault edit credentials.yml + ansible-vault edit credentials-cloud-api.yml -Step 3a: Specify security settings and credentials. --------------------------------------------------------------- +Step 3a: Specify security settings and credentials to access MiCADO. +-------------------------------------------------------------------- -MiCADO master will use these security-related settings and credentials during provisioning. +MiCADO master will use these security-related settings and credentials to authenticate its users for accessing the REST API and Dashboard. :: - cp sample-security-cred.yml security-cred.yml - vi security-cred.yml + cp sample-credentials-micado.yml credentials-micado.yml + edit credentials-micado.yml Specify the provisioning method for the x509 keypair used for TLS encryption of the management interface in the ``tls`` subtree: @@ -112,34 +117,40 @@ Specify the default username and password for the administrative we user in the Optionally you may use the Ansible Vault mechanism as described in Step 2 to protect the confidentiality and integrity of this file as well. -Step 3b: (Optional) Specify details of your private Docker repository. ----------------------------------------------------------------------- +Step 3b: (Optional) Specify credentials to use private Docker registries. +------------------------------------------------------------------------- -Set the Docker login credentials of your private Docker registries in which your personal containers are stored. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the docker registry details in a file, called docker-cred.yml. Please, do not modify the structure of the template! +Set the Docker login credentials of your private Docker registry in which your private containers are stored. We recommend making a copy of our predefined template and edit it. MiCADO expects the docker registry credentials in a file, called credentials-docker-registry.yml. Please, do not modify the structure of the template! :: - cp sample-docker-cred.yml docker-cred.yml - vi docker-cred.yml + cp sample-credentials-docker-registry.yml credentials-docker-registry.yml + edit credentials-docker-registry.yml -Edit docker-cred.yml and add username, password, and repository url. To login to the default docker_hub, leave DOCKER_REPO as is (a blank string). +Edit credentials-docker-registry.yml and add username, password, and registry url. To login to the default docker_hub, leave DOCKER_REPO as is (a blank string). Optionally you may use the Ansible Vault mechanism as described in Step 2 to protect the confidentiality and integrity of this file as well. Step 4: Launch an empty cloud VM instance for MiCADO master. ------------------------------------------------------------ -This new VM will host the MiCADO master core services. Use any of aws, ec2, nova, etc command-line tools or web interface of your target cloud to launch a new VM. We recommend a VM with 2 cores, 4GB RAM, 20GB disk. Make sure you can ssh to it (password-free i.e. ssh public key is deployed) and your user is able to sudo (to install MiCADO as root). Store its IP address which will be referred as ``IP`` in the following steps. The following ports should be open on the virtual machine: +This new VM will host the MiCADO core services. + +**a)** Default port number for MiCADO service is ``443``. Optionally, you can modify the port number stored by the variable called ``web_listening_port`` defined in the ansible playbook file called ``micado-master.yml``. + +**b)** Configure a cloud firewall settings which opens the following ports on the MiCADO master virtual machine: :: TCP: 22,2377,7946,8300,8301,8302,8500,8600,[web_listening_port] UDP: 4789,7946,8301,8302,8600 -**NOTE:** ``web_listening_port`` is defined in the ansible inventory file called ``hosts`` and defaults to ``443`` +**NOTE:** replace ``[web_listening_port]`` with the actual value specified in Step 4a. **NOTE:** MiCADO master has built-in firewall, therefore you can leave all ports open at cloud level. +**c)** Finally, launch the virtual machine with the proper settings (capacity, ssh keys, firewall): use any of aws, ec2, nova, etc command-line tools or web interface of your target cloud to launch a new VM. We recommend a VM with 2 cores, 4GB RAM, 20GB disk. Make sure you can ssh to it (password-free i.e. ssh public key is deployed) and your user is able to sudo (to install MiCADO as root). Store its IP address which will be referred as ``IP`` in the following steps. + Step 5: Customize the inventory file for the MiCADO master. ----------------------------------------------------------- @@ -148,7 +159,7 @@ We recommend making a copy of our predefined template and edit it. Use the templ :: cp sample-hosts hosts - vi hosts + edit hosts Edit the ``hosts`` file to set ansible variables for MiCADO master machine. Update the following parameters: @@ -158,8 +169,6 @@ Edit the ``hosts`` file to set ansible variables for MiCADO master machine. Upda * **ansible_become**: specifies if account change is needed to become root, defaults to "True" * **ansible_become_method**: specifies which command to use to become superuser, defaults to "sudo" * **ansible_python_interpreter**: specifies the interpreter to be used for ansible on the target host, defaults to "/usr/bin/python3" -* **docker_cred_path**: sets the path of file storing the credentials for private docker registries, defaults to "./docker-cred.yml" -* **web_listening_port**: specifies the listening port of the management interface including the MiCADO dashboard and the REST interface, defaults to the default HTTPS port (443/TCP) Please, revise all the parameters, however in most cases the default values are correct. @@ -190,8 +199,8 @@ Check the logs You can SSH into MiCADO master and check the logs at any point after MiCADO is succesfully deployed. All logs are kept under ``/var/log/micado`` and are organised by components. Scaling decisions, for example, can be inspected under ``/var/log/micado/policykeeper`` -Accessing user service -====================== +Accessing user-defined service +============================== In case your application contains container exposing a service, you have two alternatives to access its endpoint: From d9c4730d78745b708c6844021e6159246d6d829f Mon Sep 17 00:00:00 2001 From: Jozsef Kovacs Date: Fri, 12 Oct 2018 10:42:56 +0000 Subject: [PATCH 6/7] explain IP:HOST by referring to step4 --- user_documentation/rst/dashboard.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/user_documentation/rst/dashboard.rst b/user_documentation/rst/dashboard.rst index 387c3d6..abb85f5 100644 --- a/user_documentation/rst/dashboard.rst +++ b/user_documentation/rst/dashboard.rst @@ -3,7 +3,11 @@ Dashboard ********* -MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[port]``. This port number is configured during initial setup of the MiCADO master *(Docs > Deployment > Installation > Step 5 > web_listening_port)* and defaults to port 443. +MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[PORT]``, where + +* [IP] is the ip address of MiCADO master, the virtual machine you have launched in Step 4 of :ref:`deployment` + +* [PORT] is the port number configured during Step 4 of :ref:`deployment`, its value is held by the ``web_listening_port`` variable specified in the ``micado-master.yml`` ansible file. The following webpages are currently exposed: From ecbc41847594ad9798f8358ed44aa0ba45126171 Mon Sep 17 00:00:00 2001 From: Attila Farkas Date: Mon, 15 Oct 2018 14:39:48 +0200 Subject: [PATCH 7/7] MiCADO 0.6.1 release --- user_documentation/rst/deployment.rst | 2 +- user_documentation/rst/release_notes.rst | 17 ++++++++++++++++- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/user_documentation/rst/deployment.rst b/user_documentation/rst/deployment.rst index f6d8456..4379141 100644 --- a/user_documentation/rst/deployment.rst +++ b/user_documentation/rst/deployment.rst @@ -70,7 +70,7 @@ Step 1: Download the ansible playbook. git clone https://github.com/micado-scale/ansible-micado.git ansible-micado cd ansible-micado - git checkout v0.6.0 + git checkout v0.6.1 Step 2: Specify cloud credential for instantiating MiCADO workers. ------------------------------------------------------------------ diff --git a/user_documentation/rst/release_notes.rst b/user_documentation/rst/release_notes.rst index a753c82..17742d8 100644 --- a/user_documentation/rst/release_notes.rst +++ b/user_documentation/rst/release_notes.rst @@ -1,6 +1,21 @@ Release Notes ************* +**v0.6.1 (15 Oct 2018)** + +- enable VM-only deployments +- add support for special characters in SSL credentials +- fix missing vm instance number reset at undeployment +- add option to disable auto-updates on worker nodes +- modify default launch-order of TOSCA adaptors +- add cloud-specific TOSCA templates and improve helper scripts for stressng +- flatten CPU scaling policies +- improve virtual machine build time +- fix Zorp starting dependency +- fix Docker login timing issue +- remove unnecessary port from docker compose file +- enable Prometheus DB export + **v0.6.0 (10 Sept 2018)** - introduce documentation repository and host its content at http://micado-scale.readthedocs.io @@ -13,7 +28,7 @@ Release Notes - fix issue with worker node deployment in EC2 clouds - fix issue with user-defined Docker networks in OpenStack clouds - make Submitter response message structure uniform -- add 'nodes' and 'services' query methods to REST API +- add 'nodes' and 'services' query methods to REST API - improve 'stressng' and 'cqueue' test helper scripts - add more compose properties to custom TOSCA definition - fix floating ip issues in the Dashboard component