Occopus crash when quota is exceeded

[fsattila]

Prerequisites

• Can you reproduce the problem?
• Did you perform a cursory search to see if your bug or enhancement is already reported?

Description

Occopus crash when the user's cloud quota is exceeded. The issue occured on OpenNebula with EC2 interface.

Steps to Reproduce

1. Create MiCADO-master on OpenNebula
2. Set the maximum number of worker nodes to exceed the cloud quota
3. Deploy the application

Expected behavior: Occopus logs that the cloud quota is exceeded.

Actual behavior: Occopus REST service crashed

Versions

MiCADO 0.5.0

Additional context

Occopus logs:

...
** 2018-08-08 07:12:45,203	Start maintaining the infrastructure micado_worker_infra
** 2018-08-08 07:12:45,426	Health checking for node 'worker'/'ebdf1e4b-8072-452a-bb63-08c8d668b464'
** 2018-08-08 07:12:45,552	  Checking node reachability (ebdf1e4b-8072-452a-bb63-08c8d668b464):
** 2018-08-08 07:12:45,588	    192.168.154.95 => ready
** 2018-08-08 07:12:45,590	Health checking result: ready
** 2018-08-08 07:12:45,741	Finished maintaining the infrastructure micado_worker_infra
** 2018-08-08 07:12:46,169	172.19.0.11 - - [08/Aug/2018 07:12:46] "POST /infrastructures/micado_worker_infra/scaleto/worker/2 HTTP/1.1" 200 -
** 2018-08-08 07:12:55,757	Start maintaining the infrastructure micado_worker_infra
** 2018-08-08 07:12:55,934	Health checking for node 'worker'/'ebdf1e4b-8072-452a-bb63-08c8d668b464'
** 2018-08-08 07:12:56,042	  Checking node reachability (ebdf1e4b-8072-452a-bb63-08c8d668b464):
** 2018-08-08 07:12:56,071	    192.168.154.95 => ready
** 2018-08-08 07:12:56,072	Health checking result: ready
** 2018-08-08 07:12:56,301	Creating node 'worker'/'9f60f4d2-7570-4c99-8c8d-93db38a601d6'
** 2018-08-08 07:12:56,609	Error while creating node '9f60f4d2-7570-4c99-8c8d-93db38a601d6':'EC2ResponseError: 403 Forbidden\n<Response><RequestId/><Errors><Error><Code>UnsupportedOperation</Code><Message>[VirtualMachineAllocate] User [4] : user [4] limit of 8 reached for VMS quota in VM.</Message></Error></Errors></Response>'
** 2018-08-08 07:12:56,767	Critical error occured:
Traceback (most recent call last):
  File "/occopus.src/enactor/occo/enactor/__init__.py", line 351, in make_a_pass
    self.enact_delta(delta)
  File "/occopus.src/enactor/occo/enactor/__init__.py", line 331, in enact_delta
    self.ip.push_instructions(self.infra_id, instruction_list)
  File "/occopus.src/infra-processor/occo/infraprocessor/__init__.py", line 80, in push_instructions
    rv = self.strategy.perform(self, instruction_list)
  File "/occopus.src/infra-processor/occo/infraprocessor/strategy.py", line 72, in perform
    return self._perform(infraprocessor, instruction_list)
  File "/occopus.src/infra-processor/occo/infraprocessor/strategy.py", line 293, in _perform
    self._process_one_result()
  File "/occopus.src/infra-processor/occo/infraprocessor/strategy.py", line 269, in _process_one_result
    error['value'] = yaml.load(error['value'], Loader=yaml.Loader)
  File "/usr/local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 86, in load
    return loader.get_single_data()
  ...
  File "/usr/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 177, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 84, in construct_object
    "found unconstructable recursive node", node.start_mark)
ConstructorError: found unconstructable recursive node
  in "<byte string>", line 82, column 3:
    - &id002 !!python/object/apply:bot ...
      ^ (line: 82)
** 2018-08-08 07:12:56,773	Unexpected error:
** 2018-08-08 07:12:56,776	found unconstructable recursive node
  in "<byte string>", line 82, column 3:
    - &id002 !!python/object/apply:bot ...
      ^ (line: 82)
** 2018-08-08 07:13:01,405	172.19.0.11 - - [08/Aug/2018 07:13:01] "POST /infrastructures/micado_worker_infra/scaleto/worker/2 HTTP/1.1" 200 -
** 2018-08-08 07:13:16,648	172.19.0.11 - - [08/Aug/2018 07:13:16] "POST /infrastructures/micado_worker_infra/scaleto/worker/2 HTTP/1.1" 200 -