ReadTimeout when attempting to remove ASD via GUI

[ovstmoens]

Problem description

Tried to remove two asds of which the mountpoint was full. Got ReadTimeOut message in the GUI

Logs:

workers:

2017-11-09 10:30:40 89600 +0100 - pocops-voldrv01 - 1684/140547671979776 - celery/celery.worker.job - 77273 - ERROR - Task albanode.reset_asd[3238c3c5-3691-4439-93e9-77c9da23e21e] raised unexpected: ReadTimeout(
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanode.py", line 368, in reset_asd
    expected_safety=expected_safety)
  File "/usr/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 439, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 461, in new_function
    output = f(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanode.py", line 322, in remove_asd
    for device_info in node.client.get_disks().itervalues():
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 95, in _call
    response = method(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: None: None

asd-manager:

2017-11-09 10:32:25 16800 +0100 - pocops-voldrv01 - 6527/140339688146688 - extensions/ovs_extensions.generic.sshclient - 9683 - ERROR - Command "'rm' '/mnt/alba-asd/bWpBWNePj5xGLpsI/1510219945.12'" failed with o
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/ssl.py", line 743, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 709, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('10.100.200.10', 54884)
----------------------------------------
2017-11-09 10:32:25 48700 +0100 - pocops-voldrv01 - 6527/140339679753984 - asd-manager/api - 9686 - INFO - GET /asds - Entering
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/ssl.py", line 743, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 709, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/ssl.py", line 743, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 709, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/ssl.py", line 743, in sendall

2017-11-09 10:27:18 40200 +0100 - pocops-voldrv01 - 6527/140339696801536 - asd-manager/api - 9639 - INFO - GET /disks - Entering
2017-11-09 10:27:18 49500 +0100 - pocops-voldrv01 - 6527/140339696801536 - extensions/ovs_extensions.generic.sshclient - 9640 - ERROR - Command "'touch' '/mnt/alba-asd/hlGeOtrpeOmJ0Nps/1510219638.49'" failed wit
h output "" and error "touch: cannot touch '/mnt/alba-asd/hlGeOtrpeOmJ0Nps/1510219638.49': No space left on device"
2017-11-09 10:27:18 60300 +0100 - pocops-voldrv01 - 6527/140339722241792 - asd-manager/api - 9642 - INFO - GET /service_status/<name> - Entering with [] {"name": "alba-maintenance_flash-10-De1PnrZRDk32l6Zp"}
2017-11-09 10:27:18 69400 +0100 - pocops-voldrv01 - 6527/140339696801536 - asd-manager/api - 9645 - INFO - POST /disks/<disk_id>/asds/<asd_id>/delete - Entering with [] {"asd_id": "egkpwc7OZNb9d4TGqGnMU0ZkRjMdma
t8", "disk_id": "ata-DENRSTE251M45-0100_OCZ-D8Z08V39CS9JAL02"}

[wimpers]

@ovstmoens is there a workaround? Did you eventually manage to remove the ASDs?

[ovstmoens]

@wimpers The workaround we used was to remove the directory first via CLI and then remove the ASD in the GUI.

[sjovdnbos]

This ticket seems to be no longer relevant in the current develop code: changes have been made in the remove_asd (now remove_osd)

• The call node.client.get_disks is no longer used in the remove_osd code
• Concerning the touch command in the asd-manager which caused the timeout through an error raise: this code is not executed in the current develop in GET calls and should therefore not occur, even with full disks.
  @JeffreyDevloo , any thoughts?

[JeffreyDevloo]

This bug is most likely still present.