Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Too many open files #599

Open
openvstorage-ci opened this issue Jun 19, 2018 · 3 comments
Open

Too many open files #599

openvstorage-ci opened this issue Jun 19, 2018 · 3 comments
Labels
type_feature
Milestone
Backlog

Comments

@openvstorage-ci
Copy link
Contributor

From @jeroenmaelbrancke on February 13, 2018 8:35

Can you add a proper error state for the 'too many open files' error and let us know which node suffers from this?

root@SV4SRV0021:~# ovs healthcheck alba backend-test
[INFO] Storagerouter Id: KjVk832AslAaVwZ4
[INFO] Environment Os: Ubuntu 16.04 xenial
[INFO] Hostname: SV4SRV0021
[INFO] Cluster Id: JoQXf3AYDGVMBZtb
[INFO] Storagerouter Type: MASTER
[INFO] Starting OpenvStorage Healthcheck version 3.4.5-1
[INFO] ======================
[INFO] Checking available ALBA backends.
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

Exception in thread Thread-38:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

No handlers could be found for logger "ovs_extensions.generic.volatilemutex"
Exception in thread Thread-57:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

Exception in thread Thread-83:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

[SUCCESS] We found 5 backend(s)!
[INFO] Checking the ALBA ASDs.
[INFO] Checking separate ASDs for backend sv4-hddbackend03:
[SUCCESS] ASD test with DISK_ID L7jO0m3ROyabs9OMyaWvJkJV0hPjF8XV succeeded!
[SUCCESS] ASD test with DISK_ID zJ0TTqWbMWs1xV2hF3IdG1m570JXP3kQ succeeded!
[SUCCESS] ASD test with DISK_ID 15XUfbDfB9Q4meTJWEi87QotX7KRnyJy succeeded!
[SUCCESS] ASD test with DISK_ID keV4fmSxSRjzr8ODfRjezyfOyqDH48qO succeeded!
[SUCCESS] ASD test with DISK_ID sGugAUYqDUPo0c0kZbJszxsgWCFapKvn succeeded!
[SUCCESS] ASD test with DISK_ID YC904ZjtBpAsVk1TFVeRQID5r33WbbgG succeeded!
[WARNING] ASD test with DISK_ID PWaZTXaCoRXIPjaC4j1D5OzxoK4i2dvR failed because: nodedown
[WARNING] ASD test with DISK_ID ePqC4Srjy27angOiNmGn3jDuKEe7fpuP failed because: nodedown
[WARNING] ASD test with DISK_ID r8KuO7zE6jhglz04Q9PYzewt6ILIi0kp failed because: nodedown
[SUCCESS] ASD test with DISK_ID GEXsOxgzBoNPZZ0wqqVNRSEntnO4HeT3 succeeded!
[SUCCESS] ASD test with DISK_ID 3r5TcVBXN0DNjupiD4MVL8XZofvHdF5v succeeded!
[SUCCESS] ASD test with DISK_ID t2GMTsZAhXFtFkKydXCQ9iq9dV7gpfhQ succeeded!
[SUCCESS] ASD test with DISK_ID p4JO1OfBjfQ08ViGqhLBW1GNILln92AW succeeded!
[SUCCESS] ASD test with DISK_ID bSioAHQ5BreyKml1oE7Y0S3LmD14CRIE succeeded!
[SUCCESS] ASD test with DISK_ID j4ivKa7oFJXHLZif7BJeK5bZeFQ3FiJe succeeded!
[SUCCESS] ASD test with DISK_ID 6RXyP1kI9rydffOFMzCMIfz7XgbqoZw2 succeeded!
[SUCCESS] ASD test with DISK_ID pOAv0WFxleQHWiISTD14YZdkiZFqbuIf succeeded!
[SUCCESS] ASD test with DISK_ID gl2dVxGWJv7uSp5fQouYSFTjDb347OTC succeeded!
[SUCCESS] ASD test with DISK_ID o4KJ3iuH3RRLUP1FXBLhbskGr5N7rRf8 succeeded!
[SUCCESS] ASD test with DISK_ID EWLF7kdcdW6r4MnLHCNhbZUYSkYhNWFr succeeded!
[SUCCESS] ASD test with DISK_ID 9jBcEGVBRMoIuV6SkhS2nCOCfjOp2tMY succeeded!
[SUCCESS] ASD test with DISK_ID cWQc77Atk65plNxFfKGFD76FRUmvI4gM succeeded!
[SUCCESS] ASD test with DISK_ID uX6FwieXQU1CKQtSQYvIbBlccuftaewt succeeded!
[SUCCESS] ASD test with DISK_ID RMGp2z74qkt4QzzBqK9bZuM2lwYgckfi succeeded!
[SUCCESS] ASD test with DISK_ID cTIwN0SViK2epSHiI5JqbeDQVSY298nM succeeded!
[SUCCESS] ASD test with DISK_ID 6HBGz0RmO1N9ZRueLlnXJPvTV8WR5oEu succeeded!
[SUCCESS] ASD test with DISK_ID 95dAjfjUy99Nq4CTLuLYZyOjSgNoHWNQ succeeded!
[SUCCESS] ASD test with DISK_ID WcHHnOI1rw6DfajjGZZaMBPJiriojtWk succeeded!
[WARNING] ASD test with DISK_ID Xpo5IzC1q0xmHWml3OyQvGml6iXOaxJF failed because: unknown
[WARNING] ASD test with DISK_ID i1nE6N1j5qPBkfMMROqGemFd9F5pNveU failed because: unknown
[WARNING] ASD test with DISK_ID FeniKSyezD0pqgSnHKPtoQyGxVIL9KrP failed because: unknown
[WARNING] ASD test with DISK_ID kN510d0JXbyL9uXJLtpQ403ScigJ8L7D failed because: unknown
[WARNING] ASD test with DISK_ID UH5dEOW5DwgPKDn4Mm785J58JKuypeek failed because: unknown
[WARNING] ASD test with DISK_ID XWxgvUk9vaZ1gDiQzBzV45FG0Gyldlta failed because: unknown
[WARNING] Alba backend sv4-hddbackend03 should be available for VPool use with 25 asds, but there are 9 defective asds: PWaZTXaCoRXIPjaC4j1D5OzxoK4i2dvR, ePqC4Srjy27angOiNmGn3jDuKEe7fpuP, r8KuO7zE6jhglz04Q9PYzewt6ILIi0kp, Xpo5IzC1q0xmHWml3OyQvGml6iXOaxJF, i1nE6N1j5qPBkfMMROqGemFd9F5pNveU, FeniKSyezD0pqgSnHKPtoQyGxVIL9KrP, kN510d0JXbyL9uXJLtpQ403ScigJ8L7D, UH5dEOW5DwgPKDn4Mm785J58JKuypeek, XWxgvUk9vaZ1gDiQzBzV45FG0Gyldlta
[INFO] Checking separate ASDs for backend sv4-hddbackend02:
[SUCCESS] ASD test with DISK_ID Jobp5M99EvZ02gYIRGJ0nh6DPsQyFujA succeeded!
[SUCCESS] ASD test with DISK_ID tL0IJ5QOMTINnaVvfAJsGuP8tTtMmdjM succeeded!
[SUCCESS] ASD test with DISK_ID wCJ2EmJiRM9cIaAiD0PWu7ReABYiCnW2 succeeded!
[SUCCESS] ASD test with DISK_ID ugCH4EB4aGML5uViVrFOG7rtrSsLHBlf succeeded!
[SUCCESS] ASD test with DISK_ID Fdh8R8GtpQNlqV3Wq0CdWl7oNyNM3Wjt succeeded!
[SUCCESS] ASD test with DISK_ID VcrrKUhaLOOGI7bTbovVnzd1jccqM7Vr succeeded!
[WARNING] ASD test with DISK_ID JSlAW8XvG2GDzGGZNgXMkO8cdYooKEZd failed because: nodedown
[WARNING] ASD test with DISK_ID rvl7pYCsJxi0kTCgVRQKsx9zZ1w0P540 failed because: nodedown
[WARNING] ASD test with DISK_ID S3KkojHnGMq8mp6Rz8LtFY9mi6jGqFFg failed because: nodedown
[SUCCESS] ASD test with DISK_ID sr4JcRvu9mVY6qWVrXE4wnDXiupdxYMt succeeded!
[SUCCESS] ASD test with DISK_ID k66UmwP9DpFnTM3GKvgbO85UsnnzU72c succeeded!
[SUCCESS] ASD test with DISK_ID P8kh8WhTgVydWZOq97nLxJfMeA8j06WJ succeeded!
[SUCCESS] ASD test with DISK_ID F4Pdx4nQOBqM67DeBJh9NzUmBmdeqgxt succeeded!
[SUCCESS] ASD test with DISK_ID oMa65aKlADJw8rCsq5y8MXWVGpIn24mB succeeded!
[SUCCESS] ASD test with DISK_ID DWf27znUjm0hJRfPQeP2lP2m9dIjKgi1 succeeded!
[SUCCESS] ASD test with DISK_ID UngscLez1PW2jqonmgTprtdbN6mwv00N succeeded!
[SUCCESS] ASD test with DISK_ID UAKvS4u2F9yrADfZW1PtxHL9w2DireWf succeeded!
[SUCCESS] ASD test with DISK_ID fRmvSancb471LzIqkkfc9wYv8eQgq5zl succeeded!
[SUCCESS] ASD test with DISK_ID kmc0eRkyANrSkBcsldNS0N4Q7oDIuaXt succeeded!
[SUCCESS] ASD test with DISK_ID tbhhmO7Tj2PKkRIM4Hg2HnbRRkdRrfJU succeeded!
[SUCCESS] ASD test with DISK_ID AbKKEFAHYTnyvnmQvl1dJRlUsUv6I1JB succeeded!
[SUCCESS] ASD test with DISK_ID Dvj3STv8mVzVPohyR80VOI0BT2uhhRws succeeded!
[SUCCESS] ASD test with DISK_ID CIkQ63CFyKWed7DsXbY9IKSMiqLemnKV succeeded!
[SUCCESS] ASD test with DISK_ID uoYCC5sBzsSj4pPnWaLPziWRFFjovo48 succeeded!
[SUCCESS] ASD test with DISK_ID Bh0Wp4v22is1as4rMvAFiwhMsdTaS8nb succeeded!
[SUCCESS] ASD test with DISK_ID DOyQspBcA31UeBAuh27Wimzpe26kKrF6 succeeded!
[SUCCESS] ASD test with DISK_ID 3LMEHBoeod4cMmVCSGOBiodGt61cLlYy succeeded!
[SUCCESS] ASD test with DISK_ID nwX6ux8UJhSevMaerBWnBnjMAdEZRCqm succeeded!
[SUCCESS] ASD test with DISK_ID VAjRAOU5Z6j5ifYQcC2YjOCk4NMnoU4n succeeded!
[WARNING] ASD test with DISK_ID JpGu4NkSqefyN0Wjzy2IPTw27pk0vOqT failed because: unknown
[WARNING] ASD test with DISK_ID 7DhygVpgHrJK0q6X9xuB9BndjcoMDxL8 failed because: unknown
[WARNING] ASD test with DISK_ID Fo6KLMCQQoFAzHF0u1bcOjFRPY6srysa failed because: unknown
[WARNING] ASD test with DISK_ID H8OcJwwlets4kSzkN1xOsLultK41W4QR failed because: unknown
[WARNING] ASD test with DISK_ID YzSMOncTbiixJ9WIDuFccfzKxfZKNxSf failed because: unknown
[WARNING] ASD test with DISK_ID IeWoGB5FGuCjwgjXcjwkaPRc8TSkkJ37 failed because: unknown
[WARNING] Alba backend sv4-hddbackend02 should be available for VPool use with 26 asds, but there are 9 defective asds: JSlAW8XvG2GDzGGZNgXMkO8cdYooKEZd, rvl7pYCsJxi0kTCgVRQKsx9zZ1w0P540, S3KkojHnGMq8mp6Rz8LtFY9mi6jGqFFg, JpGu4NkSqefyN0Wjzy2IPTw27pk0vOqT, 7DhygVpgHrJK0q6X9xuB9BndjcoMDxL8, Fo6KLMCQQoFAzHF0u1bcOjFRPY6srysa, H8OcJwwlets4kSzkN1xOsLultK41W4QR, YzSMOncTbiixJ9WIDuFccfzKxfZKNxSf, IeWoGB5FGuCjwgjXcjwkaPRc8TSkkJ37
[INFO] Checking separate ASDs for backend sv4-hddbackend01:
[SUCCESS] ASD test with DISK_ID oUVIVpt7Xck5hTTdMrECOc199lX6T7dP succeeded!
[SUCCESS] ASD test with DISK_ID EXDm9aMGsu456yCBvqvM3yi3MZBGRUWm succeeded!
[SUCCESS] ASD test with DISK_ID bIr4nGYn7joK26sLlide7I67Uuzx53R4 succeeded!
[SUCCESS] ASD test with DISK_ID MNc8m5yk1wv2WWeQ3O7YxNJdGiK9gS17 succeeded!
[SUCCESS] ASD test with DISK_ID A43W8IeXhxEYFU6sU5xGkCV8Q7jM5OH4 succeeded!
[SUCCESS] ASD test with DISK_ID syns6L4nau6TIidH39vQ38if0Ssq4Ugh succeeded!
[SUCCESS] ASD test with DISK_ID eO5nIPuL2Q1026oaPebLXkTgpqBqLLMg succeeded!
[SUCCESS] ASD test with DISK_ID 7hOPrIhLcpaOKmjauoYYMUNobM42lvgA succeeded!
[SUCCESS] ASD test with DISK_ID KKpm8QC9nuk5jgyw30kcnniftIYkKuXd succeeded!
[SUCCESS] ASD test with DISK_ID 0nE1fzBt3DAR5TB3dPekrZT9eIHVSkfE succeeded!
[SUCCESS] ASD test with DISK_ID 5Lkf1PDkgBuj3DXnEq7tDs87Vas2s5qB succeeded!
[SUCCESS] ASD test with DISK_ID g6SVUg51VXo0MklC72GJ18s8YAzag5kT succeeded!
[SUCCESS] ASD test with DISK_ID 2I5Ky2mtRWKcIA5VV1Mh2DMiQhm673BJ succeeded!
[SUCCESS] ASD test with DISK_ID HrAhMyug1VlMfn1VyDUW1R7u6oTaN2po succeeded!
[SUCCESS] ASD test with DISK_ID zFxHBj3sGSy73NZhDsj0KFDmRNewrrab succeeded!
[SUCCESS] ASD test with DISK_ID SFq8WMRQvhyMXykGY1LLTmaFryjUcRzR succeeded!
[SUCCESS] ASD test with DISK_ID zAzGE5kiL30zAYXZGzuangO11iG7PKUE succeeded!
[SUCCESS] ASD test with DISK_ID yuhM3wRicK3w501oOYzMyLjWyLPLkMU5 succeeded!
[SUCCESS] ASD test with DISK_ID wG5s2v4xKcgchFmQmLuGDD7HM0gB5xVW succeeded!
[SUCCESS] ASD test with DISK_ID Zq3G9pCjc9EY7j0dK7g1w9XLBNAtvo52 succeeded!
[WARNING] ASD test with DISK_ID tW2z3zvWpVH5abyj1stnEkv33j23kf6n failed because: nodedown
[WARNING] ASD test with DISK_ID 3FhkQEjZmc5HOR77c1JLUguZWynlbOIn failed because: nodedown
[WARNING] ASD test with DISK_ID 3t1Q5aYAIvmqQDOjjriXIBxYMgFd98SR failed because: nodedown
[WARNING] ASD test with DISK_ID StnMHJf4vq2HhESO1qgvcW63vR9ZLbJl failed because: nodedown
[SUCCESS] ASD test with DISK_ID Bd4HOyIIA53UJAU7kDYBylrzLD4QRBUd succeeded!
[SUCCESS] ASD test with DISK_ID mq7eDIz9FT9WyXeK7wLhk9MyKdmK9w9e succeeded!
[SUCCESS] ASD test with DISK_ID IrtKR3cdyLOwX8r9bFOA279fMj1SHj9i succeeded!
[SUCCESS] ASD test with DISK_ID 122KNhftcOJLxK2jnRZ0wC6JJG13u86Y succeeded!
[SUCCESS] ASD test with DISK_ID Q6SduC1xOiUQ1ZSq1nqIMrpKwA4Wbn3y succeeded!
[SUCCESS] ASD test with DISK_ID jBuVbfYHzLh4RUN4yK3EqVPK3QyYF9AR succeeded!
[SUCCESS] ASD test with DISK_ID Schqo9pHu7vezRkOBV99e6k9GyRKsc04 succeeded!
[SUCCESS] ASD test with DISK_ID BVPpG8KvJqwGj2VNSZvYbLSu0kIMCiZo succeeded!
[SUCCESS] ASD test with DISK_ID 6ugtFQSabKfy90eTCEwATfed6FsmaZsR succeeded!
[SUCCESS] ASD test with DISK_ID DNTpdrwpn8P3f2IxUnsgXDm2c5PtcMs8 succeeded!
[SUCCESS] ASD test with DISK_ID y63aqEtTDjlRsE6uolVXOuJLebtpHv11 succeeded!
[SUCCESS] ASD test with DISK_ID ty5VAkBu5odEMZhw4aoYLLzcRI3vuNXs succeeded!
[SUCCESS] ASD test with DISK_ID bNVeQa9yvwkuen9gsvZp2PRl0OoBJyEW succeeded!
[SUCCESS] ASD test with DISK_ID 1T446PGGchlLArRCepG4DuSOuKxe4FSw succeeded!
[SUCCESS] ASD test with DISK_ID Pb8R60dXKPoowovY6wWJ9Sw75sdV4tLA succeeded!
[SUCCESS] ASD test with DISK_ID f3pW4W8vMiz2AxW7nMAyuFyBIRihOQec succeeded!
[SUCCESS] ASD test with DISK_ID lDictyEmV4kUxnxGUFsuQd7568JCUj5p succeeded!
[SUCCESS] ASD test with DISK_ID VuNreXP10aSpdRKurVorpzDUYO12TJ9F succeeded!
[SUCCESS] ASD test with DISK_ID f3zfNBtdo0LLiLrBYSKYZ2IJHOHqZkpt succeeded!
[SUCCESS] ASD test with DISK_ID fSH77yWgKMzhzH4M8h2s4yBbmxIfNiUA succeeded!
[SUCCESS] ASD test with DISK_ID RCeDqL5APlCKg2etiUygVkPQnswMSL20 succeeded!
[SUCCESS] ASD test with DISK_ID aPujLp1bx7R6nK3SOjGh8Q1KKEyO21US succeeded!
[SUCCESS] ASD test with DISK_ID He2LWyGkpiCYaqJzBpxVBFdDGt5vlK0J succeeded!
[SUCCESS] ASD test with DISK_ID wtIrhuY2GbfSwTvmuGOLqiQ9LruD5q0a succeeded!
[SUCCESS] ASD test with DISK_ID SphP5xLHqeIjfHP2exWvcKhgtxOLAcLi succeeded!
[SUCCESS] ASD test with DISK_ID 6vJ6fTJkunoakyvHKlzpYw2X47Zs7Lfo succeeded!
[SUCCESS] ASD test with DISK_ID EwnL3vhRmhs4g6GexNEV6N4Un6Mm3w7A succeeded!
[SUCCESS] ASD test with DISK_ID 6hFS3rFEr4l1h9pMA7WrNchZdZaS4qs5 succeeded!
[SUCCESS] ASD test with DISK_ID F5qGOMdcH1DDxBgkCQxdQd1YLNAy8QoE succeeded!
[SUCCESS] ASD test with DISK_ID VEoza14ASGBPuHURkpqkevryIJ4IWOfj succeeded!
[SUCCESS] ASD test with DISK_ID 0ozDPSMfCCpqF8Az0NA5m709pF9Ve3c2 succeeded!
[SUCCESS] ASD test with DISK_ID bOLrYNo4Vq7OseSSMSSkGwSfKw5Tn5bX succeeded!
[SUCCESS] ASD test with DISK_ID qphQ9dyq9qxkWZ2HYxSSOo8qTxHb9b5b succeeded!
[SUCCESS] ASD test with DISK_ID NtDi5ULVbSR9XvyJBtk2HJ2MIFLwaw3d succeeded!
[SUCCESS] ASD test with DISK_ID AHoTgoXQducNqc4Xz7ayk3sKSBR0nTtZ succeeded!
[SUCCESS] ASD test with DISK_ID 0bDyMVOnJ017Sr8AKNA6LXCznH3AUvZx succeeded!
[SUCCESS] ASD test with DISK_ID fSNi6N0HHNyV53QbZUEAamGY4pBIdZid succeeded!
[SUCCESS] ASD test with DISK_ID pncz3dBPA1BQI0d0deIz8RpX73EN5LYa succeeded!
[WARNING] ASD test with DISK_ID C8O0sa7DQdgQWluSObL8TJJGudfVzOOn failed because: unknown
[WARNING] ASD test with DISK_ID R32nqOZCuJndKJPvchvDZOBW1XTJxQ3r failed because: unknown
[WARNING] ASD test with DISK_ID jsCrAIlcQsZykWzDsQDUfhjc6uDImXpS failed because: unknown
[WARNING] ASD test with DISK_ID 9x9PuLbP8zyRAT6x42k6j8Fg9kow5mdK failed because: unknown
[WARNING] ASD test with DISK_ID TDZlEEO9mrPesnPvOluR89JRH7bvQnAC failed because: unknown
[WARNING] ASD test with DISK_ID sY653FPRoieu4PSgeW8P3fSvq5SgrqQ9 failed because: unknown
[WARNING] ASD test with DISK_ID YVuzlp8UAulH6cVjFJxuzdL2jJ406sOu failed because: unknown
[WARNING] ASD test with DISK_ID nyk6rnnCyUyO50leEjCoSQXRuMW7IwWq failed because: unknown
[WARNING] Alba backend sv4-hddbackend01 should be available for VPool use with 58 asds, but there are 12 defective asds: tW2z3zvWpVH5abyj1stnEkv33j23kf6n, 3FhkQEjZmc5HOR77c1JLUguZWynlbOIn, 3t1Q5aYAIvmqQDOjjriXIBxYMgFd98SR, StnMHJf4vq2HhESO1qgvcW63vR9ZLbJl, C8O0sa7DQdgQWluSObL8TJJGudfVzOOn, R32nqOZCuJndKJPvchvDZOBW1XTJxQ3r, jsCrAIlcQsZykWzDsQDUfhjc6uDImXpS, 9x9PuLbP8zyRAT6x42k6j8Fg9kow5mdK, TDZlEEO9mrPesnPvOluR89JRH7bvQnAC, sY653FPRoieu4PSgeW8P3fSvq5SgrqQ9, YVuzlp8UAulH6cVjFJxuzdL2jJ406sOu, nyk6rnnCyUyO50leEjCoSQXRuMW7IwWq
[INFO] Checking separate ASDs for backend sv4-ssdbackend01:
[SUCCESS] ASD test with DISK_ID Ri00AwSaKhDv3lPkJ8uRad8TUhbq9iQH succeeded!
[SUCCESS] ASD test with DISK_ID VHoynWJwb0FhXuzf9tvBbDS78c2P73sO succeeded!
[SUCCESS] ASD test with DISK_ID E8oobKsfaJJhpwnr95Ud2QYRKYX1DrCy succeeded!
[SUCCESS] ASD test with DISK_ID IPWz28JwBoTvMf6e0gjocIkrU11ySisS succeeded!
[SUCCESS] ASD test with DISK_ID 1Wlu3Mrq0psaCCx1BGFKtBPtsuyCP8Ko succeeded!
[SUCCESS] ASD test with DISK_ID P5u40kvwHesaJDhFftUM5OYQV8QdZ8k9 succeeded!
[SUCCESS] ASD test with DISK_ID MSi5oNV5X0lmMeTldq181eO49jySa7Ig succeeded!
[SUCCESS] ASD test with DISK_ID Gn2vyw34EnNzwDu3QhjX0TRnhjUB6ejY succeeded!
[SUCCESS] ASD test with DISK_ID AzorLFjYcwazpFjH9e2kiBeJ7PmhbMlo succeeded!
[SUCCESS] ASD test with DISK_ID WMemhleQdvhlqupkiPCgbKDSJgTWfFT7 succeeded!
[SUCCESS] ASD test with DISK_ID 9E8rOHzYkZVW30Bv04kyUf9EJ9QS1dVz succeeded!
[SUCCESS] ASD test with DISK_ID Tuku9yExBSqV0BRNNaLNMkcQkneNLvQ8 succeeded!
[SUCCESS] ASD test with DISK_ID 91zmoIpvPHASXq5Fehut1HTcE7S6FIZL succeeded!
[SUCCESS] ASD test with DISK_ID EcWxGTzfzOBrHEHg0yPbBqjmMLtAoIlq succeeded!
[SUCCESS] ASD test with DISK_ID vJP0kXYpIb0qb8j7WVdTeRPlejh8Lryi succeeded!
[SUCCESS] ASD test with DISK_ID 5eoLARgL1ftjAATOO3Hs7RepVSmVWr56 succeeded!
[SUCCESS] ASD test with DISK_ID xQnL6Y7pFmWizhtxfYXzX1whbTjN4HN8 succeeded!
[SUCCESS] ASD test with DISK_ID eDu3Hlgj8VLmBhCAoBEUr8kBdMgfkLbO succeeded!
[SUCCESS] ASD test with DISK_ID Ycp8Yy7QBOMVaEt0JW5Rcdbr1qbxED1Y succeeded!
[SUCCESS] ASD test with DISK_ID 7OO53i2KEK8Vt5OrdIRenjucaXde3lyg succeeded!
[SUCCESS] ASD test with DISK_ID FclnxRtjQvXskaSbueqHtxgl6B2sdetl succeeded!
[SUCCESS] ASD test with DISK_ID cvvu4Rh0KMt2ToJjo93wiYMnezKrRBp5 succeeded!
[SUCCESS] ASD test with DISK_ID CvQJ8gBD19C7neuV4PtXsDedelMyqwnr succeeded!
[SUCCESS] ASD test with DISK_ID TAo90qrfM7xd62dLGxQut8xI6nh3Mkpy succeeded!
[SUCCESS] ASD test with DISK_ID 186lDrtv9ptx8P2oWoGBLAaWt5D33Yym succeeded!
[SUCCESS] ASD test with DISK_ID G6rEQX9DOmpNzsDEirhQ2kWpTyq5W81W succeeded!
[SUCCESS] ASD test with DISK_ID vxTDWaoCa7Z9EQe4Rw8ZmNMsIlnAwtmr succeeded!
[SUCCESS] ASD test with DISK_ID B9XUkwECXvtGCxXcXvorBjx2kd0loPeq succeeded!
[SUCCESS] ASD test with DISK_ID DxYokjdZXnGH9zVrWfXyMe2mVW9LwJAe succeeded!
[SUCCESS] ASD test with DISK_ID MZXhBzr6quRnDhPhFiTDOvWomAhP1EVk succeeded!
[SUCCESS] ASD test with DISK_ID RWEclgsIcijGmAP7txXI1TzPA4cTEHrw succeeded!
[SUCCESS] ASD test with DISK_ID kzxemoocRSecis4In1u355Z64F4gBWQE succeeded!
[SUCCESS] ASD test with DISK_ID Ip1rcjPf888yNLrGmtqUSS0DI8OeYFbc succeeded!
[SUCCESS] ASD test with DISK_ID 9VqB1ZGW9UpbZs8aYRME4Ei9ufmbRXza succeeded!
[SUCCESS] ASD test with DISK_ID sW5fPit16Z6DtPO6eCpY7kKWNNK8mcQU succeeded!
[SUCCESS] ASD test with DISK_ID LZI1ANavF748VTzSucF8aCkJoaIwBCIF succeeded!
[SUCCESS] ASD test with DISK_ID oXkgmS8HLoZy5uNbjvZBalVLu31LoVnG succeeded!
[SUCCESS] ASD test with DISK_ID ejALQJim51vO3QEsSmNE69vhJBiGyflO succeeded!
[SUCCESS] ASD test with DISK_ID f7dhQhuPmbM8wgctWQOwrGnJhymL23Zl succeeded!
[SUCCESS] ASD test with DISK_ID sNDoOyV6qYb4hAnAuV0dv65vjEuiKxhs succeeded!
[SUCCESS] ASD test with DISK_ID 2UXQ72C3PzXNXgTF8OOCRkUIsQdkwJyS succeeded!
[SUCCESS] ASD test with DISK_ID eWVUzU7RwYd6peJCKFRHazYTb1aqEHRF succeeded!
[SUCCESS] ASD test with DISK_ID Q32CcBP1vncOJ01r6MG5whRd5wnTk5WD succeeded!
[SUCCESS] ASD test with DISK_ID oGOnpqMiZ84RkAKpN26SdYUcI9IHNR65 succeeded!
[SUCCESS] ASD test with DISK_ID C0DH0mkncBGiROFlXM4Xk826zdjHxwR8 succeeded!
[SUCCESS] ASD test with DISK_ID mV6EjsuGGGPz6uSp1KkbzN9LWKa6imq1 succeeded!
[SUCCESS] ASD test with DISK_ID HVm33pHx68DKs6bGMOykhuGzaIyCzq5J succeeded!
[SUCCESS] ASD test with DISK_ID 9qEItHb08qcvPk2dgx357vn3Szy9CfRu succeeded!
[SUCCESS] ASD test with DISK_ID mR3t6lUgHfTZ5PJdc4P1CCBIyWWTmkRQ succeeded!
[SUCCESS] ASD test with DISK_ID LpLWaP36eap3j3rHcd5vERyFJsPqXgJO succeeded!
[SUCCESS] ASD test with DISK_ID 6RHJjOPtiyc5IdPjEEccRGWOuiBaXXOd succeeded!
[SUCCESS] ASD test with DISK_ID Ily33U8h8nygmNNkv0aZ4GZhwvoDGtok succeeded!
[SUCCESS] ASD test with DISK_ID kn36O0HGleTiAbk491x5bTLuIkiJU7Jt succeeded!
[SUCCESS] ASD test with DISK_ID CJzGUh7cvfZOk7Zew6zBjuiWJInGtD6o succeeded!
[SUCCESS] ASD test with DISK_ID iag3vGWXAg0O1U9o2BCBFDnPu2j1Lau1 succeeded!
[SUCCESS] ASD test with DISK_ID araX7RKXhOpHSzvYm9pDaoZoaO2EbqdU succeeded!
[SUCCESS] ASD test with DISK_ID 2sMSKSTqnNPnQSUfpACI69ZXGMNCBmj5 succeeded!
[SUCCESS] ASD test with DISK_ID DWXNGkI0ZmZnyG3CZWF551XrJrA9wN5G succeeded!
[SUCCESS] ASD test with DISK_ID my4NNubJVe0YLo64V2TPF8VcbxRXJfpr succeeded!
[SUCCESS] ASD test with DISK_ID Demrk5YCUUrVEiOimGpOD7fw0lDDqkZo succeeded!
[SUCCESS] ASD test with DISK_ID 02xpv9bwHOOAaex85cQz4WgO71VlYjfF succeeded!
[SUCCESS] ASD test with DISK_ID xSxZiNDvESKAAMVaWx6qJtJLSKgYeyiI succeeded!
[SUCCESS] ASD test with DISK_ID mNDPGiYqeEh9dIVrEQk2noRwJF1jRIMs succeeded!
[SUCCESS] ASD test with DISK_ID yV6PR9FHoxJBnYEjXgHBiIDUg0uq0eYz succeeded!
[SUCCESS] ASD test with DISK_ID 68Y1duTumEiGAX8zKOfxPbZJ39zlRC5Y succeeded!
[SUCCESS] ASD test with DISK_ID EcWw26QNZFDe5MxFRMaqy8Fhx0WiODIl succeeded!
[SUCCESS] ASD test with DISK_ID B80UdgvYES89sPX5SKObFg6uR7SnMjdI succeeded!
[SUCCESS] ASD test with DISK_ID 5evaYKlQZHcytJ0TsVkrCFBYpjdkcm99 succeeded!
[SUCCESS] ASD test with DISK_ID tvNjEQy29AaDOKh2GPpQTsAS1IFnj59z succeeded!
[SUCCESS] ASD test with DISK_ID IEKn4UbniK8AYogjPdQFuTEcz2VxaYfs succeeded!
[SUCCESS] ASD test with DISK_ID mWladaizsBQji4JpeNLsvg8Kf6nOfKLZ succeeded!
[SUCCESS] ASD test with DISK_ID 4BfLzOyznfd3kkwfDUAnvvkqbSd4MV7s succeeded!
[SUCCESS] ASD test with DISK_ID 5IZjpDMfWFPFjuuhq9CjtUaiEak7nxgw succeeded!
[SUCCESS] ASD test with DISK_ID Okk1JCtp4LbV8wqIi1kPXS45GaEWoGCG succeeded!
[SUCCESS] ASD test with DISK_ID vf5i0PEQyrjIN9Iszvy5y99RM1fxUsbJ succeeded!
[SUCCESS] ASD test with DISK_ID UtvYZuw4zUt9ZyGtel0NhnYRUtUKi12G succeeded!
[SUCCESS] ASD test with DISK_ID YshMo0S5EuJDySXaaYROCTYW64HggzWL succeeded!
[SUCCESS] ASD test with DISK_ID CfBN9Waq2gDff7Y1nf3m8VFVAD97oPsU succeeded!
[SUCCESS] ASD test with DISK_ID EfG1uvWpqOoY5C2Ywd9We889bw37cDpJ succeeded!
[SUCCESS] ASD test with DISK_ID odU2grCWkgiEyRQg5qsPpzo1VczwZ0x3 succeeded!
[SUCCESS] ASD test with DISK_ID DxAlMnITbsUbhCGwnkPvOIpWrtuDzpaT succeeded!
[SUCCESS] Alba backend sv4-ssdbackend01 should be available for VPool use. All asds are working fine!
[INFO] Checking separate ASDs for backend sv4-ssdbackend02:
[SUCCESS] ASD test with DISK_ID bahkcd6tMo5QrpF9WQJwHpcalfQdG20r succeeded!
[SUCCESS] ASD test with DISK_ID klcXtlJDbQn32g8GwfNzffRWpQCWpmAZ succeeded!
[SUCCESS] ASD test with DISK_ID ifwovUzyq0GGz2y9C26fzFUQpsvSQeYd succeeded!
[SUCCESS] ASD test with DISK_ID fmYAosgs1KAvsm3IZxVVeW8fpMWEt0GC succeeded!
[SUCCESS] ASD test with DISK_ID LLdImGr5xDGbAYyGl6TkBmFuWVcQyYxG succeeded!
[SUCCESS] ASD test with DISK_ID S2fFelPj5RJ2Vz64kkkfSc5qVuoKAmYz succeeded!
[SUCCESS] ASD test with DISK_ID vtg2xTSyYex7GjCUWWAKmqmj43dvlPYs succeeded!
[SUCCESS] ASD test with DISK_ID vwiPLcKqO4CKDxOglAef2HOkwZezRqqn succeeded!
[SUCCESS] ASD test with DISK_ID NYvZNQdnGTVIUIx41a0d41o96RMz5CSn succeeded!
[SUCCESS] ASD test with DISK_ID FofTlu4eRGlg0ANAxOWpr9jvlYLHKwnH succeeded!
[SUCCESS] ASD test with DISK_ID OLrgCDqmFjaC0SjvPK96zdI8fibC6DGE succeeded!
[SUCCESS] ASD test with DISK_ID w742vl9UMhtWPZ6xK4DEUEGu3zxfHCQM succeeded!
[SUCCESS] ASD test with DISK_ID Me2EzOLniVzqmhsufJgZ0P8TCDIWCcyt succeeded!
[SUCCESS] ASD test with DISK_ID 5K5KvJgEqoHVub0buwgdpxAlz8FKXL0O succeeded!
[SUCCESS] ASD test with DISK_ID KRzIEW8ZJmGwEDuLaitopTZIk0oDCWZ9 succeeded!
[SUCCESS] ASD test with DISK_ID t7HJgCUcd3j8j91qt1eBHVYotZXf4cw1 succeeded!
[SUCCESS] ASD test with DISK_ID tLISxfSxqEvaBckLFIxBouNGF76pkdkM succeeded!
[SUCCESS] ASD test with DISK_ID CVUfP3iPW5a4WMCrp0KjYNqbAk54Y5EV succeeded!
[SUCCESS] ASD test with DISK_ID nVBkTtzHXgw3cabH1YUztZBlKEtQsrwZ succeeded!
[SUCCESS] ASD test with DISK_ID 0tX8hHuOzCGUXKbnZlYofRGnfQLc6kBC succeeded!
[SUCCESS] ASD test with DISK_ID aBmWhaKBAplJ66ORRQoJEHFIKbXrQeMe succeeded!
[SUCCESS] ASD test with DISK_ID cxcPp8n9s0N4QWowPe3eMPCJhkw5rklL succeeded!
[SUCCESS] ASD test with DISK_ID gaPsJUcSPRGLjiGsGBMgLnYCusxmoy8z succeeded!
[SUCCESS] ASD test with DISK_ID zJTvxnQHvHURHYOcYHHMTtrIvMWgALSj succeeded!
[SUCCESS] ASD test with DISK_ID xswlfpgBznVNbEWtf6RyKV3WNXxPblPr succeeded!
[SUCCESS] ASD test with DISK_ID iopIMVP5hOsg8FpJHXPF301VKzTyp7Og succeeded!
[SUCCESS] ASD test with DISK_ID I44WArlkPmE3hsVyMnky0geXwgGbeFnf succeeded!
[SUCCESS] ASD test with DISK_ID xZeMhIYtKUPf9QAA8452SR9rvXT8ijhp succeeded!
[SUCCESS] ASD test with DISK_ID 6LsazgDXFnA4envKOz1y2957kQKURnE1 succeeded!
[SUCCESS] ASD test with DISK_ID d0fEeEoFrOGHdIjkIRSjmufUJLvlx9CM succeeded!
[SUCCESS] ASD test with DISK_ID E64s9mvjxnACC8M7Y70CulVWgQg3MXVK succeeded!
[SUCCESS] ASD test with DISK_ID Yeo9RkNULMrmmXEoEgR2dRVQk1B4ixUI succeeded!
[SUCCESS] ASD test with DISK_ID sVnRBHX9FqwOGbRZtk3XLgUjRZk2sWuL succeeded!
[SUCCESS] ASD test with DISK_ID rx6mQSBWvHDCIG3Ko0qGxqCL0EIU2hIe succeeded!
[SUCCESS] ASD test with DISK_ID koKhSG954pPMMacvNJm5PokccgvR8gK3 succeeded!
[SUCCESS] ASD test with DISK_ID QTIW0yJMplUSflk0J2NKkTgXm1fQSJoQ succeeded!
[SUCCESS] ASD test with DISK_ID tCMDoBXkYAG3aEpAGFzOdWXk8XxqeJ4f succeeded!
[SUCCESS] ASD test with DISK_ID hYgj6sQdB5ZccXXakbzuYTSbjXb1mARY succeeded!
[SUCCESS] ASD test with DISK_ID hN9FaHJ1OgepUblFuaEzI0yxhtS0wu7f succeeded!
[SUCCESS] ASD test with DISK_ID GavEWQM4ajFtG9qkBGPAx5wX1bp8qrbx succeeded!
[SUCCESS] ASD test with DISK_ID orRidPXAP6SrlcKSY144QYvAUEbC5iaZ succeeded!
[SUCCESS] ASD test with DISK_ID qqy2q5AlvDhEkUcsTQd3U3Rhn316dbPJ succeeded!
[SUCCESS] ASD test with DISK_ID 7gdM9BZFNOZtj1Y4ajn2SvGE8AIPucEg succeeded!
[SUCCESS] ASD test with DISK_ID 9asoK86MdPxAZ0aUasFstxAPSOOuCnmo succeeded!
[SUCCESS] ASD test with DISK_ID Xc2SeVVxf7DDcbR4EhFLt9mnO5MCGwSG succeeded!
[SUCCESS] ASD test with DISK_ID 2Bs5QZEyCHfbq1qk6LVQ5WpKkiS08q0E succeeded!
[SUCCESS] ASD test with DISK_ID DAxORxGdzFDuikHZ5b7oh04cZrxqjRLY succeeded!
[SUCCESS] ASD test with DISK_ID aXfvbyrtfsKBomCA5qIKcwiIPALHEpqs succeeded!
[SUCCESS] ASD test with DISK_ID Z9wTIziaTAJBkt0CWBcc5KjPnK7TZovR succeeded!
[SUCCESS] ASD test with DISK_ID rZ2mrxtwbytUK2dah96pRSbzqP5qtcT3 succeeded!
[SUCCESS] ASD test with DISK_ID uhchCxdUJ7Yj4xLAH5RESvH2fmSmBM5z succeeded!
[SUCCESS] ASD test with DISK_ID B30z87Q4TXyPDIyrFtGPY8W7Q3nywkws succeeded!
[SUCCESS] ASD test with DISK_ID RMSw1L3bUvsn3OzuKoL4DquEqDw9zten succeeded!
[SUCCESS] ASD test with DISK_ID yVh2a323THyRe2fLAPpKEcjCqfGFl5ax succeeded!
[SUCCESS] ASD test with DISK_ID Af5EWvxoLJ873udllOLrLxb7dTY8ktKB succeeded!
[SUCCESS] ASD test with DISK_ID veyfsfpTRgdTVvuQGy6RDHE7qtGBGZf3 succeeded!
[SUCCESS] ASD test with DISK_ID 9cD3JnKaHFfg83OFSUrHCocwFuCHlRp4 succeeded!
[SUCCESS] ASD test with DISK_ID 3Ej54cLMsmy9Qk6Or1gNPDKtmDR7fsvd succeeded!
[SUCCESS] ASD test with DISK_ID VDCvIpL9oRTLtPJMlCmnLBTldakHC2dy succeeded!
[SUCCESS] ASD test with DISK_ID 5zIdYLaKRJWSEdKszSbW9LOmLyTqZcV0 succeeded!
[SUCCESS] ASD test with DISK_ID pubF0BZC9wqEKCMGc5tNMUs40PzBMb3b succeeded!
[SUCCESS] ASD test with DISK_ID gcwEAHaofkpctBVXrBhX49mn43GErpiE succeeded!
[SUCCESS] ASD test with DISK_ID UaDs27COHtDQlIWj6KfndPgf5E1kEJl6 succeeded!
[SUCCESS] ASD test with DISK_ID xWeW2BnumZ7YEmtsjqN95PGueV5ItmCl succeeded!
[SUCCESS] ASD test with DISK_ID VWSiIhBJl0w5tUlYMEOH2LG9iWPQqaZN succeeded!
[SUCCESS] ASD test with DISK_ID fsZqTf6ELNBOOLuTPOV8edpPdkz0Ejw6 succeeded!
[SUCCESS] ASD test with DISK_ID LKiAZD9XE6nItxIuDPjQWM2P3TvUgQzt succeeded!
[SUCCESS] ASD test with DISK_ID vEOtn1EbWItv7zWzcan1AfmB0gbbpYgC succeeded!
[SUCCESS] ASD test with DISK_ID 7UjIWyEYR73Vjxz3XC1GoYURQz1IhYPJ succeeded!
[SUCCESS] ASD test with DISK_ID ptjP6e51FnIVWaQPGi1sjWdUnpaMyXkP succeeded!
[SUCCESS] ASD test with DISK_ID tRxSgNiLS2hBxurOthgSOBMTbxoahari succeeded!
[SUCCESS] ASD test with DISK_ID 4EWM8coeLx1hpZK7Uo4SbK6gNW1o9v4x succeeded!
[SUCCESS] ASD test with DISK_ID WTcLPFttmd46hVeUehJNa8Oo4I1b3gL8 succeeded!
[SUCCESS] ASD test with DISK_ID 3ecmEDwcrXDEaxoL6dsv1U3bXyXhZiXp succeeded!
[SUCCESS] ASD test with DISK_ID 7WkxO7GoqudvwxdXWTstk18WsI3HhuuW succeeded!
[SUCCESS] ASD test with DISK_ID mubxaRQFsO84Vr6wBo7aVj2P1nzhLFUF succeeded!
[SUCCESS] ASD test with DISK_ID dUpuzurYX9sQ5NvZVZRWmQGbOZ3ICt2S succeeded!
[SUCCESS] ASD test with DISK_ID 7qcrAmUf9iHrKGzQpNt0yDnAhuJpRRAM succeeded!
[SUCCESS] ASD test with DISK_ID Nyd0SgbiYIxez92rKhxOhOfWmMaW577T succeeded!
[SUCCESS] Alba backend sv4-ssdbackend02 should be available for VPool use. All asds are working fine!
[INFO] Recap of Health Check module alba test backend-test!
[INFO] ======================
[INFO] SUCCESS=272 FAILED=0 SKIPPED=0 WARNING=33 EXCEPTION=0

Copied from original issue: openvstorage/openvstorage-health-check#434
@openvstorage-ci
Copy link
Contributor Author

From @JeffreyDevloo on February 13, 2018 9:39

Looks like an issue of the Framework Alba Plugin instead. Fetching disk information was not possible.

@openvstorage-ci openvstorage-ci mentioned this issue Jun 19, 2018
Closed
@wimpers wimpers added this to the Roadmap milestone Jun 19, 2018
@matthiasdeblock
Copy link

Had the same issue on the SV4 cluster. Restarting the asd-manager on the nodes which showed red ASD's temporary fixed the problem.

Error from the asd-manager:

2018-07-07 03:54:42 29700 -0400 - SV4SRV0065 - 10344/140292296378112 - asd-manager/api - 279164 - ERROR - Unexpected exception
Traceback (most recent call last):
  File "/opt/asd-manager/source/app/decorators.py", line 77, in new_function
    return_data = f(*args, **kwargs)
  File "/opt/asd-manager/source/app/api.py", line 94, in list_disks
    DiskController.sync_disks()
  File "/opt/asd-manager/source/controllers/disk.py", line 57, in sync_disks
    for path_type in DiskController._local_client.dir_list(directory='/dev/disk'):
  File "/opt/asd-manager/source/tools/localclient.py", line 266, in dir_list
    return os.listdir(directory)
OSError: [Errno 24] Too many open files: '/dev/disk'

@JeffreyDevloo
Copy link
Contributor

We might be leaking file descriptors in the asd-manager. Could you provide the lsof information if it occurs again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type_feature
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@wimpers @openvstorage-ci @matthiasdeblock @JeffreyDevloo