paho.mqtt.testing fails

[yqylovy]

Hi, I tried to test the broker with python3 paho.mqtt.testing/interoperability/client_test5.py
And it failed on some test
here it is

['client_test5/TopicA/+', 'client_test5/+/C', 'client_test5/#', 'client_test5//#', 'client_test5//+', 'client_test5/+/+', 'client_test5/TopicA/#']
argv ['client_test5.py']
clean up starting
clean up finished
.....E/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py:643: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 64853)>
  outcome.errors.clear()
..F..F...F.FERROR 20181213 180527 mqtt-client call: unexpected exception (<class 'ConnectionResetError'>, ConnectionResetError(54, 'Connection reset by peer'), <traceback object at 0x1059c6348>)
ERROR 20181213 180527 mqtt-client call: unexpected exception (<class 'ConnectionResetError'>, ConnectionResetError(54, 'Connection reset by peer'), <traceback object at 0x1059c6348>)
ERROR:mqtt-client:call: unexpected exception (<class 'ConnectionResetError'>, ConnectionResetError(54, 'Connection reset by peer'), <traceback object at 0x1059c6348>)
Traceback (most recent call last):
  File "/Users/yqylovy/thirdparty/src/paho.mqtt.testing/interoperability/mqtt/clients/V5/internal.py", line 162, in __call__
    self.receive(callback)
  File "/Users/yqylovy/thirdparty/src/paho.mqtt.testing/interoperability/mqtt/clients/V5/internal.py", line 47, in receive
    packet = MQTTV5.unpackPacket(MQTTV5.getPacket(self.socket))
  File "/Users/yqylovy/thirdparty/src/paho.mqtt.testing/interoperability/mqtt/formats/MQTTV5/MQTTV5.py", line 273, in getPacket
    buf = aSocket.recv(1) # get the first byte fixed header
ConnectionResetError: [Errno 54] Connection reset by peer
FTraceback (most recent call last):
  File "client_test5.py", line 342, in test_subscribe_failure
    assert callback.subscribeds[0][1][0].value == 0x80, "return code should be 0x80 %s" % callback.subscribeds
AssertionError: return code should be 0x80 [(39, [<mqtt.formats.MQTTV5.MQTTV5.ReasonCodes object at 0x11a9a45f8>])]
FF......
======================================================================
ERROR: test_flow_control2 (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 1023, in test_flow_control2
    receiver.receive(testcallback)
  File "/Users/yqylovy/thirdparty/src/paho.mqtt.testing/interoperability/mqtt/clients/V5/internal.py", line 85, in receive
    self.socket.send(self.pubrel.pack())
BrokenPipeError: [Errno 32] Broken pipe

======================================================================
FAIL: test_offline_message_queueing (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 252, in test_offline_message_queueing
    self.assertTrue(len(callback.messages) in [2, 3], len(callback.messages))
AssertionError: False is not true : 1

======================================================================
FAIL: test_publication_expiry (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 530, in test_publication_expiry
    self.assertEqual(len(callback2.messages), 2, callback2.messages)
AssertionError: 0 != 2 : []

======================================================================
FAIL: test_server_keep_alive (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 912, in test_server_keep_alive
    self.assertTrue(hasattr(connack.properties, "ServerKeepAlive"))
AssertionError: False is not true

======================================================================
FAIL: test_session_expiry (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 409, in test_session_expiry
    self.assertEqual(connack.sessionPresent, False)
AssertionError: True != False

======================================================================
FAIL: test_shared_subscriptions (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 1161, in test_shared_subscriptions
    self.assertEqual(len(callback.messages), count)
AssertionError: 0 != 1

======================================================================
FAIL: test_subscribe_failure (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 347, in test_subscribe_failure
    self.assertEqual(succeeded, True)
AssertionError: False != True

======================================================================
FAIL: test_subscribe_identifiers (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "client_test5.py", line 663, in test_subscribe_identifiers
    self.assertEqual(len(callback2.messages), 1, callback2.messages)
AssertionError: 2 != 1 : [('TopicA', b'sub identifier test', 1, False, 1, <mqtt.formats.MQTTV5.MQTTV5.Properties object at 0x11a9a47f0>), ('TopicA', b'sub identifier test', 1, False, 2, <mqtt.formats.MQTTV5.MQTTV5.Properties object at 0x11a9a4198>)]

----------------------------------------------------------------------
Ran 27 tests in 155.103s

FAILED (failures=7, errors=1)

[troian]

Thanks, I'll take a look

[troian]

Looks for MQTT V5.0 testing is not ready yet. There are bunch of issues opened for it.
I'll wait till all tests comply with spec