Query about extending XMTZC05HM decoder

[DigiH]

@1technophile @h2zero With my comments in

1technophile/OpenMQTTGateway#760
and
1technophile/OpenMQTTGateway#1166

and trying to implement all these findings in the XMTZC05HM decoder I have some queries.

Is there a possibility to have nested & and | conditions to achieve something like

"condition":["uuid", "contain", "181b", "&",("servicedata", "index", 2, "2", "|", "servicedata", "index", 2, "a")],

if both those stabilised (load still on or off) conditions want to be accepted?

"servicedata", "index", 2, "2" only gets caught when still standing on the scale and the display is still on, the advantage being that once the scale display switches off the service data is not recognised any longer during BLE scans, but if the TimeBtwRead is too long it might not get recognised at all, then only the "servicedata", "index", 2, "a" is being recognised and also published during any consecutive scan process - which it already is with just "condition":["uuid", "contain", "181b"]

Any ideas on these issues?

Thanks

[DigiH]

Got this working nicely for kg only in

https://github.com/DigiH/decoder/tree/DigiH-kg_only_test

with additional mode (person or object) to allow for backend HA or openHAB to only record person relevant measurements.

Example outputs are

Object measurement
{"id":"AA:BB:CC:DD:EE:FF","mac_type":0,"name":"MIBFS","manufacturerdata":"57010c9541d1e284","rssi":-92,"servicedata":"06a4b20701010508120000fc03","brand":"Xiaomi","model":"Miscale_v2","model_id":"XMTZC05HM","mode":"object","unit":"kg","weight":5.1}

Person measurement with shoes/socks on - no impedance reported
{"id":"AA:BB:CC:DD:EE:FF","mac_type":0,"name":"MIBFS","manufacturerdata":"57010c9541d1e284","rssi":-85,"servicedata":"02a4b2070101051c100001c3e","brand":"Xiaomi","model":"Miscale_v2","model_id":"XMTZC05HM","mode":"person","unit":"kg","weight":79.5}

Person measurement barefoot - impedance reported
{"id":"AA:BB:CC:DD:EE:FF","mac_type":0,"name":"MIBFS","manufacturerdata":"57010c9541d1e284","rssi":-85,"servicedata":"02a6b2070101051c106a01f43D","brand":"Xiaomi","model":"Miscale_v2","model_id":"XMTZC05HM","mode":"person","unit":"kg","weight":79.3,"impedance":362}

Trying to implement the same for all kg/lbs options in

https://github.com/DigiH/decoder/tree/DigiH-XMTZC05HM-Extension

resulted in some messy recognition of always object or lbs. Need to try and find out if this is due to some messed up formatting or something else tomorrow with a clearer head :)

[DigiH]

Ok, would't let me rest as to why the complete decoder gave me such problems ;)

This is the comparison of just adding the two lbs mode options to the correctly working kg only branch

https://github.com/DigiH/decoder/compare/DigiH-kg_only_test...DigiH:DigiH-kg_plus_lbs?expand=1

I just add the additional two mode options and voila, the decoding doesn't work again constantly showing "mode":"object" now even with service data:02a4b2070101073307fdff0447

where

"mode":{
         "condition":["servicedata", 1, "2"],
         "decoder":["static_value", "person"]
      },

should definitely set it to person.

What am I missing with the additional "__mode" and "___mode"?

[h2zero]

I don't see anything wrong with your changes. I will pull your branch and test locally when I get a few minutes. How are you testing this? I don't see the unit tests being updated.

[DigiH]

I was just doing real world testing with different people, barefoot and with shoes or socks on, and small objects under 10 kg for the time being. I don't actually know the 'small' object limit under which weight the scale recognises them as objects and not small people/children.

The tests I've added in the DigiH-kg_only_test branch should all run fine if I didn't make any copy/paste errors. I currently don't actually know how to run them, short of a PR to theengs/decoder ;)

But exactly that puzzles me, that these tests should still run fine even after making the small changes/additions in the diff above.

Just preparing for the person/object recognition for scales set to lbs, with the additional

   ```

"__mode":{
"condition":["servicedata", 1, "3"],
"decoder":["static_value", "person"]
},
"___mode":{
"condition":["servicedata", 1, "7"],
"decoder":["static_value", "object"]
},

should not actually affect the current (real world) tests, which are all with kg, but it does.

Since I'm using the initial additional **"servicedata", "index", 2, "a"** for **stabilised, stepped off scale** the last measurement is recognised and transmitted during every scan for quite a while, so it's interesting to observe how building and uploading with the DigiH-kg_only_test decoder branch everything is fine, then rebuilding and uploading with DigiH-kg_plus_lbs messes up the exact same regular scale transmissions.

[DigiH]

Ok, I've created two new branches, along with @1technophile's hint to split the 2 and a variants so that the nested OR condition mentioned above isn't needed any longer.

All working fine for kg only in

https://github.com/DigiH/decoder/tree/kg_split_only

(with corrected test cases now) but again, as soon as I add the appropriate lbs additions in

https://github.com/DigiH/decoder/tree/kg_split_and_lbs

, this time only to the a variant for the time being, the 2decoder is obviously still fine, but for the a the decoder always seems to pick the last entry for the mode, unit and weight decoding. Could it be that evaluating the same

"condition":["servicedata", 1, …

twelve times for all the different options is tripping something over? ;)

An other alternative trial could be to split up the decoder even more, adding the evaluation of

"__unit":{
"condition":["servicedata", 1, "3"],
"decoder":["static_value", "lbs"]
},

and

  "___unit":{
     "condition":["servicedata", 1, "7"],
     "decoder":["static_value", "lbs"]
  }, 

separately with another AND into the model condition, but this would create an unnecessary large amount of split decoders.

Any idea what is blocking me here to implement the new features for XMTZC05HM for both kg and lbs?

Thanks

[DigiH]

Ok, I've done the latter idea, splitting the possible options in the model condition, only for the a variant for now.

https://github.com/DigiH/decoder/blob/kg_lbs_a4_split/src/devices/XMTZC05HM_json.h

and it is working fine now for a broadcasting kg and lbs decoding for all the different possibilities.

If you can confirm that this is the only possibility for this feature implementation at the moment and that having 8 split decoders in the end is fine, I'll make the same changes to the 2 broadcasting option and append and amend the unit tests.

Thanks

[1technophile]

@h2zero do you think there are some possible decoder optimizations here?

[DigiH]

@1technophile, sorry I didn't comment

DigiH@60181ce

didn't work, never really evaluating the second OR case(s).

Or where did you expect just pulling together the unit condition for kg?

  "unit":{
         "condition":["servicedata", 1, "2", "|","servicedata", 1, "6"],
         "decoder":["static_value", "kg"]
      },

where now in the 3 split version (with) the unit kg case doesn't need evaluating at all?

_XMTZC05HM_json_vkga

      },"unit":{
         "decoder":["static_value", "kg"]
      },

Did I completely missunderstand the proposal?

[1technophile]

Or where did you expect just pulling together the unit condition for kg?

This was just for kg indeed, but from what I understand OR or AND into a property condition doesn't work ?

[DigiH]

"&" and "|" I've only seen in other model conditions so far, not in property conditions, and the trial with the commit above failed in my testing.

Also in my submission for https://github.com/theengs/decoder/pull/41/files being able to string together several OR options would have made the power and battery cases a lot more concise and cleaner.

Was this supposed to work, at least for servicedata property conditions already?

[1technophile]

ok, got it, we will need to analyze this

[DigiH]

Let me just quickly go back and double check, but I'm sure I would have loved that to work as it would have definitely allowed for a lot more elegant solution than now.

[1technophile]

@DigiH could we group these 2:

const char* _XMTZC05HM_json_vkgap 
const char* _XMTZC05HM_json_vkgao

by leveraging a condition into the mode:

      "mode":{
         "condition":["servicedata", 1, "2"],
         "decoder":["static_value", "person"]
      },
      "_mode":{
         "condition":["servicedata", 1, "6"],
         "decoder":["static_value", "object"]
      },

[DigiH]

Pulling back things together.

But the unit and weight will then have to be doubled and conditioned as well

…
      "unit":{
         "condition":["servicedata", 1, "2"],
         "decoder":["static_value", "kg"]
      },
      "_unit":{
         "condition":["servicedata", 1, "6"],
         "decoder":["static_value", "kg"]
      },
      "weight":{
         "condition":["servicedata", 1, "2"],
         "decoder":["value_from_hex_data", "servicedata", 22, 4, true, false],
         "post_proc":["/", 200]
      },
      "_weight":{
         "condition":["servicedata", 1, "6"],
         "decoder":["value_from_hex_data", "servicedata", 22, 4, true, false],
         "post_proc":["/", 200]
      }, 
…

and then having the identical model condition for _XMTZC05HM_json_vkga and _XMTZC05HM_json_vlbsa, might that cause an issue?

"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a"],

or then just the kg combined and keeping the lbs split with the additional conditions ...

Will try it out :)

Thanks

[DigiH]

So the pulling together both kg and lbs didn't work, likely due to the identical model condition

"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a"],

Splitting the lbs decoders again as with the previous 4 split still requires the combined kg to have the more universal above model condition, which works fine for recognising kg measurements, but the two more restrictive lbs model conditions

"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a", "&", "servicedata", "index", 1, "3"],
and
"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a", "&", "servicedata", "index", 1, "7"],

don't kick in and the more universal kg condition already catches and only publishes

… "servicedata":"07a4e60702100f240c00002602","brand":"Xiaomi","model":"Mi_Body_Scale","model_id":"XMTZC05HM"}

Now trying to reorder the two more specific lbs decoders before the more universal kg decoder, to see if it makes any difference …

[DigiH]

Yes, the reordering of the more specific two lbs decoders before the more universal kg decoder

_XMTZC05HM_json_vlbsap
"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a", "&", "servicedata", "index", 1, "3"],

_XMTZC05HM_json_vlbsao
"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a", "&", "servicedata", "index", 1, "7"],

only then

_XMTZC05HM_json_vkga
"condition":["uuid", "contain", "181b", "&", "servicedata", "index", 2, "a"],

works.

There don't seem to be other decoders where the order is an issue/needs to be followed.

My question now is, is it worth pulling _XMTZC05HM_json_vkgap and _XMTZC05HM_json_vkgao together, just to save one extra decoder (two when finally applied to the 2 variants as well), when now the evaluation order of the decoders in devices.h - const char _devices* has to be observed?

https://github.com/DigiH/decoder/blob/kg_lbs_split_min/src/devices/XMTZC05HM_json.h

or would the previous 4 (8) split not be more robust and unambiguous for now?

[1technophile]

I'm worried about memory for embedded devices so I would prefer to reduce the size of the decoders when possible.

[DigiH]

@1technophile I'm totally on for short and concise as possible decoders, but just tested again and multiple OR (never tested AND there) property conditions with servicedate (and manufacturerdata from my iBeacon experience) do not work. I would not have gone down the rabbit hole of many multiple property variants again or the multiple decoder versions, after you given me that hint.

I've left the 2 variant in the decoder how I'd love to have it condensed with multiple OR conditions, so ready for any testing if this can be implemented. Then just one second a variant decoder would be needed, or just one universal one, if nested conditions in the model conditions might be possible sometime in the future :)

I'd also like to resubmit the iBeacon decoder once this is working.

Thanks

[1technophile]

Got it we will find what is going on here

[h2zero]

I'd like to walk this back a bit so it's objectively clearer. What would the ideal JSON decode input be?
I'm not familiar with this device, and the data is getting messy with all the iterations/posts. If you could provide a few test data sets and expected outputs it would greatly help to provide a timely solution.

[DigiH]

Apologies for me probably not being clearer from the start. My ideal JSON would have been the current 2 variant together with the query in my initial posting about the possibility to have the nested OR in the model condition.

"condition":["uuid", "contain", "181b", "&",("servicedata", "index", 2, "2", "|", "servicedata", "index", 2, "a")],

obviously with escaped paranteheses or however it could be implemented.

That would have allowed for one clean concise single decoder

 {
   "brand":"Xiaomi",
   "model":"Mi_Body_Scale",
   "model_id":"XMTZC05HM",
   "condition":["uuid", "contain", "181b", "&",("servicedata", "index", 2, "2", "|", "servicedata", "index", 2, "a")],
   "properties":{
      "mode":{
         "condition":["servicedata", 1, "2", "|","servicedata", 1, "3"],
         "decoder":["static_value", "person"]
      },
      "_mode":{
         "condition":["servicedata", 1, "6", "|","servicedata", 1, "7"],
         "decoder":["static_value", "object"]
      },
      "unit":{
         "condition":["servicedata", 1, "2", "|","servicedata", 1, "6"],
         "decoder":["static_value", "kg"]
      },
      "_unit":{
         "condition":["servicedata", 1, "3", "|","servicedata", 1, "7"],
         "decoder":["static_value", "lbs"]
      },
      "weight":{
         "condition":["servicedata", 1, "2", "|","servicedata", 1, "6"],
         "decoder":["value_from_hex_data", "servicedata", 22, 4, true, false],
         "post_proc":["/", 200]
      },
      "_weight":{
         "condition":["servicedata", 1, "3", "|","servicedata", 1, "7"],
         "decoder":["value_from_hex_data", "servicedata", 22, 4, true, false],
         "post_proc":["/", 100]
      },
      "impedance":{
         "condition":["servicedata", 3, "6"],
         "decoder":["value_from_hex_data", "servicedata", 18, 4, true, false]
      }
   }
}

When then during my initial testing the first OR property condition already failed (DigiH@de30d86) - just switching the scale from kg ("servicedata", 1, "2") to lbs ("servicedata", 1, "3") and not getting the "mode":"person" JSON any longer, and with the more recent re-testing still the "unit":"lbs" and "weight":xx, with the latter two having "servicedata", 1, "3" as the first condition - and remembering the same situation with trying multiple OR conditions for metadata in the iBeacon split power/battery situation, I didn't even question the possibility of multiple OR property conditions not being possible.

So the above option would be the nicest solution for the decoder, or having two separate decoders for the 2 and a variants, if nested model conditions cannot easily be implemented.

Thanks

[DigiH]

So with the above all the 2 varaints should come back exactly like the currently working a variants with the awkwardly split 3 decoders.

servicedata":"0624e60702110e1e280000f401" produces
{"id":"0C:95:41:D1:E2:84","mac_type":0,"name":"MIBFS","rssi":-79,"servicedata":"0624e60702110e1e280000f401","brand":"Xiaomi","model":"Mi_Body_Scale","model_id":"XMTZC05HM"}

where it should be the same as
servicedata":"06a4e60702110e1e280000f401"
{"id":"0C:95:41:D1:E2:84","mac_type":0,"name":"MIBFS","rssi":-81,"servicedata":"06a4e60702110e1e280000f401","brand":"Xiaomi","model":"Mi_Body_Scale","model_id":"XMTZC05HM","mode":"object","unit":"kg","weight":2.5}

save for the 2 (stabilised - scale still on) vs. a (stabilised - scale back in standby) difference, hence the initial nested model condition request.

And the currently implemented three V2 tests should work:

{"Miscale_v2", "0x181b", "servicedata", "0226e5070c170c301df7019a38"},
{"Miscale_v2", "0x181b", "servicedata", "0224e5070c170c301d00009a38"},
{"Miscale_v2", "0x181b", "servicedata", "0624b20701010508120000fc03"},

with results

"{\"brand\":\"Xiaomi\",\"model\":\"Mi_Body_Scale\",\"model_id\":\"XMTZC05HM\",\"mode\":\"person\",\"unit\":\"kg\",\"weight\":72.45,\"impedance\":503}",
"{\"brand\":\"Xiaomi\",\"model\":\"Mi_Body_Scale\",\"model_id\":\"XMTZC05HM\",\"mode\":\"person\",\"unit\":\"kg\",\"weight\":72.45}",
"{\"brand\":\"Xiaomi\",\"model\":\"Mi_Body_Scale\",\"model_id\":\"XMTZC05HM\",\"mode\":\"object\",\"unit\":\"kg\",\"weight\":5.1}",

[DigiH]

@1technophile @h2zero I've started from scratch and cleanly implemented all the changes I think necessary, with currently two decoders for 2 and a stabilisation, including all possible 12 test cases. I hope this helps for clarification.

https://github.com/DigiH/decoder/tree/XMTZC05HM-Extended

Thanks

[h2zero]

Thanks I was just looking into this, that helps :)

[h2zero]

@DigiH I believe I have resolved this with a decoder modification. I have created a branch here: https://github.com/theengs/decoder/tree/ext-prop-conditions

This also incorporates your JSON decode specs (removed the extra "va" type ) and fixes the tests to allow them to pass. Please test this with the actual device and let me know if it's suitable to your needs.

[DigiH]

I've just done some quick testing before bedtime and things are looking great so far :)

I'll do some more thorough testing tomorrow.

Thanks a lot @h2zero!

[DigiH]

@h2zero @1technophile Tested all 12 possible cases multiple times with my device and all is working perfectly as envisaged :)

Thanks a lot for implementing this for the model and property conditions.

I do have some questions about the implementation, hopefully without being too much of a pain.

• Model condition

"condition":["servicedata", "index", 2, "a", "|", "servicedata", "index", 2, "2", "&", "uuid", "contain", "181b"],

as the definition is not unambiguous without using any parenthesis, but the implementation here has been made to recognise it as

"condition":[("servicedata", "index", 2, "a", "|", "servicedata", "index", 2, "2",) "&", "uuid", "contain", "181b"],

wouldn't it get confusing for someone who'd want to implement a model condition where something like this is needed

"condition":["servicedata", "index", 2, "a", "|"(, "servicedata", "index", 2, "2", "&", "uuid", "contain", "181b")],

and very likely two separate decoders would be needed with the current implementation?

I'm just thinking about future extended and more complex decoders and a clear definition for such condition AND OR nesting.

Also with finding that the other Mi scale model XMTZC04HM also supports person/object distinction, but there this differentiation is on the same index as 'stabilised', it would need a model condition with currently four (kg only), but very likely 8, once lbs testing has been done. Would this case already be covered with the current implementation as well?

"condition":["servicedata", "index", 0, "a", "|", "servicedata", "index", 0, "2", "|", "servicedata", "index", 0, "e", "|", "servicedata", "index", 0, "6", "|", "servicedata", "index", 0, "b", "|", "servicedata", "index", 0, "3", "|", "servicedata", "index", 0, "f", "|", "servicedata", "index", 0, "7", "&", "uuid", "contain", "181d"],

as in ( "b", "3", "f" and "7" currently only assumptions for the 'lbs' variations)

"condition":[("servicedata", "index", 0, "a", "|", "servicedata", "index", 0, "2", "|", "servicedata", "index", 0, "e", "|", "servicedata", "index", 0, "6", "|", "servicedata", "index", 0, "b", "|", "servicedata", "index", 0, "3", "|", "servicedata", "index", 0, "f", "|", "servicedata", "index", 0, "7", ) "&", "uuid", "contain", "181d"],

• Property conditions

The current decoder with the mostly dual OR property conditions works really well, but thinking about resubmitting the iBeacon decoder to reduce its size by pulling together the power and battery conditions will the current implementation already allow for a longer OR chain?

"condition":["manufacturerdata", 48, "8", "|", "manufacturerdata", 48, "9", "|", "manufacturerdata", 48, "a", "|", "manufacturerdata", 48, "b", "|", "manufacturerdata", 48, "c", "|", "manufacturerdata", 48, "d", "|", "manufacturerdata", 48, "e", "|", "manufacturerdata", 48, "f"],

It does look to me like it does briefly glancing at the changed code, but I'd need a lot longer to fully understand it all ;)

Thanks

[h2zero]

"condition":["servicedata", "index", 2, "a", "|", "servicedata", "index", 2, "2", "&", "uuid", "contain", "181b"],
as the definition is not unambiguous without using any parenthesis, but the implementation here has been made to recognise it as
"condition":[("servicedata", "index", 2, "a", "|", "servicedata", "index", 2, "2",) "&", "uuid", "contain", "181b"],
wouldn't it get confusing for someone who'd want to implement a model condition where something like this is needed
"condition":["servicedata", "index", 2, "a", "|"(, "servicedata", "index", 2, "2", "&", "uuid", "contain", "181b")],

Yes, nesting conditions will likely be needed at some point in the future but it was not my intention for today :) It would require a fairly large change to the code and I'd prefer to wait until there is a case where it's required to facilitate better testing.

Also with finding that the other Mi scale model XMTZC04HM also supports person/object distinction, but there this differentiation is on the same index as 'stabilised', it would need a model condition with currently four (kg only), but very likely 8, once lbs testing has been done. Would this case already be covered with the current implementation as well?

It should work the same.

will the current implementation already allow for a longer OR chain?

It will keep looping on however many conditions you can fit :)

[DigiH]

Yes, nesting conditions will likely be needed at some point in the future but it was not my intention for today :) It would require a fairly large change to the code and I'd prefer to wait until there is a case where it's required to facilitate better testing.

Thanks for the confirmation of that thought :) and how it is currently it really is working very nicely for the above case, especially if it should also cover the long XMTZC04HM model condition.

It will keep looping on however many conditions you can fit :)

That will nicely shorten the iBeacon decoder :)

This also incorporates your JSON decode specs (removed the extra "va" type )

Now the XMTZC05HM decoder can go back to its original _XMTZC05HM_json, no _va or even _v2 needed any longer.

Thanks again for everything!

[h2zero]

@DigiH If you would like to rebase your code with that branch and PR it with any further modifications please feel free to do so. Might as well keep it all in 1 PR.

[DigiH]

@h2zero thanks, yes, I've already made a few adjustments and will do that.

[DigiH]

Not quite what I expected, diggin into the tests ;)

[h2zero]

Expected: {"brand":"Xiaomi","model":"Mi_Body_Scale_2","model_id":"XMTZC05HM","weighing_mode":"person","unit":"kg","weight":72.45,"impedance":503}
Got JSON: {"brand":"Xiaomi","model":"Mi_Body_Scale_2","model_id":"XMTZC05HM","wgmode":"person","unit":"kg","weight":72.45,"imp":503}

Theres the issue, "wgmode" != "weighing_mode"

[DigiH]

Thanks, yes , I've already changed and submitted that, also imp-impedance, but stillm failing. Just checking the order of the tests now, if things got mixed up there