Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] NAS-WR01ZE: kWh meter value displayed as negative number #2692

Closed
4 of 7 tasks
andre-richter opened this issue May 24, 2021 · 34 comments
Closed
4 of 7 tasks

[bug] NAS-WR01ZE: kWh meter value displayed as negative number #2692

andre-richter opened this issue May 24, 2021 · 34 comments
Labels
cannot fix ❌ This is something out of our control - like invalid device behavior

Comments

@andre-richter
Copy link

Version

Checklist:

  • I am not using HomeAssistant. Or: a developer has told me to come here.
  • I have checked the troubleshooting section and my problem is not described there.
  • I have read the changelog problem was not mentioned there.

Build/Run method

  • Docker
  • PKG
  • Snap package
  • Manually built (git clone - yarn install - yarn run build )

zwavejs2mqtt version: 4.3.1
zwave-js version: 7.5.1

Describe the bug

The kWh meter of the NAS-WR01ZE (https://devices.zwave-js.io/?jumpTo=0x0258:0x0200:0x1027:2.32) smart plug is reported/displayed as a negative number.

After resetting the accumulated meter values, sane positive values are reported for sometime, but at some point it flips to negative.

zwavejs2mqtt.log:

2021-05-24 20:46:26.806 DEBUG SOCKET: Event ZWAVE_API emitted to FHb1fJMDSyU3lbGJAAAB
2021-05-24 20:46:26.808 INFO ZWAVE: Calling api refreshCCValues with args: [ 42, 50, [length]: 2 ]
2021-05-24 20:46:26.971 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.13 => -21474836.13
2021-05-24 20:46:27.120 INFO ZWAVE: Node 42: value updated: 50-0-value-66049 27.85 => 27.49
2021-05-24 20:46:27.266 INFO ZWAVE: Node 42: value updated: 50-0-value-66561 234.62 => 234.93
2021-05-24 20:46:27.410 INFO ZWAVE: Node 42: value updated: 50-0-value-66817 0.25 => 0.25

zwavejs.log:

2021-05-24T20:46:26.811Z CNTRLR » [Node 042] querying meter value (type = Electric, scale = kWh, rate type = Con
                                  sumed)...
2021-05-24T20:46:26.975Z CNTRLR » [Node 042] querying meter value (type = Electric, scale = W, rate type = Consu
                                  med)...
2021-05-24T20:46:27.123Z CNTRLR » [Node 042] querying meter value (type = Electric, scale = V, rate type = Consu
                                  med)...
2021-05-24T20:46:27.268Z CNTRLR » [Node 042] querying meter value (type = Electric, scale = A, rate type = Consu
                                  med)...

The negative value also gets picked up by Home Assistant.

To Reproduce

Let the device report for some time.

Expected behavior

A sane number being reported.

Additional context

I am unsure if this is a SW issue or a quirk with the device itself, since it looks like bit 31 is being flipped.
The screenshot below was made not too long after a reset of the accumulated values. If I take the negative number and flip bit 31 from 1 to 0, we get 35, which would be 0.35 kWh I presume, and that would make sense given that a reset was done earlier.

This is happening on multiple of the same devices I own, not only a single one.

image

@andre-richter
Copy link
Author

Update:

Caught in the act on another node:

2021-05-24 21:14:30.202 INFO ZWAVE: Node 41: value updated: 50-0-value-65537 0.27 => 0.29
2021-05-24 21:19:30.187 INFO ZWAVE: Node 41: value updated: 50-0-value-65537 0.29 => 0.33
2021-05-24 21:24:30.172 INFO ZWAVE: Node 41: value updated: 50-0-value-65537 0.33 => 0.37
2021-05-24 21:29:30.156 INFO ZWAVE: Node 41: value updated: 50-0-value-65537 0.37 => 0.39
2021-05-24 21:34:30.142 INFO ZWAVE: Node 41: value updated: 50-0-value-65537 0.39 => -21474836.05

@andre-richter
Copy link
Author

Thanks @rickk88!

From what I could see yesterday, it might really just be bit 31 randomly toggling between 1 and 0. Anyone agrees?

@robertsLando would zwave_js be open to implement masking this bit out as a quirk for this particular device?

@robertsLando
Copy link
Member

I think this should be handled on zjs side.
CC
@AlCalzone @blhoward2

@robertsLando robertsLando transferred this issue from zwave-js/zwave-js-ui May 25, 2021
@robertsLando robertsLando removed their assignment May 25, 2021
@AlCalzone
Copy link
Member

@andre-richter any chance you could catch this in a zwave-js log with loglevel debug? I wonder if it is only that bit.

@AlCalzone AlCalzone added the investigate 🔎 Not sure what's going on here - need to investigate label May 25, 2021
@andre-richter
Copy link
Author

@AlCalzone, are you particularly interested in the event of flipping from sane to negative value or something else?
I could set up that experiment later, and increase the reporting interval to report every minute or so.

In the meantime, another experiment completed, giving more insight. I have one of those devices connected to a consumer that constantly drains ~27 W. I had set the reporting interval to 10 minutes before resetting:

2021-05-25 08:38:43.108 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0 => 0
2021-05-25 08:48:43.034 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0 => 0
2021-05-25 08:58:42.973 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0 => 0
2021-05-25 09:08:42.914 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0 => -21474836.46
2021-05-25 09:18:42.857 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.46 => -21474836.46
2021-05-25 09:28:42.795 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.46 => -21474836.44
2021-05-25 09:38:43.739 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.44 => -21474836.44
2021-05-25 09:48:43.679 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.44 => -21474836.42
2021-05-25 09:58:43.618 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.42 => -21474836.42
2021-05-25 10:08:43.557 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.42 => -21474836.4
2021-05-25 10:18:43.503 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.4 => -21474836.4
2021-05-25 10:28:43.444 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.4 => -21474836.4
2021-05-25 10:38:43.388 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.4 => 0.1
2021-05-25 10:48:43.329 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0.1 => 0.1
2021-05-25 10:58:43.271 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0.1 => -21474836.36

Exactly after 2 hours, it incidentally switched back to a sane value.
We would expect to see 2 h * 27 W = 54 Wh, or 0.05 kWh respectively. The reported value is double, which seems in line with what was reported here: https://community.home-assistant.io/t/energy-consumption-for-neo-coolcam-plug/219844/8.

Taking the last timestamp, which is after 140 minutes, we would expect 140 minutes * 27 W = 63 Wh, aka 0.06 kWh

The reported value is -21474836.36.
Naively converting this (not sure if this is valid, aka what the actual number representation used in the zwave value is):

-21474836.36 * 100 = -2147483636
-2147483636 & 0x7fff_ffff = 12
12 / 100 = 0.12 kWh

Which would again be double the expected value.

@AlCalzone
Copy link
Member

I'm interested in the raw commands that get sent - best case some correct ones and some flipped ones. Make sure to set the loglevel of the zwave-js logger to debug, so we can see this.

Naively converting this (not sure if this is valid, aka what the actual number representation used in the zwave value is):

Seems correct, in Z-Wave floats are just encoded as integers plus precision (# of decimal places).

Based on the history of values I think this device has some big problems with encoding values that aren't divisible by 0.1.

        0.00 => 0.00 ✔
-21474836.46 => 0.02
-21474836.44 => 0.04
-21474836.42 => 0.06
-21474836.40 => 0.08
        0.10 => 0.10 ✔
-21474836.36 => 0.12

Which would again be double the expected value.

Not much we can do about that if the device reports this.

@andre-richter
Copy link
Author

Based on the history of values I think this device has some big problems with encoding values that aren't divisible by 0.1.

        0.00 => 0.00 ✔
-21474836.46 => 0.02
-21474836.44 => 0.04
-21474836.42 => 0.06
-21474836.40 => 0.08
        0.10 => 0.10 ✔
-21474836.36 => 0.12

Some more from this run:

2021-05-25 11:08:43.212 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.36 => -21474836.36
2021-05-25 11:18:43.157 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.36 => -21474836.34
2021-05-25 11:28:43.095 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.34 => -21474836.34
2021-05-25 11:38:43.037 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 -21474836.34 => 0.16
2021-05-25 11:48:42.981 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0.16 => 0.16
2021-05-25 11:58:42.924 INFO ZWAVE: Node 42: value updated: 50-0-value-65537 0.16 => 0.18

Which would again be double the expected value.

Not much we can do about that if the device reports this.

So some correcting math on the value on retrieval is not feasible? I can hack it into my home-assistant installation, but before I go there, I'm trying to understand to what extend zwave_js is the correct place to correct such awful quirks.

Trying to set up your experiment now.

@AlCalzone
Copy link
Member

I'm trying to understand to what extend zwave_js is the correct place to correct such awful quirks.

Generally we do not try to fix application level device quirks in the driver for two reasons:

  1. While these values seem obviously wrong, it is impossible to tell automatically whether the value was reported like this on purpose or not
  2. It is not our job to clean up the manufacturers' mess. Sure, we work around some things to make the devices usable at all, but fiddling around with the reported values doesn't belong in this project.

Other projects (like SmartThings) have done these things by writing device-specific integrations, which in the end just leads to the manufacturers saying "but it works in XYZ 🤷🏻‍♂️" when in reality they are just too lazy to fix their shit. The stuff above would have been caught by QA if they bothered to test it. Maybe having a majority of their users complain because their stuff does not work with a standards-compliant software will change that mentality.

Some manufacturers do care - I've already seen a case where we had a beta firmware version to test just hours after a bug report.

@andre-richter
Copy link
Author

I don't know what is different this time. Currently having a run from 0 to 0.14 kWh for almost 3 hours without any negative value. Will update once there is a repro.

@andre-richter
Copy link
Author

@AlCalzone
Copy link
Member

Oh, the messages are encrypted. I need your network key to decode them - you can send it via mail to the one in my profile.

@AlCalzone
Copy link
Member

It actually is just the one bit that flips. This seems to be the same in the other bug reports that can be found all over Google.

If you ask me, you should complain to the manufacturer about this and ask for a firmware update or return it or something.

@AlCalzone AlCalzone added cannot fix ❌ This is something out of our control - like invalid device behavior and removed investigate 🔎 Not sure what's going on here - need to investigate labels May 26, 2021
@andre-richter
Copy link
Author

Fair enough, I'll work around it in SW on my local HA overlay. I have little hope to see FW updates for such budget devices, but lets see (the first of the two support email addresses already replied my email wasn't delivered because the inbox quota was full 😅 )

@Minims
Copy link

Minims commented Dec 30, 2021

@AlCalzone Do you think a workaround like that can be implemented ?
domoticz/open-zwave#3

@blhoward2
Copy link
Collaborator

blhoward2 commented Dec 30, 2021

As a rule we don't implement device-specific workarounds. Doing so is part of what led to OZW's bloated code. You can use an automation or template a sensor entity in HA on your own to fix this.

@ahochsteger
Copy link
Contributor

Generally we do not try to fix application level device quirks in the driver for two reasons:

  1. While these values seem obviously wrong, it is impossible to tell automatically whether the value was reported like this on purpose or not
  2. It is not our job to clean up the manufacturers' mess. Sure, we work around some things to make the devices usable at all, but fiddling around with the reported values doesn't belong in this project.

As a rule we don't implement device-specific workarounds. Doing so is part of what led to OZW's bloated code. You can use an automation or template a sensor entity in HA on your own to fix this.

@AlCalzone, @blhoward2 I can fully understand and support these statements so zwave-js can keep its high quality and speed of development.
Nevertheless there might be a better approach from a overall smart home architecture perspective than making it a problem of the users (either buy new devices or implementing workarounds that create much more and duplicated complexity on other parts of the smart home system (solving this using template sensors in HA for every affected entity is a huge code duplication mess and a nightmare to maintain).
Wouldn't it be possible to support a kind of layer or hooks to do some device-specific processing before passing the values on to the applications?

@robertsLando, wouldn't the device values configuration of zwavejs2mqtt already allow something like this? But I'm not sure, if this helps, if e.g. Home Assistent is connected directoly to the zwave-js server port of zwavejs2mqtt.

If not, maybe providing something like that as part of the zwave-js server but on top of the zwave-js driver would solve the need (not only to fix device bugs but also to do some other useful pre-processing or conversion of data) while keeping the the concerns of clearly separated from the core.
Maybe the device configuration could be extended to provide device-specific implementations for hooks that are called by the driver or server.

Since I don't know enough about the (existing or still to be created) possibilities on the driver/server level I'd like to hear your opinion on that.

@andre-richter
Copy link
Author

andre-richter commented Jan 14, 2022

I would love to have something like that as well. 👍👍

In the current case, all we would need is to apply some logic and math on the raw device value before passing it on to home assistant or whoever reads it.

A hook in zwavejs2mqtt sounds reasonable?

@andre-richter
Copy link
Author

I have a similar use case for a roller shutter that never reaches fully closed or open in some situations.
I keep o fixing that locally in HA (https://github.com/andre-richter/core/commit/485b3e78a8be12d54c20d75fc0b7891b325bd310) but a hook in the server would do here as well.

@AlCalzone
Copy link
Member

AlCalzone commented Jan 14, 2022

there might be a better approach from a overall smart home architecture perspective than making it a problem of the users (either buy new devices or implementing workarounds)

This may sound cynical but I disagree. It really should become a problem of the manufacturers who release these buggy devices and don't offer any support whatsoever. The huge bunch of workarounds that other drivers implemented (some still do, see SmartThings) make it too easy for manufacturers to reject responsibility and claim "but it works in XYZ, cannot be our fault!"

I can understand the sentiment of making things work when you're using unofficial APIs etc, but Z-Wave is a standard, devices have to be certified and these things still happen too much. Only when we stop solving the manufacturers' mistakes for them and make them fix them, then we can get to a point where it becomes a benefit for the smart home community.

Unfortunately, I don't have the power to make manufacturers do that, so all I can do here is stay strong and say "I will not fix this" (and neither should the other projects), even if it is annoying for users. In the end, the users have the power to make a change:

  1. When there are issues with a device, contact the manufacturer support. Open Source projects are not free tech support for manufacturers.
  2. Hold them accountable. If there is clearly an issue with the device firmware, demand them to fix it. Even if they initially claim its not their fault. European law recently even made it mandatory to provide updates for ~5 years.
  3. Vote with your wallet:
    • Don't buy from manufacturers that don't provide any support
    • Don't buy from manufacturers that don't provide firmware updates or make you pay for them. Nowadays there is really not a reason not to provide them online.
    • Don't buy from manufacturers that tend to seek fault elsewhere ("The device is certified, cannot be our fault...")
    • DO buy from manufacturers that have reliable devices and give good support.
    • Inform yourself before you buy

We can help with the latter - we recently added the ability to display comments in our device configuration DB, but it isn't used much yet:
grafik
https://devices.zwave-js.io/?jumpTo=0x0258:0x0100:0x1027:0.0


Side note: I currently have devices from 3 large manufacturers. Of these 3, I'd only buy from one again and my DO-NOT-BUY list is growing rapidly just by looking at the issues that are reported here.

@andre-richter
Copy link
Author

andre-richter commented Jan 14, 2022

I just checked https://zwave-js.github.io/zwavejs2mqtt/#/usage/setup?id=device-values-configuration some more and it seems this would be almost a solution. I think not all problems could be solved with math operations only, though. At least not as comfortable.

And sometimes you might not want this for ALL devices of the same type (that's what it does?), but just some specific devices?

@AlCalzone since this is in the realm of zwavejs2mqtt, would you be opposed to exploring this more as well? Or is this anyways not "your domain" but @robertsLando's ? :)

@AlCalzone
Copy link
Member

@robertsLando Can do what he likes

@robertsLando
Copy link
Member

For the ones using mqtt devices configuration table can already be used to create a custom send and write functions

@ahochsteger
Copy link
Contributor

For the ones using mqtt devices configuration table can already be used to create a custom send and write functions

@robertsLando would that only affect values sent via MQTT or also affect values sent to HA using the server port integration (posibly not).

@robertsLando
Copy link
Member

It only affect values sent/received via mqtt, I haven't any control on what is done by zwavejs server, it is only another listener of zwave driver events

@knthm
Copy link
Contributor

knthm commented Jan 25, 2022

I've created a Template Sensor that does corrective calculations on the values reported by NAS-WR01ZE devices:

https://gist.github.com/KloudJack/4489ffcbeae58b88072f391dca55b40d

I've been using this successfully with my power sockets until a firmware fix is released, not holding my breath though as this issue has been known for at least 2 years.

@heapxor
Copy link

heapxor commented Mar 22, 2022

Hi,
i am not sure if the template fix the issue? Once applied i can see results as
image

the bottom one is the value from the template.

@MarijnS95
Copy link

@heapxor That gist also mentions this in the top-level comment:

# In addition to random most significant bit flips in the 32-bit energy consumption value reported
# which results in massive negative values, the NAS-WR01ZE also generally computes the energy consumed incorrectly.
# In brief practical tests the actual consumption is actually ~1/4 of the value reported.

It seems to have deduced a correction factor of 0.26 which when multiplied with the original value of 18.48 results in 4.8048:

        {% set correctionFactor = 0.26 %}
...
          {{ ( value * correctionFactor ) | round(roundDecimals) }}

@heapxor
Copy link

heapxor commented Mar 22, 2022

@MarijnS95 ah okay, now i see the point, in that case it seems to be okay. thx.

@heapxor
Copy link

heapxor commented Mar 28, 2022

hello,
i think this is another bug node-zwave-js bug? https://community.home-assistant.io/t/fgwp102-zw5-v3-2-bug-in-reports/406169

@blhoward2
Copy link
Collaborator

blhoward2 commented Mar 28, 2022

Don't comment on a closed issue. Open a new bug report and fill out the template.

@heapxor
Copy link

heapxor commented Mar 28, 2022

okay its here > #4408

thanks

@joshuisken
Copy link

I've created a Template Sensor that does corrective calculations on the values reported by NAS-WR01ZE devices:

https://gist.github.com/KloudJack/4489ffcbeae58b88072f391dca55b40d

I've been using this successfully with my power sockets until a firmware fix is released, not holding my breath though as this issue has been known for at least 2 years.

This gist has disappeared, anyone have a copy?

@knthm
Copy link
Contributor

knthm commented Jul 8, 2024

@joshuisken updated link:
https://gist.github.com/knthm/4489ffcbeae58b88072f391dca55b40d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cannot fix ❌ This is something out of our control - like invalid device behavior
Projects
None yet
Development

No branches or pull requests