diff --git a/README.md b/README.md index 00819cc6..ddc6452c 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,9 @@ # *Highlights* -`monitor` sends a JSON-formatted MQTT message including a confidence value from 0 to 100 to a specified broker when a specified Bluetooth device responds to a `name` query. By default, `name` queries are triggered after receiving an anonymous advertisement from a previously-unseen device. +`monitor` sends a JSON-formatted MQTT message including a confidence value from 0 to 100 to a specified broker when a specified Bluetooth device responds to a `name` query. By default, `name` queries are triggered after receiving an anonymous advertisement from a previously-unseen device (e.g., a device in peripheral mode advertising an ability to connect). -Example: +Example JSON package: ``` topic: monitor/{{name of monitor install}}/{{mac address}} message: { @@ -29,8 +29,7 @@ message: { In addition, optionally, a JSON-formatted MQTT message can be reported to the same broker whenever a publicly-advertising beacon device or an iBeacon device advertises. -Example: - +Example JSON package: ``` topic: monitor/{{name of monitor install}}/{{mac address or ibeacon uuid}} message: { @@ -54,7 +53,7 @@ ___ # *Oversimplified Analogy of the Bluetooth Presence Problem* -Imagine you're blindfolded in a large room with other people. We want to find out who of your friends **is** there and who of your friends **isn't** there: +Imagine you're blindfolded in a large room with other people. We want to find out who of your friends **is** present and who of your friends **isn't** present: ![First Picture](https://i.imgur.com/FOubz6T.png) @@ -62,92 +61,94 @@ Some of the people in the room periodically make sounds (e.g., eating a chip, sn ![Second Picture](https://i.imgur.com/UwPJIMM.png) -Here's the problem. You can’t just shout “WHO’S HERE” because then everyone would say their name at the same time and you couldn’t tell anything apart. That won't work. Obviously, you also can't simply ask "WHO ISN'T HERE." That's a pretty foolish suggestion... - -So, what about taking attendance like in a classroom? Great idea! Everyone in the room agrees beforehand to respond to an attendance call **only** when their own name is shouted. That way you can actually hear whether someone responds. Neat! +Here's the problem. You can’t just shout “WHO’S HERE” because then everyone would say their name at the same time and you couldn’t tell anything apart. Similarly, for obvious reasons, you can't simply ask "WHO ISN'T HERE?" -Since you're blindfolded, you don't know how big the room is. That means you have take attendance loudly because we don't want our friends to not hear their name being called - taking attendance quietly simply won't do. +So, you take attendance like in a classroom. Everyone in the room responds **only** when their own name is shouted. ![Third Picture](https://i.imgur.com/VCW8AmH.png) -Ok, now how can you figure out which of your friends are in the room? If your friends say a name out loud randomly ("[Alan!](https://www.youtube.com/watch?v=RFhNJ8ozDBk)") it’s easy to know if they’re present or absent - all you have to do is listen for them in the din of the crowd. For most of your friends though, you need to take attendance from a list shouting names one at a time. All other sounds you hear in the room are totally anonymous ... you have no idea who made what sound. - -So, one way to take attendance is to shout for each friend on a list by name, one at a time, repeatedly. Shout, get a response, wait for a moment, and ask again. +So, one way to take attendance is to shout for each friend on a list by name, one at a time, repeatedly. Ask for someone, get a response, wait for a moment, and ask again. Once a friend stops responding (for some period of time), you presume that he or she has left: ![Simple Loop](https://i.imgur.com/ijGw2qb.png) -This technique should work just fine, but there's a problem. You're constantly shouting into the room, which means that it's difficult for you to hear quiet responses and it's difficult for other people to carry on conversations. What else can we do? +This technique should work just fine, but there's a minor problem. You're constantly shouting into the room, which means that it's difficult for you to hear quiet responses and it's difficult for other people to carry on conversations. What else can we do? Can we use those random sounds for anything? -A smarter approach is to wait for an anonymous sound, *then* start asking whether your friend is there: +Yes! A smarter approach is to wait for an anonymous sound, *then* start asking whether a friend *you know isn't present* has just arrived: ![Complex Loop](https://i.imgur.com/9Ugn27i.png) +This way, you're not constantly asking the room for all of your friends. Efficient! + This technique is a very simplified description of how `montior` works for devices like cell phones (friends on a list) and beacons (announce a name out loud). This also gives an idea of how `monitor` uses anonymous sounds to reduce the number of times that it has to send inquiries into the Bluetooth environment. ___ -# *Oversimplified Technical Description* +# *Oversimplified Technical Background* -The Bluetooth Low Energy 4.0 spec was designed to make connecting Bluetooth devices simpler for the user. No more pin codes, no more code verifications, no more “discovery mode” - for the most part. It was also designed to be much more private than previous Bluetooth specs. But it’s hard to maintain privacy when you want to be able to connect to an unknown device without user intervention, so a compromise was made. The following is oversimplified and not technically accurate in most cases, but should give the reader a gist of how `monitor` determines presence. +The Bluetooth Low Energy spec was designed to make connecting Bluetooth devices simpler for the user. No more pin codes, no more code verifications, no more “discovery mode” - for the most part. It was also designed to be more private than previous Bluetooth implementations. That said, it’s hard to maintain privacy when you want to be able to connect to an unknown device without intervention. ## Name Requests -A part of the Blueooth spec is a special function called a `name` request that asks another Bluetooth device to send back a human-readable name of itself. In order to send a `name` request, we need to know a private (unchanging) address of the target device. +A part of the Blueooth spec is a special function called a `name` request that asks another Bluetooth device to send back a human-readable name of itself. In order to send a `name` request, however, we need to know a private (unchanging) address of the target device. -Issuing a `name` request to the same private mac address every few seconds is a reliable - albeit rudimentary - way of detecting whether that device is "**present**" (it responds to the `name` request) or "**absent**" (no response to the `name` request is received). However, issuing `name` requests too frequently (*e.g.*, every few seconds) uses quite a bit of 2.4GHz spectrum, which can cause substantial interference with Wi-Fi or other wireless communications. +Issuing a `name` request to the same private mac address every few seconds is a reliable - albeit rudimentary - way of detecting whether that device is "**present**" (it responds to the `name` request) or "**absent**" (no response to the `name` request is received). However, issuing `name` requests too frequently (*e.g.*, every few seconds) uses quite a bit of 2.4GHz spectrum, which can cause interference with Wi-Fi or other wireless communications. -## Connectable Devices +Not all devices respond to `name` requests, however. For example, beacon devices do not respond. -Blueooth devices that can exchange information with other devices (almost always) advertise a random/anonymous address that other devices can use to negotiate a secure connection with that device's real, private, Bluetooth address. +## Connectible Devices -Using a random address when publicly advertising prevents baddies from tracking people via Bluetooth monitoring. This is because monitoring for anonymous advertisement is not a reliable way to detect whether a device is **present** or **absent**. However, nearly all connectible devices respond to `name` requests if made to the device's private Bluetooth address. +Blueooth devices that can exchange information with other devices (almost always) advertise a random/anonymous address that other devices can use to negotiate a secure connection and receive the first device's real, private, Bluetooth address. Using a random address in this way when publicly advertising prevents bad actors from tracking via passive Bluetooth monitoring. -## Beacon Devices +## Beacon/Advertising Devices -The Bluetooth spec has been used by Apple, Google, and others to create additional standards (e.g., iBeacon, Eddystone, and so on). These devices generally don't care to connect to other devices, so their random/anonymous addresses don't really matter. Instead, these devices encode additional information into each advertisement of an anonymous address. For example, iBeacon devices will broadcast a UUID that conforms to the 8-4-4-4-12 format defined by [IETC RFC4122](http://www.ietf.org/rfc/rfc4122.txt). +The Bluetooth spec has been used by Apple, Google, and others to create additional standards (e.g., iBeacon, Eddystone, and so on). These devices generally don't care to connect to other devices, so use of random/anonymous addresses doesn't really matter. Instead, these devices encode additional information into each advertisement of an anonymous address. For example, iBeacon devices will broadcast a UUID that conforms to the 8-4-4-4-12 format defined by [IETC RFC4122](http://www.ietf.org/rfc/rfc4122.txt). -Beacons do not respond to `name` requests, even if made to the device's private Bluetooth address. So, issuing periodic `name` requests to beacons is not a reliable way to detect whether a beacon device is **present** or **absent**. However, monitoring for beacon advertisement is a reliable way to detect whether a beacon device is **present** or **absent**. +As noted above, most beacons do not respond to `name` requests, even if made to the device's private Bluetooth address. So, issuing periodic `name` requests to beacons is not a good way to detect whether a beacon device is **present** or **absent**. However, monitoring for beacon advertisement is a reliable way to detect whether a beacon device is **present** or **absent**. _____ # *How `monitor` Works* -This script combines `name` requests, anonymous advertisements, and beacon advertisements to logically determine (1) *when* to issue a `name` scan to determine whether a device is **present** and (2) *when* to issue a `name` scan to determine whether a device is **absent**. The script also listens for beacons. +This script combines `name` requests, anonymous advertisements, and beacon advertisements to logically determine (1) *when* to issue a `name` request to determine whether a device is **present** and (2) *when* to issue a `name` request to determine whether a device is **absent**. The script also listens for beacons. ##### Known Static Addresses -More specifically, `monitor` accesses private mac addresses that you have added to a file called `known_static_addresses`. These are the addresses for which `monitor` will issue `name` requests to determine whether or not these devices are **present** or **absent**. +`monitor` uses unchanging/static mac addresses for your devices that you have added to a file called `known_static_addresses`. These are the addresses for which `monitor` will issue `name` requests to determine whether or not these devices are **present** or **absent**. -Once a determination of presence is made, the script posts to an mqtt topic path defined in a file called `mqtt_preferences` that includes a JSON-formatted message with a confidence value that corresponds to a confidence of presence. For example, a confidence of 100 means that `monitor` is 100% sure the device is present and present. Similarly, a confidence of 0 means that `monitor` is 0% sure the device is present (*i.e.*, the `monitor` is 100% sure the device is absent). +Once a determination of presence is made, the script posts to an mqtt topic path defined in a file called `mqtt_preferences` that includes a JSON-formatted message with a confidence value that corresponds to a confidence of presence. For example, a confidence of 100 means that `monitor` is 100% sure the device is present. Similarly, a confidence of 0 means that `monitor` is 0% sure the device is present (*i.e.*, the `monitor` is 100% sure the device is absent). -To minimize the number of times that `monitor` issues `name` requests (thereby reducing 2.4GHz interference), the script performs either an ***ARRIVAL*** scan or a ***DEPART*** scan, instead of scanning all devices listed in the `known_static_addresses` each time. More specifically: +To minimize the number of times that `monitor` issues `name` requests (thereby reducing 2.4GHz interference), the script performs either an ***ARRIVAL*** scan or a ***DEPART*** scan, instead of scanning all devices listed in the `known_static_addresses` each time. -* An ***ARRIVAL*** scan issues a `name` request *only* for devices from the `known_static_addresses` file that are known to be **absent**. +More specifically: -* Similarly, a ***DEPART*** scan issues a `name` request *only* for devices from the `known_static_addresses` file that are known to be **present**. +* An ***ARRIVAL*** scan issues a `name` request, sequentially, for each device listed in the `known_static_addresses` file that is known to be **absent**. + +* Similarly, a ***DEPART*** scan issues a `name` request, sequentially, for each device listed in the `known_static_addresses` file that is known to be **present**. For example, if there are two phone addresses listed in the `known_static_addresses` file, and both of those devices are **present**, an ***ARRIVAL*** scan will never occur. Similarly, if both of these addresses are **absent** then a ***DEPART*** scan will never occur. If only one device is present, an **ARRIVAL** scan will only scan for the device that is currently away. -To reduce the number of `name` scans that occur, `monitor` listens for anonymous advertisements and triggers an ***ARRIVAL*** scan for every *new* anonymous address. +To reduce the number of `name` requests that occur, `monitor` listens for anonymous advertisements and triggers an ***ARRIVAL*** scan for every *new* anonymous address. The script will also trigger an ***ARRIVE*** scan in response to an mqtt message posted to the topic of `monitor/scan/arrive`. Advertisement-triggered scanning can be disabled by using the trigger argument if `-ta`, which causes `monitor` to *only* trigger ***ARRIVAL*** scans in response to mqtt messages. If `monitor` has not heard from a particular anonymous address in a long time, `monitor` triggers a ***DEPART*** scan. The script will also trigger a ***DEPART*** scan in response to an mqtt message posted to the topic of `monitor/scan/depart`. Expiration-triggered scanning can be disabled by using the trigger argument if `-td`, which causes `monitor` to *only* trigger ***DEPART*** scans in response to mqtt messages. -To reduce scanning even further, `monitor` can filter which types of anonymous advertisements are used for ***ARRIVE*** scans. These are called "filters" and are defined in a file called `behavior_preferences`. The filters are bash RegEx strings that either pass or reject anonymous advertisements that match the filter. There are two filter types: +To reduce scanning even further, `monitor` can filter which types of anonymous advertisements are used for ***ARRIVE*** scans. These are called "filters" and are defined in a file called `behavior_preferences`. The filters are bash RegEx strings that either pass or reject anonymous advertisements that match the filter. + +There are two filter types: * **Manufacturer Filter** - filters based on data in an advertisement that is connected to a particular device manufacturer. This is almost always the OEM of the device that is transmitting the anonymous advertisement. By default, because of the prevalence of iPhones, Apple is the only manufacturer that triggers an ***ARRIVAL*** scan. Multiple manufacturers can be appended together by a pipe: `|`. An example filter for Apple and Samsung looks like: `Apple|Samsung`. To disable the manufacturer filter, use `.*`. * **Flag Filter:** filters based on flags contained in an advertisement. This varies by device type. By default, because of the prevalence of iPhones, the flag of `0x1b` triggers an ***ARRIVAL*** scan. Like with the manufacturer filter, multiple flags can be appended together by a pipe: `|`. To disable the manufacturer filter, use `.*`. ##### Beacons & iBeacons -In addition, once installed and run with the `-b` beacon argument, `monitor` listens for beacon advertisements that report themselves as "public", meaning that their addresses will not change. The script can track these by default; these addresses do not have to be added anywhere - after all, `monitor` will obtain them just by listening. +In addition, when run with the `-b` beacon argument, `monitor` listens for beacon advertisements that report themselves as "public", meaning that their addresses will not change. The script can track these by default; these addresses do not have to be added anywhere - after all, `monitor` will obtain them just by listening. Since iBeacons include a UUID and a mac address, two presence messages are reported via mqtt. ## Known Beacon Addresses -In some cases, certain manufacturers try to get sneaky and cause their beacons to advertise as "anonymous" (or "random") devices, despite that their addresses do not change at all. By default, `monitor` ignores anonymous devices, so to force `monitor` to recognize these devices, we add the "random" address to a file called `known_static_beacons`. After restarting, `monitor` will know that these addresses should be treated like a normal beacon. +In some cases, manufacturers try to get sneaky and cause their beacons to advertise as "anonymous" (or "random") devices, despite that their addresses do not change at all. By default, `monitor` does not report presence of anonymous advertisement devices, so to force `monitor` to recognize these devices, we add the "random" address to a file called `known_static_beacons`. After restarting, `monitor` will know that these addresses should be treated like a normal beacon. ___ @@ -258,7 +259,7 @@ git clone git://github.com/andrewjfreyer/monitor #enter `monitor` directory cd monitor/ -#switch to beta branch for latest updates and features (may be instable) +#(optional) switch to beta branch for latest updates and features (may be unstable) git checkout beta ``` @@ -473,7 +474,7 @@ Similarly, we can create a negative filter. If you or your neighbors use Google PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="Google" ``` -Filters are a great way to minimize the frequency of `name` scanning, which causes 2.4GHz interference and can, if your values are too aggressive, dramatically interfere with Wi-Fi and other services. +Filters are a great way to minimize the frequency of `name` requestning, which causes 2.4GHz interference and can, if your values are too aggressive, dramatically interfere with Wi-Fi and other services. 2. **Standard configuration options:** @@ -498,7 +499,7 @@ In addition to the options described above, there are a number of advanced optio | **Option** | **Default Value** | **Description** | |-|-|-| -PREF_INTERSCAN_DELAY|3|This is a fixed delay between `name` scans. Increasing the value will decrease interference, but will decrease responsiveness. Decreasing the value will risk a Bluetooth hardware fault.| +PREF_INTERSCAN_DELAY|3|This is a fixed delay between `name` requests. Increasing the value will decrease interference, but will decrease responsiveness. Decreasing the value will risk a Bluetooth hardware fault.| PREF_RANDOM_DEVICE_EXPIRATION_INTERVAL|75|This is the interval after which an anonymous advertisement mac address is considered expired. Increasing this value will reduce arrival scan frequency, but will also increase memory footprint (minimal) and will decrease the frequency of depart scans.| PREF_RSSI_CHANGE_THRESHOLD|-20|If a beacon's rssi changes by at least this value, then the beacon will be reported again via mqtt.| PREF_RSSI_IGNORE_BELOW|-75|If an anonymous advertisement is "farther" away (lower RSSI), ignore the advertisement @@ -541,33 +542,6 @@ message: -99 through 0 If an rssi measurement cannot be obtained, the value of -99 is sent. -3. Using the rssi data for something: - -I strongly recommend using a filter to smooth the rssi data. An example for Home Assistant follows: - - -```yaml -sensor: - - - platform: mqtt - state_topic: 'location/first floor/34:08:BC:15:24:F7/rssi' - name: 'Andrew First Floor RSSI raw' - unit_of_measurement: 'dBm' - - - platform: filter - name: "Andrew First Floor RSSI" - entity_id: sensor.andrew_first_floor_rssi_raw - filters: - - filter: outlier - window_size: 2 - radius: 1.0 - - filter: lowpass - time_constant: 2 - - filter: time_simple_moving_average - window_size: 00:01 - precision: 1 -``` - Anything else? Post a [question.](https://github.com/andrewjfreyer/monitor/issues/new) diff --git a/monitor.sh b/monitor.sh index da7fa01a..ab192f20 100644 --- a/monitor.sh +++ b/monitor.sh @@ -25,7 +25,7 @@ # ---------------------------------------------------------------------------------------- #VERSION NUMBER -export version=0.2.187 +export version=0.2.196 #COLOR OUTPUT FOR RICH OUTPUT ORANGE=$'\e[1;33m' @@ -1103,7 +1103,7 @@ while true; do "type=KNOWN_MAC" done - elif [[ $mqtt_topic_branch =~ .*NEW\ STATIC\ DEVICE.* ]] || [[ $mqtt_topic_branch =~ .*DELETE\ STATIC\ DEVICE.* ]]; then + elif [[ $mqtt_topic_branch =~ .*ADD\ STATIC\ DEVICE.* ]] || [[ $mqtt_topic_branch =~ .*DELETE\ STATIC\ DEVICE.* ]]; then if [[ "${data_of_instruction^^}" =~ ([A-F0-9]{2}:){5}[A-F0-9]{2} ]]; then #GET MAC ADDRESSES @@ -1111,7 +1111,7 @@ while true; do if [ ! ${known_public_device_name[$mac]+true} ]; then #HERE, WE KNOW THAT WE HAVE A MAC ADDRESS AND A VALID INSTRUCTION - if [[ $mqtt_topic_branch =~ .*NEW\ STATIC\ DEVICE.* ]]; then + if [[ $mqtt_topic_branch =~ .*ADD\ STATIC\ DEVICE.* ]]; then #WAS THERE A NAME HERE? name=$(echo "$data_of_instruction" | tr "\\t" " " | sed 's/ */ /gi;s/#.\{0,\}//gi' | sed "s/$mac //gi;s/ */ /gi" ) diff --git a/support/README.md b/support/README.md index 517585b2..fdc02999 100644 --- a/support/README.md +++ b/support/README.md @@ -1,39 +1,32 @@ -# *Frequenty Asked Questions:* -[***Basics***](#monitor-basics) +## *Basics* -[***Wi-Fi Interference & Performance***](#wi-fi-interference--performance-issues) +
Will this be able to track my Apple Watch/Smart Watch?

-[***Debugging***](#monitor-logs--debugging) +Yes, with a caveat. Many users, including myself, have successfully added Apple Watch Bluetooth addresses to the `known_static_addresses` file. In my personal experience, an Apple Watch works just fine [once it has connected to at least one other Bluetooth device, apart from your iPhone](https://github.com/andrewjfreyer/monitor#my-phone-doesnt-seem-to-automatically-broadcast-an-anonymous-bluetooth-advertisement-what-can-i-do). Other users have reported that the Apple Watch will occasionally not respond to this script. Your mileage using the Apple Watch and/or other low-power connectible Bluetooth devices may vary. I strongly recommend tracking phones. -[***Filters***](#filters) +

-[***Miscellaneous***](#other-questions) - -____ - -## *Monitor Basics* - -#### Will this be able to track my Apple Watch/Smart Watch? - -Yes, with a caveat. Many users, including myself, have successfully added Apple Watch Bluetooth addresses to the `known_static_addresses` file. In my personal experience, an Apple Watch works just fine [once it has connected to at least one other Bluetooth device, apart from your iPhone](https://github.com/andrewjfreyer/monitor#my-phone-doesnt-seem-to-automatically-broadcast-an-anonymous-bluetooth-advertisement-what-can-i-do). Other users have reported that the Apple Watch will occasionally not respond to `monitor`. Your mileage using the Apple Watch and/or other low-power connectible Bluetooth devices may vary. I strongly recommend tracking phones. - -#### What special app do I need on my phone to get this to work? +
What special app do I need on my phone to get this to work?

None, except in rare circumstances. The only requirement is that Bluetooth is left on. Works best with iPhones and Android phones that have peripheral mode enabled. +

-#### Does `monitor` reduce battery life for my phone? +
Does this script reduce battery life for my phone?

Not noticeable in my several years of using techniques similar to this. +

-#### How can I trigger an arrival scan? +
How can I trigger an arrival scan?

Post a message with blank content to `monitor/scan/arrive` +

-#### How can I trigger an depart scan? +
How can I trigger an depart scan?

Post a message with blank content to `monitor/scan/depart` +

-#### How can I trigger an arrive/depart scan from an automation in Home Assistant? +
How can I trigger an arrive/depart scan from an automation in Home Assistant?

For an automation or script (or other service trigger), use: @@ -48,20 +41,24 @@ For an automation or script (or other service trigger), use: data: topic: location/scan/depart ``` +

-#### How can I add a known device without manually entering an address? +
How can I add a known device without manually entering an address?

Post a message with the mac address separated from an alias (optional) by a space to: `monitor/setup/add known device` +

-#### How can I delete a known device without manually editing an address? +
How can I delete a known device without manually editing an address?

Post a message with the mac address to: `monitor/setup/delete known device` +

-#### How can I upgrade to the latest version without using ssh? +
How can I upgrade to the latest version without using ssh?

Post a message with blank content to `monitor/scan/update` or `monitor/scan/updatebeta` +

-#### How can I restart a `monitor` node? +
How can I restart a this script node?

Via command line: @@ -70,94 +67,115 @@ sudo systemctl restart monitor ``` Or, post a message with blank content to `monitor/scan/restart` +

-#### Why don't I see RSSI for my iPhone/Andriod/whatever phone? +
Why don't I see RSSI for my device?

-See the RSSI section of this FAQ. You'll have to connect your phone to `monitor` first. +For phones, you'll have to connect to `monitor` first using the `-c` flag. +

-#### How do I force an RSSI update for a known device, like my phone? +
How do I force an RSSI update for a known device, like my phone?

Post a message with blank content to `monitor/scan/rssi` +

+ ____ -## *Wi-Fi Interference & Performance Issues* +## *Performance* -#### I'm running 5GHz Wi-Fi, I don't use Bluetooth for anything else, and I don't care whether I interfere with my neighbor's devices. Can't I just issue a name scan every few seconds to get faster arrival and depart detection? +
Can't I just issue a name scan every few seconds to get faster arrival and depart detection?

Yes, use periodic scanning mode with `-r`. +

-#### Can I use other Bluetooth services while `monitor` is running? +
Can I use other Bluetooth services while this script is running?

No. Monitor needs exclusive use of the Bluetooth radio to function properly. This is why it is designed to run on inexpensive hardware like the Raspberry Pi Zero W. +

-#### Can `monitor` run on XYZ hardware or in XYZ container? +
Can this script run on XYZ hardware or in XYZ container?

Probably. The script has been designed to minimize dependencies as much as possible. That said, I can't guarantee or provide support to all systems. +

-#### Does `monitor` interfere with Wi-Fi, Zigbee, or Zwave? +
Does this script interfere with Wi-Fi, Zigbee, or Zwave?

-It can, if it scans too frequently, especially if you're running `monitor` from internal Raspberry Pi radios. Try to use all techniques for reducing `name` scans, including using trigger-only depart mode `-tdr`. When in this mode, `monitor` will never scan when all devices are home. Instead, `monitor` will wait until a `monitor/scan/depart` message is sent. Personally, I use my front door lock as a depart scan trigger. +It can, if it scans too frequently, especially if you're running this script from internal Raspberry Pi radios. Try to use all techniques for reducing `name` scans, including using trigger-only depart mode `-tdr`. When in this mode, this script will never scan when all devices are home. Instead, this script will wait until a `monitor/scan/depart` message is sent. Personally, I use my front door lock as a depart scan trigger. +

-#### How can I check if a `monitor` node is up and hasn't shut down for some reason? +
How can I check if a this script node is up and hasn't shut down for some reason?

Post a message to `monitor/scan/echo`, and you'll receive a response at the topic `$mqtt_topicpath/$mqtt_publisher_identity/echo` +

-#### I *still* have interference and/or my ssh sessions to the raspberry pi are really slow and laggy. What gives? +
I have interference and/or my ssh sessions are really slow and laggy. What gives?

Cheap Wi-Fi chipsets and cheap Bluetooth chipsets can perform poorly together if operated at the same time, especially on Raspberry Pi devices. If you still experience interference in your network, switching to a Wi-Fi dongle can help. +

+ +
I use a Bluetooth dongle, and this script seems to become non-responsive after a while - what's going on?

-#### I use a Bluetooth dongle, and `monitor` seems to become non-responsive after a while - what's going on? +Many Bluetooth dongles do not properly filter out duplicate advertisements, so this script gets overwhelmed trying to filter out hundreds of reports, when it expects dozens. I'm working on a solution, but for now the best option is to switch to internal Bluetooth or, alternatively, you can try another Bluetooth dongle. +

-Many Bluetooth dongles do not properly filter out duplicate advertisements, so `monitor` gets overwhelmed trying to filter out hundreds of reports, when it expects dozens. I'm working on a solution, but for now the best option is to switch to internal Bluetooth or, alternatively, you can try another Bluetooth dongle. ___ -## *Monitor Logs & Debugging* +## *Debugging* -#### I keep seeing that my Bluetooth hardware is "cycling" in the logs - what does that mean? +
I keep seeing that my Bluetooth hardware is "cycling" in the logs - what does that mean?

If more than one program or executable try to use the Bluetooth hardware at the same time, your Bluetooth hardware will report an error. To correct this error, the hardware needs to be taken offline, then brought back. +

-#### I can't do **XYZ**, is `monitor` broken? +
How do I access logs?

Run via command line and post log output to github. Else, access `journalctl` to show the most recent logs: ```bash journalctl -u monitor -r ``` +

-#### My Android phone doesn't seem to send any anonymous advertisements, no matter what I do. Is there any solution? +
My Android phone doesn't seem to send any anonymous advertisements, no matter what I do. Is there any solution?

-Some phones, like the LG ThinQ G7 include an option in settings to enable file sharing via bluetooth. As resported by Home Assistant forum user @jusdwy, access this option via Settings >Connected Devices > File Sharing > File Sharing ON. For other android phones, an app like [Beacon Simulator](https://play.google.com/store/apps/details?id=net.alea.beaconsimulator&hl=en_US) may be a good option. You may also be able to see more information about Bluetooth on your phone using [nRF Connect](https://play.google.com/store/apps/details?id=no.nordicsemi.android.mcp&hl=en_US). +Some phones, like the LG ThinQ G7 include an option in settings to enable file sharing via bluetooth. As resported by Home Assistant forum user @jusdwy, access this option via Settings >Connected Devices > File Sharing > File Sharing ON. For other android phones, an app like [Beacon Simulator](https://play.google.com/store/apps/details?

id=net.alea.beaconsimulator&hl=en_US) may be a good option. You may also be able to see more information about Bluetooth on your phone using [nRF Connect](https://play.google.com/store/apps/details?

id=no.nordicsemi.android.mcp&hl=en_US). Unfortunately, until Android OS includes at least one service that requires bluetooth peripheral mode to be enabled, Android devices will probably not advertise without an application running in the background. In short, as I understand it, Android/Google has been slow to adopt BTLE peripheral mode as an option in addition to the default central mode. [Here is a decently comprehensive list of phones that support peripheral mode](https://altbeacon.github.io/android-beacon-library/beacon-transmitter-devices.html), should an application choose to leverage the appropriate API. It does not appear as though the native OS has an option (outside of the file sharing option mentioned above on LG phones) to enable this mode. Unfortunately, it seems to me that absent an application causing an advertisement to send, Android users will not be able to use monitor in the same way as iOS users or beacon users. +

-#### My phone doesn't seem to automatically broadcast an anonymous Bluetooth advertisement ... what can I do? +
My phone doesn't seem to automatically broadcast an anonymous Bluetooth advertisement ... what can I do?

Many phones will only broadcast once they have already connected to *at least one* other Bluetooth device. Connect to a speaker, a car, a headset, or `monitor.sh -c [address]` and try again. +

-#### I have connected my phone to Bluetooth devices before but my phone doesn't seem to automatically broadcast an anonymous Bluetooth advertisement ... what can I do? - -See above. - -#### Why does my MQTT broker show connection and disconnection so often? +
Why does my MQTT broker show connection and disconnection so often?

This is normal behavior for `mosquitto_pub` - nothing to worry about. +

-#### I updated and `monitor` is no longer working ... what gives? +
I updated and this script is no longer working ... what gives?

Make sure you've updated `mosquitto` to v1.5 or higher. In order to support a wider userbase, backward compatibility for old versions of `mosquitto` was dropped. It is alos strongly recommended that you upgrade to bash 4.4+. +

+ +
I keep seeing MQTT Broker Offline messages in the this script log. What's going on?

+ +mosquitto fails to connect to a broker if your password has certain special characters such as: `@`, `:`,`/` - if this is the case, the easiest solution is to create a new user for this script with a different password. +

-#### I keep seeing MQTT Broker Offline messages in the `monitor` log. What's going on? +
Can I use a certfile for mosquitto instead of my password?

+ +Yes, specify a path for `mqtt_certificate_path` in mqtt_preferences. +

-mosquitto fails to connect to a broker if your password has certain special characters such as: `@`, `:`,`/` - if this is the case, the easiest solution is to create a new user for `monitor` with a different password. ____ ## *Filters* -#### What filters do you personally use? +
What filters do you personally use?

```bash @@ -169,12 +187,14 @@ PREF_PASS_FILTER_MANUFACTURER_ARRIVE=\"Apple\" PREF_FAIL_FILTER_MANUFACTURER_ARRIVE=\"Google|Samsung\" PREF_FAIL_FILTER_MANUFACTURER_ARRIVE=\"NONE\" ``` +

-#### What are the default filters for the PDU filter option? +
What are the default filters for the PDU filter option?

```ADV_IND|ADV_SCAN_IND|ADV_NONCONN_IND|SCAN_RSP``` +

-#### How do I use this as a device_tracker, in addition to the standard confidence messages? +
How do I use this as a device_tracker, in addition to the standard confidence messages?

Set the option `PREF_DEVICE_TRACKER_REPORT` in your `behavior_preferences` file to true. If it's not there, add a line like this: @@ -184,7 +204,7 @@ PREF_DEVICE_TRACKER_REPORT=true Then, an additional mqtt message will be posted to the topic branch ending in `/device_tracker` -So, as an example for a `monitor` node named "first floor", a device tracker configuration for Home Assistant can look like: +So, as an example for a this script node named "first floor", a device tracker configuration for Home Assistant can look like: ```yaml @@ -195,26 +215,28 @@ device_tracker: ``` The standard confidence report will also send. +

-#### How do I determine what values to set for filters? +
How do I determine what values to set for filters?

-Try using the verbose logging option `-V` to see what `monitor` sees when a new bluetooth device advertisement is seen. Then, power cycle the bluetooth radio on the device you'd like to track - you'll probably see a pattern develop with flags or manufacturers. Use these values to create your arrival filters! +Try using the verbose logging option `-V` to see what this script sees when a new bluetooth device advertisement is seen. Then, power cycle the bluetooth radio on the device you'd like to track - you'll probably see a pattern develop with flags or manufacturers. Use these values to create your arrival filters! Similarly, to set exclude filters, you can observe bluetooth traffic for a period of time to see what devices you simply do not care about seeing. +

____ -## *Other Questions* +## *Other* -#### It's annoying to have to keep track of mac addresses. Can't I just use a nickname for the mac addresses for MQTT topics? +
It's annoying to have to keep track of mac addresses. Can't I just use a nickname for the mac addresses for MQTT topics?

-Yes, this is now default behavior. All you have to do is provide a name next to the address in the `known_static_addresses` file. For example, if you have a known device with the mac address of 00:11:22:33:44:55 that you would like to call "Andrew's Phone": +Yes, this is default behavior. All you have to do is provide a name next to the address in the `known_static_addresses` file. For example, if you have a known device with the mac address of 00:11:22:33:44:55 that you would like to call "Andrew's Phone": ```bash 00:11:22:33:44:55 Andrew's iPhone ``` -Then restart the `monitor` service. The script will now use "andrew_s_iphone" as the final mqtt topic path component. +Then restart the this script service. The script will now use "andrew_s_iphone" as the final mqtt topic path component. ***Important:*** @@ -229,12 +251,14 @@ The same is true for beacons in the `known_beacon_addresses` file as well: ``` To disable this feature, set `PREF_ALIAS_MODE=false` in your `behavior_preferences` file. +

-#### I don't care about a few devices that are reporting. Can I block them? +
I don't care about a few devices that are reporting. Can I block them?

Yes. Create a file called `address_blacklist` in your configuration directory and add the mac addresses you'd like to block (or uuid-major-minor for iBeacons) one at a time. +

-#### I can't use the `device_tracker` platform with the default status strings of `home` and `not_home` with my home automation software. What can I do? +
I can't use the device_tracker platform with the default status strings of `home` and `not_home` with my home automation software. What can I do?

Set these options in `behavior_preferences`: @@ -268,4 +292,6 @@ Generic: PREF_DEVICE_TRACKER_HOME_STRING='home' PREF_DEVICE_TRACKER_AWAY_STRING='away' PREF_DEVICE_TRACKER_TOPIC_BRANCH='anything you like' -``` \ No newline at end of file +``` + +

diff --git a/support/log b/support/log index 868f4b86..323f1d59 100644 --- a/support/log +++ b/support/log @@ -45,9 +45,6 @@ log_listener () { #IS THIS LINE DUPLICATED? if [ "$last_log_line" == "$line" ] ; then duplicate_log_count=$((duplicate_log_count + 1 )) - line_append=" (x$duplicate_log_count)" - should_repeat=$REPEAT - else duplicate_log_count=1 fi @@ -58,7 +55,7 @@ log_listener () { fi #ECHO LAST LINE - printf "${should_repeat}$launch_flag $version $(date "+%d-%m-%Y %I:%M:%S %p") $line$line_append\n" + printf "%s\n" "$launch_flag $version $(date "+%d-%m-%Y %I:%M:%S %p") $line" last_log_line="$line" done < log_pipe diff --git a/support/mqtt b/support/mqtt index 31ec3e9e..373faae5 100644 --- a/support/mqtt +++ b/support/mqtt @@ -16,6 +16,10 @@ # UTILITY FUNCTION FOR JOINING STRINGS # ---------------------------------------------------------------------------------------- +#GLOBALS +last_error_message="" +duplicate_error_count=1 + #UTILITY FUNCTIONS function join_string () (IFS=$1; shift; printf "$*"); function json_keypair () (join_string ":" "\"$1\"" "\"$2\"") @@ -44,13 +48,14 @@ function json_format (){ # MQTT ANNOUNCE ONLINE # ---------------------------------------------------------------------------------------- mqtt_announce_online(){ + #ANNOUCEE HEALTH - $mosquitto_pub_path \ + mqtt_error_handler $($mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $mqtt_version_append \ $mqtt_ca_file_append \ -L "$mqtt_url$mqtt_topicpath/$mqtt_publisher_identity/status" \ - -m "online" -q "2" 2>&1 + -m "online" -q "2" 2>&1) } # ---------------------------------------------------------------------------------------- @@ -58,13 +63,13 @@ mqtt_announce_online(){ # ---------------------------------------------------------------------------------------- mqtt_echo(){ #ANNOUCEE HEALTH - $mosquitto_pub_path \ + mqtt_error_handler $($mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $mqtt_version_append \ $mqtt_ca_file_append \ -L "$mqtt_url$mqtt_topicpath/$mqtt_publisher_identity/echo" \ -m "ok" \ - -q "2" 2>&1 + -q "2" 2>&1) } @@ -80,7 +85,7 @@ mqtt_broker_clean (){ while read instruction; do #ERROR HANDLING - [[ ${instruction^^} =~ .*REFUSED.* ]] && return 0 + mqtt_error_handler "$instruction" && break #EXTRACT TOPIC PATH FROM FORMATTED MQTT MESSAGE topic_path_of_instruction="${instruction%%|*}" @@ -113,6 +118,8 @@ mqtt_broker_clean (){ mqtt_listener (){ #ANNOUNCE ONLINE PRESENCE mqtt_announce_online + + #DEFINE LOCALS local topic_data local topic_path @@ -121,7 +128,7 @@ mqtt_listener (){ while read instruction; do #ERROR HANDLING - [[ ${instruction^^} =~ .*REFUSED.* ]] && break + mqtt_error_handler "$instruction" && break #PRINT THE INSTRUCTION BACK TO THE MAIN THREAD printf "MQTT$instruction\n" > main_pipe @@ -137,14 +144,60 @@ mqtt_listener (){ --will-topic "$mqtt_topicpath/$mqtt_publisher_identity/status" \ --will-payload "offline" 2>&1 ) - #LOG ERROR - log "[CMD-ERRO] ${RED}mqtt broker went offline, or a password/username combination was rejected.${NC}" - + #NEED TO RESUBSCRIBE sleep 10 done } +# ---------------------------------------------------------------------------------------- +# MQTT ERROR HANDLER +# ---------------------------------------------------------------------------------------- +mqtt_error_handler () { + local received="$*" + local return_value + local print_message="" + + #SET RETURN VALUE TO TRUE + return_value=1 + + if [ -n "$received" ]; then + #ERRORS + [[ ${received^^} =~ .*REFUSED.* ]] && return_value=1 && print_message="mqtt broker refused connection - check username, password, and host address" + [[ ${received^^} =~ .*NETWORK.* ]] && return_value=0 && print_message="network is down. enqueuing command to try again after a delay" + + if [ -n "$last_error_message" ] && [ "$last_error_message" == "$print_message" ]; then + #HERE, WE HAVE A REPEATED ERROR + duplicate_error_count=$((duplicate_error_count + 1 )) + last_error_message="$print_message" + + if [ "$duplicate_error_count" -gt "3" ]; then + log "${RED}[CMD-ERRO]${NC} ${RED}fatal mqtt error - messages may not be delivered as intended ($print_message / $duplicate_error_count)${NC}" + duplicate_error_count=0 + last_error_message="" + fi + + #SET TO TRUE TO CAUSE A LOOP OF + return_value=0 + + elif [ -n "$print_message" ]; then + #MESSAGE IS NOT REPEATED, SO FEEL FREE TO LOG IT + log "${YELLOW}[CMD-MQTT]${NC} ${YELLOW}warning: $print_message ${NC}" + duplicate_error_count=0 + last_error_message="$print_message" + + #SET TO TRUE TO CAUSE A LOOP OF + return_value=0 + fi + else + duplicate_error_count=0 + last_error_message="" + fi + + #RETURN VALUE + return $return_value +} + # ---------------------------------------------------------------------------------------- # PUBLISH RSSI MESSAGE # ---------------------------------------------------------------------------------------- @@ -156,11 +209,10 @@ publish_rssi_message () { local address local message local mqtt_topic_branch - + #SET ISOLATED ADDRESS address="$1" message="$2" - mqtt_topic_branch="$address" #ALIASES? @@ -168,15 +220,17 @@ publish_rssi_message () { local topic="$mqtt_topicpath/$mqtt_publisher_identity/$mqtt_topic_branch/rssi" - $PREF_VERBOSE_LOGGING && log "${YELLOW}[CMD-POST]${NC} ${YELLOW}$topic${NC}" + $PREF_VERBOSE_LOGGING && log "${YELLOW}[CMD-MQTT]${NC} ${YELLOW}$topic${NC}" #POST TO MQTT - ($mosquitto_pub_path \ + while mqtt_error_handler $($mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $mqtt_version_append \ $mqtt_ca_file_append \ -L "$mqtt_url$topic" \ - -m "$message" 2>&1) + -m "$message" 2>&1); do + sleep 15 + done fi } @@ -194,8 +248,10 @@ publish_presence_message () { local message local existing_alias local mqtt_topic_branch + local confidence_printable local device_tracker_message + #SET ISOLATED ADDRESS isolated_address="${1##*=}" mqtt_topic_branch="$isolated_address" @@ -249,71 +305,104 @@ publish_presence_message () { #ONLY POST A MESSAGE TO DEVICE TRACKER BOARD IF WE HAVE #A COMPLETE CONFIDENCE MESSAGE if [ -n "$device_tracker_message" ]; then - #PRINT FOR DEVICE TRACKER - log "${YELLOW}[CMD-POST]${NC} ${YELLOW}$topic/$PREF_DEVICE_TRACKER_TOPIC_BRANCH${NC}" - $PREF_VERBOSE_LOGGING && printf "%s\n" "${YELLOW}$topic/$PREF_DEVICE_TRACKER_TOPIC_BRANCH $device_tracker_message${NC}" - #POST TO MQTT - ($mosquitto_pub_path \ + while mqtt_error_handler $($mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $should_retain \ $mqtt_version_append \ $mqtt_ca_file_append \ -L "$mqtt_url$topic/$PREF_DEVICE_TRACKER_TOPIC_BRANCH" \ - -m "$device_tracker_message" 2>&1) + -m "$device_tracker_message" 2>&1); do + sleep 10 + done + + #PRINT FOR DEVICE TRACKER + if [ $PREF_VERBOSE_LOGGING == true ]; then + log "${YELLOW}[CMD-MQTT]${NC} ${YELLOW}$topic/$PREF_DEVICE_TRACKER_TOPIC_BRANCH $device_tracker_message${NC}" + + elif [[ "${message,,}" =~ \"confidence\":\"([0-9]{1,3})\" ]]; then + + #GET CONFIDENCE VALUE + confidence_printable="${BASH_REMATCH[1]}" + + #EXTRACT CONFIDENCE VALUE TO POST + log "${YELLOW}[CMD-MQTT]${NC} ${YELLOW}$topic/$PREF_DEVICE_TRACKER_TOPIC_BRANCH { ... confidence : $confidence_printable ... } ${NC}" + fi fi fi - #DEBUGGING WITH SIMPLE - log "${YELLOW}[CMD-POST]${NC} ${YELLOW}$topic${NC}" - $PREF_VERBOSE_LOGGING && printf "%s\n" "${YELLOW}$(json_unformat "$message")${NC}" - #POST TO MQTT - ($mosquitto_pub_path \ + while mqtt_error_handler $($mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $should_retain \ $mqtt_version_append \ $mqtt_ca_file_append \ -q 2 \ -L "$mqtt_url$topic" \ - -m "$message" 2>&1) + -m "$message" 2>&1); do + sleep 5 + done + + + #MESSAGE PRINTING + if [ $PREF_VERBOSE_LOGGING == true ]; then + + + log "${YELLOW}[CMD-MQTT]${NC} ${YELLOW}$topic${NC}" + #REDACTIONS? + if [ "$PREF_REDACT" == true ]; then + printf "%s\n" "${YELLOW}$(json_unformat "$message" | sed "s/\([0-9A-Fa-f]\{2\}:\)\{5\}/ [REDACTED]:/gi;s/\([0-9A-Fa-f-]\{36\}\)/ [REDACTED]/gi" )${NC}" + else + printf "%s\n" "${YELLOW}$(json_unformat "$message")${NC}" + fi + + #EXTRACT RAW CONFIDENC + elif [[ "${message,,}" =~ \"confidence\":\"([0-9]{1,3})\" ]]; then + + #GET CONFIDENCE VALUE + confidence_printable="${BASH_REMATCH[1]}" + + #EXTRACT CONFIDENCE VALUE TO POST + log "${YELLOW}[CMD-MQTT]${NC} ${YELLOW}$topic { ... confidence : $confidence_printable ... } ${NC}" + fi fi } publish_cooperative_scan_message () { + #ANNOUNCE ONLINE mqtt_announce_online if [ -n "$1" ] && [ -z "$2" ]; then #POST TO MQTT - ($mosquitto_pub_path \ + $mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $mqtt_version_append \ $mqtt_ca_file_append \ -q 2 \ -L "$mqtt_url$mqtt_topicpath/scan/$1" \ - -m "{\"identity\":\"$mqtt_publisher_identity\"}" 2>&1) + -m "{\"identity\":\"$mqtt_publisher_identity\"}" 2>&1 elif [ -n "$1" ] && [ -n "$2" ]; then #POST TO MQTT - ($mosquitto_pub_path \ + $mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $mqtt_version_append \ $mqtt_ca_file_append \ -q 2 \ -L "$mqtt_url$mqtt_topicpath/$2/$1" \ - -m "{\"identity\":\"$mqtt_publisher_identity\"}" 2>&1) + -m "{\"identity\":\"$mqtt_publisher_identity\"}" 2>&1 else - ($mosquitto_pub_path \ + $mosquitto_pub_path \ -I "$mqtt_publisher_identity" \ $mqtt_version_append \ -q 2 \ $mqtt_ca_file_append \ -L "$mqtt_url$mqtt_topicpath/scan" \ - -m "{\"identity\":\"$mqtt_publisher_identity\"}" 2>&1) + -m "{\"identity\":\"$mqtt_publisher_identity\"}" 2>&1 fi }