Severe bug in Bgeigie Import viewer! Imports are incorrectly rejected because of false high values

[fobrs]

If you look at this import: https://api.safecast.org/en-US/bgeigie_imports/39752

You are greeted with a message that this import contains high values:
Attention! This log file contains: 「A possible radiological anomaly. Max µSv/h: 0.68.」
The moderator writes:

Comment
Moderator comment: Holding approval pending confirmation of high dose rates.

But at the same time there is displayed:

Highest Measurement
124

This last values is correct if you examine the actual LOG data. 124 CPM is 0.37 µSv/h

So where comes the Max µSv/h: 0.68 from?

Could it be that a lot of imports are incorrectly rejected because of this?

Another example.

My own import: https://api.safecast.org/en-US/bgeigie_imports/47818 has strange high values too.
Max is 108 CPM ( 0.32 µSv/h)

But in the following image the selected measurement has 156 CPM. But the actual line in the LOG file of this point shows 106 CPM:

$BNRDD,2735,2020-06-29T10:42:11Z,106,13,1769,A,5313.0698,N,00633.5683,E,-6.30,A,9,91*54

(text continues after image)

The Log File Stats in the next image are incorrect too. Max value was 108 but in the image 156 CMP is shown as max.

[sasharevzin]

@matschaffer @auspicacious from ruby code perspective I didn't find anything wrong. would take a look?

[matschaffer]

The viewer is JS - @Frangible can probably confirm but I suspect there’s a cp5s causing the anomaly alert.

[fobrs]

Yes, the use of the cp5s value at those points in the viewer is the problem. When LOG data is imported in your database however, the cp5s values are not used.

[Frangible]

This is by design and not a bug. Note that the viewer’s log checking is a displayed advisory of anomalies detected only, and it’s up to the moderator reviewing the log whether they interpret the anomaly as bad/invalid data or not. Also, a high cp5s reading is not a “false” high value. It is data collected by the instrument. While other layers of the API may only use an avaerage, an error in a cp5s will still cause elevated measurements for the time period it is being integrated into. These elevated averaged values may not be above an error threshold, but it is still enough to introduce and error and reduce the precision and accuracy of the dataset if that high cp5s value shifts others to be statistically significantly higher than they truly are. This introduces error and variance into the dataset as a whole and is not ideal.

…

On Jul 1, 2020, at 4:17 PM, Mat Schaffer ***@***.***> wrote:  The viewer is JS - @Frangible can probably confirm but I suspect there’s a cp5s causing the anomaly alert. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[fobrs]

I think it's a 'false high' if you multiply the cp5s value by 12 and take that as the CPM value of a measurement.

[anaavu]

We noticed the same discrepancy, but it was our impression that the values on the viewer were the correct ones, while due to a conversion problem with cp5s to CPM, they don't show up on the log files properly. It would be great if someone can clarify!
Here's our previous posting about this #654 (comment)

[juhele]

Hi,
I do not see any problem with Safecast API or the data. I downloaded the data and displayed them:

The values are slightly higher and there is one over 0.5 microSv/h so I understand that visual alert. Does not say it is dangerous or man-made anomaly - most of the anomalies are natural or related with natural materials. Looks like Nairobi just has higher natural bacground - information about local geology might tell you more.

But you can find such anomalies also in Europe - Roma (Italy) has higher natural background and this anomaly is in Berlin, for example:
https://api.safecast.org/en-US/bgeigie_imports/47045

[seanbonner]

As it appears everything is working as designed and it was just a misunderstanding of the design goals I'm going to close this.