Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Serial collisions cause LS240 to respond with garbage and crashes the agent #418

Open
BrianJKoopman opened this issue Feb 23, 2023 · 2 comments · May be fixed by #430
Open

Serial collisions cause LS240 to respond with garbage and crashes the agent #418

BrianJKoopman opened this issue Feb 23, 2023 · 2 comments · May be fixed by #430
Labels
agent: ls240 bug Something isn't working

Comments

@BrianJKoopman
Copy link
Member

BrianJKoopman commented Feb 23, 2023 •
edited
Loading

If multiple programs try to communicate with the 240, junk might be sent back when the agent is querying the sensors. This junk then can't be cast to float here:

socs/socs/Lakeshore/Lakeshore240.py

Line 337 in 9ab5c5e

return float(response)

This causes the acq process to crash. We should catch these instances and log the error, but just move onto the next iteration of the loop and continue querying.

Here are some examples:

2023-02-22T23:53:34+0000 startup-op: launching init_lakeshore
2023-02-22T23:53:34+0000 start called for init_lakeshore
2023-02-22T23:53:34+0000 init_lakeshore:0 Status is now "starting".
2023-02-22T23:53:34+0000 init_lakeshore:0 Status is now "starting".
2023-02-22T23:53:34+0000 attempt 0
2023-02-22T23:53:34+0000 Initialized Lakeshore module: Lake Shore Model 240-2P          (LSA21OT)
2023-02-22T23:53:34+0000 start called for acq
2023-02-22T23:53:34+0000 init_lakeshore:0 Lakeshore initialized with ID: LSA21OT
2023-02-22T23:53:34+0000 acq:1 Status is now "starting".
2023-02-22T23:53:34+0000 init_lakeshore:0 Lakeshore module initialized.
2023-02-22T23:53:34+0000 init_lakeshore:0 Status is now "done".
2023-02-22T23:53:34+0000 acq:1 Status is now "running".
2023-02-23T05:34:52+0000 acq:1 CRASH: [Failure instance: Traceback: <class 'ValueError'>: could not convert string to float: '0292.4\r+00.5789'
/usr/lib/python3.8/threading.py:932:_bootstrap_inner
/usr/lib/python3.8/threading.py:870:run
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_threadworker.py:47:work
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_team.py:182:doWork
--- <exception caught here> ---
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:244:inContext
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:260:<lambda>
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:117:callWithContext
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:82:callWithContext
/usr/local/lib/python3.8/dist-packages/socs/agents/lakeshore240/agent.py:221:acq
/usr/local/lib/python3.8/dist-packages/socs/Lakeshore/Lakeshore240.py:337:get_reading
]

2023-02-22T23:00:16+0000 startup-op: launching init_lakeshore
2023-02-22T23:00:16+0000 start called for init_lakeshore
2023-02-22T23:00:16+0000 init_lakeshore:0 Status is now "starting".
2023-02-22T23:00:16+0000 init_lakeshore:0 Status is now "starting".
2023-02-22T23:00:16+0000 attempt 0
2023-02-22T23:00:16+0000 Initialized Lakeshore module: Lake Shore Model 240-2P          (LSA21OT)
2023-02-22T23:00:16+0000 start called for acq
2023-02-22T23:00:16+0000 init_lakeshore:0 Lakeshore initialized with ID: LSA21OT
2023-02-22T23:00:16+0000 acq:1 Status is now "starting".
2023-02-22T23:00:16+0000 init_lakeshore:0 Lakeshore module initialized.
2023-02-22T23:00:16+0000 init_lakeshore:0 Status is now "done".
2023-02-22T23:00:16+0000 acq:1 Status is now "running".
2023-02-22T23:06:57+0000 acq:1 CRASH: [Failure instance: Traceback: <class 'ValueError'>: could not convert string to float: 'D-670S-.L,6SN  235000,\r+0.00000'
/usr/lib/python3.8/threading.py:932:_bootstrap_inner
/usr/lib/python3.8/threading.py:870:run
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_threadworker.py:47:work
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_team.py:182:doWork
--- <exception caught here> ---
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:244:inContext
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:260:<lambda>
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:117:callWithContext
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:82:callWithContext
/usr/local/lib/python3.8/dist-packages/socs/agents/lakeshore240/agent.py:221:acq
/usr/local/lib/python3.8/dist-packages/socs/Lakeshore/Lakeshore240.py:337:get_reading
]
2023-02-22T23:06:57+0000 acq:1 Status is now "done".
2023-02-22T23:53:13+0000 Received SIGTERM, shutting down.
@BrianJKoopman BrianJKoopman added bug Something isn't working agent: ls240 labels Feb 23, 2023
@davidvng
Copy link
Contributor

This error looks exactly like a connection error I'm getting while testing connection drops (unplugging the USB). In my fix, I am able to address this by moving onto the next iteration without the process crashing. For documentation purposes, I label this error as a connection error, but I can add something like: "This might also be multiple programs trying to communicate with the device."

@davidvng davidvng linked a pull request Apr 3, 2023 that will close this issue
Open
6 tasks
@BrianJKoopman
Copy link
Member Author

Interesting. I do like that additional log message you suggest.

I'm catching up on emails/PRs, and hope to get to #430 soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
agent: ls240 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed LS240 connection drop crashing acq process simonsobs/socs
2 participants
@BrianJKoopman @davidvng