Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate / silence Kafka errors #86

Open
martinjohndyer opened this issue Jul 30, 2024 · 0 comments
Open

Investigate / silence Kafka errors #86

martinjohndyer opened this issue Jul 30, 2024 · 0 comments
Labels
-sentinel Issues/PRs related to the sentinel alert listener

Comments

@martinjohndyer
Copy link
Member

The new hop client (#84) works well it seems, but the logs get full of rather annoying errors:

2024-07-29 14:11:32.085:DEBUG - Received notice: ivo://gwnet/LVC#MS240729o-1-Preliminary
2024-07-29 14:11:32.186:DEBUG - Processing new notice: ivo://gwnet/LVC#MS240729o-1-Preliminary
2024-07-29 14:11:32.187:DEBUG - Ignoring test notice
2024-07-29 14:16:07.133:DEBUG - Received notice: ivo://gwnet/LVC#MS240729o-2-Preliminary
2024-07-29 14:16:07.135:DEBUG - Processing new notice: ivo://gwnet/LVC#MS240729o-2-Preliminary
2024-07-29 14:16:07.135:DEBUG - Ignoring test notice
2024-07-29 14:21:27.187:DEBUG - Received notice: ivo://gwnet/LVC#MS240729o-3-Initial
2024-07-29 14:21:27.257:DEBUG - Processing new notice: ivo://gwnet/LVC#MS240729o-3-Initial
2024-07-29 14:21:27.380:DEBUG - Ignoring test notice
2024-07-29 14:31:27.331:ERROR - internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="GroupCoordinator: kafka1.gcn.nasa.gov:9092: Failed to connect to broker at [2600:1f10:4f61:4a00:79d8:1a49:534a:ee9]:9092: Network is unreachable (after 24ms in state CONNECT)"}
2024-07-29 14:41:28.150:ERROR - internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="GroupCoordinator: kafka1.gcn.nasa.gov:9092: Failed to connect to broker at [2600:1f10:4f61:4a00:79d8:1a49:534a:ee9]:9092: Network is unreachable (after 138ms in state CONNECT)"}
2024-07-29 14:51:28.817:ERROR - internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="GroupCoordinator: kafka1.gcn.nasa.gov:9092: Failed to connect to broker at [2600:1f10:4f61:4a00:79d8:1a49:534a:ee9]:9092: Network is unreachable (after 32ms in state CONNECT)"}
2024-07-29 15:11:30.195:ERROR - internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="GroupCoordinator: kafka1.gcn.nasa.gov:9092: Failed to connect to broker at [2600:1f10:4f61:4a00:79d8:1a49:534a:ee9]:9092: Network is unreachable (after 38ms in state CONNECT)"}
2024-07-29 15:11:32.714:DEBUG - Received notice: ivo://gwnet/LVC#MS240729p-1-Preliminary
2024-07-29 15:11:32.784:DEBUG - Processing new notice: ivo://gwnet/LVC#MS240729p-1-Preliminary
2024-07-29 15:11:32.785:DEBUG - Ignoring test notice
2024-07-29 15:16:08.763:DEBUG - Received notice: ivo://gwnet/LVC#MS240729p-2-Preliminary
2024-07-29 15:16:08.840:DEBUG - Processing new notice: ivo://gwnet/LVC#MS240729p-2-Preliminary
2024-07-29 15:16:08.840:DEBUG - Ignoring test notice
2024-07-29 15:21:27.818:DEBUG - Received notice: ivo://gwnet/LVC#MS240729p-3-Initial
2024-07-29 15:21:27.865:DEBUG - Processing new notice: ivo://gwnet/LVC#MS240729p-3-Initial
2024-07-29 15:21:27.866:DEBUG - Ignoring test notice

Those internal kafka error lines seem to come from here:
https://github.com/astronomy-commons/adc-streaming/blob/b1a5a81c52d2752a9fd63b9717c121285646d09b/adc/errors.py#L12-L21

The system seems to be running fine, and the errors aren't fatal. It's not easy to research, https://stackoverflow.com/a/60389747 leads to a GitHub issue mentioning "At this time an application should consider all errors informational, and not terminal." In which case I'd rather just shut the errors up, I'm not even sure how they're finding their way into our logger (and stdout for that matter).

@martinjohndyer martinjohndyer added the -sentinel Issues/PRs related to the sentinel alert listener label Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-sentinel Issues/PRs related to the sentinel alert listener
Projects
None yet
Development

No branches or pull requests

1 participant