Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new mime-types for debug/warning/info logging return channels for boefjes #1373

Open
underdarknl opened this issue Jul 11, 2023 · 2 comments
Assignees
Labels
boefjes Issues related to boefjes technical-design

Comments

@underdarknl
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Currently we dont have a return channel for info, warning and debugging info from inside the boefjes. And, with the containerized option we have no other route than either std-out, or std-err. Std-err currently signals that all output is either a stacktrace or at least something unstructured, (so can only contain 1 datablob), and std-out is always structured to contain a set of mime-type(s)+rawfile combinations.

It would fairly simple to specify a few 'special case' mime-types in that structured std-out to hold various levels of debugging output.

Describe the solution you'd like
I propose we standardize a couple of new mime-types next to error/boefje:

  • debug/boefje
  • warning/boefje
  • info/boefje

These could then be used as a side-channel to get data out of the boefje that is not meant for parsing by the usual normalizers, but would provide more context on what happened inside the boefje.

A direct usecase for this would be the shodan boefje which currently silently ignores local ip's. the boefje then completes successfully, without any user visible output. An error for this boefje might also be reasonable I' guess, but that would be indicative of something you'd need to fix.

The scheduler currently special cases raw files that has a mimetype starting with error/ this would for now need to be extended to also exclude the above mimetypes.

Describe alternatives you've considered
No other options come to mind. Other mime-types could be considered, but these seem to be inline with the current error mime-type, and it would be easier to keep them separate when (in the future) we can target boefje/* or info/* as a consumable mime-type for a normalizer.

Additional context
Depending on some security questions, we might even consider showing the contents of some of these mime-types directly in side user-interface. (as plain text). (see https://docs.openkat.nl/guidelines/ideas.html#safe-viewing-of-boefjes-data-dutch)
There is some issue with boefjes describing their own mime-types, as this would open the door to (for example) the download boefje being tricked into 'downloading' a file from the web with one of these mime-types, and then returning it as is.

@underdarknl underdarknl added the boefjes Issues related to boefjes label Jul 11, 2023
@github-project-automation github-project-automation bot moved this to Incoming features / Need assessment in KAT Jul 11, 2023
@Darwinkel Darwinkel removed their assignment Oct 31, 2023
@underdarknl underdarknl moved this from Incoming features / Need assessment to To be discussed in KAT Feb 20, 2024
@underdarknl
Copy link
Contributor Author

related to: #1378

@dekkers
Copy link
Contributor

dekkers commented Feb 20, 2024

Results from discussion today:

  • We should have a specific channel for boefje to return that there wasn't an error but it also didn't execute
  • Logging is useful information to present to the user for troubleshooting and it shouldn't be a problem to show raw text to a user
  • We should make sure that we don't save too many raw files in bytes

@dekkers dekkers moved this from To be discussed to Approved features / Need refinement in KAT Feb 20, 2024
@dekkers dekkers moved this from Approved features / Need refinement to Backlog / Refined tasks in KAT Feb 20, 2024
@zcrt zcrt moved this to Stability in OpenKAT @ Z-CERT Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
boefjes Issues related to boefjes technical-design
Projects
Status: No status
Status: No status
Development

No branches or pull requests

4 participants