Add context to hosts, run bits with that context

[noamblitz]

Requirement

We aim to add context to IP addresses. By doing so, port classification mechanisms can better identify the type of server and determine which ports should be expected to be open.

Solutions

I identify two ways of doing that:

• Add an attribute to the IPAddress that can be a list of types (if Octopoes allows that), or a comma-separated list of types. This context can then be used by bits. Pro: Easy to implement. Con: Not that clean, the attribute is not part of the primary key. Also, I am not sure whether other normalizer/bits that yield this OOI after setting the type will overwrite this attribute.
• Add an OOI with server type. This OOI will be called IPAddressType and can be user,nameserver,webserver etc. In my opinion, this is the cleaner way.

Proposal

My proposal is to go with the second option. I have figured out a bit that creates these types of OOIs. The sub graph looks like this:

stateDiagram-v2
DNSNSRecord --> Hostname|mispo.es
DNSNSRecord --> Hostname|ns1.example.com
DNSARecord --> Hostname|ns1.example.com
DNSARecord --> IPV4Address

Loading

The bit can run on n1.example.com to decide the type, and fetch the IPV4Address because it is an attribute of the DNSARecord.

[noamblitz]

In hindsight, I think it's better to add information as an attribute. It reduces the amount of OOIs and has the same pros.

Just need to check if everything will work as expected.

[dekkers]

I agree with using attributes. XTDB primarly stores documents and as far as I know has fast mechanism to filter/query on the fields in those documents.

I am also not sure if the way Octopoes works does what we want/what is expected. Maybe we should implement a way to yield just one or more specific attributes of an object but not the object itself? That might make it clearer what the source of the objects and attributes are.

[noamblitz]

I will test this today! I am not sure how exactly the overwriting mechanisms work but the expected is:

• When only primary key is yielded by another normalizer/bit, the attribute that is not part of the primary key (the server type) should not be overwritten
• When that attribute is specifically part of the yielded OOI, the attribute should be overwritten

[noamblitz]

Since the attribute is not part of the primary key, the bit does not have access to the content. So the chronological order of bits influences the attribute. I am trying to look at other bit param paths to achieve this (which will also be very beneficial for other bits in the future). The new query_from_path() might help (used for bit configs)

[noamblitz]

After playing around with octopoes, we could consider allowing Path queries for bit params. This will make the bits more useful because it allows for both outward and inward queries. It may cost a lot of compute though, so we should consider it carefully. Example of the Path query that solves this specific problem is:

path = Path.parse("Hostname.<hostname[is DNSARecord].address")
client.query(str(Query.from_path(path).where(Hostname, primary_key="Hostname|internet|mx.wijmailenveilig.nl")))

Which gives all info on all IP addresses and their context.

We need to keep in mind though, that bits need to keep track of whether one of their bit params changes, or whether a new ooi is added that is part of the bit params, so this feature will need a lot of refinement.