[FEATURE] Checksum for downloaded files

[k8ieone]

• Server 0.10.1
• Python client 0.4.0
• Hashcat 5.1.0
• Command independent
• Debug not needed

Problem description:
Each time an agent downloads a task, it could calculate an MD5sum (or something similar) of the dictionary and check with the server if the download went correctly. The same could be applied for hashlists. The server would have to calculate the sum itself or the user would have to manually calculate it and provide it to the server. Also if implemented, there should be an option to disable the sum checking (for example when dealing with very large files).

[s3inlc]

I understand the motivation of this. The question is, if this is really needed. The client currently already checks that the filesize matches exactly, this already covers most of the cases where the data somehow got messed up.
As you already mention somehow, the problem with the checksum is, that for larger files this might take quite some time to run on the server when importing a file.
So for the final decision for me is difficult, I'm not sure if it's worth the work to have this additional check. Or do you have another good reason why it should be added? :)

[evilmog]

File hash prevents bit flips or other issues, I would prefer to have configurable file hashing as a client option, md5, sha1, sha256, with rsync it removes some of the problem

[k8ieone]

I did not know that Hashtopolis already checks if the file sizes match. This means that most of the cases where the file would get damaged (interrupted download, etc.) are already covered. The only thing that could happen is as @evilmog mentions bit flips... Not sure if the checksums are necessary then...

[s3inlc]

@evilmog if it is configurable on the client, that would mean that the server would always have to do it, which might not be ideal for big files. Or how do you mean having it as client option.
And if I understand you correctly, you would still like to have the checksum check added?

[evilmog]

I would like 2 levels of configurability, the first being on the server side. For the server being able to select: always checksum, checksum selected files only, or never checksum, with the default being never checksum.

For the client side have the following options: ignore checksums/don't run checksums on the client side [default], perform checksums on files where a checksum and type are set by the server (returned in json via some sort of api), and lastly require checksums match on all files

Also for the clients enable checksum on client startup or just post transfer checksum as an option

[s3inlc]

Ok, makes sense.

I cannot follow the last sentence, what should be enabled on the client startup?

[jessevz]

This is a bug I recently got where the Wordlist on the server was not the same anymore as the wordlist on the agent. In cases like that it seems that there is no length check because i think the agent just assumes that the wordlist it has locally is the correct wordlist. My suggestion as a fix is to make use of etags where the hash of the file is the etag. This way we can solve the problem by cheaply checking if the file has been changed without redownloading. We could also use the size of the file as an etag or even store the hash, if hashing takes too long for big files.