New encryption version V002 proposal

[clach04]

Basically the same as V001 but adopt iteration count recommendation from OWASP. Could potentially also implement U002 as well for older devices.

See previous comment on this in #1378 (comment)

I have an experimental version implemented in Python at clach04/jenc-py#7

NOTE as expected it is slower to encrypt/decrypt (well, slow KDF, actually encryption/decryption time is the same).

From initial implementation.

Before V002 Python https://github.com/clach04/jenc-py/ test suite run time under 3.12.1 (tags/v3.12.1:2305ca5, Dec 7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] was sub 0.2 seconds:

Ran 11 tests in 0.193s

With V002 and 210K iterations instead of 10K, about 1.6 secs:

Ran 11 tests in 1.613s

If this is adopted, older Markor installation will NOT be able to opened newer Markor encrypted files.

Thoughts?

[gsantner]

Regardless if it's a different algorithm or different revision...not a good idea to immediate make the new one by default, and break all other installations that don't support it yet.

So if it has to be a new revision, the existing implementation should still be the default - but be able to read both.
When a while has passed, we can always think on defaulting to the new. At this point older and newer installations/tools alike support both.

With V002 and 210K iterations instead of 10K, about 1.6 secs:

What is most important is that it not makes Markor extreme slow in real use. Do you notice a notable difference/slowdown?

[clach04]

Agreed on default. This would mean we need a UI option and config for picking the preferred version (which I think is a great idea), but I may not be able to implement this myself 😿

RE Speed, I don't know if I have the answer you looking for. TL'DR is that for my test case of viewing/editing a file interactively on my DESKTOP (not phone) I can't perceive/measure a difference with wall-time. There is a measurable difference for a single file, here is an example:

(py3.12.5venv) C:\code\py\jenc-py_main>echo hello| python -m jenc -e -p geheim --jenc_version V001 - -o output.V001.jenc
Read in from stdin...
took 0.008994 secs
(py3.12.5venv) C:\code\py\jenc-py_main>echo hello| python -m jenc -e -p geheim --jenc_version V002 - -o output.V002.jenc
Read in from stdin...
took 0.164001 secs

I can not perceive this whilst using it.

However, this is where I'm not fully answering your question:

1. I do not have a build for Android. I have proposed changes, that are 100% untested 😆 See branch https://github.com/clach04/markor/tree/issue2471_V002_proposal
2. I've only tested one file at a time and my (tiny) Python test suite (which does more than one file, but only a small handful).

I think to answer this question fully, we need:

1. an Android build with this change in, and the default switched to V002 (in getVersionForAndroid() https://github.com/clach04/markor/blob/2e2958e86d028b5aa50dee2307cfef85d2cd8630/app/thirdparty/java/other/de/stanetz/jpencconverter/JavaPasswordbasedCryption.java#L79).
2. Test viewing/editing a file on a phone - my prediction is that this will imperceptible but we need to confirm/deny this
3. Test searching across many files - my prediction is that it will be fine BUT noticeable - again needs testing. Attempting to guess an answer, I do not think it will be an extremely slow impact (see question below)

Even if it is 🐢 slower, it would make sense to have the option, even if it's not the default so end users can make an informed decision.

Questions:

1. is there a test suite for number 3, for performance benchmarking? I'm not sure what I can do here. The best I can think of at the moment is test with my Python app
2. What percentage of files are people encrypting?
   • for myself today with Markor, it's zero (I've been using Markor on my phone with unencrypted files). I use a different tool at the moment which I'm looking to transition to a new encryption format. I only have just over 8% files encrypted (and for my work notes under 3% are encrypted). Even though I've not ran my full tests yet, I'm not expecting to have an increase in time spent that will bother me

[clach04]

I forgot to add, both ptgrep (plain text and encrypted file grep tool) and ptig (interactive plain text and encrypted file search tool) both report how long a search takes. There is support for the jenc format so I can run desktop tests very quickly if there is already a corpus of test data, I can then re-encrypt and re-test.

> ptgrep --note-root=. --search_encrypted -p geheim -t -i word_that_does_not_exist_in_files

Query time: 0.01 seconds

[gsantner]

no gui selection. always write with v1

[clach04]

...

What is most important is that it not makes Markor extreme slow in real use. Do you notice a notable difference/slowdown?

I finally got a build to test. I had a really hard time using Android Studio 😭. In the end, even after I finally got the debugger/emulator working on my PC I just used your GitHub action to build and download an APK, hat worked great (really nice job there!).

I can see a noticeable delay in opening a v002 file compared with v001 and u001. It's not crazy long but it is noticeable.

If anyone wants to have a play with it see the APK at the bottom of the page of https://github.com/clach04/markor/actions/runs/11992540595 - this reads v002 but won't write them. I'd like to hear other opinions on the performance difference. You can get a test file from https://github.com/clach04/jenc-py/tree/issue_7_V002_experiment/jenc/tests/data the password is "geheim" (no quotes):
https://github.com/clach04/jenc-py/blob/6b8279b27c8147ff1bb6834fdb1bfc2cc9c09558/jenc/tests/testsuite.py#L62

From my quick tests I've been tearing my hair out with the same issues reported in #1250 - the UX for failed encryption loads is not great. The error message is a little bit vague, and not being able to clear the password (the only option appears to be to set it to something that isn't the right password or blank) was really irritating compared to other tools that handle secrets. I'll think about this some more and join in the conversation over there :-) Sadly as I'm not an android dev I'm not sure I can do anything other than moan and make suggestions rather than a concrete change.

[clach04]

I still do not have an android build (even though, I've written the code 😆).

What I have done is implement some testing with a desktop/command-line, python implementation. This implementation is single-threaded, it does NOT attempt to do anything in parallel.

I ran the tests at least 3 times ('cos this is supposed to be a fun project). I have an SSD and I will ignore the first run so as to not test disk-IO timing, and instead focus on the CPU processing time.

I have essentially done 2 benchmarks with 4 different encryption implementations (2 jenc).

1. is 10K files, just under 50Mb taken from https://github.com/Zettelkasten-Method/10000-markdown-files - this test encrypts EACH of the 10,000 files - I consider this a stress-test. This does NOT reflect my personal use case. It is significantly more notes/files that I have, also I do NOT encrypt all my notes only a subset
2. My real "work" notes, I do have a larger set of notes for personal use (I've just not had chance to benchmark this yet). This is 1587 *.txt files and 42 encrypted files.

The benchmark searches both the plain text and the encrypted files so one can argue that case 2 above is not solely testing crypto impact, but it matches my real use cases :-)

Stats for 2:

txt
1587 File(s)      4,710,616 bytes

tombo - chi / Tombo / Blowfish derived
42 File(s)         82,856 bytes

jenc v001
41 File(s)         85,456 bytes

jenc v002
42 File(s)         86,228 bytes

tombo_aes256_zip - aes zip
49 File(s)        336,841 bytes
NOTE there are 42 encrypted notes in .aes.zip, and 7 regular zip files without encrypted notes.
ptgrep will start to search ALL of them

Results - 1

Results 1 - Raw

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist C:\code\notes\10000-markdown-files\10k_v001_jenc

Query time: 94.72 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v001_jenc

Query time: 49.17 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v001_jenc

Query time: 47.97 seconds


(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 1055.48 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 1000.81 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 987.80 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 1005.16 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_aes256_zip

Query time: 10.42 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_aes256_zip

Query time: 10.44 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_aes256_zip

Query time: 10.35 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_tombo_chi

Query time: 10.60 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_tombo_chi

Query time: 10.60 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_tombo_chi

Query time: 10.41 seconds

Results 1 - Summary

1. 10.42 seconds - 10k_aes256_zip
2. 10.60 seconds - Tombo
3. 49.17 seconds - under a minute - jenc v001, as used by Markor
4. 1055.48 seconds - ~17 minutes - jenc v002 proposal/experiment

Results - 2

Results 2 - Raw

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo

Query time: 0.14 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo

Query time: 0.13 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo

Query time: 0.13 seconds

(py3.12.5venv) C:\tmp\work_notes>echo.


(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_aes256_zip

Query time: 5.54 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_aes256_zip

Query time: 0.16 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_aes256_zip

Query time: 0.16 seconds

(py3.12.5venv) C:\tmp\work_notes>echo.


(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v001_jenc

Query time: 0.32 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v001_jenc

Query time: 0.31 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v001_jenc

Query time: 0.31 seconds

(py3.12.5venv) C:\tmp\work_notes>echo.


(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v002_jenc

Query time: 4.20 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v002_jenc

Query time: 4.16 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v002_jenc

Query time: 4.18 seconds

Results 1 - Summary

1. 0.13 seconds - Sub-second - My current, very old Blowfish based from the Tombo project
2. 0.16 seconds - Sub-second - aes256_zip, AE-1 only Zip file with AES-256 - Standard WinZip/7z (not the old ZipCrypto!)
3. 0.32 seconds - Sub-second - jenc v001, as used by Markor
4. 4.20 seconds - over 4 seconds - jenc v002 proposal/experiment

Background Notes

I used a new feature in puren_tonbo to generate the test data, there is a new tool for bulk converting files:

For 10K notes:

git clone https://github.com/Zettelkasten-Method/10000-markdown-files
cd 10000-markdown-files
python -m puren_tonbo.tools.ptrecrypt --cipher .v001_jenc  --new_extension .jenc -p password  --destination_directory 10k_v001_jenc "10000 markdown files"
python -m puren_tonbo.tools.ptrecrypt --cipher .v002_jenc  --new_extension .jenc -p password  --destination_directory 10k_v002wip_jenc "10000 markdown files"
python -m puren_tonbo.tools.ptrecrypt --cipher .aes256.zip --new_extension .aes256.zip  -p password  --destination_directory 10k_aes256_zip "10000 markdown files"

For my notes:

Copy tombo folder a few times and name using dirs below, re-encrypt in-place:

REM prompt for password, these are my real work notes. Do not leak secret ;-)
ptrecrypt --cipher .v001_jenc  --new_extension .jenc --skip_unencrypted --existing-files=delete    tombo_v001_jenc
ptrecrypt --cipher .v002_jenc  --new_extension .jenc --skip_unencrypted --existing-files=delete    tombo_v002_jenc
ptrecrypt --cipher .aes256.zip                       --skip_unencrypted --existing-files=delete    tombo_aes256_zip
ptrecrypt --cipher .zip                              --skip_unencrypted --existing-files=delete    tombo_zip        

[harshad1]

@gsantner have you ever thought about adding plugin support to markor?

[gsantner]

For that you need enough developer (code contribution) interestors. As time has told so far, there are only handful of people doing that.

When plugin stuff happens in whatever way, it means we need to supply a API - and by that have another level of thing to constant maintain and watch out to not break.

So humbly said, I rather prefer people to contribute to Markor itself, then trying to make some externalization API that may barely get people interest either, and at same time holds back Markor by "not breaking this or that plugin".