Performance

[pawamoy]

I'm having performance issues with inline-snapshot on Python < 3.11. I added statements to the module using inline-snapshot to print timestamps to a file:

# tests/test_end_to_end.py
log("Loading snapshots", "time.txt")
snapshots_signatures = snapshot(...)  # redacted for brevity

log("Parametrizing tests", "time.txt")
@pytest.mark.parametrize(...)
@pytest.mark.parametrize(...)
...  # redacted for brevity

log("Finished", "time.txt")

python3.8: 2024-05-22 15:48:41.444515: Loading snapshots
python3.8: 2024-05-22 15:48:54.558690: Parametrizing tests
python3.8: 2024-05-22 15:48:54.559026: Finished
python3.9: 2024-05-22 15:49:15.362138: Loading snapshots
python3.9: 2024-05-22 15:49:28.414378: Parametrizing tests
python3.9: 2024-05-22 15:49:28.414719: Finished
python3.10: 2024-05-22 15:49:49.733597: Loading snapshots
python3.10: 2024-05-22 15:50:12.119295: Parametrizing tests
python3.10: 2024-05-22 15:50:12.119664: Finished
python3.11: 2024-05-22 15:50:34.953705: Loading snapshots
python3.11: 2024-05-22 15:50:35.173075: Parametrizing tests
python3.11: 2024-05-22 15:50:35.173476: Finished
python3.12: 2024-05-22 15:51:00.978469: Loading snapshots
python3.12: 2024-05-22 15:51:01.282534: Parametrizing tests
python3.12: 2024-05-22 15:51:01.284369: Finished

As you can see, on Python 3.8 and 3.9 it took approximately 13 seconds to load the snapshots. On Python 3.10 it took 23 seconds. On Python 3.11 and 3.12 it's almost instantaneous. Subsequent runs give the same results. Not sure why the times differ this much. I'm running pytest without any inline-snapshot options. Snapshots are already generated, and tests pass.

Note that I have a lot of snapshots keys: 469 (equal to the number of parametrized tests), but not a lot of snapshot files: only 20 or so.

While debugging other things, I think I had noticed that the same code was run twice or more when checking snapshot flags.

[15r10nk]

I can look into it tomorrow.

It might be that the algorithm which is used by executing (the library which I use internally) is the reason for your problem.
It changed in python 3.11 to a complete new implementation.

You can use --inline-snapshot=disable which disables the snapshot logic ... But this does not help you if you want to fix the snapshots.

Can you tell me which commit/file caused this performance problems?

[15r10nk]

It should. It is a bug if it does not.

[pawamoy]

Adding --inline-snapshot=disable doesn't help 😕 I'm getting the same perfs with or without it.

[pawamoy]

And it's indeed executing who takes most of the time ☺️

[15r10nk]

Adding --inline-snapshot=disable doesn't help 😕 I'm getting the same perfs with or without it.

I can not reproduce your problem in your branch.

I added this line:

log("Parametrizing tests", "time.txt")
log(str(type(snapshots_signatures)),"time.txt") # added this

and got the following:

python3.10: 2024-05-26 07:27:44.608515: Loading snapshots
python3.10: 2024-05-26 07:28:00.159804: Parametrizing tests
python3.10: 2024-05-26 07:28:00.159880: <class 'inline_snapshot._inline_snapshot.UndecidedValue'>
python3.10: 2024-05-26 07:28:00.160141: Finished
python3.10: 2024-05-26 07:28:33.884355: Loading snapshots
python3.10: 2024-05-26 07:28:33.885501: Parametrizing tests
python3.10: 2024-05-26 07:28:33.885549: <class 'dict'>
python3.10: 2024-05-26 07:28:33.885809: Finished

The first run was done with: python -m pytest tests/test_end_to_end.py

and the second with: python -m pytest tests/test_end_to_end.py --inline-snapshot=disable

You can see the difference in the timing and the different return type of snapshot()

I used a new virtualenv with python3.10 and make setup.

Can you reproduce my results?

[pawamoy]

Indeed, I deleted my envs, recreated them, and it works now.

One small issue:

$ pytest tests --color yes -c config/pytest.ini -n auto --inline-snapshot=disable
ERROR: inline-snapshot can not be combined with xdist

Since inline-snapshot is disabled, maybe it shouldn't error out because xdist is enabled?

Also at the end of the run it says:

INFO: inline-snapshot was disabled because you used xdist

But that's not really true. I disabled it explicitly, and also didn't enable xdist explicitly because inline-snapshot wouldn't let me 😉