Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom file comparison #97

Closed

Conversation

BenjaminHabert
Copy link

Allow user to provide a custom comparison function to FileSnapshot.

This can be usefull if the file being snapshot-tested contains meta-data that is irrelevent to the user. An example of usage was added to exemples/pytest/test_demo.py.

Closes: #96

@gurunars
Copy link

gurunars commented Feb 3, 2020

As arrogant as it may sound but you are applying a fix to a wrong place.

You need to fix the approach you use to generate and store the screenshots and audio-files so that they store only the data that is relevant for your tests.

By definition, if during verification you are able to say which parts of the payload are important, you also are able to say it during a serialization phase.

I.e. you need to extract the entity that you would otherwise compare using a custom comparison routine and serialize that entity instead of the whole thing.

I wouldn't introduce this feature.

@paulmelnikow
Copy link
Collaborator

Hi! Thanks for the contribution. Like @gurunars I'm not thrilled about adding this feature. Although, I can see some justification for it, for example normalizing newlines, or stripping/normalizing metadata.

Would it be possible to implement this as a transform function that is run on the file?

If for your use case it really needs to be a custom comparator, I wonder if that kind of functionality absolutely needs to be in this library.

Or alternatively, is there something snapshottest could expose to help you load the data from the snapshot, with which you could then write your own assertions?

@BenjaminHabert
Copy link
Author

BenjaminHabert commented Aug 6, 2020

Hello. I understand if you prefer not to include this feature and close the pull request, no problem.

Your suggestion of an easy way to load data from the snapshot would also solve my problem.

To give you a better understanding of why I added the feature, here is the use case I was working with:

  • my python application code generates an html file containing images and graphs
  • I want to test that the webpage generated stays visually the same; if it changes I want to see the diff in the pull request
    • this makes it easy to accept / reject the change when discussing with the team or client
  • I use a third-party tool to create an image (screenshot) of the webpage : this is want is compared by snapshottest
  • because I do not have full controle on this third-party tool, the image is allways slightly different:
    • metadata changes
    • slight pixel variations of the image; for instance everything is shifted by 1 pixel
  • consequently, I needed a custom comparison function that can compare two images and decide if they are at leat 99% similar

@paulmelnikow
Copy link
Collaborator

It makes sense that a transform will not work and that you need to write your own comparator function

It's not quite clear from the diff how this would make writing that type of test more convenient. How would you invoke this functionality from your tests?

@paulmelnikow
Copy link
Collaborator

See a similar request in #38.

@paulmelnikow
Copy link
Collaborator

Let's discuss in #38 and circle back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Custom comparison for files
3 participants