-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature request: "animation mode" to detect and avoid blocky frames from mismatching #30
Comments
I realise this is years old, but can you send me a clip so I can play around with it? TIVTC has lots of quirks for example PP is bugged and have to workaround with PP=3/6 + clip2=bwdif(). There are false positives on scenechange detection which animation triggers a lot due to the high luma diffs vs live action and scenechange can determine matching behaviour. The hint system is bugged and should be turned off, and some other stuff like TFM still uses MI & cthresh as part of the decision making process in which field to match even when PP is disabled. It looks like your RecoverOrphanFields script has probably solved your issue if I'm not mistaken but I'm still curious to play around with the clip to see if my understanding of TIVTC's quirks can avoid the issue. |
Oh and slow=2 is critical as well. |
hey yeah, so I've ended up "solving" this by using YATTA (ugh) to manually go through frame by frame and choose appropriate matches. it takes forever to achieve anything so I've not made a lot of progress with this. so any way of improving the automated processes would be most welcome, as currently no automated process I've found can be trusted to find good matches for telecine animation without producing thousands of these blocky frames my RecoverOrphanFields script does something else - I've found in these Simpsons DVDs that there's quite a few frames of animation that only exist as a single field with no matching fields to build a full frame out of, so any field matching process (even a manual one) discards these. this can result in jerky animation at times as often these come up in scenes where the background pans across the screen, so if one frame of animation is dropped it's really noticeable. so these are the "orphan" fields my script is designed to find, and when it finds one it uses interpolation make a full frame out of it. I've found it works very well as it's quite easy to detect these orphan fields I've not yet found any way to detect these blocky frames though. as far as I can tell on these frames each field is close to identical which is why the frame looks blocky, but I've not managed to find a way of detecting this in AVS without producing more false positives than anything here's a lossless clip that demonstrates the problem, run this through TFM on any setting and you'll get a couple of the blocky frames in there, when there are better matches that would not do this. happy to send more/longer clips if needed https://drive.google.com/file/d/1zr1Ftrygayvarbu1ylEhpjg6N4QAY7IS/view?usp=sharing and yeah - region 1 simpsons is the only way to get watchable results, I'm in UK so they're not super cheap here either but I've managed to find them all on ebay over the years. however they're still a mess that require lots of processing, and unfortunately so far the only frame matching solution I've found is manually checking and selecting every single frame one by one, which isn't really a sustainable process for 10 seasons lol |
Thanks for the clip, but to truly figure out what is going on I'd need the remuxed MPEG2 stream which includes the repeat field flags in the stream. If you can be bothered doing that I would recommend MakeMKV as it's easy to use and gives you individual remuxed .mkv files for each episode. If you could upload an entire episode that would be fantastic as that would allow me to confirm every scene is getting processed correctly, including scenechanges, cadence breaks and various field cadences the show might be using. Alternatively you can clip it shorter at the ffmpeg command line for example in Windows it would be I'm cautiously optimistic I might have a "one size fits all" solution for you. I have been working on it for the last 9 months, and it has worked on all my NTSC and PAL DVDs so far. |
I've done some pull requests recently for TIVTC so if the root cause is an issue internally with TIVTC I may be in a position to help with that too. |
I have VOB files which I index to D2V that way I can also pass the D2V file directly to TFM I do have MakeMKV which I use for BluRays but I wasn't exactly sure what was happening to some of the DVD specific stuff like flags etc when using that, or what happens to that information when loaded into AVS via MKV rather than D2V, also there's a lot of switching between 23 and 29fps in these DVDs and from memory MakeMKV seems to make a decision on handling this stuff which I didn't like I figured D2V would definitely contain and correctly load all the relevant information, and give me the most control, and it means working on the raw VOBs without any changes, so that's what I've been using. I've ended up decimating to VFR to keep both 23 and 29fps sections smooth without having any unneccesary dups or dropped frames (a lot of people will disagree with doing this but watching on a 120fps display it can manage this just fine and doing 2-pass decimation has worked well for this) https://drive.google.com/file/d/1HYFn4juqQl8eC0SVbcdSFaGI_8HmM5Z3/view?usp=sharing let me know when you've got the files so I can pull them |
Got it, thank you. |
Alright, I've had a play around with it and can see the aliased frames you're talking about like this one: In my opinion these individual aliased frames appear to be baked into the source rather than being a result of faulty field matching. TFM doesn't look at the quality of the resulting frame in terms of how much resolution it contains, it only looks to see if the result is combed, and these frames aren't combed they're just aliased so TFM has no reason to look for a better match. Now you have suggested there is a better match to not produce an aliased frame. Well sort of, but it's kind of coincidental due to there being so many duplicate frames in animation. However there's film gate jitter so it's not really a true duplicate. But if the gate jitter is low enough you could consider it a duplicate and match the bottom field from a different neighbour and get a better quality output frame. But the direction of matching has to dynamically change depending on how much aliasing there is in the image.
The way I would deal with this issue is with QTGMC's progressive repair mode - not to be confused with its default mode which is deinterlacing (which I'm not a fan of). This progressive repair mode is in my opinion perfect for repairing this kind of line twitching and bob shimmer which is typically a result of field interpolation caused by deinterlacing. However you may not like it as it slightly denoises the image and stabilises the film jitter due to the way it works temporally. Script used for the above screenshot:
Here's a demo of it working extremely well on this opening scene from Sliders Region 1 |
Another thing is TFM is comparing fields for similarity and the top and bottom fields might actually have more similarity when the frame is aliased like that, resulting in a stronger match metric. I'll have a look through the source code to see if it can be reversed to match the one with less similarity when there are multiple candidates. |
when selecting matches manually the resulting "good frame" is almost always an exact pixel perfect dup of a neighbouring frame. I don't think any film jitter affects this because if it did, there'd be cases where I could not choose a better match - "so it's not a true duplicate, but if the jitter is low enough you could consider it a duplicate". well somehow the jitter is always low enough for me to choose a perfect match every time, without a single instance of failure across 9 episodes around 30,000 frames each. my manually chosen matches are 100% clean on every frame, without any aliased frames or any frames that aren't a solid field match. in almost every case the selected match is a pixel perfect dup of a neighbouring frame, and so will often get decimated it may well be the case that this is just baked in and that's how it is, but despite that it's clearly possible to choose better matches that don't look aliased - I'm able to do this myself with 100% success rate - so what I was asking for in my orignal feature request was a way to make TFM do this as well. even if it's not technically using "correct" matches, our goal is always better looking footage and this would produce exactly that I will investigate the qtgmc script but looking at the screenshot it doesn't look as good as manually choosing matches, and I've had plenty other issues with qtgmc on these (not least destruction of fine details in scenes with close together lines). admittedly it's probably a little bit faster than spending 20 hours going over each frame manually... |
You can check it by looking at the raw frames with just
Yes you are right there doesn't appear to be any gate jitter either side of the aliased frames in the Simpsons. As the other frames do have gate jitter I'm guessing these aliased frames were created during editing after it was already telecined. Maybe their editing software/equipment back then was doing some kind of slight time stretch to make the scene fit to an exact length and added some duplicate fields here and there, I don't really know. I've seen these kind of shenanigans refered to as "bad edit".
I was looking into that and the only idea I've had so far is to look at the resulting frame after field matching and do a FieldDiff() and if the 2 fields are so similar within a threshold then try to match with a different frame. eg. if match is "c" and FieldDiff() < thresh, try to match from "p/n" instead.
That's right it's not as good, but did you see the Sliders demo? It's about as good as theoretically possible in terms of repairing it after the fact. Frankly I'd call it miraculous.
Yes QTGMC destroys fine details in its default mode but there are different modes which don't do that... InputType=0 is the default deinterlacing mode (scrubs lots of detail) Note the screenshots need to be viewed at 1:1 pixel mapping to see what I'm talking about. Script used for the above screenshots:
|
The FieldDiff function appears bugged - if you do A simpler and more reliable solution might be adding an option to simply preference p/n over c. So for example in mode=0 it tries to only match from c or p, and if both c and p are a match, just use p instead of c. Since the top field of the aliased frame (c) matches the bottom field of both c and p, if we chose p instead of c we'd be choosing bottom field from the previous frame which isn't aliased. This would only work for single-frame aliased frames - if there are strings of 2 or more aliased frames in a row you'd get a good frame on the first one aliased on the second. This sounds simple enough that I might actually be able to implement it. It looks like the field matching is done inside It's possible it might not work properly if the compareFields function isn't as accurate as I'm imagining it to be - for all I know it might match with p instead of c on frames where c is the real match and p is a false positive match. |
Seem to have hit a dead end, because The block that gets done on aliased frame 33079 when mode=0 is:
I believe the conditional of that block means there's a big difference in match quality between |
Oh also there's a black row of pixels flashing at the bottom of frame which is artificially making the previous frame an even worse match - can avoid it with something like |
Just to update I am still working on this issue daily and have made some small progress - the FieldDiff value is still useful in that I can scale it to a value such that it gives a value less than 0 when weaving those fields would result in an aliased frame, otherwise greater than 0. It's not going to be a perfect metric though - I'd bias it towards false negatives to prevent bad field matches (better to have an aliased progressive frame than a combed frame, I think). Will keep trying but make no guarantees. |
hey thanks for trying on this, I am still here just been caught up with other stuff that conditional block - it looks to me like mtn1 and mtn2 are quality metrics for two possible matches, with lower values being better quality, so it looks for the lowest "mtn" to choose the match, but there's scaling applied to this so that at higher quality metrics there needs to be a greater distance between the two values to influence the match, it doesn't just pick the best quality metric and it doesn't just look for a large difference between them, but the amount of difference it looks for is increased if the metrics are higher quality (lower values) so when either mtn1 or mtn2 are >= 250 (vey good quality) then the lower (better) metric must also be less than 1/4 of the higher (worse) metric, to find a good match. so for example if mtn1=250 (very high quality) then mtn2 must also be >1000 to find a match, this is a very big difference but if the values are >= 1000 (poor quality) the lower metric only has to be less than 2/3 of the higher one to find a match, so here if mtn1=1000 then mtn2 must be >1500 to find a match, not such a big difference and if the values are >= 2000 then the difference only has to be less than 4/5, so if mtn1=2000 then mtn2>2500 will find a match so this does mean a big difference in match quality IF the best quality match is a very high quality match - but if the best quality match is a low quality match then the difference does not need to as big. without knowing what mtn1 and mtn2 values are for this frame you have inspected, it's impossible to say what's actually happening. I think you need to find a way to discover these values 😄 looking at those numbers again, it seems a strange way of doing it - based purely on the conditions they have given, at those thresholds (250, 375, 500, 1000, 2000) when the best metric is < 500 then the difference between them needs to be >750, and when the best metric is >=500 then the difference needs to be >500, which could be two lines of conditions instead of 5... their way means scaling (using a fixed multiplication) within each of these bounds, and that scaling changing at each threshold, which seems a strange way of doing it, especially when if the metrics are > 500 then all of that scaling seems to result in the difference value being approximately 500 anyway, why not just choose 500 at this point instead of moving around that value a bit what I'd also be curious to see to fully understand this, is what happens outside of this block - this block checks the difference between these two metrics, then if this difference is big enough it returns a match chosen from the best quality metric - but what happens if the difference is not enough? it won't go into the "if... ret=match2 else ret=match1" block at all, so it won't return a match at all. what happens then? maybe the next line after this block is just "ret=match1" or something but I am curious to know haha I have tried to look through this code myself in the past and I agree it's difficult - I haven't tried to compile any avs plugins myself but I can code c# which isn't too different, so I can usually read and make sense of c++ stuff, but as you say there's no comments or labels and it's very hard to understand. I will try to find this bit of code you've highlighted though and see if I can make any better sense of it another thing : it's been a long time since I did any work on these dvds (over a year) as my progress was so slow so I got bored of it, so some aspects of this I've forgotten, until I opened up some files to have a dig around. so I've now remembered something - actually not every aliased frame has a better match, there are quite a few of these which do not have any other matches at all, and in these cases they have no duplicate frames either side. so you get a single animation frame that only exists as a single aliased video frame. in these cases cycling through the matches usually produces a good frame but which is a different animation frame,so if you choose this match then you are dropping a frame of animation in favour of a dup, which results in more jerky animation. in my opinion it's better to keep every animation frame even if they're "bad" as this still looks better than dropping animation frames, and by default TFM is picking these anyway so that's not a problem. it's just a thing I remember about manually choosing matches, that sometimes you do have a single frame that only exists as an aliased frame so yes this problem is heavily baked in to these encodes what I feel would be really useful is if TFM had some way of detecting these (which I think is what you're still working on?) which was able to pass this on to the post-processing filter as well as influencing the matches - so then if we think a frame has this aliasing even after matching then we can throw it out to QTGMC or whatever, without having to run that across every frame so in short if there's an "animation=true" setting to enable this behaviour, then TFM will attempt to avoid aliased frames in the matching, and will then flag "combed" for frames it thinks are still aliased, so that the PP filter can deal with them (I always use a clip2 for PP). this would be the ideal behaviour |
I like this idea as it means a false positive would just produce a QTGMC'd frame instead of a potentially bad field match. TIVTC already provides the CFieldDiff() runtime function for use inside Avisynth ScriptClip() so we can do that without having to modify TIVTC:
But the result is poor - the CFieldDiff value gives way too many false positives when I scale it so that frame 26463 results in fieldDiff=0 and outputting a QTGMC'd frame: Earlier I wrote "the FieldDiff value is still useful in that I can scale it to a value such that it gives a value less than 0 when weaving those fields would result in an aliased frame, otherwise greater than 0" but I'm unable to reproduce this behaviour to the accuracy I claimed here. I'm not sure why. I tried the opt=0 argument as that gives a different value, but the false positives are just as bad. I then tried using So what we need is a metric that can more accurately detect when the frame is aliased. I think this would be fairly trivial if the two fields were carbon copies of eachother (for example doing a PointResize to half height and back up to double height) but in the Simpsons example it's not really the case as the aliasing bleeds across neighbouring lines. It's not trivial to detect I don't think. |
I'm processing a lot of telecined animation from DVD and find that any IVTC process, including TFM with access to the D2V file, sometimes produces blocky looking frames where it has combined top and bottom fields that consist of identical pixels, producing an output that's the same as if it had thrown away one field entirely, and scaled the remaining field to fill the image. the result is a blocky image that looks like the vertical resolution is halved. here's a screenshot showing the sort of thing I mean - lines 0 and 1 of the image are the same, lines 2 and 3 are the same, etc
and here's a better version of this frame produced by using an override file to choose a different match (I'm using mode 2 because on this dvd, every good match is always either c p or u)
the frequency of these bad frames varies depending on the show and the episode, but it happens to some extent with pretty much every episode of any DVD of telecined animation I have (I've got several well-known shows and they all exhibit this to some extent), at the moment I'm finding hundreds of bad frames in each episode around 40,000 frames long, but in some really bad cases there can be sections with a bad frame on every cycle of 5 frames. not only do these bad frames look awful and distracting when they appear, but they also mess up the decimation as they are almost always a blocky version of a neighbouring frame, so while they should be a candidate for decimation they instead look like unique frames. in the example above, the next frame is identical to the good frame I've shown, so that should be two identical frames that are candidates for decimation, but instead we have two unique frames
I've tried using overrides to fix this problem, and this is very successful - I can override almost every bad frame this way (on the two episodes I've done, I've removed hundreds and hundreds of bad frames leaving only 3-5 max), showing that good matches do exist but TFM is not choosing them. I've tried a myriad of modes and settings in TFM and while the results do vary, nothing stops these frames from being outputted in significant numbers. I think the problem is fundamentally that they are not combed frames, so there's nothing to identify them as a bad match, however to the eye they look terrible, and are very difficult to deal with using filters across the entire clip, without using extreme processing that does more harm than good to the rest of the clip. it's also difficult to identify and treat them separately in AVS, I've had some success with this but it feels like taking a hammer to fix hundreds of bad frames, when the better option is to output better frames in the first place. unfortunately using manual overrides to the extent required is an enormous process, it's taking around 10 hours to manually scan through and override all the bad frames in a single 23 minute episode, and I've got hundreds of episodes in each of multiple shows, which is why I'm here making this suggestion!
so - what would be really useful is if there was an "animation mode" for the field matching, where it would try to identify these frames and treat them the same way it does combed frames - rejecting them as matches, and passing through any leftover blocky frames that couldn't be avoided for post-processing (ideally I'd like to offer a clip3 for fixing blocky frames the same way I already use clip2 for combed frames, I have found that 2-3 stacked calls to QTGMC deals with the blockiness pretty effectively, but I obviously don't want to destroy the entire clip this way). I'm pretty sure these frames should be easy to detect - an algorithm similar to the one that detects combing by looking for significant pixel differences across several neighouring lines of pixels, could instead look for pixel differences that are aligned to odd/even pairs of lines only
I've never seen this issue with telecined live action stuff, it's only ever with animated material, which is why I suggested making it an optional "animation mode" but maybe it would work fine leaving this in place for all material, either way I think it would be a really useful feature for TIVTC resulting in better output for some difficult material
The text was updated successfully, but these errors were encountered: