[ Linux ] malloc(): invalid size (unsorted)

[TRPB]

I'm trying to get QTGMC deinterlacing working on linux which requires a bit of work.

I've installed the arch linux package avisynth-plugin-tivtc-git and if I run the following script:

LWLibavVideoSource("test.mkv")
TDeint()

and then try to run it through ffplay:

ffplay -i test.avs
ffplay version n4.4 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 11.1.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-openssl --enable-shared --enable-version3 --enable-libfdk_aac --enable-nonfree
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
malloc(): invalid size (unsorted)  0KB vq=    0KB sq=    0B f=0/0   
Aborted (core dumped)

Without TDeint() it works fine, so it seems to be a TDeint issue, I'm not sure where to start on debugging it as it's my first time using AviSynth.

[pinterf]

There must be an allocated memory over-addressing somewhere. Without other debug info unfortunately we cannot go any further.
I'm not using linux, usually just make a build for a short test if it runs or not. Debugging there is beyond my knowledge other than enabling debug build.

[marukuru]

I had a similar issues (free(): invalid pointer & random Segmentation faults on startup) with AviSynth+ releases from 3.7.0 and newer. I did some testing in a fresh Ubuntu VM yesterday, and found, that the problem was with a broken AviSynth+ installation.

I followed the guide (posix.rst) and used the check install way. Adding sudo ninja install && sudo ldconfig fixed the problem for me. Still strange, that (for me) only TFM & TDeint were effected, every other plugin I used worked without problems.

I found that something was fishy, when I compiled 3.6.0 and it was also broken. I did build 3.6.0 and 3.7.0 to before, and they were both working.

@pinterf let me know which kind of debug information might help on this and I'll try my best to get them.

[pinterf]

Thanks, it's not easy, the final error message unfortunately means nothing because the memory overwrite occurs somewhere else.
Only valgrind that I am aware of https://valgrind.org/ , but had never seen, used, or smelled :) it. Maybe I would ask my sons about that but I'd be happy when this task does not fall on my shoulders.

[knuxyl]

With current master I am getting

Error: TFM: malloc failure (cArray)!

I checked all the values being passed in the initialization of cArray and they are all valid. I've tried AI to redo this section but always end up with same error. Removing the throw condition just ends up with the same error for tbuffer later in TFM.cpp.
I'm using D2VSource() and then TFM() in my AVS script. Removing TFM makes everything work just fine.

[pinterf]

Could you please check if this method works for you?

[knuxyl]

Could you please check if this method works for you?

This gave a ton of errors during build
tivtc.log

[pinterf]

Than must be a different issue. including smmintrin.h gives you this huge error dump.

Do you use cmake option -DENABLE_INTEL_SIMD:bool=off or on?
EDIT: Irrelevant question. This project is not yet prepared for non-intel or C-only build, the INTEL_INTRINSIC define which is set by ENABLE_INTEL_SIMD cmake option is not used anywhere.

Did you make any other changes in your build process or environment? The errors are not related to the changes made recently.

[knuxyl]

Yeah I'm not sure what happened with that build, sorry about that. Anyways I started over completely and it built fine but I still get

Error: TFM: malloc failure (cArray)!

Also can you fix common/fixedfonts.h and add #include <cstdint> ? I have to add this or it won't build. I don't believe this library is linux specific.

I'm not using any cmake defines, I just started building exactly how the readme said because I was having problems.

I am on a intel xeon w1290-t 64gb ecc ram debian testing machine using gcc 13.2.0-25.

This is the script I've been using. I have changed and removed TFM's parameters to debug but nothing changed.

SetMemoryMax(48000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
D2VSource(d2v="1.d2v", idct=5)
tfm(mode=1, slow=2)

Removing TFM completely I get a perfectly fine video output.

[pinterf]

I was trying in minGW with an updated gcc (just for the syntax). Seems I have to resurrect my WSL Ubuntu... Stay tuned.

[pinterf]

Also can you fix common/fixedfonts.h and add #include <cstdint> ? I have to add this or it won't build. I don't believe this library is linux specific.

I've already added it in the previous commit. I'm now on Linux and gcc 13.1 (instead of 9.4) and it builds just fine. Now I try to make a working avisynth environment as well and I hope I would see the magic error message.

[knuxyl]

Awesome thanks. My build script auto adds the line so I didn't notice. Avisynth should be quick but debian and ubuntu should set -DCMAKE_INSTALL_PREFIX=/usr because it defaults to /usr/local and nothing will be able to find it.

[knuxyl]

I should note also that instead of rebuilding the entire ffmpeg binary to include avisynth support I just compiled avs2yuv and pipe it to ffmpeg. it can also just directly output to y4m (vlc compatible). ffmpeg in debian repo from what i remember doesnt include avs support so i suspect neither does ubuntu.

[pinterf]

Thanks, I'm using linux once a year and even forget what sudo and apt-get is, I always need the learning curve. I've installed a fresh 22.04, gcc, g++13, cmake (and saw your comment after there). Then got a stock 6.1 ffmpeg (22.04 was stuch with an earlier 4.x version) but it came w/o avs support, so I built it from qyot27's repo. Compiled a fresh avisynth and tivtc, and now the only problem I got (on my workbench) that at the very end of the encoding I get a segmentation fault. So I'm on the right trace, there must be a solution, I ask a little patience.

[knuxyl]

I'm on the right trace, there must be a solution, I ask a little patience.

Of course 😁. If u need any help/suggestions with linux just let me know. I noticed with a lot of the avisynth plugins if I compiled with optimizations such as O3 or LTO I would get segfaults. I just stopped trying and compile debug builds.

[pinterf]

Well, it works on my Linux, yeah.
(Segfaults are not normal, if you find one, you can report it, maybe the bug is there on Windows as well but is fortunately hidden)

[knuxyl]

Well, it works on my Linux, yeah. (Segfaults are not normal, if you find one, you can report it, maybe the bug is there on Windows as well but is fortunately hidden)

You are awesome, it builds and run perfect now. Thank you so much!
I'm using this avs script

SetMemoryMax(48000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
D2VSource(d2v="1.d2v", idct=5)
tfm(mode=1, slow=2, hint=true, pp=7)
tdeint()
tdecimate(mode=1, hint=true)
Spline64Resize(640,480)

And this command to process a DVD video.

./avs2yuv -raw input.avs - | ffmpeg -pix_fmt yuv420p -s 640x480 -f rawvideo -framerate 24000/1001 -i - -c:v libx264 -preset veryslow -crf 21 -y output.mkv

I just need nnedi3 for QTGMC to compile for linux and my entire conversion process will work on linux.

I haven't experimented with this tdeint() though but it didn't fail malloc like the original bug report said, so I think this bug is closed. Thanks again!

[pinterf]

Your welcome, happy encoding :)

[Asd-g]

off topic

I just need nnedi3 for QTGMC to compile for linux and my entire conversion process will work on linux.

You can try nnedi3cl (the latest QTGMC(p) versions can work with it).

/off topic

[knuxyl]

off topic

I just need nnedi3 for QTGMC to compile for linux and my entire conversion process will work on linux.

You can try nnedi3cl (the latest QTGMC(p) versions can work with it).

/off topic

Thank you for the suggestion but I use mesa's amd driver and it doesn't include opencl support (listed as WIP) and amd's official driver has broken too many of my previous systems so I don't mess with it. Maybe I'll get back around to it. I'm waiting for AI to catch up and convert the code for me lol. And I haven't heard of qtgmcp, what is that?

Update: 6/29
I tried with intel opencl drivers on my main pc and it worked fine with parameter EdiMode=NNEDI3CL but it does me no good because my encode machine has a threadripper.

[Asd-g]

And I haven't heard of qtgmcp, what is that?

https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/QTGMC%2B.avsi
https://github.com/Dogway/Avisynth-Scripts/tree/master/MIX%20mods
https://rawcdn.githack.com/Dogway/Avisynth-Scripts/f3eba6028ba464ba80cda20dd299c55fbd93b318/MIX%20mods/QTGMC%20Docs%20(v3.33).html

[knuxyl]

Thanks. I'm getting "Error: Script error: There is no function named 'ex_bs'.", so I guess requirements also changed for it.

[Asd-g]

https://rawcdn.githack.com/Dogway/Avisynth-Scripts/f3eba6028ba464ba80cda20dd299c55fbd93b318/MIX%20mods/QTGMC%20Docs%20(v3.33).html#Requirements