§ ¶AVI files and common problems
I thought today I'd actually post some advice that was relevant to desktop video. But first, some commenters to my "I hate Windows" story asked for me to post my port of the Samba 4.x "editreg" tool, so here it is: editreg_src_win32.zip. I've only included the source code because the port is VERY rough and doesn't do full argument passing, and realistically you have to hack it to use it; also, it doesn't actually allow you to edit the registry, only dump its contents. Still, it works. At the time of this writing, you can get the original source code from the official Samba CVS web browser.
Now, about AVI files....
AVI stands for "Audio/Video Interleaved" and holds both audio and video together in a playable format. The basic structure of an AVI file comes from a general file structure called the Resource Interchange File Format (RIFF), which is in turn based off of the Electronic Arts IFF specification that had its roots on the Amiga. RIFF files consist of a series of chunks, each prefixed by a four-character chunk ID and a four-byte length. These chunks may in turn be nested to form a structured file. This chunk structure is nice because any unrecognized chunk can simply be skipped, allowing a file format to be extended without breaking backwards compatibility. The Portable Network Graphics (PNG) standard has an even more evolved tagging system that also encodes "should-copy" and "must-understand" bits so that old programs even have an idea of what they should do with an unrecognized chunk.
Here's what the structure of a sample AVI file looks like:
RIFF 'AVI ' Audio/Video Interleaved file
LIST 'hdrl' Header LIST
'avih' Main AVI header
LIST 'strl' Video stream LIST
'strh' Video stream header
'strf' Video format
LIST 'strl' Audio stream LIST
'strh' Audio stream header
'strf' Audio format
LIST 'movi' Main data LIST
'01wb' Audio data
'00dc' Video frame
- AVI files are composed of general streams; this one consists of one video and one audio stream. You can actually store text streams and even multiple video and audio streams in a single AVI file. (VirtualDub's AVI parser supports multi-video and multi-audio stream AVIs; it's the UI and output modules that don't.)
- Audio and video data is stored in the 'movi' LIST chunk in chronological order. (This isn't actually true of MUSTUSEINDEX is set in the stream header, but I've never seen this used.) Combined with the header being at the front this means that AVI files can actually be played in streaming mode across a network. They won't play that well given that AVI doesn't interleave data finely enough, doesn't have scrambling mechanisms for enhanced audio error masking and doesn't have a rigid enough structure for good error recovery, but you can do it.
- Audio and video data are broken into blocks and mixed together; each video frame has its own chunk, but audio data is organized in small packets. This is the "interleaved" part of the file. The reason for it is so that a player can read the AVI file sequentially and pick up the audio and video it needs without seeking all over the place. This isn't too important for playback from a hard disk, and modern players are very good at handling non-interleaved or badly interleaved files, but proper interleaving is critical for proper playback in embedded devices and CD-ROM playback. Note that despite common belief, the interleaving of audio and video chunks has nothing to do with the timing of the streams, and thus no effect on sync.
- The 'idx1' index lists all of the chunks in the main 'movi' LIST, which in turn holds all audio and video frames. Without this chunk, the only way to find frame 400 would be to run forward through all the chunks in the 'movi' LIST and count them until you hit the 401st video frame. That would be very sloooooooowwwww. The index chunk also says whether video frames are key frames. In fact, it is the ONLY chunk in this file that does. More on that later.
- Note that the header and index are at the beginning and end of the file. Lose the index at the end, and your AVI file will become unseekable or even refuse to play in some players. Lose the headers at the start, and your file is unusable.
If you load an AVI file into VirtualDub's hex editor, the Show RIFF Tree (Ctrl+R) command will display the RIFF tree for the file.
Overall, AVI isn't a very complex file format, and in particular it's very easy to write -- so it's not surprising that a lot of programs support it. Often AVI gets a lot of flak for not supporting a lot of newer video storage functionality, such as subtitles, timestamps, and more complex frame dependencies. Keep in mind, however, complexity has a cost, and not every file format has to, or should, support everything. The MPEG-1 file format is improved in a number of ways but is significantly more complex to support with its precise per-byte timing and bandwidth constraints and lack of a central index. In general, the more features that are put into a file format, the nicer it becomes to write and the harder it is to read... and the harder a file format is to read, the less universally it tends to be supported properly. If a file format is way too complex but still becomes popular, generally the number of incomplete and broken clients using it grows and the usable part of the file spec contracts to the part that people actually need. So file format designers beware!
Now, as for AVI files, there are a lot of mistakes that can be made in creating or transferring AVI files, and VirtualDub tries to be tolerant. Here are some of the problems I've seen:
- Misaligned or broken data chunks: Data chunks in the 'movi' LIST are supposed to be properly tagged and aligned to two-byte boundaries like any chunk, but since their data is pointed to by the index, it is possible to totally scramble the headers in that chunk and still have a playable file.
- Bad RIFF size: Some AVI files don't have the correct size in the top-level RIFF chunk.
- Bad header sizes: When programs crash, they sometimes leave an AVI file that has mostly valid headers and data, except that the sizes on some of the LIST chunks are incorrect. For this reason VirtualDub doesn't validate LIST chunk sizes. This means that the hierarchical structure of an AVI file is lost, but fortunately the order of the LIST chunks is sufficient to decipher the file.
- Old header fields: Some of the header fields in AVI are no longer supported and must be ignored, despite their original meaning. For instance, dwStreamSize=0 is supposed to indicate a stream that has one chunk per sample, which permits VBR -- but in fact it is ignored for audio streams. More on this in a later article.
- Wild headers: A video game emulator was writing out video stream formats that were 150K-600K in size instead of the usual ~40 bytes, due to mixing up header size and frame sizes... whoops. It's fixed now (thanks, Pete!), but I had to fix VirtualDub too because it was crashing trying to Direct-stream those files. (VirtualDub was using a fixed 64K buffer to hold the output AVI header; it's now dynamic.)
- Junk at the end of the file: Some AVI files have garbage after the 'idx1' chunk; for this reason VirtualDub stops parsing an AVI file the minute it has seen headers, the 'movi' LIST chunk, and an index.
- Absolute vs. relative index: The 'idx1' chunk is supposed to point to data chunks using relative offsets within the LIST 'movi' chunk; however, some AVI files have absolute file offsets here instead. VirtualDub's parser, like that of many players, will detect and accept either. Actually, VirtualDub used to write absolute file offsets too, but I fixed that a long time ago.
- Missing or truncated index: This one is very common as it happens with any incomplete file. The problem here is that the offsets and sizes of the index can be recovered by scanning the whole file, which VirtualDub does when it detects a missing index, but the key frame flags that are required for seeking and decoding can't. VirtualDub recovers these by decoding the frames both in forward and reverse order and watching for artifacts; this drives some codecs nuts -- it's not exactly correct usage -- but usually it recovers the key frame information safely. If this recovery process is not done, then VirtualDub has to assume that only the first frame is a key frame, which makes seeking very painful -- potentially requiring every frame in the video stream to be decoded to retrieve the last frame.
- Damaged chunks in 'movi' LIST chunk: This one is fairly common for files that have been transferred over *ahem* distributed networks, due to small instances of corruption here and there, or worse, attempting to play an AVI file for which not all segments have been received. The index mostly allows such files to be played, but it still causes problems in a couple of cases. One problem is that it confounds a player that attempts to stream through the 'movi' LIST rather than using the index; I think the OS/2 player does this. A more serious problem is that it makes recovery more difficult if the index is also missing. When VirtualDub finds a problem scanning through the file it switches to "aggressive mode" and begins brute-force searching through the remainder of the file, looking for consistent chunk headers in order to detect the good parts of the streams. Unfortunately, while this recovers the valid data, it can't determine what times correspond to that data due to the lack of timestamps, leading to a loss of sync. There are probably ways I can determine approximate timings heuristically using regression algorithms similar to what I use in the capture module, based on the chunk interleave, but VirtualDub doesn't do this yet.
In case it's not clear yet, VirtualDub shouldn't be used to determine that an AVI file is valid, because in the interests of compatibility and recovery it allows a number of violations. It also doesn't support a few of the lesser-used parts of the file format, most notably streams with a non-zero start time and in-stream palette changes. However, I've tried to make the parser accept most damaged AVI files and flag warnings for files that it should but can't support, to improve the usefulness of the program.
By the way... what are OpenDML AVI files?
Earlier, I said that chunks are tagged with a four-byte length. This limits the size of any chunk to ~4GB (4,294,967,295 bytes), and in fact, the practical limit is a lot lower due to compatibility concerns, around 1-2GB. The index also has the same limitation and this limits the size of standard AVI files to 2GB. A group called the OpenDML AVI M-JPEG File Format Subcommittee devised a semi-backwards-compatible way to extend this limit, by appending additional structure to a standard AVI file. The result is that legacy applications still can't read beyond 2GB, but the rest of the data is appended after the standard AVI and pointed to by a new type of two-level index. VirtualDub calls this the AVI2 format; by default it writes standard AVI files until it hits 2GB, at which point it switches to the new format.
Finally, there is one major feature of common AVI files that I haven't mentioned yet: VBR audio. In particular, I mean VBR MP3 audio. For various reasons VBR audio both is and isn't supported in the file format, and the way that it is implemented in practice is also both supported and not supported. In a future post I'll go into the technical details of how VBR MP3 audio is popularly implemented and the reasons for VirtualDub's behavior with it. I already wrote on this a long time ago, but it's a common enough question that the answer bears repeating.
Ah... things are clicking into place in my brain now :-)
Many thanks for the intelligence within VirtualDub. Its saved (for the most part) many avi's from scratched CDs.
Very informative reading these posts about VirtualDub and a/v media. Thanks for sharing with us.
SonOfAdam - 11 11 04 - 07:09
You may find AVIMaster (http://www.thozie.de/avimaster
) useful, which focuses on parsing and reconstructing the structure of AVIs (and I'm quite proud what it ingests) and issuing warnings on conformance. So you get a report on the file, statistics and all, and if necessary an as good as possible remake. Warning: command line utility...
Thomas Ziegler (link) - 11 11 04 - 13:23
Don't forget applications that write bad dwDuration values into the indx chunk :D :D
Concerning "VBR MP3", it seems to work really nicely, as the same thing works with AAC and HE-AAC as well, even with M$'s b0rked AVI splitter. "Frame-wise" is not necessary; AVI-Mux GUI successfully puts several MP3 VBR frames into one chunk now if the user wants to.
Also, "Open DML 1.02" actually requires that the first RIFF list be not larger than 1 GB. I don't know which idiot set the limit to 1 GB, but it is there
Alexander Noé - 11 11 04 - 14:11
I did fix that dwDuration bug as soon as I heard about it. :)
Sadly, in practice the compatibility features of OpenDML hierarchical indexing are not very useful because older programs balk when they can't read the whole file (dwLength in the stream header points beyond valid data in the legacy index).
VBR audio does not work really nicely -- it breaks compatibility with several parsers. More on that next time.
Phaeron - 12 11 04 - 00:07
Again, I just have to say I love reading these posts. I really enjoy the details and information contained in them. I am learning, rather quickly, about things that would take me months, perhaps, to learn elsewhere.
Abu Hurayrah (link) - 12 11 04 - 22:16
Good Document. Helped to know the basics of AVI format
Xavier - 06 02 05 - 22:45
On another page you wrote "I'm not going to speak on the properness of VBR audio in AVI because almost everyone knows how I feel about this."
- Unfortunately despite spending a long time of googling for qualified answer about whether or not VBR audio in .avi files is "acceptable" or not, i still haven't really found any good answers, except for some _very_ technical (and more or less unconcluded) discussions on the topic. Additionally it seems that DivX Networks are encouraging - or at least not warning against - using VBR audio in AVI. They even require hardware vendors to support VBR audio in AVI to become "divx certified"! But is this reasonable?
So i'd be very happy if you could write some of your personal views on this subject. I'm quite concerned about the portability issues, and if this is as bad an idea as i suspect, someone should stop all those people currently using this format!
JoaCHIP (link) - 21 03 05 - 07:01
I am recieving an error message that i have never seen before and i cant seem to find out the answer into resolution. if you could please assist with with this issue - if there is a way around it i would greatly appreciate it.
error -- couldnt locate decompressor for format 'wmv3' (unknown)
virtualdub requires a video for windows (vfw) compatible codec to compress video. directshow codecs, such as those used by windows media player are not suitable.
Kristina Mount - 28 07 05 - 01:34
Kristina Mount: I don't know if you fixed this problem by now, however, I think your solution would be to install the “WMV3
” (or Windows Media Video v. 3, some time Microsoft calls it v. 9) codec
. Go to Microsoft's web-site and download the latest Windows Media Player and install it even if you don't want to use their application. It should install the required codec
. I hope this helps you.
Note that I do NOT
use/used Windows for the last five years!
Ziyad A. M. AL-Batly - 13 09 05 - 23:34
i want to know how to extract RGB from the avi file in the data chunk.
praveen - 06 12 05 - 01:22
Hey everybody. I have got a problem. I've deleted avi file (compressed from DV video to DivX by DrDivX). Then i noticed, it was deleted. I recovered it, but it can't be played. The proview doesn't show in windows explorer. When i drop it into Virtual Dub is starts to reconstruct missing index block, but after 10 minutes of working i can see only first frame. When svaing Direct stream copy, size doesn't change, but player plays only first frame. It is rather important video for me, and it'll be bad if i loose it. Does anyone knows probable resolution for it? Thanks a lot.
Max - 08 01 06 - 06:28
praveen - for a visual look at how RGB data appears in an uncompressed (full frames) AVI file, open the file in a hex editor (look up "frhed" in google - great free editor), and search for chunks beginning with either "00db" or "01db"; the RGB data for each frame is preceeded by this + a 4 byte value that indicates the size of the frame.
If the video frames in the file are compressed, then you will need to use the appropriate codec to decompress them.
Hope this helps.
Jon - 19 02 06 - 16:52
Earlier someone wrote about problems reading wmv3 files. As another poster noted, this is the Windows Media 9 codec. Go to the Microsoft site and look for the Video for Windows codec for Windows Media 9. Whenever I install a machine I have to poke around a bit to find it and usually its not until the 2nd or 3rd download that I actually get the right one.
Gerald - 13 07 06 - 18:28
I've got a problem:
When saving to avi-file the following error occurs:
'audio samples 477xxxx-477xxxx could not be read from source. The file may be corrupted'
I have to delete a part of the video and that's not what I want.
Is there a way to save the file without deleting this part of the video (with audio!)?
Dusty - 06 08 06 - 14:04
I have an important AVI file taken on a Fuji Finepix F30 that was inadvertently corrupted by plugging in the power cord during a transfer. No data recovery software has been able to extract a working AVI file. The video is actually encoded using M-JPEG and the audio is mono wave. I was trying to find out if there are any utilities that would work better than what I've tried:
Easy Recovery – Ver 6.1
R-Studio – Ver 3.0
BinaryBiz VirtualLab Data recovery v5.5.
File Scavenger 3.exe
GetDataBack for FAT V3.03
PC Inspector Smart Recovery 4.43.exe
Restorer2000 Pro 3 build 123007.exe
Stellar Phoenix Windows (FAT & NTFS).exe
Zero Assumption Recovery 8.0.exe
In a few instances an AVI file was recovered that was unplayable. Also, in one instance FIL0.AVI was found immediately followed by FIL0.MOF and FIL1.MOF. I think the MOF files may correspond to the video MJPEG stream, though I’m not sure.
I also tried these recovery programs that were specifically designed for recovering pictures/camera data:
Magic Photo Recovery PRO v3.0.exe
FileRecovery for xD-Picture Card
PhotoRescue 2.1 Demo wizard.exe
PhotoRescue 2.1 Expert Demo Version 682
None was especially helpful.
For the unworkable AVI files I was able to recover, I tried the following programs in hopes of correcting the internal inconsistencies:
ASF-AVI-RM-WMV Repair v1.82.exe
Divx Avi Asf Wmv Wma Rm Rmvb Fix Joiner
DivX Repair 1.3.exe
Video fixer 3.23.exe
What the file signature is and detailed info relevant to the MJPEG stream that might allow me to recover the data manually? I’m not afraid of getting dirty with a hex editor if need be.
John Whitlow - 09 11 06 - 10:28
I looked into divx avi created in direct show and see 00db and 00dc data chhunks. Similar file created by other program has only 00db. What is the difference between 00db and 00dc data chunks.
anatolik - 20 11 06 - 02:40
What is the best way to capture a video stream from a vcr tape, and encode it to avi and clean it up a bit, bu tkeep the file size near vcd levels, WHEN armed with virtual dub? I have been capturing files with virtual dub, but get alot of sync problems but this is my processor running at or near 100% it seems, so now i have been capturing uncompressed and no more sync problems. so now what, i have read alot of articles... i am guessing to use virtual dub to a lamemp3/divxmp4 format. i notice that when i capture into divx at 2500kbps on the divx codec filter, the video states to be capturing at near 700kbps on VD. this is confusing. So i hear we can nicely fit atleast 2hrs video onto a cd-r, so i assume that near 800kbps max on the video encoding and using mp3 audio is all that is needed to nicely achieve this.
If thats all, then what about any filtering though? Inverse Telecine? and any extra interleaving, smoothing a good idea? I sometimes like a 15% sharp as the only needed filter, when capturing into divx. But my processor cant handle this, so i have to introduce another step that adds hours to the process... Of course that extra step was to capture raw, then filter after on a second pass. I thought virtual dub would be good for a one run, capture into a filtered finalized avi. DO i just need a good processor? Or is the payoff just WAY better to capture raw, then encode after?
Brian (link) - 06 02 07 - 03:00
by the way, im using a pinnacle capture card, and have composite in from the vcr
Brian - 06 02 07 - 03:04
ok, im back at it, i have been recapturing with a super processor now(1,5Ghzfsb,pc6400DDR2-2GB-am2), and no more processor clipping problems. i find that i have been able to squeeze video and audio compression and still maintain a grip on sync, but still no cigar. its not quite good enough. Maybe theres a good format out there for capturing? i try even raw capture, with no audio or video compression, and still end up with sync problems. whats the deal? is it just the way vcr tapes screw up the sync rates? i believe capturing from the tv signal has NO sync problems, then when i push play ont he tape, i watch my sync meters go off the screen for a bit until they funally go into a wavy oscillation. the resample rate gets all the way around +4 to +4, and at something like .97 times, and the resync goes near 100ms, i know this is not too bad, and my processor seems to be hanging on to the sync, but the end result is still out of sync. again, when i capture from the regular tv tuner through the vcr, the lines are all merged and synced. still learning...
p.s. thanks for the awesome virtual dub support, i couldnt be digitizing my entire vhs library without you!! hats off!
brian (link) - 02 05 07 - 13:53
I've been having problems with my avi's I try to import them into my Windows movie maker and adobe premier and they can't find some sort of codec. It worked perfectly before I have no idea what's happening!! could someone help me?
Rose - 02 06 07 - 00:03
I have been unsuccessful in recombining four AVI files. I am able to open up the first file, but then I receive a series of error messages:
[!] AVI: Index not found or damaged -- reconstructing via file scan.
[!] AVI: Invalid chunk detected at 272618258. Enabling aggressive recovery mode.
[!] AVI: Keyframe flag reconstruction was not specified in open options and the video stream is not a known keyframe-only type. Seeking in the video stream may be extremely slow.
What do I do?
[!] AVI: Variable bitrate (VBR) audio detected. VBR audio in AVI is
non-standard and you may encounter sync errors up to 7154ms when
attempting to extract WAV files or processing the audio in Direct Stream
Copy mode. Full Processing mode is recommended to decompress or recompress
the audio. (bitrate: 127.9 ± 18.3 kbps)
George - 19 06 07 - 16:49
Very interesting to learn about AVIs and how they work. I am having troubles exactly like Max's where i accidentally deleted some AVI's and recovered them. Some of the AVI's work, the preview shows up and they open just great. Others, not so much. It says they are an AVI but they have no thumbnail preview and when I try to open them Windows says something like a codec not found. These AVI's worked fine before. I have tried several AVI recovery softwares with no avail. Some say they "fixed" the files but they still wont open. One said it was due to bad frames at random points throughout the file and it fixed the files by skipping those frames, it still did not work after.Is there something i'm missing here? Help, i really want to get these AVI's back!
Jeremy - 03 07 07 - 11:06
I wanted to know how seeking can be made possible in AVI files. This is required for the support of fast forward & rewind. As far as my knowledge is there i dont think there is any timestamp information is available in the AVI file. If it is , then please someone let me know where or how can i calculate.
G Bansidhar - 23 07 07 - 11:30
I'm using virtualdub 1.9.9 to capture AVI files. I then use software I have written to do processing on said files. My program opens the AVI file and looks at the main 'avih' header at the start of the file to extract the number of frames. It seems that virtualdub is writting incorrect values to the dwTotalFrames, I have to seek into the 'strh' vids chuck and extract the correct value from dwLength. I tried recording with fraps and hypercam which produce correct files so I have narrowd down the problem to virtualdub. Its not a major problem but still, it is incorrect. Sorry if this is the wrong place to post, I hope somebody reads this.
Mark - 12 09 10 - 06:43
Ignore my above post, it seems its correct behaviour for AVI 2.0 RIFF file format, if the avi contains AVIX chunks.
Mark - 13 09 10 - 00:50
I use Virtual dub 1.9.11 to dub a movie. But as soon as i open a media file, it shows an error. The error is as follows :
Couldn't locate decompresser for format 'XVID' (unknown)
Virtual Dub requires a Video for windows (VFW) compatible codec to decompress video. Direct Show codecs, such as those used by Windows Media Player, are not suitable.
Kindly help me with this problem and post here only. Please help fast.
Yogesh Prasher - 11 04 11 - 16:51
I have donload a 997mb *.avi file and it plays the first 18secs then opens up some spam website !! I have tried cutting the avi but the programs only identify that there are 18secs of play tim eout of the 997mb ?? and help on this matter or is it a dump file
Adam - 01 05 11 - 01:25
Yogesh are you converting MOV to AVI? try to convert using MJPEG. (FormatFactory)
Bob - 24 06 11 - 15:02
is there a list of codecs that virtualdub is compatible with? i would really like to use the program but it doesnt open half the video files i try to feed it. could such a list appear in a "compatibility" section somewhere in the documentation or the website? thanks.
i also get the VFW error by the way. google brought me here.
josh - 26 01 12 - 04:06
my video keeps freezing after i've apprehend some videos together and save in avi...is there any way to make sure it doesnt freeze when i play it in avi? (using virtualdub 1.9.11)
PROBLEM - 15 02 12 - 00:55
I have a problem on .srt file it says that the first sub start on 0.0.8.495 and after I saved the video with the subs the same sub starts earlier what is the problem???
George - 04 06 12 - 01:20
I produce videos for our church. I receive AVI files that virtual dub reports "Invalid AVI file: The main 'movi' block is missing." When I use the hex editor, I see the movi block starting at 0x10000 with 'movi' characters just before the data at 0x10000. I have to use Super to convert from AVI to AVI in order to be able edit them with virtual dub. I would like to eliminate the conversion step as it doubles the raw disk space I need to process and takes hours. These are big files up to 90 minutes in length. Can you suggest anything I could try to debug the problem. I am wondering if it is a codec problem The files work fine in Windows Media Player, but not in VLC.
Rich - 18 08 12 - 04:58
It looks like the idx1 section is missing on the file. Is there a way to regenerate it other than using conversion software (like Super) to do it?
Rich - 19 08 12 - 01:02