You could sum it all up with the Israeli saying - don't be smart, be right. It's usually applied to things like driving a car. For example, if a mad driver tries to bypass you, it might be better to let him/her through than risk an accident just because the law is on your side.
With that said, what you should have done - like other programmers do - is put a big "If your antivirus detects Virtualdub, than know it's just a false positive" message. I know it doesn't really prove anything, but it is reassuring to know you don't have a bad/fake version of the program on your system.
Lior - 29 03 09 - 05:06
I meant a message not in the program itself, of course, but in the front page of your site.
Lior - 29 03 09 - 05:07
Thatís just one more reason not to use UPX in particular and executable packers in general.
Yuri Khan - 29 03 09 - 05:19
Why whould i refuse to use UPX just because some AV suck. UPX is wonderfull piece of software and support more platforms, then any antivirus. Can't see why not use one high quality program(VirtualDub) with another one(UPX)
chornobyl - 29 03 09 - 07:00
FWIW you can use http://www.virustotal.com/
to upload files and get back a list of scan results from almost 40 different AV programs.
I usually use it when someone posts something I'm not sure about to a forum I help run, just in case there's malware in it which my personal AV program didn't detect.
Of course, with 40 AV programs you almost always seem to get false positives. Some of the AV programs are really crap but at least VirusTotal will let you know what to expect (and it includes non-crap AV programs as well). With UPX files you always get a lot of false positives, too.
Leo Davidson (link) - 29 03 09 - 09:23
Antivirus vendors that do this have size of the brain equal to those malware writers who thought they can hide in UPX. :)
GrofLuigi - 29 03 09 - 14:24
Using UPX to pack executables is often not as helpful as it may seem at first. If you pack the executable with UPX (or any other executable packer), you have the advantage that the file uses less space on disk. Which is a good thing, right? Well not always, because when you run an UPX'ed executable, it has to be loaded into memory completely, to be correctly uncompressed. So you don't get the cool benefits of demand-paging that you normally have with EXE files (i.e. windows only loads the parts into memory that are actually used). That is probably not noticeable on a rather small EXE file like VirtualDub, but it is one of the reasons why projects with larger EXE files don't compress it (take MAME for an extreme example)
Darkstar - 29 03 09 - 18:53
Honestly, I haven't seen a UPX-related issue in a long time, since I think at this point all major AVs now have logic to unpack UPX executables. UPX does make the final archive smaller than just .zip alone, so it hasn't been worth removing.
Arranging the executable so that it can be demand paged isn't a benefit for memory consumption -- Windows will trim your code pages to the working set whether it can page directly from the EXE or not. The main benefit of doing so is allowing Windows to use the EXE directly instead of the page file. You're correct in implying that VirtualDub isn't optimized for low memory conditions. If it were, then I probably would compile the entire application with /Os, optimize the link order, and use memory mapping for I/O. As it is, though, it's a bit silly to do so when VirtualDub's entire code segment fits in two video frames (and it's not very fun work).
One thing that enlarges VirtualDub's code segment is all of the padding that the VC++ compiler puts between functions, which is due to /Ot (optimize for speed). For small functions it can contribute as much as 30-40% of the raw footprint, and it's useless if you have an optimized core that doesn't call functions in critical paths. Unfortunately, there's no option to turn off only this padding. I used to compile some portions of VirtualDub as /O1 to reduce this, and it shrunk the raw executable by almost one-third, but I stopped doing it because it was a pain to maintain at the project level and UPX compressed it all out anyway.
Phaeron - 29 03 09 - 20:40
Use Linux guys ... ;)
(Ok I still am not happy that Linux has its own share of problems... I still want Avisynth 3.0 ... but as far as Viruses is concerned, I never had any problem whatsoever on Linux with viruses. Sure, other problems... but no viruses anywhere.)
mark - 29 03 09 - 21:12
Personally I think that packing the executable was convenient back in times when you needed your code to fit onto one floppy disk or you needed to count every byte on your hard drive. Currently a lot of VirtualDub copies fit onto a DVD and I think I have low disk space when there's less than 10 gigabytes free so I think I wouldn't mind at all if VirtualDub ceased to stop using UPX just to reduce number of false positives. Size of the zip archive or VirtualDub load time seem to be quite unimportant factors to me.
Kasuha - 30 03 09 - 08:13
@Kasuha: a 500 Kb .EXE is fast to read, and is unlikely to fragment. Due to the way UPX works, it is also very fast at unpacking said binary (CPU-wise). This, in turn, allows VirtualDub to be included in stuff like free software compilations on a USB key, to load instantly no matter what the PC is doing or how badly fragmented the system hard disk drive is (loading 1.8 Mb of data from a HDD, when fragmented AND busy, can be incredibly long; something like a few dozen seconds - while loading 500 Kb is fast, no matter what), and also is much less costly bandwidth-wise, as UPX can compress executable binaries much better than, say, ZIP.
As such, using UPX should be pretty much mandatory, and not the exception; "smart" AVs are able to recognize UPX as it is, and indeed scan the unpacked binary; lousy ones don't, and assume only virii authors use UPX; since UPX 'signs' the binaries packed with it, it is seen as much less time consuming.
For the sake of testing, I once packed the Mozilla Firefox 2 main EXE: it disallowed small automatic patches. But damn, did it load faster!
Mitch 74 (link) - 30 03 09 - 08:37
You can read the detailed arguments on my blog: http://hype-free.blogspot.com/2006/11/to..
But the gist of it is: it is (almost) never useful to pack executables. Any speedup you get from it is purely imaginary :-), since (as the blogpost describes in details) you actually have to read more from the disk, not less, during application startup.
Cd-MaN (link) - 30 03 09 - 10:04
I've had my own share of problems with lousy antivirus programs - I use Inno Setup as the installer for GIMP, and for a while I'd be receiving at least one report per month that the uninstaller is infected. I'd put a warning on the download page every time it happened, and last year I decided to start doing a hall of shame of all AV products that did this - except that since I started that, it only happened once more (and that time, it was one of the plugins that was triggering the false alarm instead).
ender - 30 03 09 - 14:09
did some tests
1256127 unp-u7z.7z smallest package
1332790 upx-5s.rar good choise imho
1346695 upx-mix.zip compatible as current
1594047 unp-mix.zip max compability(zip no upx)
5s-winrar best solid
u7z-ultra 7z optimizer
mix-zipmix 7-zip deflopt
chornobyl - 30 03 09 - 14:10
The startup time issue is too vague to make a decision just based on the compression. While it is possible that the OS faults in fewer pages when starting an uncompressed executable, the fault pattern will be less efficient than UPX's decompression.
Note that for the main distribution, I only use .zip because that's the only archive that everyone's basically guaranteed to be able to open. I do use 7-zip for the source archive, and it does compress significantly better.
Phaeron - 30 03 09 - 14:26
Two comments regarding the antiviruses:
1. Yes, most antiviruses are able to unpack UPX and scan the inner content. However, it means that the compressed data have to be unpacked twice - first by the antivirus (which has to use slower code, with lots of range checking, to prevent possible security problems), second by the loader itself. So, startup of the executable is certainly slowed down.
2. The [in]ability to unpack UPX in an antivirus engine doesn't have anything to do with false alarms on UPX. The antivirus may be able to unpack this format, but a (unrelated) virus signature in its database may have been chosen (incorrectly) such that it detects part of the UPX unpacking stub. Or, it detects a common compressed snippet (e.g. MS CRT or Borland VCL, packed by LZO - even that happens). The companies receive thousands of packed samples every day, many of them are packed by UPX - but not just pure UPX, but further scrambled/crippled with custom UPX scramblers - so mistakes like that can easily happen.
And, the truth is that since the biggest companies (Microsoft, Adobe, ...) don't pack their executables (where false alarms would be very serious), so false alarms on packed executables have somehow lower priority to solve.
i_g - 30 03 09 - 14:46
You just reiterated my point: Valid programming techniques are being invalidated due to interference from third parties that are largely unaccountable if you are a small-time developer.
Phaeron - 30 03 09 - 14:53
a whole post for little old me?
I'm still very much for a change from registry to .ini, it might be a old api, but its not going away, Vista still uses desktop.ini files all over the place. ini files are not only good for portable apps, but they reduce registry bloat (I don't run VDub 24/7) On some systems (w2k IIRC) the registry is in the nonpaged pool so keeping the size down is a good thing, not that VDub is the big problem, its more the stupid stuff stored in there, mostly by MS. I'm looking at you .NET and MSI!, you also have stuff like ...Windows\CurrentVersion\SharedDlls that should have an API to modify the list, and that list should not be in the registry etc...
asf - 30 03 09 - 21:51
More false positives:
I think anti-virus vendors need to tread carefully - too many false positives and people may just go "The Boy Who Cried Wolf"-style and stop believing them all together.
yawnmoth - 31 03 09 - 11:42
I would characterize that last ZSNES case as particularly irresponsible. Instead of putting up a dialog saying that a program is using Microsoft DirectInput and asking the user to confirm, the AV just put up a cryptic warning about dinput8.dll and a keylogger, leaving it to the software vendor to explain:
1) What dinput8.dll is
2) What DirectInput is
3) Why the AV is mentioning dinput8/DirectInput instead of the application
DirectInput is used by so many full-screen games it's not even funny.
Phaeron - 31 03 09 - 14:30
*annoy-mode* Maybe a good point to think about using the users applicationdata folder files instead of the registry. You once said it is about database-like features - not that there is so much concurrent access to VirtualDubs config file... */annoy-mode*
DJT-Rex - 01 04 09 - 02:12
Avery is right as usual:
1. UPX packing is completely legit
2. AV vendors are jerkwads
3. Too many false positives as of late (some stuff like keygens is even being added on purpose and I believe that AV vendors are getting a nice profit for allowing arbitrary entries to their blacklists)
4. That said, if VirtualDub were digitally signed it could have stayed compressed by whatever means. It seems almost magical that signature from trusted CA practically means "lower the shields and let me in"
5. Once everyone starts using trusted CAs, we will again have no way to distinguish the good guys from the bad ones. Ouch, back to square one.
Needless to say, that will have the largest impact on small independent vendors.
I never use AV because I don't want to support them -- they are selling snake oil, not protection. There is always a time window between virus outbreak and updated signature database during which you are vulnerable like everyone else and you are paying for that privilege. Ah, what to say, sheeps are for fleecing.
Igor Levicki (link) - 04 04 09 - 16:20
I'm not opposed to signing the VirtualDub executable, but I am opposed to paying a yearly fee to be able to do it, and more opposed to the apparent requirement that you need to be a company to be able to sign code.
Antivirus software was more effective, I think, in the days when people wrote viruses for fun. The viruses still did a lot of damage, but in that case the AVs were frequently able to stay far enough ahead, especially when people still managed to get infected with OLD viruses, mainly out of carelessness. Now that malware is being often exploited for gain, I think we're seeing more sudden outbreaks.
To clarify, I'm not totally opposed to the idea of antivirus software, especially where they are able to fill gaps in application or operating system security models or where they can catch previously unknown threats through heuristics. What I'm opposed to is the lack of checks and balances where third parties can be improperly accused of wrongdoing and there are no established procedures for getting mistakes resolved.
Phaeron - 05 04 09 - 01:07
> 4. That said, if VirtualDub were digitally signed it could have stayed compressed by whatever means. It seems almost magical that signature from trusted CA practically means "lower the shields and let me in"
It doesn't actually. All my installers for GIMP for Windows, and all binaries in the installer are signed by a certificate donated by Certum, but that never prevented random antiviruses from flagging it as malware at random times.
ender - 05 04 09 - 16:30
@Igor: as someone who worked for years in the AV industry, the statement "some stuff like keygens is even being added on purpose and I believe that AV vendors are getting a nice profit for allowing arbitrary entries to their blacklists" is completely without foundation. The real reasons for keygens being detected is:
- some of them are fake and actually are malware
- some of the them are bundled with malware (thing SFX which extracts two executables - a trojan and the keygen - and runs them both). This is actually quite common
- quite a lot of keygen writers have the impression that packing is "cool", and many of them use packers mainly used by malware. For the AV it is easy to blacklist the packer, since it catches a large amount of malware and doesn't give FPs on legitimate software.
So let me repeat it again: I never ever seen any AV companies signing cracks or keygens on purpose or because some vendor asked for it (also, it would be a bad financial decision from the vendors part: if there are N AV companies and M crack versions, you have to convince each AV company to blacklist all cracks to be effective - and also you need to stay on top of the new cracks appearing).
Cd-MaN (link) - 06 04 09 - 04:46
> For the AV it is easy to blacklist the packer, since it catches a large amount of malware and doesn't give FPs on legitimate software.
...and that's the main problem with AVs nowadays: they do the easy thing, and don't care about collateral damage. How many times have NSIS and Inno-based installs been flagged as malware because somebody did the easy thing, and blacklisted all files that came with the malware, even if they were completely innocent 3rd party installers, or the program was simply packed with a free executable compression tool?
ender - 06 04 09 - 05:52
"For the AV it is easy to blacklist the packer, since it catches a large amount of malware and doesn't give FPs on legitimate software."
I'm not sure what message you're sending here. Are you trying to say that any packed executables aren't legitimate software, or are you trying to say that software is only legitimate if it's digitally signed?
Bandwidth is fairly cheap these days, but still not free. There are plenty of people out there still stuck with dial-up connections. Sites like SourceForge may be free for projects like this, however they still have hosting costs that increase with bandwidth consumption.
As for the second possible message, there are countless OSS (or free closed-source) projects out there that are much more legitimate than their commercial counterparts.
The day that AV software starts flagging apps like the Google Toolbar as Malware is they day I start having even a small amount of faith in them.
subhuman - 09 04 09 - 17:46
The message here is that packed executables are suspicious.
If I have an executable I am uncertain about (which is most of the files you download from web these days, considering the number of sites hacked every day) and want to check what it's about, I simply open it in a hex editor and quickly browse through it. Usually, it's possible to get a pretty good idea according to the strings inside (yes, I know this is nothing really conclusive, bad stuff can be hidden, but it's just a quick check). Now, if the file is packed... it looks like it's trying to hide something - for no good reason.
Yes, digital signatures certainly help - but it still depends on who issued the certificates (I'd say you can trust Verisign certificates, but there is quite some malware signed with Comodo certificates, for example).
So, except for very special cases, packing your product will do more harm than good, both for you and your users.
>...and that's the main problem with AVs nowadays: they do the easy thing, and don't care about collateral damage.
Depends on the point of view; AVs care about their users. So, the question is what damage is bigger - having a FP (which affects the particular AV users as well, of course, so it's not something to ignore) or allowing their users to get infected because of undetected malware.
I don't believe an AV simply blacklists the whole NSIS/Inno installer on purpose (which I'm not sure you're suggesting, but it sounds that way) - when it happens, it's a mistake caused by the incredible number of malware samples that have to be processed each day. Hard to do much about it, I'm afraid.
> Bandwidth is fairly cheap these days, but still not free. There are plenty of people out there still stuck with dial-up connections.
If you want to save bandwidth, you pack the file with ZIP/RAR/7-ZIP, possibly self-extracting, before you put it on web - not with an executable packer.
i_g - 10 04 09 - 08:11
So I'm supposed to anticipate third party interference with a version of my program that didn't have any problems when I released it a year ago, with rules that deliberately aren't published and change with each version of AV signatures? I'm sorry, but that doesn't scale on my end.
You got it correct when you said that AVs care about their users. They don't care about MY users. False positives don't hurt them unless their users know that they're occurring, which is almost never the case because the user is depending on the AV's expertise.
If you personally don't want to run my executable just because it's packed, that's fine. I'd hope that if you wrote some software that monitored other executables that you'd take a little more care before firing false accusations.
Phaeron - 10 04 09 - 15:21
I wasn't talking about VirtualDub in particular and I am not making any accusations.
All I'm saying is that packing doesn't bring any advantage these days - and since, on the other hand, it brings quite a few problems with false alarms (which is not your fault, of course, but that's how it is and I don't think there's any way out), I find it simply a bad idea to do so.
Btw, it's not really true that false positives don't hurt the companies; if the false positive occured before the user attempts to download/install the program, then maybe (they'll think it's a real warning and don't even try to get the program) - but if it occurs later, after the program is installed for a long time, quite a few users notice that something strange is happening and report the problem. Of course, the situation in corporate environment (where the AV is usually set-up to remove the "infected" stuff automatically) is even worse - the customers are not very happy when the antivirus interferes with the tools they need for their job. But, as Cd-MaN was trying to say when speaking about "legitimate software" - software you normally find in corporate environment is not packed, i.e. the risk of similar incidents is significantly smaller.
i_g - 10 04 09 - 18:32
[..] software you normally find in corporate environment is not packed, i.e. the risk of similar incidents is significantly smaller. [..]
Well, it may not be packed as often - but on the other hand most of the time business or expensive corporate software uses something much worse (I think) -> copy protection of some sort. For example scrambled/crypted executables etc. .. nowadays there's quite some "evil" stuff out there that relies on so many OS/driver specifics etc.pp. it's not much fun any more. Something like UPX is a god sent in comparison. ;)
Sorry for ranting.. I just downloaded/installed the Cineform NeoHD trial and I am quite nerved out by there strange "ensure that 14 days trial really means 14 days and not one more" protection mechanism. It relies on several cryptic strange registry keys (some under HKCR\CLSID, some as subkeys under other legit regkeys, some obviosly "fake" etc.etc.) - it basically gobbled much of the registry with it's "trial protection" keys which obviously won't be removed when I uninstall it to ensure I won't use it again for another 14 days. (add to that a challange/response like combination of invisbly-called-by-the-installer activation/deactivation programs and whatnot - I'm not a hacker so I will never fully know what else it did to my OS installation)
What that means for me personally: a simple _CODEC_ that is as expensive as Cineform and obviously had so much work put into their stupid protection system(s) (which perhaps adds to the price? ;o) is just to unreliable for my low/middle budget production work.. when I quickly need it on another computer or my OS installation/registry gets somehow corrupted I risk getting stuck with a non-working codec that cost me several hundred $$$...
The strange thing is (to bring this post slightly on topic again ;o) ... all this "hidden" modifications the program did, and not once did the antivirus complain!
However, when I used sysinternals Tools for tracking what this codec/it's installer did.. yeah.. suddenly my AV woke up again and bombarded me with warnings that I was using evil hacker tools from hell.. great!
Soeren - 13 04 09 - 12:45
Phaeron obviously puts a lot of effort into making Virtualdub easy for people to use, even beginners. So I applaud his pragmatism in making a (relatively) minor change to try and avoid potential false positives from AV, which naive computer users (i.e. most users) wouldn't be able to tell from a real positive. It's all very well being pigheaded and insisting AV vendors do something about the problem, or that his audience somehow become less naive about false positives, but all that would do is hurt his users. The goal isn't to be a purist, it's to make sure people can keep using Virtualdub with as few hassles as possible.
Re the comment about packed executables being suspicious... it would be a pretty poor virus or malware that was able to be detected through looking at plain text strings in the executable. Using that method, virtually every piece of software with any copy protection at all would be deemed suspicious since they use much more determined methods to obfuscate executables than simply packing them. If you're that worried about what downloaded programs are going to do, it would be much quicker to just run them in a sandbox and track what they do with simple tools like those from sysinternals. If it is something that you just want to run once then never again, use sandboxie.
ElTorqiro - 14 04 09 - 10:44
> it would be a pretty poor virus or malware that was able to be detected through looking at plain text strings in the executable.
That's indeed the case for a vast majority of malware. Don't expect most of the malware authors to be very bright; they take an existing module, possibly do some small changes, repack it with a few packers (sometimes not even noticing that the combination of packers they had used resulted in an executable that cannot even be started) - and start to spread their "creation" on web.
Malware that is interesting / good from a programming point of view - is quite rare. Today's malware world is much more about quantity than quality, so to say.
i_g - 14 04 09 - 14:33
"The day that AV software starts flagging apps like the Google Toolbar as Malware is they day I start having even a small amount of faith in them."
Are you joking? IMO, this is so ridiculous that I misread this at first.
Yuhong Bao - 15 04 09 - 01:22
This sounds like a very mature attitude to take... and one that I wish other authors would consider.
Kentaro (link) - 16 04 09 - 16:51
AntiVirus? Hmmm? Oh yeah, that silly thing you need on Windows... (-;>
Yogarine (link) - 17 04 09 - 06:52
Comming back in a few word to your original post. I'm in mind that you took too much care of your users :-) if even one percent of them would have take care of you as much as you do for them; you would have been very rich since now.
By the way, you shouldn't consider some isolated cases as a problem, it's just some natural error that arise. Considere that for each complaint you receive, there are all around the world another 20 or even more guys that heapilly use virtualdub without any issue.
And if there remains some lazy AV users, just tell them to turn it off while they use virtualdub. If they really want to use it, don't worry, they will find a way to do so.
bork - 17 04 09 - 15:51
You're assuming they have the option of turning it off. In a corporate environment, IT often has settings locked down so that this is not possible.
Phaeron - 17 04 09 - 16:01
If you have problems with packed executable - just unpack it!
Millions like VirtualDub and hundreds can't launch it. Hundreds have to use The Utility (UPX) to play with this great program. I think, it's fair.
Yorik.sar - 19 04 09 - 08:12
I agree with "i_g - 10 04 09 - 08:11".
Packing in UPX these days is a sign of suspicious behavior. Download sizes and disk usage are hardly prime concerns for anyone, rendering the primary purpose of UPX a moot point.
I consider EXE packers to be a gimmick (when not being used to obfuscate malicious code, obviously.)
One must ask: Does the benefit outweigh the risk? What exactly is the benefit?
Josh Straub (link) - 05 12 09 - 09:28
> Download sizes and disk usage are hardly prime concerns for anyone, rendering the primary purpose of UPX a moot point.
This is simply not true. Program size is a concern anytime you're dealing in either a small storage (think netbook or USB stick), or in a quick bootstrap scenario (installers, particularly web-based). Just because you don't have this problem doesn't mean that no one does.
> Packing in UPX these days is a sign of suspicious behavior.
That's self-reinforcing thanks to the false positive problem. When I and others started using it, there was no such problem, and such UPX-packed programs are now retroactively affected.
Phaeron - 05 12 09 - 10:01
I am a small-time developer, I use UPX to pack my executables. My most popular download is 928KB unpacked, 436KB packed. That's a 53% saving; once upon a time I was getting an 80% saving, though at that time it was ~800KB unpacked.
With several thousand downloads per day, it adds up. I think it is worth using UPX to save bandwidth and to provide an EXE that doesn't have to be unpacked with winzip or winrar.
When I first started using it 1 or 2 AV's would flag it purely because it was using UPX, this is no longer a problem. I still have false positives, but they are unrelated to UPX and more related to the type of software and what it does.
Macka - 17 05 10 - 23:03