Current version

v1.10.4 (stable)

Navigation

Main page
Archived news
Downloads
Documentation
   Capture
   Compiling
   Processing
   Crashes
Features
Filters
Plugin SDK
Knowledge base
Contact info
Forum
 
Other projects
   Altirra

Search

Archives

01 Dec - 31 Dec 2013
01 Oct - 31 Oct 2013
01 Aug - 31 Aug 2013
01 May - 31 May 2013
01 Mar - 31 Mar 2013
01 Feb - 29 Feb 2013
01 Dec - 31 Dec 2012
01 Nov - 30 Nov 2012
01 Oct - 31 Oct 2012
01 Sep - 30 Sep 2012
01 Aug - 31 Aug 2012
01 June - 30 June 2012
01 May - 31 May 2012
01 Apr - 30 Apr 2012
01 Dec - 31 Dec 2011
01 Nov - 30 Nov 2011
01 Oct - 31 Oct 2011
01 Sep - 30 Sep 2011
01 Aug - 31 Aug 2011
01 Jul - 31 Jul 2011
01 June - 30 June 2011
01 May - 31 May 2011
01 Apr - 30 Apr 2011
01 Mar - 31 Mar 2011
01 Feb - 29 Feb 2011
01 Jan - 31 Jan 2011
01 Dec - 31 Dec 2010
01 Nov - 30 Nov 2010
01 Oct - 31 Oct 2010
01 Sep - 30 Sep 2010
01 Aug - 31 Aug 2010
01 Jul - 31 Jul 2010
01 June - 30 June 2010
01 May - 31 May 2010
01 Apr - 30 Apr 2010
01 Mar - 31 Mar 2010
01 Feb - 29 Feb 2010
01 Jan - 31 Jan 2010
01 Dec - 31 Dec 2009
01 Nov - 30 Nov 2009
01 Oct - 31 Oct 2009
01 Sep - 30 Sep 2009
01 Aug - 31 Aug 2009
01 Jul - 31 Jul 2009
01 June - 30 June 2009
01 May - 31 May 2009
01 Apr - 30 Apr 2009
01 Mar - 31 Mar 2009
01 Feb - 29 Feb 2009
01 Jan - 31 Jan 2009
01 Dec - 31 Dec 2008
01 Nov - 30 Nov 2008
01 Oct - 31 Oct 2008
01 Sep - 30 Sep 2008
01 Aug - 31 Aug 2008
01 Jul - 31 Jul 2008
01 June - 30 June 2008
01 May - 31 May 2008
01 Apr - 30 Apr 2008
01 Mar - 31 Mar 2008
01 Feb - 29 Feb 2008
01 Jan - 31 Jan 2008
01 Dec - 31 Dec 2007
01 Nov - 30 Nov 2007
01 Oct - 31 Oct 2007
01 Sep - 30 Sep 2007
01 Aug - 31 Aug 2007
01 Jul - 31 Jul 2007
01 June - 30 June 2007
01 May - 31 May 2007
01 Apr - 30 Apr 2007
01 Mar - 31 Mar 2007
01 Feb - 29 Feb 2007
01 Jan - 31 Jan 2007
01 Dec - 31 Dec 2006
01 Nov - 30 Nov 2006
01 Oct - 31 Oct 2006
01 Sep - 30 Sep 2006
01 Aug - 31 Aug 2006
01 Jul - 31 Jul 2006
01 June - 30 June 2006
01 May - 31 May 2006
01 Apr - 30 Apr 2006
01 Mar - 31 Mar 2006
01 Feb - 29 Feb 2006
01 Jan - 31 Jan 2006
01 Dec - 31 Dec 2005
01 Nov - 30 Nov 2005
01 Oct - 31 Oct 2005
01 Sep - 30 Sep 2005
01 Aug - 31 Aug 2005
01 Jul - 31 Jul 2005
01 June - 30 June 2005
01 May - 31 May 2005
01 Apr - 30 Apr 2005
01 Mar - 31 Mar 2005
01 Feb - 29 Feb 2005
01 Jan - 31 Jan 2005
01 Dec - 31 Dec 2004
01 Nov - 30 Nov 2004
01 Oct - 31 Oct 2004
01 Sep - 30 Sep 2004
01 Aug - 31 Aug 2004

Stuff

Powered by Pivot  
XML: RSS feed 
XML: Atom feed 

§ How to fix an incorrectly converted build rule in VS2010

One of the rites of passage when converting a Visual Studio 2005 or 2008 project to VS2010 is learning how to fix custom build rules. The MSBuild system uses completely different custom build rule files than VS2005/2008, and the converter is dicey at best. Here I'll outline some of the issues I've encountered and ways to fix them.

First, some background: a custom build rule file (.rules) in VS2005/2008 tells the project system how to invoke an external tool to do a build step. It contains not only the instructions for how to run the build step, but also which properties to expose in the UI for setting or overriding at project and file level, and dependencies to integrate into the incremental build. They're manipulated from the Custom Build Rules menu option on the project context menu. Visual Studio comes with a few custom build rules, including one for the Microsoft Macro Assembler (MASM).

A foo.rules file in VS2005/2008 is converted to three separate files in VS2010 for MSBuild consumption:

  • foo.xml: This file lists the properties that show up in the UI and that can be overridden from the .vcxproj property files. Usually this is not a source of problems and can be left alone.
  • foo.props: This file contains the default properties for items subjected to the custom build rule, as well as a few derived properties, such as the command line. A number of issues arise from expansions not working correctly on the derived properties.
  • foo.targets: This is the meat of the custom build rule and is the one that actually contains the MSBuild-based logic for invoking the custom build tool. This is the file on which you'll usually be doing heavy surgery.

There are two additional complications with fixing custom build rules in VS2010. The first is that the UI for custom build rules has been removed, so fixing the rules involves wading into raw XML. The second problem is that the MSBuild documentation is quite poor, not describing basic concepts adequately and frequently mixing API and build file concepts. The information I provide below has mostly been gleaned through trial and error, so those of you familiar with MSBuild can probably find many places to correct terminology or provide better solutions.

Oh, one more thing: we'll need a sample project to begin. This is a VS2005 project with a correctly working build rule: http://www.virtualdub.org/downloads/CustomBuildRule-VS2005.zip

If you have VS2010 or VC++ 2010 Express, convert the project to 2010 format and follow along.

Starting out

Building the VS2005 project produces the following happy output:

1>------ Build started: Project: VS2005-custombuildrule, Configuration: Debug Win32 ------
1>Preprocessing: hello.foo -> hello.cpp (Include name: main)
1>Copying: d:\projwin\VS2005-custombuildrule\main.bar
1>Adding: d:\projwin\VS2005-custombuildrule\hello.foo
1>Compiling...
1>hello.cpp
1>Linking...
1>Embedding manifest...
1>VS2005-custombuildrule - 0 error(s), 0 warning(s)
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

Building the converted VS2010 version produces less happy output:

1>------ Build started: Project: VS2005-custombuildrule, Configuration: Debug Win32 ------
1> Preprocessing: hello.foo -> hello.cpp (Include name: [IncludeName])
1> hello.cpp
1> VS2005-custombuildrule.vcxproj -> E:\t\Debug\VS2005-custombuildrule.exe
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

There are a few issues immediately apparent in this output and more lurking under the surface. Let's take them one by one.

Missing output

The first thing you might notice when building the VS2010 version of the project is that it doesn't print the Copying and Adding lines like it does in VS2005. Custom build rules converted in VS2010 don't output anything to the log at normal logging levels. Zero, zilch, nada. This is a pretty lame oversight, and it makes it a bit difficult to figure out why a custom build tool failed.

It turns out that this output is actually captured, but is normally output at too low of a level to appear. The execution description does appear, though, and the Message task has an Importance="High" property. We can therefore apply something similar to the actual build task in the targets file:

<Foobar_builder
Condition="'@(Foobar_builder)' != '' and '%(Foobar_builder.ExcludedFromBuild)' != 'true'"
CommandLineTemplate="%(Foobar_builder.CommandLineTemplate)"
IncludeName="%(Foobar_builder.IncludeName)"
AdditionalOptions="%(Foobar_builder.AdditionalOptions)"
StandardOutputImportance="High"
StandardErrorImportance="High"
EchoOff="true"

Inputs="%(Foobar_builder.Identity)" />

The EchoOff property is also needed to prevent the command line from being echoed along with the build step output.

Unexpanded tokens

Next problem: the execution description of the tool looks a bit weird.

Preprocessing: hello.foo -> hello.cpp (Include name: [IncludeName])

Looks like the [IncludeName] property reference is not getting expanded. The reason is that the expansion is only performed by the command line task, and not by other tasks such as the Message task. Therefore, we will need to perform the expansion manually:

<Message
Importance="High"
Text="Preprocessing: %(Foobar_builder.Filename).foo -> %(Foobar_builder.Filename).cpp
(Include name: %(Foobar_builder.IncludeName))
" />

The ExecutionDescription property is no longer needed and can be deleted from the .props file afterward, if desired.

Additional dependencies

The way the build tool was originally set up, the hello.foo file also depended upon main.bar, and optionally main.blah if it existed. Does this dependency checking still work? To review, we are using a custom build rule to process (hello.foo; main.bar; main.blah) -> (hello.cpp), which is processed by the C++ compiler: (hello.cpp) -> (hello.obj). Therefore, as you should have learned in Make 101, the following should hold:

  • If hello.cpp is missing or older than hello.foo, main.bar, or main.blah, then the custom build rule is invoked to build it.
  • If hello.obj is missing or older than hello.cpp, then the C++ compiler is invoked to build it.
  • If neither of these hold, then neither the custom build rule nor the C++ compiler should be invoked.

Let's modify hello.foo... builds once... doesn't build again. Good. Now let's modify main.bar and hit Build....

Drat, nothing built.

As you might have guessed, the main.bar dependency isn't actually getting tracked. However, if you delete the executable to force the project to rebuild, you'll notice something else that's screwy, which is that the hello.foo file does rebuild. There are actually two build systems involved here, one being the dependency check in the IDE, and the other being MSBuild itself. They don't use the same mechanisms, and therefore you can get into four different situations:

  • VS2010 sees outdated files, and invokes MSBuild, which builds the outdated files. This is how we should start.
  • VS2010 and MSBuild both see project as up to date. This is how we should end up.
  • VS2010 sees outdated files, but MSBuild doesn't. When this happens, you get projects in the to-build list on a Run command, but all MSBuild does is run through them and determine everything is up to date. Annoying, but livable.
  • VS2010 sees the project as up to date, but MSBuild doesn't. This is the broken case we have now. You'll know you're running into this case if you try debugging it by raising the verbosity and you don't get any MSBuild output at all. There is a way to enable logging on the VS project side via the devenv config file, but I've never found it useful.

The reason why this happens is a bit obscure: VS2010 doesn't actually run through the MSBuild scripts, but instead processes the .tlog files in the intermediate directory and checks timestamps on files. In this case, the converted custom build rule has produced a VS2005-custombuildrule.write.1.tlog file indicating what hello.foo produces, but not a VS2005-custombuildrule.read.1.tlog file indicating what it consumes. Therefore VS2010 doesn't see the dependency from hello.foo to main.bar, and it doesn't even bother to invoke the MSBuild engine.

What we need to do is tell Visual Studio about the additional dependencies. We can do this by generating a new .tlog file based on the AdditionalDependencies property:

<ItemGroup>
<Foobar_builder_inputs Include="%(Foobar_builder.AdditionalDependencies)"/>
</ItemGroup>

<ItemGroup>
<Foobar_builder_tlog
Include="%(Foobar_builder.Outputs)"
Condition="'%(Foobar_builder.Outputs)' != '' and '%(Foobar_builder.ExcludedFromBuild)' != 'true'">
<Source>@(Foobar_builder, '|')</Source>
<Inputs>@(Foobar_builder_inputs, ';')</Inputs>
</Foobar_builder_tlog>
</ItemGroup>
<Message
Importance="High"
Text="Preprocessing: %(Foobar_builder.Filename).foo ->
%(Foobar_builder.Filename).cpp (Include name: %(Foobar_builder.IncludeName))" />
<WriteLinesToFile
Condition="'@(Foobar_builder_tlog)' != '' and '%(Foobar_builder_tlog.ExcludedFromBuild)' != 'true'"
File="$(IntDir)$(ProjectName).write.1.tlog"
Lines="^%(Foobar_builder_tlog.Source);@(Foobar_builder_tlog->'%(Fullpath)')" />
<WriteLinesToFile
Condition="'@(Foobar_builder_tlog)' != '' and '%(Foobar_builder_tlog.ExcludedFromBuild)' != 'true'"
File="$(IntDir)$(ProjectName).read.1.tlog"
Lines="^%(Foobar_builder_tlog.FullPath);%(Foobar_builder_tlog.Inputs)" />

What this does is build an item list called @(Foobar_builder_inputs) out of the additional dependencies, combine it into a string with semicolons as separators, and then pass that to a WriteLinesToFile task which then splits it into lines as semicolons. This is a bit roundabout, but I couldn't get it to work either by referencing %(AdditionalDependencies) directly or by doing the merging directly in the task attributes.

If you actually try this, you may run into another weakness of the dual dependency check setup, which is that the .tlog files themselves are written incrementally, and therefore there may be old dependencies in the files that screw up the dependency. When verifying a change to a build setup, a rebuild is prudent to ensure that the .tlog files are actually being generated correctly.

Conditional dependencies

At this point, VS2010 and MSBuild now agree on what needs to be built... both incorrectly. The hello.foo file is now building all of the time. The best way to diagnose this that I know of is to go to Tools | Options | Projects and Solutions | Build and Run and to set the MSBuild output verbosity to Diagnostic level. Unfortunately the output at this level is rather ridiculous and badly formatted, and mostly full of useless spew -- and when I say spew, I mean spew, because for a real project you may get over 50,000 lines of it! However, it's not too long on this test project, and a bit of searching reveals this line:

Input file "D:\projwin\VS2010-custombuildrule\[IncludeName].bar" does not exist.

Ugh. As with ExecutionDescription, the property tags are not being expanded in the AdditionalDependencies property either. Also, for some bass-ackward reason, MSBuild believes that a missing source warrants a rebuild of the target. (Making a build tool that writes to sources is usually a floggable offense.) We can try embedding the property expansion into the target:

<Target
Name="_Foobar_builder"
BeforeTargets="$(Foobar_builderBeforeTargets)"
AfterTargets="$(Foobar_builderAfterTargets)"
Condition="'@(Foobar_builder)' != ''"
DependsOnTargets="$(Foobar_builderDependsOn);ComputeFoobar_builderOutput"
Outputs="%(Foobar_builder.Outputs)"
Inputs="%(Foobar_builder.Identity);
%(Foobar_builder.RootDir)%(Foobar_builder.Directory)%(Foobar_builder.IncludeName).bar;
%(Foobar_builder.RootDir)%(Foobar_builder.Directory)%(Foobar_builder.IncludeName).blah
;
$(MSBuildProjectFile)"
>

This solves the problem, but introduces a new one:

Input file "D:\projwin\VS2010-custombuildrule\main.blah" does not exist.

The main.blah file doesn't exist, and unlike VS2005/2008, this trips a rebuild. What we need to do is inject this dependency only if that file actually exists. We can't do the existence check in the Inputs property of the target, so we'll have to back that change out, and instead dynamically generate the %(AdditionalDependencies) property from an earlier target:

<Target
Name="ComputeFoobar_builderOutput"
Condition="'@(Foobar_builder)' != ''">
<ItemGroup>
<Foobar_builder>
<AdditionalDependencies>%(RootDir)%(Directory)%(IncludeName).bar</AdditionalDependencies>
</Foobar_builder>
<Foobar_builder Condition="Exists(@(Foobar_builder->'%(RootDir)%(Directory)%(IncludeName).blah'))">
<AdditionalDependencies>%(AdditionalDependencies);%(RootDir)%(Directory)%(IncludeName).blah</AdditionalDependencies>
</Foobar_builder>
</ItemGroup>

Check the VS2010-custombuildrule.read.1.tlog file, and the dependencies should now be getting generated correctly whether the main.blah file is present or not. (Remember to rebuild after adding or removing that file, so that the .tlog files are regenerated.)

The final result

At this point, the VS2010 project should be building cleanly with working dependency checking. For a copy of the fixed version of the project: http://www.virtualdub.org/downloads/CustomBuildRule-VS2010.zip.

Comments

Comments posted:


Rebuilding the target when one of the source files is deleted is actually the only correct way to do builds - at least if the source file is not itself autogenerated. Otherwise you get the behavior you have in VS2008 and before - deleting the header file does not force a recompile, so until you do a rebuild you can think that everything compiles fine.

Arseny Kapoulkine (link) - 26 02 11 - 18:50


That would be fine if the build tool actually produces an error. When it doesn't, all you have is broken dependency checking, and a build system should really flag that instead of just rebuilding inscrutably. That happens at the project system level too, and it's annoying to track down -- I actually had to write a tool to crawl the project files and report which missing files are causing rebuild problems.

Phaeron - 27 02 11 - 12:05


I wanted to port my application to vs2010 for better os compatibility. One look at vs2010 and I stumped back into newbiness. I wish there was an easy way to import old vb6 projects.

evropej - 28 02 11 - 15:36


You're trying to port from VB6 to VS2010?? You've been postponing the pain for quite a while -- Microsoft cut the legs out from under you guys eight years ago....

Phaeron - 28 02 11 - 15:41


I tinker with vb6 here and there. I have one free app out there which I wanted to upgrade so I thought it would be a good motivational factor to learn a dot net platform. Boy was I in for a surprise lol. I mean, what a surprise. I posted a simple question about control arrays in vbforums and got a dissertation instead of a code snip. How do they expect big corporations to rewrite all their source code every time they release a new version of vb? Is this an attempt by Microsoft to slow down the development of open source technology so they stay ahead lol?

I guess I should just stop whining and open up some tutorials instead :)

evropej - 02 03 11 - 10:38


@evropej
As has been said by others, the curious thing about .Net is that it allows you to use any language you want, as long as it's C#.

TheZeusJuice - 06 03 11 - 09:44


Hello,

For "ordinateurs" with no visual C 2010 cappabilities, the project could be a library extension (compiled in conjuction with virtualdub`s main.obj) for implement-/adding say, a virtualdub rec pause button. It should fit some other function-require-VS2010 purposes of Microsoft | know not

`bout to forget. You should call "compile command" once again from menu brfore "buiding all" the whole projekt. | bounced it since VS6

Happy new year 2011,
mita patratica cuciumita
paul sgr

troianii@gmail.com - 10 03 11 - 11:37


Phaeron, MS has released Windows 7 SP1 and Visual Studio 2010 SP1.

Win7 SP1
http://www.microsoft.com/downloads/en/de..

VS2010 SP1
http://www.microsoft.com/downloads/en/de..

Unfortunately there's a really stupid installer bug in VS2010 SP1, you might want to read this first before installing.

http://msdn.microsoft.com/en-us/visualc/..

SP1 - 12 03 11 - 01:33


Yeah, I've been holding off installing it for that reason. I already have SP1 beta installed, though.

Phaeron - 13 03 11 - 08:07


MS has fixed the installer bug with an update.

http://www.microsoft.com/downloads/en/de..

SP1 - 29 03 11 - 00:46


Getting a "not found" page for that link at the moment.

Phaeron - 29 03 11 - 17:27


Thank You a lot for the great help!!

ati - 12 04 12 - 03:21

Comment form


Please keep comments on-topic for this entry. If you have unrelated comments about VirtualDub, the forum is a better place to post them.
Name:  
Remember personal info?

Email (Optional):
Your email address is only revealed to the blog owner and is not shown to the public.
URL (Optional):
Comment: /

An authentication dialog may appear when you click Post Comment. Simply type in "post" as the user and "now" as the password. I have had to do this to stop automated comment spam.



Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.