Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Sunday 20 May 2007

Speeding Up Mkdepend

I did a little bit of research into the problem of mkdepend.exe being slow under VMWare with sources in VMWare's shared file system.

The first thing to do was to measure the problem. This was pretty easy: running mkdepend on nsPresShell.cpp consistently takes 32 seconds. Actually compiling the file takes only a few seconds. So clearly something is very wrong.

Poor-man's profiling in the debugger shows that mkdepend is spending all its time in stat() calls. These stat calls are just checking to see whether a file exists; on a hunch I wondered if there's a faster way to check for file existence than stat(). Some Web searches suggested that GetFileAttributesEx is a good fast way. So I refactored the code to use a does_file_exist function which calls GetFileAttributesEx on Windows. Ta da! Now mkdepend on nsPresShell.cpp takes 1.5 seconds. GAME OVER. Patch submitted.

I conjecture that stat() is so slow because it fills in a bunch of information about devices, inode numbers, and access permissions, and on Windows that may require several filesystem operations to get that information. Perhaps VMWare's file system is particularly slow at some of those extra operations.


I'm going to ask a naive question (given I don't know the material at all) for a bit - did you check if similar timesaving stuff exists on Mac or Linux? This sounds like a boon which will be good to have on any platform, really!
Shawn Wilsher
what bug was that?
Unsurprisingly nsLocalFileWin::Exists already uses GetFileAttributesEx().
Brendan Eich
Yay! But the same argument applies to Unix: access(2) is faster than stat(2). Care to test?
Robert O'Callahan
Shawn: 381247
Brendan, Gijs: on Mac and Linux I believe we don't use mkdepend normally, we generate the information with gcc -MD as a byproduct of normal compilation.
Brendan Eich
Oh, I was misremembering http://benjamin.smedbergs.us/blog/2006-11-22/depths-of-the-mozilla-build-system/ -- amazing the lengths we go to for always-fresh dependencies.
Diego Calleja
Amazing. How much it speeds up the whole compilation of firefox?
Can't wait to see what this does for windows build times for the release team since we use VMWare tehre as well...
Robert O'Callahan
schrep: might help a bit but I think those builds have the sources on a local filesystem, so we won't see the huge penalty.