You are viewing hughsient

Technical Blog of Richard Hughes - GNOME using waf?

Richard Hughes
Date: 2007-01-04 21:03
Subject: GNOME using waf?
Security: Public
First, apologies about the long blog.

The programs autoconf, automake are not the easiest friends to get to know. Scripted in odd languages, and with seemingly many ways to do the same thing they are confusing. Most developers I know simply copy/paste configure.in code from one project to another with little understanding of how it works. I call it auto-foo.

People working on GNOME have looked at many replacements over the years, and for gnome-power-manager I've even looked at using scons as a build system in the past. Scons is great for building a program, but not suitable for managing a project. Something bigger and better is required.


waf example

I've also been following the progress of a project called waf quite closely for the last few months, and even getting involved with providing gnome example code and filing bugs. All the helper modules are python, so it's quite easy to extend and hack. As a build system it's looking quite promising, and I have even got a local copy of gnome-power-manager built using it. It takes about half the time to configure and build compared to ./configure && make and also has pretty output formatting.

I have found a few architectural design choices in waf that in my mind make it less suitable as a general build system:
  • It creates project cache files in ~/.wafcache, which is obviously not okay in an automated build machine. In my opinion it should create the configure cache in the project directory under .waf/_cache_.
  • There is a full tree of wafadmin (waf python source code) in every project, which for managing many projects is a no-no. That's like having a copy of the m4 interpreter in every module (okay, it's pretty small, but still a bad idea)
  • Waf has no stable branch, and the API is not stable yet. I keep having to change my wscripts to keep up with svn head.
  • You have to set the WAFADMIN path for each user if you have installed a shared copy in /usr/local/share - no other paths are permitted. waf's ability to work with a shared wafadmin path is poor.
  • By default waf outputs details in console colour text mode - and the yellow text is unreadable.
  • Checking for headers is not simple, requiring a few lines of python for each include, where autoconf could do this in one line.
  • You have to decompress a semi-binary waf-light file to waf before you can start. This also smatters files into ~/.waf-version/wafadmin/. This is confusing on so many levels. If waf installed a single system shared copy then we wouldn't need all this madness.
  • You can have lots of different input files, like wscript, wscript_build, wscript_configure etc. I don't think the extra suffix saves anything at all except a layer of indentation, and it certainly confuses people.
  • Handling of the .po, .xml and .in files for GNOME is basic at best (mainly my fault).
waf does do a lot of stuff right in my opinion:
  • All generated files are put into a _build_ directory rather than clutter up the source tree.
  • Configure checking is really quick and cached for the next run
  • Building is fast due to parallel building
  • Dependency resolution is really quick, REALLY quick.
  • Can change the prefix and destpath for rpms and debs.
  • Lots of support for lots of languages (qt, c++, c, java, tex and more) and modules ready to go
  • Written in 100% python so no messing about with custom scripting languages - it also means you can do some funky python stuff in the wscript.
This is simple stuff should just work. An application developer shouldn't have to be playing with manually setting WAFDIR or debugging m4.
In my ideal world (with my GNOME "just works" hat on), waf would install the common python files in /usr/share/wafadmin and there would be one main file waf in /usr/bin. waf would just run the top level wscript in each project and then follow subdirs looking for other wscripts. The author isn't keen on this, and would prefer one complete copy of waf in each project. In my opinion there's a fair bit that needs to be changed before it can be used with GNOME software.
What can we do about waf? It's BSD licensed, so we could fork the project and make it more suitable for GNOME (and then try to stay in sync or contribute the new stuff back to waf). We could just re-write the frontend (waf-gnome?) so that it works better with stuff like jhbuild and is easier to use. Or we could just start pestering the maintainer to modify the way development is going.
So, the purpose of this blog. What do you think about the way forward? Do we keep limping on with auto-foo? Is it just me who doesn't understand all this m4 stuff? Is waf a viable system to try? Has anybody else got any ideas for a new build system? Am I asking too many questions?

Thanks.
Post A Comment | 18 Comments | Share | Link






User: kamstrup
Date: 2007-01-04 21:56 (UTC)
Subject: autotools :-(
I hate autotools and I can safely say that they have raised the level of entry of new contributors. I was lost myself for a long time where all the m4 and autotools foo scared the shit out of me. I was thinking that I couldn't possible contribute without grokking all this (which I now know is totally untrue, since I'm a deskbar maintainer :-))

I had a small flirt with QT4 back when it was originally released. It took me about 10 minutes to get going with the project/building tools related to this - I don't remember their names anymore, but I clearly remember thinking: "Wow - can things really be this easy?".

I realize and appreciate that autotools has a lot of nifty things that make them great tools, I just really dislike working with them.
Reply | Thread | Link



User: (Anonymous)
Date: 2007-01-05 00:33 (UTC)
Subject: Re: autotools :-(
I don't have any answers but you hit the nail on the head: autotools raises the entry barrier a lot.

Then again, if there are good GNOME bindings to your favorite language (PyGTK is great), the need for autotools is virtually eliminated. This won't help people wanting to get involved with most existing projects, but does make it easier for people writing new apps.
Reply | Parent | Thread | Link



User: kamstrup
Date: 2007-01-05 06:09 (UTC)
Subject: Re: autotools :-(
PyGTK doesn't solve the problem as such. Deskbar for instance uses autotools, practically any respectable pygtk app I can think of uses autotools.

I've had small hobby projects in pygtk a few times, and each and every time (yes) people have requested autotooled versions.

There are several reasons why people want the autotooled versions. Some being old habit (the configure; make; make install dance), others being easier integration with known package building systems.

Integration with package building systems is actually one of the most important factors in my mind (besides from ease of use).
Reply | Parent | Thread | Link



User: (Anonymous)
Date: 2007-01-05 21:14 (UTC)
Subject: Re: autotools :-(
True, PyGTK doesn't *necessarily* solve the problem. But whereas in C it's practically required, in PyGTK it's not as necessary, or dominant. You can use distutils, for example, which sucks far less.

I've never had people ask for an autotooled PyGTK app, but that could simply be a testament to the (un)popularity of my programs.
Reply | Parent | Thread | Link



nohatmatt
User: nohatmatt
Date: 2007-01-04 22:01 (UTC)
Subject: (no subject)
Well I've not seen an autotools-like thing which looked as promising as this before, and so that alone makes it worthy of further investigation I would say.

What bothers me is that with autoconf, tarballs are distributed with generated configure scripts that just need the standard stuff to run. If a system-installed waf wasn't done properly, it could require people to install things just to get the information about how to check for dependencies for other things! I assume that's the basis behind the one copy of waf per project approach.

Of course the reason why autotools haven't been replaced long ago is that this is not an easy problem to solve.
Reply | Thread | Link



User: (Anonymous)
Date: 2007-01-04 23:21 (UTC)
Subject: Buildsystem features
You asked for ideas so here you've got some.

I think there is a lot to be done with how source and binaries are produced and distributed in the FLOSS community. Stuff like SCM, buildtools and packaging needs to integrate better.

Regarding build tools a "build" (action to derive some data from it's dependencies) should be triggered only when the data is requested. Similarily it should be triggered when the data is requested if and only if there is no cache or the dependencies are altered in a way that would produce new data.

To reach this usability the build system probably needs some file system integration. Build targets would thus be represented as files. Building a tarball would thus be "cp tarball dest" rather than "make tarball; cp tarball dest" or whatever is done.

While thinking about files as dependency targets you also realize that there are many files that are dependency targets. Even the source files have dependencies in theire SCM download+merge.

So the whole chain from coding to SCM to distribution to installation to system maintancne can, and should, be managed as a filesystem integrated buildsystem.
Reply | Thread | Link



User: Ycros [ycros.be]
Date: 2007-01-05 00:00 (UTC)
Subject: Forking?
Call it Waffle.
Reply | Thread | Link



Monument
User: marnanel
Date: 2007-01-05 04:58 (UTC)
Subject: (no subject)
KDE dumped autotools a while ago and are now using cmake. I think we should seriously consider cmake as well.
Reply | Thread | Link



User: kamstrup
Date: 2007-01-05 06:18 (UTC)
Subject: cmake
Yeah, cmake is not bad. I'd go for it any day over autotools. Waf is intriguing though.

Quick cmake example: http://www.insightsoftwareconsortium.org/wiki/index.php/CMake_Tutorial

Is this seriously the only page on waf: http://freehackers.org/~tnagy/bksys.html ?
Reply | Parent | Thread | Link



User: (Anonymous)
Date: 2007-01-14 19:37 (UTC)
Subject: Re: cmake
No:
http://code.google.com/p/waf/

The documentation is in preparation too.
Reply | Parent | Thread | Link



User: (Anonymous)
Date: 2007-01-05 10:20 (UTC)
Subject: CMake
At aKademy 2005, it was decided that KDE would switch from autotools to bksys (the former incarnation of waf). It looked promising in the end, but over time it became clear that this was not the ideal solution, there were some drawbacks:

- We wanted to switch to a buildsystem that didn't have to be maintained by KDE people themselves. bksys looked like it would get a lot of manpower from the scons devs, but in the end waf was kind of a fork, and only Thomas Nagy is working on it. If he quits, waf will essentially be unmaintained. In opposition, CMake is well-maintained by a very responsive development team from a commercial entity. (This was the main reason, afair.)

- CMake is arguably better documented as well. There's even a "Mastering CMake" book out there.

- If you're doing IDEs, providing a graphical build system interface will be very hard with waf makefiles, because they are plain Python and by being a source file of a full-fledged programming language, it's quite impossible to reliably import projects with hand-made makefiles. (Not that CMake is specifically easy to do, but it's at least feasible.)

- There were a couple of difficult unsolved problems with waf/bksys when the switch to CMake was done. I didn't follow that discussion so I don't know more details, but do read a few mails on the kde-buildsystem list, you might benefit from those experiences.

Maybe waf has advanced since, er, when was that again? Half a year ago? 10 months? Anyways, you should seriously consider CMake, it's a real pleasure to use, for big modules just as for small two-file projects.
Reply | Parent | Thread | Link



User: (Anonymous)
Date: 2007-01-14 19:41 (UTC)
Subject: Re: CMake
The capital letters and the obscure command-lines are such a pleasure indeed...

CMake also has both the make limitations (distributed builds, ..), a custom scripting language to learn (macros?), so it certainly needed more than one book, a lot of documentation and the cmake developers to get accepted by the kde people
Reply | Parent | Thread | Link



Danielle
User: dannipenguin
Date: 2007-01-05 06:26 (UTC)
Subject: (no subject)
m4 is a pain the neck, a damned legacy pain in the neck. I heard that autoconf 3 deprecates m4 in favour of straight sh.

All in all, while autotools are sometimes quite challenging to use, they do run on systems with very few tools (think legacy UNIX). What I would like is for the autotools system to better pick up on syntax errors before they get translated into shell at which point they become practically undebuggable (because debugging generated source sucks).
Reply | Thread | Link



User: (Anonymous)
Date: 2007-01-05 19:57 (UTC)
Subject: Waf ideas
I hate Gnome and I hate Python even more. But I say this not to start a flame war, but to help you understand just how happy I am that Gnome could one day be using Waf.

XMMS2, the media player project I work on, has recently switched to Waf from SCons. I don't know a lot about Waf and I had only just started to catch on to SCons when we switched over, but I've seen a lot of Waf bugs get fixed just from being the XMMS2 build system. If you're on IRC or you browse around the Waf bugtracker you've probably seen tru, one of our head devs who has been contributing a lot back to the Waf project based on our experience. So to see a big project like Gnome use it would mean that it would likely get enough contributions to propel it into the mainstream and supplant autohell on modern systems.

I do have some gripes, though, most of which are in line with yours:
- needs a config file--with sane defaults (COLORS OFF!)
- local wafadmin stuff is insane
- standalone zipped waf script is cute, but ultimately makes it harder to stay up-to-date with HEAD (needs manual updating periodically)
- build dirs are annoying, ex. _build_/default/ is a PITA to keep typing in a shell and the default directory seems extraneous; try using .waf/build, .waf/cache, etc. (Remember I don't know a whole lot about Waf still, so there could be a good reason for this, but probably not.)
- prefix and destpath still need work; things like java and ruby install directories are not easy to prefix

and as for praise:
- lightweight and fast, much nicer than SCons, comparable to autohell
- does more in the core and handles components better than SCons (for XMMS2, we had to make our own classes to abstract away components of the build, much like how wscripts work by default)
- built-in caching, eliminates the need for things like ccache and works at the buildsystem level
- configure step is easy and permanent, e.g. CC will stay set throughout the build steps rather than having to export it or specify it during every step
Reply | Thread | Link



User: (Anonymous)
Date: 2007-01-07 14:26 (UTC)
Subject: Raising the bar
autotools is one of the things making me feel 'scared' about developing for GNOME - I feel like I couldn't possibly start developing without knowing it - yet at UNI where they have a setup (on Linux) for teaching C/C++ where it isn't necessary I really enjoy the stuff we are doing

Anything to make life easier for young/interested developers would be great
Reply | Thread | Link



User: (Anonymous)
Date: 2007-01-09 02:31 (UTC)
Subject: (no subject)
I think people under-estimate the knowledge base that is called autofoo. One can consider moving on to some other system only when that system provides the equivalent of libtool to start with, and don't forget the mountains of project-specific, working, debugged, portable, autofoo macros used. I mean, you're not telling me that Waf or cmake can compile shared libs on all those legacy Unix systems, working around all the bugs that libtool does, are you?

--behdad
Reply | Thread | Link



Richard Hughes
User: hughsient
Date: 2007-01-13 17:25 (UTC)
Subject: (no subject)
Ohh I agree, well debugged autofoo is great, and also works well for cross compiling and other random stuff. What I do think is important is lowering the level of entry; autofoo was the hardest thing I had to master for writing GNOME software.
Reply | Parent | Thread | Link



User: (Anonymous)
Date: 2007-01-14 19:09 (UTC)
Subject: waf
The yellow color was changed to orange, and it is now very easy to disable colors for the whole project: either in the script or by an environment variable.




The apis for listing headers were not done by me and was kept to the minimum. The change was done; you could have made a single and meaningful bug report for the feature, or better, done it yourself (you have svn access to waf).




WAFDIR needs to be set for waf developers only.
The shared copy in /usr/local is versioned /usr/local/lib/waf-1.x (similar to autotools).
Waf uncompresses its files in its own folder (project directory) if no shared copy is found (svn for now).
Ranting about such a detail (100kb) looks irrelevant to me.




You can have lots of different input files, like wscript, wscript_build, wscript_configure etc. I don't think the extra suffix saves anything at all except a layer of indentation

Some people have explicitely asked for this feature to save a layer of indentation (replace a function named 'build' in a file name 'wscript' by a script named 'wscript_build'). Nothing forces you to use it instead of regular wscript files.




The stable branch is 1.0, but you seem to prefer head which is evolving for 2.0
svn co http://waf.googlecode.com/svn/branches/waf-1.0/ waf




You have to decompress a semi-binary waf-light file to waf before you can start

waf is generated from waf-light
waf-light can be used directly, but this practice is discouraged:


  • waf developers will use the svn folder (WAFDIR)

  • developers are likely to ship their own version of waf to avoid problems with newer/older versions

  • users are unlikely to modify waf






The build dirs are needed to create different versions of the programs (cross-compilation) and to avoid source directory pollution. There are helpers to execute the programs or to perform unit tests after they are built.




The waf cache can be redirected using WAF_HOME, and is ignored if HOME is not set (build machines). Most users are happy with the default, but tests can be forced using --nocache, which should be the default for packaging anyway.




Before forking anything, the svn repository is open for commits on
http://code.google.com/p/waf/

Feature requests are usually solved quickly, if meaningful, accurate and polite:
http://freehackers.org/bugtracker

You may join #bksys on irc.freenode.net too

Reply | Thread | Link



browse
my journal
April 2008