Unfortunately in a lot of projects this very important topic does not get the appreciation it deserves. On one hand side it's a very sad story of suffering and endurance. On the other hand side we are talking about a lot of wasted money.In my career as software engineer I experienced some different approaches to the project setup and the build process.

It started with something like find . -name '*.java' | xargs javac, which worked quite well for the first weeks of my professional career. But with a couple of software developer working on a project you want something more standardized.

So we went on with make(*). I guess most of the software engineers will know this build tool which still is a standard tool in the Unix/Linux world. make is a very powerful tools but has its peculiarities also. The make grammar is very delicate when it comes to white spaces. I quess quite a few people sat over a broken make build when a TAB char wasted your build script.

In the Java world Apache Ant was the successor to make in my career. The problem with make and the Java world is, that make is not really platform independent. There are make ports for Microsoft Windows but (at least when I made the jump from make to Ant) it does not work very well. So if you have a heterogeneous environment e.g. you develop on Microsoft Windows and you deploy into a Linux based cloud, you want your build to to be platform independent.

A nice story about platform independence. I did some consulting in a project for the Deutsche Bank Italy and a very large (and expensive) consulting company provided the BEA Weblogic build and setup. I am not 100% sure but I guess it was an Apache Ant based build. One major issue was the application server integration integration. To solve it, the complete project was placed and build in the application server which was committed in to the CVS source control management system. The delivery artifact was - of course - delivered without the application server. But this leads to another wish I have when I am using a build. You want the build to credit the current environment. If you are building for your local development environment you may have different requirements concerning the built artifact as when you are creating an artifact that will be deployed on staging or productive environment.

Back on memory lane: Apache Ant worked pretty well for some years and a couple of projects. But one thing that started to annoy me. Every project was using a slightly build setup so in the daily business there were always searching for artifacts or checking library versions.

Some former colleagues introduced me to Apache Maven. I started with Maven 1 which is a quite interesting mixture of Ant with a framework that offered the possibility to manipulate the Ant build file during the runtime. But Apache Maven had some features which I hat to get used to.
An example is the rigid directory structure defined in a Maven build: Why do I have to follow them? I want my special directory names! But nowadays I love this structure which is equal in every project. You find your files so fast now and even in foreign project. That's another thing that you want: You want your builds to follow standards, so you can get familiar with the project in very short time. If you are working in the software business a couple of years, may have already realized that you will see very much projects during a professional career and standard projects and processes will make your life much more easy.

Let's talk about some goodies now which modern build system offer. I will use Apache Maven as example but other systems like rake or buildr offer similar functions.

One thing I love/hate is dependency management. When I started with make or ant, we where committing all libraries into our CVS. Which worked quite well, in most cases you could check our the project, performed the initial build and started to work. But there where lots of minor but very nasty problems. Starting from which library version is really used to incompatibilities of the build tool version.

With the Apache Maven dependency resolution mechanism a new area begun. If you are having one independent project the mechanism is pretty easy to use. You define the libraries you want to use in your pom.xml and maven takes care of the rest. Okay if you want to use libraries which are not in the maven central repository it's a little more complicated

If you are having a multi module/project environment things will get messy pretty easy, but: They would also if you don't use the dependency management mechanism. You ask why? When you don't use this mechanism you'll normally have a huge monolithic project in your SCM system which will get even more difficult to manage or build. So you want to have a transparent dependency management in your build system. With Ivy for example you'll have an add-on which offers the possibility to have an dependency management when you are using Ant.

Another goodie is release management. It's very handy if you start a build to create your release artifact and your tag (or something similar) in the SCM system of your choice. You want your release process embedded in your build system, so you can perform the build when you are creating your release.

Another important topic is testing. You really want your build tool to support your testing facilities. At least when you create your release or deployment artifact you want your build to automatically run your test suite.

I have only one topic left in this post, that is IDE support. Unfortunately most modern IDE like Eclipe, Netbeans or Intellij IDEA offer a IDE specific build. This this very comfortable if you have a small project and what to deal only with your IDE. But you'll grow really beyond this (my experience). But integration into other build tools is sometimes really poor. So you have to check how you can bridge the gap between your IDE and the build tool. E.g. the maven directory structure troubles some IDE
standard configurations. Fortunatly more and more projects provide solutions for this.

So let's have a quick resume. This is what you should want in your build system:

• standardized process
• platform indipendence (if your need it)
• credit current environment
• standardized structures
• dependency management
• embeddable release process (including SCM support)
• support of your test tools
• IDE support

I think this list is not too lang and offers some major improvements for your daily work. So if your build process is missing some items from this list, you maybe should think about your build process. And remember: Your development resources are far to valuable to be wasted in an inefficient build process

The bottom line is that we have approximately 100 projects build with maven to build our entire platform. You don't need much imagination that we have a couple of challenges to meet.I started with Maven 1 which was a astonishing mixture of Apache Ant and Jelly. But I never really liked the idea of executable XML. In my opinion XML is not a good programming language at all but this would be a blog posting of its own

Back to to the builds. A couple of days ago a read some tweets from the the man behind tapestry to remind me about Apache Buildr. This lead to rethink some experiences I made with Maven2 in the last couple of month...

This lead to certain discoveries what I started to really dislike on Maven2:

1. The XML - The pom.xml which defines the central project object model os getting really cluttered. When you have a small project there is almost no problem but if you have an enterprise scale project (almost evereything after an example project) the pom.xml is getting huge and you don't find the things you need.

   Okay, there are "helpers" to maintain the pom.xml, but e.g. if you use Eclipse with a POM file the XML editor starts to parse it. If the file is no very small - even a small open source project like jaev has an approx. 140 lines pom.xml - it'll make Eclipse somewhat slow. If you use the m2eclipse plugin it will get even worse. The m2eclipse offers some very nice tools for POM handling and it got really good compared to the early versions. But I know a couple of guys who refuse to use it because of the negative impact on your development environment.

2. Reuse by inheritance - The only way (so far I know ) to reuse settings in Maven2 is POM inheritance. This works like prototypic inheritance (like in Javascript). You may define prototypic plugin settings in the parent POM and all POMs which inherit from this parent POM will use this settings as a default.

   So far so good, but in most object oriented programming languages inheritance to reuse code is considered to be a bad design pattern. In Maven2 you may stumble across the same bad issues as e.g. in Java when you use the POM inheritance. As a matter of fact you may only inherit from one parent POM, so your inheritance tree will get cluttered with settings and overrides.

3. Insufficient version and dependency management - The dependency management is one of Maven2 strongpoints, no doubt about it. But if you really start to use it you'll get to it's limits very fast.

   I guess Maven2 did some awesome pioneer work to version handling. Due to the fact that every artifact has a version number. Unfortunately the version numbers in Maven do not have an ordinal structure and admits special indicators like SNAPSHOT, alpha or LATEST which are adding a whole new level of complexity. So maven can't tell if a version is greater than another version, e.g. is 1.4-SNAPSHOT greater than 1.4?. This is very problematic when use version ranges and transitive dependencies. When you'll get a dependency from different sources (e.g. the commmons-lang library) there is always some magic involved which version you'll get. This leads to next next issue:

4. The build does not break early enough. - Maven2 tries to be very clever in some situations like the dependency resolution. If a problem occurs maybe a log message will be provided but the build will continue even if you have contradictory dependency declarations. So the only "hint" you'll get, if the dependency tree is faulty, are compile errors if you're lucky. If you have runtime dependencies you'll get some funny effects in your application
5. No separation of concerns. - The POM includes all concerns of a build like:
   • Dependency management
   • Compiling
   • Tesing
   • Packaging
   • Reporting
   • Deployment

   You'll have different sections in the pom.xml like <build/>, <dependencies/>, <profiles/> or <reporting/> which group the declarations a little bit. But if you take a closer look at the <build/> section you'll soon realize that it works quite different that e.g. the <dependencies/> section. In the <build/> you are more on a programming level to specify the compile and packaging issues of your build. The <dependencies/> is only declarative.

6. Only one file - To make the lack of separation more painful you are not able to break the different concerns into separate files. There are some Maven plugins which use other files like the assembly plugin, but this an exception. It would be really nice if out could include files into your pom.xml. This really could have positive effects on the inheritance issue too.

Okay to straighten this a little bit. I guess the Apache Maven did some of the most innovative work on the build sector in the last few years. And I don't consider to switch to make or Apache Ant. It makes many things a lot easier and most of my critique may be corrected in further versions I already had a peek into the Maven3 source code and I'll guess the Maven/Sonartype guys will level some of this issues. I 'am very curious about Maven3 and I hope it will ease some of my pain related to the Version 2 of Maven.

So, it was great to get some thoughts clear about what is important when you build a project and what are the different facets when you use a build tool. I had a first look on Apache Buildr maybe I get enough stuff together to create a blog entry about it ...

After a little more than a glimpse into the capabilities of Git I am quite impressed!I think I have a quite common background in source code management systems (SCM). I started with CVS at university and I really hated it. On my first job I continued with CVS and started to value a good source code management system.

When you are working with a bunch of guys on a large Java/HTML project you can't work without a SCM system. CVS was a good system with some quite nice GUI clients like the Eclipse CVS plugin or WinCVS.

I am not quite aware when, but it was 2004-2005 I initiated a project to update the SCM infrastructure to Subversion to address some problems with CVS. If you ever created a major release tag on CVS you'll know what I mean
In my opinion Subversion is more than a worthy successor to CVS but it has it's own weaknesses.

The greatest weakness is the lack of branches and tags as an additional dimension in the SCM model. This makes at least branching with SVN not a very nice task. Tracking the changes of files through branches or tags is not that easy, because SVN simulates a branch/tag in the file structure.
On my current job we came across some special SVN related problem. SVN uses the HTTP protocol for communication between SVN client and server. This solves al lot of network related issues, because HTTP is a well known and wide spread protocol. But creates a some new ones. So the synchronization of the working copy can some time, especially for for large binary assets and/or large projects.

But let me emphasize that SVN is a great SCM system with very well tool support (Eclipse, TortoiseSVN, etc.) and a large community. So let's stop SVN "bashing" and start with the real topic.

Git is a relatively new SCM system. It was started in April 2005 by Linus Torvalds. It was created due to the change in license of Bit Keeper, so that the linux kernel developers have no longer access to the SCM system.

The main difference between Git and SVN is, that SVN (like CVS) has a centralized model and Git has a decentralized model. What does this mean? In the SVN world you have one (central) repository which keeps track of all versions. In the Git world you have a local repository which can pull or push versioning data with other (remote) repositories. This has some fancy effects on the Git SCM system:

One thing that puzzled me most during my switch form CVS to SVN was the change in the version number. CVS provides for each file a version number (e.g. 1.1, 1.3.4, etc.) but SVN uses a version number for the whole repository (e.g. 47382). This works only because the is one leading instance (repository). Git does not have this central instance so the Version number is a SHA1 id (e.g. f4d267c9374da99c95222a95e203dab6c15a499). I guess you noticed the difference too But this is necessary because you cannot tell which version (number) this other guy half the way around the world created a moment ago. The version is also provided for a defined set of changes when committed. The version tracks the parent version and so you can build a version tree. This this is quite different the the linear model of SVN, where the version number is an integer.

Another effect of the design is, that there is more emphasis on branching and merging. So the Git provides a lost more support for this tasks. Like git stash which pushes the changes on the working copy into a local branch and restores the HEAD. This is quite nice, if you want to store your work to e.g. fix an urgent problem. Of course you can easily create a new named branch of your stashed changes. Needless to say that merging is quite easy in the Git world. Git also provides a good visualization of the different branches and commits. This makes cherry picking quite easy.

Since I am playing around with Git, I realized how SVN is conceptually weak in branching and merging. A former colleague mentioned this already in 2004 when the migration at my former employer started. I think I understand him now

Another nice feature of Git is that you can fetch versioning informations of SVN. So if you have a central SVN repository, you can push and pull version to this repository and keep track of you local work/changes in Git.

Last topic for this blog entry (yeah, all good things must come to an end ) is the Git tool support. Git is a quite nice command line tool. But sometimes I am quite lazy so I want a GUI tool as well. Eclipse is mandatory for any competitor in the SCM game. Let me mention some real surprise: The EGit Eclipse plugin is the first SCM eclipse plugin (I came across) which could move and rename files and keep track of the version history!
Of course there are various tools for all kind of platforms (e.g. GitX, TortoiseGit or qgit).

I guess, I will keep an eye on Git ...

It's quite sad, but if you are honest even with your source code quality is far below average the project may be a success by the given business goals. On the other hand side your source code may be excellent, your code metrics are outstanding and your architecture is state of the art but you are missing the needs of your stakeholders by miles.Programming is the most known part of software engineering to non software types. A couple of years ago I improved my skills in programming with the hope to improve my software engineering skills. But it took not very long for me to realize, that it takes a lot more to be successful in software engineering.

This is quite sad to say, but if you are honest: Even with your source code quality is far below average the project may be a success by the given business goals. On the other hand side your source code may be excellent, your code metrics are outstanding and your architecture is state of the art but you are missing the needs of your stakeholders by miles.

Realizing this made me a kind of sad, because learning to be a good programmer is task you can achieve by practicing. But learning how to e a good software engineer is a lot more difficult. It's about communication, understanding the needs of your stakeholders, making good decisions, being flexible and able to incorporating short-term requirements.

In the last years I was able to learn some important lessons, which I believe, made me a better software engineer. Working on some medium to large sized software development projects from 2001 to 2009 I learned what is important and works fine but I also had the chance to make some quite big mistakes.

During the last years I read a couple of books about different software development approaches. Starring the classics from Tom DeMarco (e.g. The Deadline) which are quite excellent. My problem with the DeMarco/Lister books is, that they soon start to repeat them self.

The next great books where from the extreme programming (e.g. from Kent Beck) and agile development guys. I love this concepts. They are great! I guess anyone dealing with waterfall style processes and monster MS Project plans will agree.

One of the highlights of the last year where the books of Mary and Tom Poppendiek:

• Lean Software Development: An Agile Toolkit"
• Implementing Lean Software Development: From Concept to Cash

The Poppendieks are bearing some resemblance between agile software development and lean production methods and offering some very easy to understand principles. I am quite curious and exited how I will be able to incorporate this principles to my daily work and our organization at scoyo.

One thing I learned the hard way. No software development process will work, if your organization is not willing to accept the consequences.