While the push to reduce the bug count in CXF has been a very welcome and important step for CXF, it has detracted from the development of the new features and stuff for 2.3. 2.3 is definitely many months behind where I would have liked it to be. Now that the count is down to something reasonable and manageable, I’m really hoping to re-concentrate on getting 2.3 ready to ship. No promises though.

Number of posts on the users list: 6614

Number of posts on the dev list: 1661

Number of messages on commits list (svn commits and wiki changes and such): 3893

Number of JIRA issues raised in 2009: 595

Number of JIRA issues resolved in 2009: 763

Number of JIRA issues raised in 2009, but still unresolved: 51 (only 19 are bugs, 3 are bugs in JAXB where we need a new release of JAXB and 5 were logged in the last 2 weeks while most devs were on vacation, so really 11 bugs unresolved).

The JIRA stats are kind of interesting. Put in a graph:

CXF issues created/resolved graph for 2009

You can kind of see that the last 4 months or so, the CXF community really tried to go through all the old JIRA entries and resolve as many as possible. The result should be a much more stable and bug free product.

Being on holidays, I haven’t had time to look much at it (I’m only 1/4 of the way through the first book), but a quick glance through provided me a nice surprise. I kind of expected that it would be geared more toward Axis2 (since it’s an 3rd edition of the Axis2 book) with small “Now here’s how you do the same thing with CXF” type sections. However, it looks to be completely the oppossite. The initial examples and screen shots and stuff in the text is using CXF and JAX-WS stuff and then a small “Now here’s how you do the same thing with Axis2 section.”

Anyway, after the holidays, I hope to review both books in more detail.

What’s particularly exciting about this is that it’s not written by any committer on the CXF project. The authors and reviewers are completely independent. They decided that CXF needed a book and set out to write one. That’s quite interesting because on most of the OpenSource projects I’ve seen, the first sets of books are usually written by people actively involved in the project. The fact that this was written by actual users, not the developers working on the project, gives my some hope that it’s targeted correctly. One of the problems I’ve seen repeatedly with books written by the same developers that are writing the project code is the book gets too technical and makes assumptions about the readers that aren’t correct. The “I know this, so eveyone else must know it too” fallacy.

Anyway, I haven’t read it yet (I just bought it and am only up to page 17 or 336) so I cannot vouch for how good/bad it is yet. However, I’m very excited as this does fill a gap in the CXF offerings. I just wanted everyone to know it’s there.

Basically, a short while ago, he was tasked with upgrading some older web services to get them off of Apache Axis. They started porting to Axis2 assuming that would be the easiest effort. After hitting bugs and spending a lot of time trying to work around issues and such, they punted on the effort and switched to CXF. For them, CXF “just worked”.

I hear the same basic thing from many users. CXF just “works”. It does what it says it will do relatively easily (relative term, nothing in WS-* is designed to be easy) and bugs, when found, are usually fixed quickly, but most users I’ve talked to having found any real bugs to log. It just “works”.

For some reason, the person I talked to yesterday got me thinking about the various Open Source SOAP stacks and how “buggy” they are and kind of wondered if that could be quantified. One of the nice things about the open source stacks is the issue trackers are also “open”. Thus, it shouldn’t be too hard. Famous last words.

There are two main issues with trying to compare bug counts:

1. Features: Each of the three major open source stacks (Axis2, CXF, Metro), while they all implement the same basic specs, they also have unique features. For example, CXF has bugs logged for the CORBA binding, JAX-RS, Distributed OSGi, etc… that the others wouldn’t have. Axis2 has bugs for the UDP transport which is not available in the others. Etc….
2. Issue tracker layout: Each of the projects handle their issue tracker differently. CXF has a single project in JIRA and all bugs for everything are logged there. Metro has their issues split into two trackers, one for the base JAX-WS implementation and another for the WS-* stuff. Axis2 splits things across a bunch of JIRAs: a core area, one for Rampart, one for Sandesha, one for Kandula, transports, etc….

With that in mind, lets look at some numbers. To make things simple, I looked at the full “bug” count from CXF including all the features including JAX-RS, CORBA, DOSGi, etc…. Thus, the CXF counts are relatively inflated compared to the others. I didn’t look at feature requests, wishes, tasks, etc… Just bugs. For Metro, I looked at the “defects” only for the JAXWS RI and WSIT trackers. For Axis2, to make things simple, I only looked at the “bugs” in the core “Axis2″ jira and didn’t bother looking at the others. I didn’t need to as you’ll see.

As of this morning, Nov 5, 2009, the counts are:

I think those numbers paint a pretty good picture of the state of stacks. However, it’s not the whole picture. Another useful stat is the number of “critical” or “blocker” bugs. (For Metro, P1 or P2 level)

Another interesting stat is how many bugs logged in the last year or 6 months remain open. This shows how well the community responds to users bug reports.

Anyway, that last table is quite interesting to me. Any non-trivial software (more than “Hello World”) has bugs. (as anyone that implements WS-* will tell you, WS-* is REALLY non-trivial) When selecting an open source project for use in your enterprise, the questions you need to ask are:

1. How likely am I to hit a bug?
2. If I do hit a bug, how serious is it likely to be?
3. More importantly, if I hit a bug, how likely will a fix be made available in a timely manner?

That last question is critical. As I mentioned, all software has bugs. Getting those bugs fixed when encountered is important.

In an enterprise, you also need to ask if there is a good company behind the project from whom you can get a support agreement, training, consulting, etc… In that case, maybe the above questions are irrelevant.

Of course, being open source, you also should ask: Can I fix it myself and submit the patch back? The communities love it when you do that.

Update: (12 hours later) I just wanted to point out that I DO agree with most of the comments posted below. Raw numbers are kind of pointless and are very hard to do comparisons. They can be interpreted many ways. For example, if you look in the last table, CXF had 441 bugs logged in the last year whereas Axis2 only had 332. Thus, an argument could easily be made that CXF is buggier than the others. Even if I pull out the JAX-RS and DOSGi bugs, it only drops to 366.

However, the one thing that I think IS comparable are the percentages on that last chart. The CXF team is resolving over 85% of the bugs that are logged. The Metro folks are resolving over 80%. Those are pretty respectable numbers. Again, that third question is important. If you do hit a bug, how likely is it to be fixed? 40%? 80%?

Of course, with Open Source, the answer can easily be 100% chance: fix it yourself and submit a patch.

Within few minutes of sitting down, the couple on the other side of us asked to get their food “To Go” because they had to “get away from all the kids” (last part said pretty loudly). That in itself was pretty rude. I can understand asking to be reseated or quietly asking to change to “To Go”, but to emphasize the why was a bit uncomfortable.

The worse part about it was our kids were REALLY on their best behavior tonight. Nathan was busy rolling a crayon back and forth with me and Ryan was coloring the ship on the kids menu and chatting with mommy. Neither was causing and scene. They were both quiet. Both sitting nicely. Etc…

Anyway, we just kind of let that slide, but when their food came and they paid their bill, things got even worse. On the way out, the man flipped us the bird. We were really shocked by that. Neither of us could believe he did that. Believe it or not, it gets even worse. The woman, who had already made it all the way to the door must have looked back and saw the look of shock in our face. She then walks all the way back to the table. I was expecting her to apologize for her husband, but instead, she said “Next time, you should take your kids to Friendy’s”. I wish I would have had a snappy comeback, but I was in complete shock. Couldn’t believe anyone was that rude. The couple next to us (with the child) couldn’t believe it either.

To their credit, the staff at John Harvard’s was excellent. Apparently, the manager was on the way in right as this all happened and saw it all. He came straight over to our table to apologize (not his fault) and the other wait staff also came by specifically to say our kids were behaving quite well and to not take it personally.

Anyway, just had to vent. What a bizarre experience. I still like John Harvard’s a lot and highly recommend it as a place to eat (and drink very good beer . I just hope those people stay away.

commit email

A lot has definitely happened in 3 years. We’ve gone from two very separate codebases (Celtix and XFire) and have produced one of the leading Web Services frameworks. We were the first Apache Licensed JAX-WS compliant implementation. We were the first Apache Licensed JAX-RS compliant implementation. CXF is embedded in MANY applications including JBoss, Geronimo, Jonas, Camel, ServiceMix, and more…

Anyway, it’s been an exciting 3 years. Looking for many more.

• Management involves power by position.
• Leadership involves power by influence.

and specifically these distinctions:

• Managers administer; leaders innovate.
• Managers ask how and when; leaders ask what and why.
• Managers focus on systems; leaders focus on people.
• Managers do things right; leaders do the right things.
• Managers maintain; leaders develop.
• Managers rely on control; leaders inspire trust.
• Managers have short-term perspective; leaders have long-term perspective.
• Managers accept the status-quo; leaders challenge the status-quo.
• Managers have an eye on the bottom line; leaders have an eye on the horizon.
• Managers imitate; leaders originate.
• Managers emulate the classic good soldier; leaders are their own person.
• Managers copy; leaders show originality.

I REALLY like that list. It definitely sums up the causes of many of my “problems” when dealing with “managers” in the past (and present). I definitely tend to fall into the “leader” roll and generally have trouble trying to see the other side of the situations. I readily admit that. But I think it’s also very important to have both types of people represented. One or the other and things don’t happen, or at least not well. It takes a good balance.

So, which do YOU strive to be? A “Leader” or a “Manager”?

However, I did decide to give the git-svn integration a whirl to start getting a feel for git. Basically, learn some of the basic commands, experiment with the tools, etc…. I must admit, I’ve been pleasantly surprised and I now feel that git-svn is nearly the perfect svn client. I don’t think I’ll be going back to normal svn clients for day-to-day work anytime soon. Why do I like it?

• It’s fast – a “git pull” is much faster than a “svn update”. What’s more, “git pull” updates all branches, not just the one I’m working on. Other options like “git blame” and “git log” and such are nearly immediate where with svn, they take forever.
• Disk space – yes, disk space is cheap these days, but it does matter. A fresh “svn checkout” of CXF trunk takes about 225MB of space. A fresh “git clone” of CXF takes about 154Mb of space. The git version also has the advantage of having all the branches and tags in that 154MB of space. With svn, I need an additional 225MB for each tag/branch.
• Works better with tools – thing like “grep -r” and “find . -name” and such work much better with git as there is only the top level “.git” dir, not a “.svn” all over the place. I had to create a bunch of silly wrappers to various things to make using those tools in svn easier.
• All tags/branches are available – if someone posts a stack trace to a JIRA, it’s trivial to flip over to whatever tag they are reporting it against to see what lines the line numbers line up with. Takes just a couple seconds. No “svn checkout” of various tags (or keeping 225MB of space per tag around) or digging through fisheye or similar.
• Saves me a TON of time by allowing me to test multiple “commits” together.

That last entry is really key. If you read my last blog post, you know I’m a firm believer in “DON’T BREAK THE BUILD”. The best way to achieve that is to run the full set of tests before committing any changes. However, the full CXF test suite takes a little over 1/2 hour to run on my machine. Thus, it’s not something you WANT to run on all the little changes. It kind of becomes a limiting factor with how many commits/changes you can do in a day if you always run it.

That’s where git-svn comes in. It’s brilliant. With svn, if you make a small change and then “svn commit”, it’s immediately sent to the central repository for all (and the CI builds) to grab. Thus, you BETTER have run all the tests. With “git commit -a”, you just commit the changes locally on your branch. Thus, I can make some changes, run just the unit tests and maybe a couple of related sys tests, commit the changes, then move to the next JIRA/bud, make some changes, run a few tests, commit, etc… Basically, collect the changes as individual commits in my local git branch. At the end of the day, run a full “mvn install” to make sure ALL the tests pass. If so, run “git svn dcommit” to push the changes back into SVN. The way “git svn dcommit” works is that each commit to the branch goes to SVN as an individual commit. Thus, each JIRA or other change that was worked on and committed locally appears in SVN as an individual commit. This is good as you may need to merge (or backout) just those individual changes. With SVN, I always ended up committing several things together as a single commit which made things harder to track/merge to the fixes branches and such. Separate commits also help boost your commit count.

There are some downsides to using git-svn:

• The egit eclipse plugin sucks, even compared to the subversion eclipse plugins. Call it a “work in progress”.
• Merge tracking – for CXF, we’ve decided to use the svnmerge.py script to track merges between the fixes branches. This doesn’t work with git so I still need the fixes branches checked out to do the merging. Ideally, everything would be “pure git” and we could all use the git merging, but that day isn’t going to come any time soon.
• No integration with maven release plugin – I have no idea how the release plugin would deal with git-svn. Releases still need a “pure” svn checkout.

Anyway, I’ve started pretty much using git-svn for most of my work. I’m definitely happy with it. I’d be happier if Apache would allow us to go 100% git and not have the svn repo, but for now, this is certainly better than svn.

Breaking a build on something you’re the only one working on is usually not a huge issue. Just commit a fix at some point and no-one will ever know. However, when working with a large team or working on OpenSource where people all over the world could be building it, break a build can be a serious issue. If the build is broken, you can easily have MANY engineers wasting a lot of time trying to debug something they shouldn’t be. That can be quite “expensive”.

For example, if I run the tests and see a test failure, I assume one of my changes is the cause of that. Thus, I start debugging and try and figure out what I did. I shouldn’t be trying to debug into tests that aren’t caused by my changes. The code base should be stable and just work.

Unfortunately and apparently, there are a lot of people that don’t believe the same as I do. The Hudson Continuous Integration Game plugin provides an interesting way to see who believes what. Basically, for each build done in Hudson, the plugin assigns a score. If the build succeeds cleanly, all developers that contributed changes get a point. If the build fails spectacularly (fails to even compile), it’s -10 points. A failing unit test is a minus point. Fixing a failing test is a plus point. etc… Thus, over time, people that tend to break things end up with less points (even negative) and those that tend to fix things and/or commit clean code end up positive. Obviously, it’s not perfect. People that commit a LOT can earn higher scores as they can get one point per commit. I commit a couple times a day. Thus, I have potential to get more points than someone who only commits a couple times a week.

Looking at the Apache Hudson Instance Leader Board, you can see quite a range of scores. IMO, there are entirely too many people with negative scores. Something for those people to start working on. Get to work!