Dan Kulp's Blog

On March 9, it would have been 10 years ago when IONA acquired Watershed, the small startup that I was working at. Over the last 10 years, I’ve been involved in many projects, done a lot of new things, met a lot of really great people, and most had a pretty good time. There were rocky patches, of course, but all in all, it’s been an exciting adventure.

As they say, all good things must end. As of today, Progress has let me know they no longer need me. Thus, I’m now unemployed. This is really the first time since I was 15 that I’m either not in school or don’t have a job. Very strange feeling.

What does this mean for me? I don’t really know at this point. My knee-jerk reaction this morning was to start pinging all my contacts to get another job ASAP. The “oh my god, I’m unemployed!” panic attack. Luckily, I’ve cooled down a bit since then. I need to take a little time and figure out what options are out there, what I really want to do, and what fits me (and my family) best. I don’t yet know what that will be. At this point, I’m hoping I can find something that allows me to stay involved in the open source projects I really believe in and stay involved with Apache. But like I said, I need a little time to calm down and gather some thoughts.

I was chatting with a good friend this morning and he stated that he believed being laid off was the best thing that could have happened to me. Right now, I’m not sure I agree, but I’m definitely taking a “wait and see” approach.

To all of you in the Boston area, I’m suddenly free for lunch pretty much anytime.

It’s now been over three months since I blogged about the status of the various Java SOAP stacks and the bug counts. When I first posted it, many people considered it a “wake up call” to the projects and were wondering how the projects would respond.

A lot has happened in the last three months so I wanted to re-examine the states of the projects.

Apache CXF

The CXF team had a busy three months mostly working on fixing bugs directly reported by users. They released 2.1.8 and 2.2.5 in November and 2.1.9 and 2.2.6 in January. Between them, over 150 JIRA issues were closed. Some bugs, some new features, etc… CXF also had two books published targetting CXF users providing additional documentation and tutorial information.

Apache Axis

The Axis folks also had an exciting quarter. They released their Axis2 1.5.1 patch that fixed around 15 bugs, one of them a critical issue that was causing a lot of problems. Their most exciting news, however, was that they have been promoted to a Top Level Project at Apache. Probably long overdue, but it’s nice to see them out of the “Web Services” umbrella and out on more equal footing with the other projects at Apache. The process of getting things moved out to the new TLP areas is still ongoing, but a major “Congrats!” goes out to them for finally getting that done!

Sun Metro
Obviously the big news for Sun is that the aquisition by Oracle is now complete. Time will tell if that ends up a good thing or bad thing for the Metro folks. In anycase, they released their new 2.0 version in the last few months which includes a bunch of new things such as JAX-WS 2.2 support.

As you can see, it’s been quite a busy three months for all three of the projects. However, I’m still interested in looking at how many “known bugs” exist in the projects. As of today (Feb 12, 2010):

So, of the three projects, only CXF reduced their bug count. That said, I certainly cannot fault the Metro folks. I would definitley expect an influx of bugs after releasing any “#.0″ version and the uncertainty of the Oracle/Sun situation I’m sure put a damper on things for quite a while.

The last table in my previous post showed what percent of bugs logged recently had been fixed. If you look at just the bugs that have been logged since my previous post on Nov 5, you see:

Last time, people complained about using raw bug count numbers. I agree that raw numbers are usually not useful since projects with more users are more likely to have users that encounter (and report) bugs. Complexities of the projects are different. Features are different. How the tracking system is managed is different. Etc…. However, it is the trends that the raw numbers expose that I think is important. Bug counts going down or staying steady is good. Projects fixing a majority of reported bugs in a timely manner is good. Projects providing new releases that fix reported bugs is good.

Of course, bug counts and releases are only a couple aspects that are important when selecting a project. Features are important (after all, if it doesn’t have a feature you need, it doesn’t matter how bug free it is). Performance is important. Documentation is important. Ease of use is important. A friendly and responsive community is important. I’m sure there are lots of other reasons that people use when evaluating projects. I’d love to see CXF “on top” in all those categories, although I have to admit, I’m pretty bad at doing documentation.

I finally managed to get Apache CXF 2.2.6 and 2.1.9 released and out the door. In a lot of ways, this was a “big” release, not because of any cool or exciting new features, but due to the concerted effort of just “fixing bugs”. At the time I did the release build, there were 19 known “bugs” (couple more bugs have been logged since then), several of them blocked on third party fixes such as in JAXB and WSS4J. Getting down that low has been a giant effort for the entire CXF team. At one point about 4 months ago, we had almost 250 open bugs. Getting to the current level required a LOT of work on everyones part to dig through JIRA, write test cases, dig into with debuggers, etc… The result is definitely all positive. We’ve fixed a TON of bugs that have been outstanding for a very long time. I definitely encourage CXF users to update to 2.2.6.

While the push to reduce the bug count in CXF has been a very welcome and important step for CXF, it has detracted from the development of the new features and stuff for 2.3. 2.3 is definitely many months behind where I would have liked it to be. Now that the count is down to something reasonable and manageable, I’m really hoping to re-concentrate on getting 2.3 ready to ship. No promises though.

Claus Ibsen wrote up a nice blog post about how Apache Camel did this year using some easily obtained metrics. I thought it was kind of interesting so I wanted to do the same for CXF.

Number of posts on the users list: 6614

Number of posts on the dev list: 1661

Number of messages on commits list (svn commits and wiki changes and such): 3893

Number of JIRA issues raised in 2009: 595

Number of JIRA issues resolved in 2009: 763

Number of JIRA issues raised in 2009, but still unresolved: 51 (only 19 are bugs, 3 are bugs in JAXB where we need a new release of JAXB and 5 were logged in the last 2 weeks while most devs were on vacation, so really 11 bugs unresolved).

The JIRA stats are kind of interesting. Put in a graph:

CXF issues created/resolved graph for 2009

You can kind of see that the last 4 months or so, the CXF community really tried to go through all the old JIRA entries and resolve as many as possible. The result should be a much more stable and bug free product.

Just a week or so after the first book for Apache CXF was released, a second book now appears. Developing Web Services with Apache CXF and Axis2 (3rd Edition) is an update to Kent Ka Iok Tong’s book about Axis2, but it now covers CXF as well.

Being on holidays, I haven’t had time to look much at it (I’m only 1/4 of the way through the first book), but a quick glance through provided me a nice surprise. I kind of expected that it would be geared more toward Axis2 (since it’s an 3rd edition of the Axis2 book) with small “Now here’s how you do the same thing with CXF” type sections. However, it looks to be completely the oppossite. The initial examples and screen shots and stuff in the text is using CXF and JAX-WS stuff and then a small “Now here’s how you do the same thing with Axis2 section.”

Anyway, after the holidays, I hope to review both books in more detail.

Someone once told me that you’ll know your product is a success when someone writes a “For Dummies” book about it. Apache CXF took a step in that direction yesterday with the release of Apache CXF Web Service Development. As far as I know, this is the first published book about CXF.

What’s particularly exciting about this is that it’s not written by any committer on the CXF project. The authors and reviewers are completely independent. They decided that CXF needed a book and set out to write one. That’s quite interesting because on most of the OpenSource projects I’ve seen, the first sets of books are usually written by people actively involved in the project. The fact that this was written by actual users, not the developers working on the project, gives my some hope that it’s targeted correctly. One of the problems I’ve seen repeatedly with books written by the same developers that are writing the project code is the book gets too technical and makes assumptions about the readers that aren’t correct. The “I know this, so eveyone else must know it too” fallacy.

Anyway, I haven’t read it yet (I just bought it and am only up to page 17 or 336) so I cannot vouch for how good/bad it is yet. However, I’m very excited as this does fill a gap in the CXF offerings. I just wanted everyone to know it’s there.

One of the fun things about attending conferences such as ApacheCon is that you get to meet with users of the software I work on and hear “real world” stories of the issues they’ve run into. I talked to one user yesterday that presented a common story that I’ve heard over and over again from many users, but it really got me thinking.

Basically, a short while ago, he was tasked with upgrading some older web services to get them off of Apache Axis. They started porting to Axis2 assuming that would be the easiest effort. After hitting bugs and spending a lot of time trying to work around issues and such, they punted on the effort and switched to CXF. For them, CXF “just worked”.

I hear the same basic thing from many users. CXF just “works”. It does what it says it will do relatively easily (relative term, nothing in WS-* is designed to be easy) and bugs, when found, are usually fixed quickly, but most users I’ve talked to having found any real bugs to log. It just “works”.

For some reason, the person I talked to yesterday got me thinking about the various Open Source SOAP stacks and how “buggy” they are and kind of wondered if that could be quantified. One of the nice things about the open source stacks is the issue trackers are also “open”. Thus, it shouldn’t be too hard. Famous last words.

There are two main issues with trying to compare bug counts:

1. Features: Each of the three major open source stacks (Axis2, CXF, Metro), while they all implement the same basic specs, they also have unique features. For example, CXF has bugs logged for the CORBA binding, JAX-RS, Distributed OSGi, etc… that the others wouldn’t have. Axis2 has bugs for the UDP transport which is not available in the others. Etc….
2. Issue tracker layout: Each of the projects handle their issue tracker differently. CXF has a single project in JIRA and all bugs for everything are logged there. Metro has their issues split into two trackers, one for the base JAX-WS implementation and another for the WS-* stuff. Axis2 splits things across a bunch of JIRAs: a core area, one for Rampart, one for Sandesha, one for Kandula, transports, etc….

With that in mind, lets look at some numbers. To make things simple, I looked at the full “bug” count from CXF including all the features including JAX-RS, CORBA, DOSGi, etc…. Thus, the CXF counts are relatively inflated compared to the others. I didn’t look at feature requests, wishes, tasks, etc… Just bugs. For Metro, I looked at the “defects” only for the JAXWS RI and WSIT trackers. For Axis2, to make things simple, I only looked at the “bugs” in the core “Axis2″ jira and didn’t bother looking at the others. I didn’t need to as you’ll see.

As of this morning, Nov 5, 2009, the counts are:

I think those numbers paint a pretty good picture of the state of stacks. However, it’s not the whole picture. Another useful stat is the number of “critical” or “blocker” bugs. (For Metro, P1 or P2 level)

Another interesting stat is how many bugs logged in the last year or 6 months remain open. This shows how well the community responds to users bug reports.

Anyway, that last table is quite interesting to me. Any non-trivial software (more than “Hello World”) has bugs. (as anyone that implements WS-* will tell you, WS-* is REALLY non-trivial) When selecting an open source project for use in your enterprise, the questions you need to ask are:

1. How likely am I to hit a bug?
2. If I do hit a bug, how serious is it likely to be?
3. More importantly, if I hit a bug, how likely will a fix be made available in a timely manner?

That last question is critical. As I mentioned, all software has bugs. Getting those bugs fixed when encountered is important.

In an enterprise, you also need to ask if there is a good company behind the project from whom you can get a support agreement, training, consulting, etc… In that case, maybe the above questions are irrelevant.

Of course, being open source, you also should ask: Can I fix it myself and submit the patch back? The communities love it when you do that.

Update: (12 hours later) I just wanted to point out that I DO agree with most of the comments posted below. Raw numbers are kind of pointless and are very hard to do comparisons. They can be interpreted many ways. For example, if you look in the last table, CXF had 441 bugs logged in the last year whereas Axis2 only had 332. Thus, an argument could easily be made that CXF is buggier than the others. Even if I pull out the JAX-RS and DOSGi bugs, it only drops to 366.

However, the one thing that I think IS comparable are the percentages on that last chart. The CXF team is resolving over 85% of the bugs that are logged. The Metro folks are resolving over 80%. Those are pretty respectable numbers. Again, that third question is important. If you do hit a bug, how likely is it to be fixed? 40%? 80%?

Of course, with Open Source, the answer can easily be 100% chance: fix it yourself and submit a patch.

We had an “interesting” experience at dinner tonight. We (myself, my wife, and 2 kids) went out to dinner at John Harvard’s in Framingham. We’ve been there as a family many times before and always enjoy it. While it is a Brew House which you wouldn’t think is the most “family friendly” environment, we almost always are seated near other families with small kids and plenty of kids are always there. Today wasn’t an exception as we were seated next to a couple with a 17 month old girl with them (just a month older than our youngest).

Within few minutes of sitting down, the couple on the other side of us asked to get their food “To Go” because they had to “get away from all the kids” (last part said pretty loudly). That in itself was pretty rude. I can understand asking to be reseated or quietly asking to change to “To Go”, but to emphasize the why was a bit uncomfortable.

The worse part about it was our kids were REALLY on their best behavior tonight. Nathan was busy rolling a crayon back and forth with me and Ryan was coloring the ship on the kids menu and chatting with mommy. Neither was causing and scene. They were both quiet. Both sitting nicely. Etc…

Anyway, we just kind of let that slide, but when their food came and they paid their bill, things got even worse. On the way out, the man flipped us the bird. We were really shocked by that. Neither of us could believe he did that. Believe it or not, it gets even worse. The woman, who had already made it all the way to the door must have looked back and saw the look of shock in our face. She then walks all the way back to the table. I was expecting her to apologize for her husband, but instead, she said “Next time, you should take your kids to Friendy’s”. I wish I would have had a snappy comeback, but I was in complete shock. Couldn’t believe anyone was that rude. The couple next to us (with the child) couldn’t believe it either.

To their credit, the staff at John Harvard’s was excellent. Apparently, the manager was on the way in right as this all happened and saw it all. He came straight over to our table to apologize (not his fault) and the other wait staff also came by specifically to say our kids were behaving quite well and to not take it personally.

Anyway, just had to vent. What a bizarre experience. I still like John Harvard’s a lot and highly recommend it as a place to eat (and drink very good beer . I just hope those people stay away.

Today marks exactly 3 years since my first commit at Apache toward Apache CXF:

commit email

A lot has definitely happened in 3 years. We’ve gone from two very separate codebases (Celtix and XFire) and have produced one of the leading Web Services frameworks. We were the first Apache Licensed JAX-WS compliant implementation. We were the first Apache Licensed JAX-RS compliant implementation. CXF is embedded in MANY applications including JBoss, Geronimo, Jonas, Camel, ServiceMix, and more…

Anyway, it’s been an exciting 3 years. Looking for many more.

On a recent thread on the general@incubator list at Apache, Niclas Hedhman pointed out a very interesting list of differences between Managers and Leaders from Wikipedia. I really like this list:

• Management involves power by position.
• Leadership involves power by influence.

and specifically these distinctions:

• Managers administer; leaders innovate.
• Managers ask how and when; leaders ask what and why.
• Managers focus on systems; leaders focus on people.
• Managers do things right; leaders do the right things.
• Managers maintain; leaders develop.
• Managers rely on control; leaders inspire trust.
• Managers have short-term perspective; leaders have long-term perspective.
• Managers accept the status-quo; leaders challenge the status-quo.
• Managers have an eye on the bottom line; leaders have an eye on the horizon.
• Managers imitate; leaders originate.
• Managers emulate the classic good soldier; leaders are their own person.
• Managers copy; leaders show originality.

I REALLY like that list. It definitely sums up the causes of many of my “problems” when dealing with “managers” in the past (and present). I definitely tend to fall into the “leader” roll and generally have trouble trying to see the other side of the situations. I readily admit that. But I think it’s also very important to have both types of people represented. One or the other and things don’t happen, or at least not well. It takes a good balance.

So, which do YOU strive to be? A “Leader” or a “Manager”?