Dan Kulp's Blog

This weekend, there was an interesting exchange of email on the Apache WebService email list that really struck a nerve with me.

To give a little context, one of the things the Apache WebServices project is doing is kind of auditing it’s subprojects to figure out which projects are healthy enough to graduate to a new top level project (TLP) at Apache, which are best served by remaining subprojects of Apache WebServices, and which projects are pretty much “dead” and are candidates to move to the Apache Attic. This is not an easy process and definitely takes time and consideration and the Apache WebServices project members should definitely be applauded for taking on this task.

As part of this, they identified two subprojects to move to the Attic, the WSIF subproject and the Muse subproject. Neither project has done a release in YEARS (early 2007) and no-one is really doing any work on either project. There hasn’t been any develpment traffic on the mailing list or even user traffic. No commits, jira’s etc… Basically, nothing going on at all. Sounds like PERFECT candidates for the Attic and the WebServices project pretty much came to the conclusion.

Then, this weekend, an email was sent to the ws mailing list basically expressing concern about Muse heading to the attic. They apparently are using and developing Muse internally. To quote directly:

“we have been maintaining an internal Muse build in which we have patched
open issues/bugs as we encountered them and improved parts of its implementation”

When I read that, I definitely had a “WTF?!?!” moment. This is a common issue I have with a lot of companies, but the Muse sitation makes it even more comical.

I’ve seen a lot of cases where patches and fixes and such are applied to an internal fork of a project. I’m defintely OK with that part. In many cases, that’s the fastest and easiest way to get an issue fixed. The problem is when those fixes never get pushed back into the upstream project. It’s even worse when issues aren’t even logged at the upstream project so others don’t even know there is a problem. What I usually see happen at this point is that the “internal project” sometime down the road tries to upgrade to the “latest and greatest” version of the upstream project, but that doesn’t go well as it doesn’t contain some fixes and such and people forgot about those issues and it pretty much just wastes everyones time.

However, in this case, the repercussions are even bigger. If they had filed JIRA’s, submitted patches, discussed the improvements on the dev lists, etc…., then the project itself may still be active. They themselves may have been voted in as committers. There wouldn’t have been thoughts about moving it to the Attic (quite the opposite, moving to TLP may have been on the table).

I guess it really boils down to my belief that if you use an open source project, it’s definitely in your best interest to participate in that project. Contributing back and helping to grow the community really helps in the long run. This example is really an extreme case in the other direction.

Sopera, the company I am working for, is in the process of ramping up it’s support for for some of the popular Apache SOA/middleware projects on which it’s products are based. For those of you not familiar with Sopera’s offerings, the main projects that would be included are:

• Apache CXF
• Apache Camel
• Apache ActiveMQ
• Apache ServiceMix

In addition to quality developers, we’re also looking for evangelists and sales folks that are familiar with Open Source and the Apache projects.

If you think you would be a good fit, please checkout the careers page on Sopera’s web site.

I’ve been unemployed for just over 9 weeks now, but as of today, that comes to an end as I start a new job. While I did spend a significant part of that time NOT thinking about working at all, I have spent the last few weeks chatting with several people about opportunities. In the end, it really came down to one of three options:

1. Independent contracting – the day I was layed off, I had SEVERAL people contacting me about possible contracts and projects needing help and such. Being one of the primary developers on the leading Java Web Services stacks, there are a LOT of opportunities with companies using CXF that would like an expert to come in an help out. I really did consider this option fairly extensively. Bunches of pros (money, hours, tax deductions for new computer, etc..) and few cons (likely some travel).
2. JBoss/RedHat – RedHat is a gigantic company and there are PLENTY of opportunities for someone with my skills. Actually finding the “right” position can sometimes be a challenge due to the size. I talked to a couple people and I know some of the other folks that Progress let go also talked to a bunch of people in RedHat before finding appropriate spots. You can literally interview for weeks before finding your one true spot. That’s good and bad. Good that there are a lot of opportunities, but kind of frustrating the sheer number of people you may need to talk to. That said, there are a LOT of good people that work for RedHat/JBoss. Joining a team there would really mean working with some good people. RedHat has also hired a bunch of the other engineers that Progress let go so I already would know a bunch of folks. With RedHat’s engineering HQ in Mass., they would also be able to easily offer benefits such as insurance and such, which is another positive.
3. Sopera – pretty much on the extreme opposite end of the spectrum from JBoss is Sopera. Sopera is a MUCH MUCH smaller, private company based in Germany that specializes in SOA expertise. Their stack is very similar to what I worked on at Progress, with projects like CXF and ServiceMix being building blocks for a more complete SOA platform. They provide a lot of value add on top of the Apache projects. However, they also understand and promote open source concepts very well. For example, they are a major driving force behind the Swordfish project at Eclipse. In a lot of ways, the stack at Sopera is much more complete than what we were able to provide at FUSE and that provides a bunch of new and exciting opportunities to expand my knowledge, yet also apply my expertise in new and exciting ways.

So, which direction did I chose? I ended up going with option #3, Sopera. At this point in time, I REALLY feel this was the best fit for me.

My wife kind of vetoed the independent contractor thing. With two small children (one entering his terrible twos), she wasn’t comfortable with some of the uncertainty around that option. I definitely thought it would be a fun challenge getting out to see real world scenarios and such, but the concern around the uncertainty and such is somewhat justified.

JBoss definitely had a bunch of positives, but there were a couple things that kind of bothered me. The number one issue is the size. After dealing with “large company politics” at Progress for a year and a half, and the “we wish we were a large company” politics at IONA for 8 years, I really wanted a break from that. Also, with the size of JBoss/RedHat and the sheer number and breadth of projects, it was kind of hard to feel “super excited” about any individual opportunity. In the end, I really felt that, at this point in time, a smaller company where the entire company is focused around doing one thing and doing it right was more what I was looking for. That said, I could just be repressing all the bad parts about working for smaller companies. Time will tell. Oh, and to the other IONA/Progress people that are now working for RedHat, I definitely wish them the best of luck and a big thanks to RedHat for stepping up and offering them positions that they are now excited about.

So, in the end, the Sopera opportunity really feels right for me. I get to continue to be highly involved with Apache CXF and continue to drive direction there. I’ll probably have to get more involved with ServiceMix, which is something I should have been doing anyway. I may also have to get more involved with the communities at Eclipse, which will be completely new for me. Being a smaller company, I’m also going to have to be highly involved in solving specific customer problems. That’s NOT a bad thing. Getting to see and participate in real world use cases is kind of exciting. Of course, there is certainly going to be new issues and challenges. Even negotiating terms was “interesting” due to language differences and the 6 hour time zone between myself and them. Nothing insurmountable though. Dublin was 5 hours off (most of the year) so I’m kind of used to at least that part.

I’m quite excited to see how this new chapter of my life unfolds.

It’s now been over three months since I blogged about the status of the various Java SOAP stacks and the bug counts. When I first posted it, many people considered it a “wake up call” to the projects and were wondering how the projects would respond.

A lot has happened in the last three months so I wanted to re-examine the states of the projects.

Apache CXF

The CXF team had a busy three months mostly working on fixing bugs directly reported by users. They released 2.1.8 and 2.2.5 in November and 2.1.9 and 2.2.6 in January. Between them, over 150 JIRA issues were closed. Some bugs, some new features, etc… CXF also had two books published targetting CXF users providing additional documentation and tutorial information.

Apache Axis

The Axis folks also had an exciting quarter. They released their Axis2 1.5.1 patch that fixed around 15 bugs, one of them a critical issue that was causing a lot of problems. Their most exciting news, however, was that they have been promoted to a Top Level Project at Apache. Probably long overdue, but it’s nice to see them out of the “Web Services” umbrella and out on more equal footing with the other projects at Apache. The process of getting things moved out to the new TLP areas is still ongoing, but a major “Congrats!” goes out to them for finally getting that done!

Sun Metro
Obviously the big news for Sun is that the aquisition by Oracle is now complete. Time will tell if that ends up a good thing or bad thing for the Metro folks. In anycase, they released their new 2.0 version in the last few months which includes a bunch of new things such as JAX-WS 2.2 support.

As you can see, it’s been quite a busy three months for all three of the projects. However, I’m still interested in looking at how many “known bugs” exist in the projects. As of today (Feb 12, 2010):

So, of the three projects, only CXF reduced their bug count. That said, I certainly cannot fault the Metro folks. I would definitley expect an influx of bugs after releasing any “#.0″ version and the uncertainty of the Oracle/Sun situation I’m sure put a damper on things for quite a while.

The last table in my previous post showed what percent of bugs logged recently had been fixed. If you look at just the bugs that have been logged since my previous post on Nov 5, you see:

Last time, people complained about using raw bug count numbers. I agree that raw numbers are usually not useful since projects with more users are more likely to have users that encounter (and report) bugs. Complexities of the projects are different. Features are different. How the tracking system is managed is different. Etc…. However, it is the trends that the raw numbers expose that I think is important. Bug counts going down or staying steady is good. Projects fixing a majority of reported bugs in a timely manner is good. Projects providing new releases that fix reported bugs is good.

Of course, bug counts and releases are only a couple aspects that are important when selecting a project. Features are important (after all, if it doesn’t have a feature you need, it doesn’t matter how bug free it is). Performance is important. Documentation is important. Ease of use is important. A friendly and responsive community is important. I’m sure there are lots of other reasons that people use when evaluating projects. I’d love to see CXF “on top” in all those categories, although I have to admit, I’m pretty bad at doing documentation.

I finally managed to get Apache CXF 2.2.6 and 2.1.9 released and out the door. In a lot of ways, this was a “big” release, not because of any cool or exciting new features, but due to the concerted effort of just “fixing bugs”. At the time I did the release build, there were 19 known “bugs” (couple more bugs have been logged since then), several of them blocked on third party fixes such as in JAXB and WSS4J. Getting down that low has been a giant effort for the entire CXF team. At one point about 4 months ago, we had almost 250 open bugs. Getting to the current level required a LOT of work on everyones part to dig through JIRA, write test cases, dig into with debuggers, etc… The result is definitely all positive. We’ve fixed a TON of bugs that have been outstanding for a very long time. I definitely encourage CXF users to update to 2.2.6.

While the push to reduce the bug count in CXF has been a very welcome and important step for CXF, it has detracted from the development of the new features and stuff for 2.3. 2.3 is definitely many months behind where I would have liked it to be. Now that the count is down to something reasonable and manageable, I’m really hoping to re-concentrate on getting 2.3 ready to ship. No promises though.

Claus Ibsen wrote up a nice blog post about how Apache Camel did this year using some easily obtained metrics. I thought it was kind of interesting so I wanted to do the same for CXF.

Number of posts on the users list: 6614

Number of posts on the dev list: 1661

Number of messages on commits list (svn commits and wiki changes and such): 3893

Number of JIRA issues raised in 2009: 595

Number of JIRA issues resolved in 2009: 763

Number of JIRA issues raised in 2009, but still unresolved: 51 (only 19 are bugs, 3 are bugs in JAXB where we need a new release of JAXB and 5 were logged in the last 2 weeks while most devs were on vacation, so really 11 bugs unresolved).

The JIRA stats are kind of interesting. Put in a graph:

CXF issues created/resolved graph for 2009

You can kind of see that the last 4 months or so, the CXF community really tried to go through all the old JIRA entries and resolve as many as possible. The result should be a much more stable and bug free product.

Just a week or so after the first book for Apache CXF was released, a second book now appears. Developing Web Services with Apache CXF and Axis2 (3rd Edition) is an update to Kent Ka Iok Tong’s book about Axis2, but it now covers CXF as well.

Being on holidays, I haven’t had time to look much at it (I’m only 1/4 of the way through the first book), but a quick glance through provided me a nice surprise. I kind of expected that it would be geared more toward Axis2 (since it’s an 3rd edition of the Axis2 book) with small “Now here’s how you do the same thing with CXF” type sections. However, it looks to be completely the oppossite. The initial examples and screen shots and stuff in the text is using CXF and JAX-WS stuff and then a small “Now here’s how you do the same thing with Axis2 section.”

Anyway, after the holidays, I hope to review both books in more detail.

Someone once told me that you’ll know your product is a success when someone writes a “For Dummies” book about it. Apache CXF took a step in that direction yesterday with the release of Apache CXF Web Service Development. As far as I know, this is the first published book about CXF.

What’s particularly exciting about this is that it’s not written by any committer on the CXF project. The authors and reviewers are completely independent. They decided that CXF needed a book and set out to write one. That’s quite interesting because on most of the OpenSource projects I’ve seen, the first sets of books are usually written by people actively involved in the project. The fact that this was written by actual users, not the developers working on the project, gives my some hope that it’s targeted correctly. One of the problems I’ve seen repeatedly with books written by the same developers that are writing the project code is the book gets too technical and makes assumptions about the readers that aren’t correct. The “I know this, so eveyone else must know it too” fallacy.

Anyway, I haven’t read it yet (I just bought it and am only up to page 17 or 336) so I cannot vouch for how good/bad it is yet. However, I’m very excited as this does fill a gap in the CXF offerings. I just wanted everyone to know it’s there.

One of the fun things about attending conferences such as ApacheCon is that you get to meet with users of the software I work on and hear “real world” stories of the issues they’ve run into. I talked to one user yesterday that presented a common story that I’ve heard over and over again from many users, but it really got me thinking.

Basically, a short while ago, he was tasked with upgrading some older web services to get them off of Apache Axis. They started porting to Axis2 assuming that would be the easiest effort. After hitting bugs and spending a lot of time trying to work around issues and such, they punted on the effort and switched to CXF. For them, CXF “just worked”.

I hear the same basic thing from many users. CXF just “works”. It does what it says it will do relatively easily (relative term, nothing in WS-* is designed to be easy) and bugs, when found, are usually fixed quickly, but most users I’ve talked to having found any real bugs to log. It just “works”.

For some reason, the person I talked to yesterday got me thinking about the various Open Source SOAP stacks and how “buggy” they are and kind of wondered if that could be quantified. One of the nice things about the open source stacks is the issue trackers are also “open”. Thus, it shouldn’t be too hard. Famous last words.

There are two main issues with trying to compare bug counts:

1. Features: Each of the three major open source stacks (Axis2, CXF, Metro), while they all implement the same basic specs, they also have unique features. For example, CXF has bugs logged for the CORBA binding, JAX-RS, Distributed OSGi, etc… that the others wouldn’t have. Axis2 has bugs for the UDP transport which is not available in the others. Etc….
2. Issue tracker layout: Each of the projects handle their issue tracker differently. CXF has a single project in JIRA and all bugs for everything are logged there. Metro has their issues split into two trackers, one for the base JAX-WS implementation and another for the WS-* stuff. Axis2 splits things across a bunch of JIRAs: a core area, one for Rampart, one for Sandesha, one for Kandula, transports, etc….

With that in mind, lets look at some numbers. To make things simple, I looked at the full “bug” count from CXF including all the features including JAX-RS, CORBA, DOSGi, etc…. Thus, the CXF counts are relatively inflated compared to the others. I didn’t look at feature requests, wishes, tasks, etc… Just bugs. For Metro, I looked at the “defects” only for the JAXWS RI and WSIT trackers. For Axis2, to make things simple, I only looked at the “bugs” in the core “Axis2″ jira and didn’t bother looking at the others. I didn’t need to as you’ll see.

As of this morning, Nov 5, 2009, the counts are:

I think those numbers paint a pretty good picture of the state of stacks. However, it’s not the whole picture. Another useful stat is the number of “critical” or “blocker” bugs. (For Metro, P1 or P2 level)

Another interesting stat is how many bugs logged in the last year or 6 months remain open. This shows how well the community responds to users bug reports.

Anyway, that last table is quite interesting to me. Any non-trivial software (more than “Hello World”) has bugs. (as anyone that implements WS-* will tell you, WS-* is REALLY non-trivial) When selecting an open source project for use in your enterprise, the questions you need to ask are:

1. How likely am I to hit a bug?
2. If I do hit a bug, how serious is it likely to be?
3. More importantly, if I hit a bug, how likely will a fix be made available in a timely manner?

That last question is critical. As I mentioned, all software has bugs. Getting those bugs fixed when encountered is important.

In an enterprise, you also need to ask if there is a good company behind the project from whom you can get a support agreement, training, consulting, etc… In that case, maybe the above questions are irrelevant.

Of course, being open source, you also should ask: Can I fix it myself and submit the patch back? The communities love it when you do that.

Update: (12 hours later) I just wanted to point out that I DO agree with most of the comments posted below. Raw numbers are kind of pointless and are very hard to do comparisons. They can be interpreted many ways. For example, if you look in the last table, CXF had 441 bugs logged in the last year whereas Axis2 only had 332. Thus, an argument could easily be made that CXF is buggier than the others. Even if I pull out the JAX-RS and DOSGi bugs, it only drops to 366.

However, the one thing that I think IS comparable are the percentages on that last chart. The CXF team is resolving over 85% of the bugs that are logged. The Metro folks are resolving over 80%. Those are pretty respectable numbers. Again, that third question is important. If you do hit a bug, how likely is it to be fixed? 40%? 80%?

Of course, with Open Source, the answer can easily be 100% chance: fix it yourself and submit a patch.

Today marks exactly 3 years since my first commit at Apache toward Apache CXF:

commit email

A lot has definitely happened in 3 years. We’ve gone from two very separate codebases (Celtix and XFire) and have produced one of the leading Web Services frameworks. We were the first Apache Licensed JAX-WS compliant implementation. We were the first Apache Licensed JAX-RS compliant implementation. CXF is embedded in MANY applications including JBoss, Geronimo, Jonas, Camel, ServiceMix, and more…

Anyway, it’s been an exciting 3 years. Looking for many more.