Dan Kulp's Blog

A long time ago (ok, only 5.5 years ago), when Apache CXF was fist being spun up, we made a decision to try and separate the “API” and the “implementation” of the various CXF components into separate module. This was a very good idea for the most part. However, we did this prior to really investigating OSGi requirements (OSGi was just beginning to become well known). Thus, we didn’t take into account the OSGi split-package issue. The common pattern we used was to put the interfaces in cxf-api, and the “impl” in the exact same package in cxf-rt-core. In OSGi, that’s a big problem (without using fragments).

A few years ago, when users started really pushing the idea of using CXF in OSGi, we had to come up with a solution. I’ll admit we kind of took the “easy road” at the time. As part of the CXF build, we already were creating an “uber bundle” which contained all of the individual CXF modules squashed together into a single jar. We primarily did that to simplify classpath issues, dependencies, etc… for users (single jar to deal with, single pom dependency, etc…). When looking at options for OSGi, we saw that bundle as an “easy” path to OSGi. By adding OSGi manifest entries, CXF can be brought up in OSGi. Since ALL the packages are in that jar, no split-package issues.

For the most part, that setup has worked “OK”, but it’s really not ideal. CXF has a wide range of functionality, much of it may not be required by every user. The bundle ended up importing a LOT of packages. The current bundle imports about 240 packages. Thus, we had to go through and mark many of the imports “optional”. Over 160 of those imports are marked optional. (and likely some of the others could as well) This type of setup, while it “works”, is really less than ideal. It doesn’t work well with things like the OBR resolvers. It requires extra refreshes to add functionality that may be required by new bundles. It requires a lot of extra dependencies to be loaded upfront to reduce refreshes. Etc…

Over the past few weeks, Christian Schneider and I have been re-evaluating the split-package issue in CXF to try and come up with a better solution. An initial evaluation back in December found 25 packages that were split across jars. I have spent quite a bit of time the last couple weeks looking at each of those 25 split packages to figure out what can be done. We did hit some challenges and I ended up talking quite a bit with Christian about possible solutions, but I’m really happy to say that as of my commits this morning, all 25 are now resolved. All of the CXF individual modules are loadable as individual OSGi bundles. A couple of very simple test cases are working.

What kind of problems did we hit:

• One of the main issues as mentioned above was an interface in API and then some sort of implementation in rt-core. For the most part, users always looked up the implementation via the bus.getExtension(..) call so the impl is really “private”. In those cases, the impl was moved to a new package in rt-core.
• Abstract base classes in rt-core – similar to above. We had the interface in api, but an abstract base class in rt-core. There were then subclasses of that base class in the other modules. This really showed that the abstract base classes should have been part of the API, which is exactly what we did. Moved the class to API.
• Promote some stuff to API – we did have a bunch of utility things in rt-core that was easily promoted to api. Things like our CachedOutputStream, mime attachment support, etc… that were easy to just promote up. They are heavily used in several submodules so it made sense to promote them anyway.
• Pushed some optional stuff out of API – particular WS-Policy support. We did have some ws-policy related stuff in API that we could not resolve the split-package issue by promoting up. Thus, we pushed down into the cxf-rt-ws-policy. This may have an end user impact as they may need to have the cxf-rt-ws-policy module on their classpath to compile their code. However, they would have needed it on their classpath anyway to run the code so impact should be very minimal. The good thing about this is that this then reduces the imports in cxf-api by not requiring things like neethi for the api module. For JAX-RS users, this is good.

There were other issues caused by promoting stuff into API, but for the most part, they ended up being resolvable by loading an implementation via a bus extension. That seems to have cleaned things up quite a bit.

Anyway, it’s been a very challenging few weeks as we worked through all the split package issues. To be honest, the end result isn’t “ideal”. There is definitely more stuff in API than I’d really like to have there, but we were able to maintain a high degree of compatibility with existing applications. That’s important. Maybe when we do a “3.0″ we can re-look at that, but for now, compatibility is very important. With the split package issue resolved, Christian and I are embarking on a new mission to examine the OSGi manifests of all the individual jars to adjust import version ranges, check for optionals, update the karaf features file, etc… One advantage of the big bundle was we only needed to do that in one place. Now we have many more touch points. However, the end result should definitely be a better experience for end users. I’m very excited about that.

After writing my post about setting up Apache CXF in OSGi, I’ve had a couple people ask about how to do the same with Apache Camel. Camel is a bit more complex due to all the libraries that it may pull in depending on which components you use, but it’s not really that much more complex.

As with CXF, the easiest way to get it up and running would be to grab a distribution that has it ready to go. Again, I’d recommend either the Talend Integration Factory or Talend ESB. Both have Apache Camel, Apache CXF and Apache ActiveMQ pre setup to run and everything well configured. The more popular Camel components are pre-installed with their dependencies making it fairly easy to get started.

That said, setting up your own instance is really not that hard.

1. Follow steps 1-3 from my post about setting up Apache CXF in OSGi. This gets Karaf setup and ready.
2. From the Karaf command line, run:

   features:addurl mvn:org.apache.activemq/activemq-karaf/5.5.0/xml/features
   features:addurl mvn:org.apache.cxf.karaf/apache-cxf/2.5.0/xml/features
   features:addurl mvn:org.apache.camel.karaf/apache-camel/2.8.2/xml/features
   

   That will add the ActiveMQ, CXF, and Camel features into Karafs feature resolver.

3. If you plan on using JMS, it’s recommended to install the ActiveMQ features first. From the Karaf command line, run:

   features:install activemq
   features:install activemq-spring    (if you use spring)
   

4. Next up would be CXF if you plan on using the CXF components. Again, from the Karaf command line, run:

   features:install cxf
   

5. Finally, install the Camel components you are interested in using:

   features:install camel-core
   features:install camel-spring
   features:install camel-jms
   features:install camel-cxf
   etc...
   

   You can get a list of the Camel features by doing a “features:list | grep camel” from the Karaf command line.

That’s pretty much all there is to it. It’s not hard. It’s now all setup! To try it out, I’d once again grabbing the samples package from Talend Integration Factory. There are several examples in there on how to do various Camel things, all running in OSGi.

This morning, I sent out an announcement that Apache CXF 2.5.0 is now released! Over 6 months of work went into this release to expand the feature set, stabilize it, and get it ready for release.

There are two major categories of new “stuff” in this release:

1. Normal “framework” features. These features expand the capabilities of CXF as a Services Framework and are more in line with the traditional CXF enhancements. In 2.5.0, these include things like:
   • Enhanced WS-RM interopability
   • WS-MetadataExchange support
   • OAuth support
   • Enhanced security options for REST
   • Enhanced JMX instrumentation
   • Enhanced OSGi and Karaf integration
2. New “Services”. This is a brand new area for CXF. With 2.5.0, the scope of CXF now expands to include some enterprise level services that are related to CXF. At this point, they include:
   • Security Token Service
   • WS-Notification Service

I’m very excited about the second category of enhancements. Traditionally, CXF has always been “just a framework” that can be used to build your services. CXF never really provided any services “out of the box”. I’m hoping that by providing some enterprise level services out of the box, it will become even easier for people to develop and deploy CXF based applications in their enterprise. Both of the new services, while “new” for CXF, are really enterprise ready. The WS-Notification Service is a port (mostly done by Guillaume Nodet) of the ServiceMix WS-Notification service which has been deployed and used in production in many places. By removing the JBI dependencies and re-writing the communications to use pure JAX-WS API’s, the WS-Notification service is now much more usable by a wider audience. The STS was developed by Talend for some of our customers and has been deployed into production. Colm has done an amazing job with the STS to get it to meet a wide range of requirements. Definitely check out his blog for more information about the capabilities of the STS.

Anyway, this is another big release for Apache CXF. Definitely give it a spin and let us know how we’re doing!

I’ve had a bunch of people asking me lately about getting Apache CXF up and running in OSGi. A lot of people have run into issues trying to find the right third party bundles, configuring things like JAX-WS and JAXB, normal “class loader” issues, etc…. Thus, I decided I need to write a blog entry.

Obviously, the EASIEST way to get CXF up and running in OSGi is to grab a preconfigured OSGi container that has CXF already setup. There are a couple of good ones out there. For “pure CXF”, the best option is the Talend Service Factory. It’s pretty much Apache Karaf and Apache CXF completely setup to run in OSGi out of the box. Very simple. If you want/need a little more functionality, you could grab the Talend Integration Factory or the Talend ESB. Both of them contain Apache Karaf, Apache CXF, and Apache Camel pre-setup to run in OSGi. If you don’t mind an older version of CXF, you can also use the latest Apache ServiceMix 4.x distributions as they would have CXF ready to run as well. However, the current downloads are based on a fairly old version of CXF. They are working on a newer version that would contain a newer version of CXF, but that may not be available for a couple weeks. Also, ServiceMix contains a lot of other things that may or may not be useful for you. For the “lightest weight” option, the Talend Service Factory is the best option.

However, not everyone wants a pre-configured thing. They may want to customize things a bit. Or maybe add some of their own things. For those types of people here are some basic instructions that I would recommend:

1) Start with Apache Karaf. Karaf is an excellent general purpose OSGi container and provides a bunch of features to make life with OSGi much easier. Grab the latest (currently 2.2.4) version, download it, and unpack it.

2) In the unpacked Karaf area, edit the etc/jre.properties file. For CXF to work we need to use OSGi enabled bundles of various API’s. The versions of the API’s built into the JDK are not OSGi enabled and thus will not work properly with CXF. In the jdk-1.6 (and 1.7 if using Java7), you need to commend out the lines:

# javax.activation, \
# javax.annotation;version="1.1", \

# javax.jws, \
# javax.jws.soap, \

# javax.xml.bind, \
# javax.xml.bind.annotation, \
# javax.xml.bind.annotation.adapters, \
# javax.xml.bind.attachment, \
# javax.xml.bind.helpers, \
# javax.xml.bind.util, \

# javax.xml.soap, \
# javax.xml.stream, \
# javax.xml.stream.events, \
# javax.xml.stream.util, \

# javax.xml.ws, \
# javax.xml.ws.handler, \
# javax.xml.ws.handler.soap, \
# javax.xml.ws.http, \
# javax.xml.ws.soap, \
# javax.xml.ws.spi, \

All of those API’s use the SPI mechanism in the JDK to find implementations. That mechanism relies on Thread contextClassLoader mechanisms that don’t work reliably in OSGi. For CXF, we have specific bundled versions of those API’s that use a proper BundleActivator/Listener method to find the implementations.

I’d also recommend editing the etc/custom.properties file to add karaf.framework=equinox to set the underlying OSGi runtime to Equinox instead of Felix. Currently, CXF is a bit better tested with Equinox. Future versions of CXF will work better with Felix, but right now, the testing is more complete with Equinox.

3) Start the container by running the bin/karaf file.

4) From the Karaf command line, type:

features:addurl mvn:org.apache.cxf.karaf/apache-cxf/2.5.0/xml/features

That will install the CXF Karaf features file. After that, you can run “features:list” and see the features that CXF provides.

5) Run

features:install cxf

To install the CXF feature. Karaf will then download and install all the required dependencies, bundles, etc… and install them into the OSGi container. It will also start the bundles. If you run “list -t 0″ you should see “Apache CXF Bundle Jar” active at the bottom.

That’s really all there is to it. At that point, CXF is installed and ready to be used. Now, USING it is a subject for another blog post. For now, your best option is to grab the Talend Service Factory examples and take a look at them. All of them are ready to run in OSGi.

One of the important differentiators between Apache CXF and some of it’s competitors has been it’s support for standards, both “on the wire” standards like SOAP and API standards like JAX-WS and JAX-RS. When Apache resigned from the JCP last year, there was a lot of concern about whether CXF would be able to continue to be involved with the defining and implementing the standards. As an Apache member, I fully support the actions that Apache took as there really wasn’t a good alternative. However, it did leave a gap in some of the projects abilities to participate in the specifications.

I’m happy to announce that Talend has joined the JCP to allow some of our contributors like myself and Sergey to continue being involved and help shape the future of the standards that CXF implements. The immediate benefit is that Sergey can jump into the JAX-RS 2.0 discussions and help make sure that JAX-RS 2.0 can meet the needs of CXF users. The combination of the JAX-WS and JAX-RS programming models in CXF is a great combination of technologies and making sure the CXF users needs are well represented is important.

For a more complete discussion of Talends thoughts, see Fabrice’s blog entry on Talends site. From my standpoint, the most important piece to understand is that this shows Talend is not just committed to supporting CXF and the other Apache projects we are involved in, but also in investing in making sure the projects continue to innovate and move forward to solve users needs and problems. That is one of the unique points that makes Talend special.

On July 22, 2005, I did my first commit into the Apache subversion repository. That commit was the first part of bringing Apache CXF (was called CeltixFire at the time) to Apache.

This morning, according to svnsearch, I committed my 7000th change. That’s a lot of changes. Kind of silly milestone, but I’m proud of it. I’m certainly happy to have had jobs that allow me to spend a lot of time working on the Apache projects as part of my job.

Off to starting working on some more commits……

After 6 months of development, the Apache CXF team is proud to announce that 2.4.0 is now released. (official announcement) This version was a fairly large undertaking and has a bunch of significant new features.

• Security: WS-Security enhancements were a major theme for this release. It mostly started as work to get a new version of Apache WSS4J (version 1.6) integrated into CXF but really turned into a bunch of major updates to WSS4J and even larger enhancements to CXF. The primary “driving feature” was support for SAML2 tokens. In the end, we support SAML2 tokens, but also have much better configurability of the keystores, more consistent callback handling, much better token validation API’s and capabilities, the ability to “fail fast” in many cases when security issues are encountered, etc… I HIGHLY recommend reading through Colm’s blog to discover all the new features in WSS4J that were developered to support CXF.
• Performance and memory footprint: A significant amount of work was put into the startup procedures in CXF to reduce startup time and lower the memory footprint. As part of this, the dependency on Spring to wire together CXF has been greatly reduced. Now, almost all features of CXF can be used without Spring. Previously, many of the advanced features like WS-SecurityPolicy and WS-RM and such required Spring. In all, a particular test case I have that deploys over 200 services within over 75 wars can now deploy and run with MUCH less memory than before (required -Xmx drops from 192M to 96M and startup time in tomcat drops from about 50 seconds to 20). Note: with this change came the removal of almost all the META-INF/cxf/cxf-extension-*.xml files. MUCH easier to configure now as well.
• HTTP transport refactoring Related to the Spring stuff, the HTTP transport was refactored to remove the requirement of Spring to select “which” http transport. Previously, the user really need to select (by importing the appropriate spring file) either the servlet version, the embedded jetty version, or the OSGi version. There is now a single http transport that handles all three cases. You can even mix/match with some some services that deploy on the servlet provided context, but others that spin up an embedded jetty instance on a different port.
• JiBX databinding: a Google Summer of Code project from last year, CXF now supports using JiBX in addition to JAXB, XMLBeans, and SDO.
• Log browser: another Google Summer of Code project resulted in a very nice web/rest based frontend to the logs produced by CXF. It’s a starting point for future admin related tasks, but is quite useful right now.
• Transformation feature: provides for a fast and effective way to transform inbound and/or outbound XML messages. There is a nice introduction to this on Sergey’s blog.
• OSGi improvements: A lot of stuff was done to make working with CXF in OSGi much better. The OSGi HTTP transport was already mentioned. We also added an Apache Karaf features file to help setup CXF in Karaf a lot quicker and easier. Finally, work was started (it’s a work in progress) to add Blueprint/Aries namespace handlers to allow deployment of services using Blueprint instead of Spring DM. Currently, the basic creation of clients and services works in Blueprint. Some of the more advanced configuration (most notable is the HTTPs configuration) is still being worked on.
• Work with Apache WebServices community: along with working closely with WSS4J as mentioned above, CXF developers also worked to drive changes in Neethi and XmlSchema. The result is much better collaboration between CXF and the WebServices communities and many hacks and workaround in CXF being removed as the fixes and changes were made in the underlying libraries.

As you can see, there was definitely a lot of exciting work put into 2.4.0. I definitley encourage forlks to read the migration guide for more information about the changes and the potential impact it has on users.

Another interesting thing I’d like to point out is how much this was a COMMUNITY effort. The features above were developed by a bunch of people in the community, not just a single person or company. While the security and performance work was primarily done by Talend employees, we also had major contributions from the GSoC students (2 students became committers), Benson Margulies did a lot of work getting XmlSchema 2.0 released and integrated, much of the Blueprint work is being done by Johan Edstrom as part of his work with Savoir Technologies, etc…

Of equal interest is who was NOT a driving force for this release: IONA/Progress/Fuse. Since CXF was started at Apache way back in 2006, the IONA/Progress/Fuse developers have been a primary driving force for new features and development in CXF. 2.4.0 really marks a major change from that. It’s not to say they haven’t been contributing to CXF, they definitely have. However, most of their contributions are more in the form of bug fixes and minor tweaks, not major new feature development. In my opinion, this a good thing for CXF and shows the value in the way Apache values the diversity of the community. In many cases, open source is about “scratching your own itch.” Basically, if you need something done or fixed, you do it and submit it to the project. Over time, in a healthy community, many of the original itches get scratched and those people move on, but new people come in with new ideas and requirements and drive furthur work. The mark of a healthy community is how open they are to such changes and how well the community adapts. CXF has adapted quite well.

Anyway, I definitely encourage everyone to upgrade to CXF 2.4.0 and experience the new features. All of this “goodness” will soon be available in Talend Service Factory and the other Talend offerings. Keep an eye out for it.

Back in December, I talked about the release of the Talend Service Factory as being the first step in the new product plans coming out of Talend. The second step occured today with the release of the Talend Integration Factory. (Click here for the official press release if you are interested.)

Talend Integration Factory is very similar to Talend Service Factory in that it builds upon one of the most successful Apache projects, in this case Apache Camel, and enhances it to meet the needs of enterprise users. For Talend Integration Factory, the major enhancements to Apache Camel are:

• OSGi container – a “container” directory is added which is a full OSGi container, including Equinox and Karaf, preconfigured with all the dependencies and bundles required for the most popular use cases of Camel in OSGi.
• Camel component for Talend Open Studio which allows jobs created from TOS to be part of a Camel route

From an enterprise standpoint, the first item is very important. Having a preconfigured, supported OSGi container setup and optimized for Camel makes it a lot easier to get started with Camel.

The second item is interesting in that it’s the first piece that really starts bridging the the Talend Data technologies with the Application Integration technologies. At this point, it’s in a very early state, but it’s a starting point to build from.

Like the Talend Service Factory release, for Talend Integration Factory we again worked very hard on creating some new documented examples that demonstrate some of the advanced features of Apache Camel. The really cool thing is that several of the examples build off of the examples from Talend Service Factory, extending them to participate in Camel routes. This is very similar to how enterprises end up using Camel. They start off with existing services or have services being developed by a particular group. Those services then need to be integrated into various EIP style routes or used from other pieces of logic. The other nice thing about building up the examples like that is it really shows how well Apache CXF and Apache Camel work together seamlessly.

Anyway, I’m definitely excited about this release. As mentioned, it’s just “Step 2″ and we have a lot of other exciting ideas and products that will be coming out this year. Stay tuned!

A month ago, it was announced that Talend had acquired Sopera. At the time, many people wondered what that really meant as Talend specialized in Data Integration and Mangement technologies and Sopera was more of an Application Integration company. Over the last couple months, I’ve defintely been involved in a lot of meetings and con calls and such to start flushing out ideas and roadmaps and such.

The first step in plan occured today with the release of Talend Service Factory. TSF is primarily a repackaged distribution of Apache CXF that is optimized more for Enterprise level use. There are two major changes from the Apache distributions:

1. Java 6 optimized – Talend is only committing to supporting Java 6 so the lib directory has been optimized for Java 6, removing a bunch of things that are already provided by the JDK.
2. OSGi container – a “container” directory is added which is a full OSGi container, including Equinox and Karaf, preconfigured with everything needed to use CXF in OSGi.

I’m actually quite excited about the second item for a few reasons. For one, it actually made me start looking at and using OSGi. I guess I’m a little behind the curve on that. Second, it fills a void between the lightweight services and samples that Apache CXF provides and supports and a full, heavyweight ESB like ServiceMix. We’ve had quite a few people ask about OSGi containers and such on the CXF lists. I’ve always felt a bit guilty about pointing them at something as huge and complex as ServiceMix when they don’t need it or want it. You can download the release here.

The OTHER thing that we worked hard on is a new set of documented examples that show off some of the more advanced features of CXF, ALL of them designed to be run standalone (like all the Apache CXF samples) or in the OSGi container. We all learned a lot writing the examples. We had to fix some bugs in CXF to get some of them to work in the OSGi container, but that’s part of the process. I think Sergey, Glen, and I may talk more about these examples in the upcoming days. Until then, feel free to download them and give them a run through. We have more in the works as well.

The important thing to remember is that this is just a start. Step 1. We have a lot of stuff in progress for 2011 that I’m quite excited about.

I kind of feel bad about not blogging this earlier in the week, but it’s been one of those weeks…

On Monday, Glen Mazza officially started at Talend. For those of you that are not familiar with him, Glen has been a long time commiter to Apache CXF getting started with CXF while it was still in the incubator. On CXF, he’s probably best known for reviewing the code and changes and such and fixing all the typos he found and enhancing the error messages and javadoc and such. While not a strict “feature contribution”, these types of changes have been invaluable in making CXF more pleasant to use.

However, he is a software engineer so he’s also contributed to CXF in other areas. He’s done a bit of work in the WS-Security stuff, especially in the area of creating test cases and identifying deficiencies and bugs. He’s also done quite a bit of testing with CXF and other WS-Security implementations (mostly Metro) to find interopability issues and has worked to get them resolved.

That all said, he’s probalby BEST known for the examples and walk throughs that he has done on his blog: http://www.jroller.com/gmazza/ His examples on his blog have been a great starting point for MANY CXF users as they start using CXF. When people ask for a getting starting tutorial or sample, many people point them off to his blog. His blog also tends to collect links to other interesting web pages and blogs and such related to Web Services. Definitely a great place to look for information.

Anyway, I just wanted to say “Welcome Aboard!” to Glen and I definitely look forward to working with him to make CXF better and make Talends products better.