Archive for December, 2010

WSO2 ESB Performance

I uploaded  an  article [1] with the latest performance numbers of WSO2 Enterprise Service Bus. It is quite a long time since WSO2 last published it’s ESB performance numbers. During that time it released performance numbers on customer request on various instances.

WSO2 ESB has undergone various feature additions during that time. It is now based on WSO2 Carbon (with ESB 2.x family of products). WSO2 Carbon[2] provides the development framework and the runtime environment for the WSO2 products including the ESB. I am not going to talk all the new features here. WSO2 Oxygen Tank library is full of articles/tutorial written on WSO2 ESB. So decided it is time again to publish some performance numbers.

In Previous articles the performance was compared with different open source and commercial competitors.That is a time WSO2 was desperately searching for a niche in the middleware market. Now WSO2 ESB has established it’s ground firmly in the middleware ESB market, it has changed it’s approach on publishing performance numbers. No comparison with other vendors. We just publish our numbers and provide a performance test framework freely for our customers(or any interested user) to download and verify the results and, also do perform their own comparison with the products of their choice. This performance testing framework will be available soon on WSO2 Oxygen Tank.

This article has taken a different approach from previous performance test articles of WSO2 ESB in various aspects. I would like to discuss some of them here

First, the way performance presented is different. To quote from the article

“For each scenario each message size load is generated with concurrency varying from 20 to 300(increasing by 40 at each stage). Then the maximum transactions per second(TPS) achieved during this concurrency range is recorded as the Transactions per second(TPS) for the corresponding scenarios message size.”

What does that mean?.  From the experience of the previous performance tests I could see that  WSO2 ESB performs best at the concurrency range from 20 to 300. So I decided that the best performance number for a particular message size within that range would be the best choice to present to the reader. My original plan was to break several ranges like

20-300, 300-900 and 900-1800. But that could overly make the results complex. So I omitted the higher concurrency results from the article to avoid unnecessary cluttering.

Besides I wanted to capture results for a wider message size range. I thought that the concurrency range mentioned in the article best suited for that purpose.

Also my approach this time is to present the numbers in best possible way, so that one could easily grasp the results in easy to read graphs. When you compare the graphs from round3 this could be easily understood.

Note that I have done the performance test without keep alive enabled. HTTP keep-alive is great to improve performance. But it is not the best way to say that your product has good performance. My understanding
is that, to emulate how the server react to real world scenario of concurrent users, the best way is to assume that each tcp connection represent a user. So my decision was to ignore how many requests each user do on a particular connection.

In the article I have provided enough information to reproduce the performance test scenarios. . I have not done any special tweaks to gain extra bit of performance, other than the things I have mentioned in the article. Providing the exact scripts, messages in files etc for somebody to immediately reproduce the results is a bonus. Yet WSO2 have planned a something more than a bonus. It will soon upload into Oxygen tank a great open source performance test framework which I sincerely hope will serve as a good platform for benchmark tests for many open source projects. It is a fun to reproduce the performance numbers provided in the article  using that tool.

[1] http://wso2.org/library/articles/2010/08/wso2-esb-performancenew

[2] http://wso2.com/products/carbon/

Read Full Post »