Axis2/C benchmark performance test results are published here
Archive for July, 2008
I mentioned that Axis2/C version 1.5 reached 25K tps performance margin running on Apache2. I hope Axis2/C running as a lighttpd module will further extend this margin. Still there is some work to be done before it could be brought to production use. But the important thing is the module is working now.
Few words about Dinesh’s effort on the work. He started on this module two weeks before he left WSO2 for his higher studies in USA(And he married last month !!). In the middle of all his preparation for new life he had time and will to bring this work to a working state. At this time of his parting I would like to remind his great contributions to Axis2/C project. He workied on lot of ares’s in Axis2/C code. He initiated Guththila Stax parser, XMPP transport and contributed to wsclient command line tool to name a few. I think he is the one so far to be the release manager for most number of Axis2/C releases. Kudos Dinesh for your great work.
I would like to talk about our programming model when it comes to Axis2/C Apache2 module. As discussed in a earlier blog suppose two requests come to Axis2/C sequentially. Say the first request is served by one apache process and the next request is served by another process. Then problem is you cannot access the previous requests configuration context from your next request because each process has it’s own configuration context. We recently solved this problem using shared memory. But as should be expected this is extremely inefficient.
The other and more effiecient solution is using a database that could be shared among your processes. Sandesha2/C uses this approach. To solve concurrency issues when using this approach we can use Apache2 golbal mutex which could be used when accessing global resources like system files and databases. But the problem is how we could pass this global mutext from Axis2 apache module to be accessible from other parts of the code, for example from a Axis2/C module. We currently pass Apache2 pools through axis2_env_t environment. My understanding is that we could use a similar method to solve the problem.
According to my performance testing on Axis2/C 1.5(Soon to be released) I could reach 25K tps with keep alive on. When tested with a empty module instead of Axis2/C where no handling of soap data is done but just set a HTTP_OK and return it, reached 40Ktps mark(This is also similar to index page request as might be expected). This means there is still room for 15K improvement. Profiling tools show that mostly the improvements needs on parser.
Also when I experiments with using a localpool instead of the request pool of apache the performance degraded by about 1000 tps. This further shows that there could be further performance gains that could be achived perhaps by improving apache2 module. Has anybody thought of implementing Axis2/C module as a apache2 filter or using filters instead as an apache2 content generator handler?. In apache2 documents it says that
‘Any form processing application is normally be implemeneted as a handler-perticulary those that accept POST data.’
However in the same documentation it says that markup processing libraries better be implemented as filters or using filters. It also says that parsers like expat and libxml2 which have parseChunk APIs work well with apache2 filters. Working with filters give direct access to bucket brigades which gives the control into the filter developer for data manipulations.
I heard somebody is developing a expat parser plugging for Axis2/C. It is a good thing to do a performance comparision with that also.
These directions are for future considerations only. Axis2/C is already ready for performance hungry deployments, may be ahead of other competing web services engines. I have actually plans for doing some performance benchmarks using Axis2/C as a soap gateway in an EC2 cloud.
Now that I have done some performance tests(Results will be published soon. In summary Axis2/C 1.5 which will be released within next week achieved 25K tps benchmark) I thought of using a profiler to see which part of codebase need further attention for improvement. I profiled with google proftools and valgrind profile tools.
proftools has a cpuprofiler and a heap profiler. You need to just download it and ./configure, make, make install.
where first one is where cpu profiler output file is created and the second one is where heap profiler output file is created. These files will be created when you run your program.
You need to build Axis2/C with
Just add those linker options into configure.ac CFLAGS.
Now you can your executable as normal. After that
For cpu profiling
./pprof –gv axis2_http_server /tmp/axis2_http_server.prof.
This will give you a nice graph. You must have installed dot, gv and perl5 installed in your system.
If you have installed kcachegrind in your system you can use it to have nice graphical view by
pprof –callgrind ./axis2_http_server /tmp/axis2_http_server.prof > axis2_http_server.callgrind
For heap profiling
–callgrind ./axis2_http_server /tmp/axis2_http_server.hprof.0001.heap > axis2_http_server.callgrind
again you can use kcachegrind to have a nice gui view.
Note:I found that proftools are only good for profileing Axis2/C client side because I could not find a way to capture profiling for request handling in server side. If you need to just profile deployment in Axis2/C then this can be used.
You can separetely install callgrind. But if you install valgring in dabian uisng apt-get install you can use it by using
valgrind –tool=callgrind < your program name>
I used callgrind to profile simple axis2 server as follows
valgrind –tool=callgrind ./axis2_http_server
Now make a request to the server.
Now you have a file generated in your current folder called something like callgrind.out.8307.
Use this as input to kcachegrind
Important: I did the following addition in my src/core/transport/http/receiver/http_svr_thread.c code towards the end of server while loop to exit the server after serving two requests. Instead if I use ctrl-c to signal the server to stop then profiling tool won’t be able to capture the requests for profiling.
if(counter == 1)
svr_thread->stopped = AXIS2_TRUE;
}// end while loop
My conclusion is that both of these tools are good for profiling in a threaded environment. Problem with gprof is that it could not be used for profiling in a threaded environment. Although there is a hack to do this I saw a user say that it is not trustworthy.
One of the best way to get you immediately get aware of the concepts
My wife was asking me for some time to install linux on her machine. I thought of this. Well it take some time, not to install ubuntu. But to answer her non-stop questions on how to do this, how to do that. Last week. Behold she has installed ubuntu on her machine herself. She says, “best replacement for windows”.
These days I am working on some benchmark testing of Axis2/C. While looking for good tools for the purpose I came across several good candidates. I would like to discuss about some of these tools.
Tools that impressed me and disscussing here are
ab(apache workbench), siege, hammerhead and javabench and http_load. I also tried tsung and curl-loader but could not get it work for me. Here it again prove that however good a software be it does not make appeal to average user if it does not provide a good impression on first try. What most developers expects from a good software is to get them immediately familier with it with some intuitive and working samples. To me it should provide a good <toolname> -h help menu, easy and upto the point samples first. A comprehensive documentations should follow next. tsung seems to be a good performance testing tool that provide a good distributed testing enviroment.
Since my perticular interest is on testing web services one of my main requirement is supporting http POST method. http_load seems to be a good tool but it does not support POST. So I am not discussing it further here.
The tool most impressed me is ab(apache bench) that ships with apache2. So naturally it is the tool I selected for my purpose.
ab -k -T “application/soap+xml; charset=UTF-8″ -p data/data.xml -n 1000 -c 32 http://localhost:80/axis2/services/Benchmark > results/result.txt
Here -k indicates whether to use keep-alive feature of http 1.1. You can use -H to add any custom http header you want.
-T options is for giving the content type. Using -p option you can give your data file which contain basically the soap envelope you need to send. Use ab -h to see the meaning of all options.
Following is a result I got after executing the tool
Concurrency Level: 10
Time taken for tests: 0.084 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100
Total transferred: 97910 bytes
Total POSTed: 77550
HTML transferred: 76900 bytes
Requests per second: 1195.69 [#/sec] (mean)
Time per request: 8.363 [ms] (mean)
Time per request: 0.836 [ms] (mean, across all concurrent requests)
Transfer rate: 1143.26 [Kbytes/sec] received
905.52 kb/s sent
2048.78 kb/s total
There is a configuration file called siegerc.
siege –rc=./siegerc -H”Content-type: application/soap+xml; charset=UTF-8″ -H”Connection: Keep-Alive”
Almost all ouf your configurations can be put into this configuration file.
Following are some of the options available there
verbose = false
logging = false
protocol = HTTP/1.1
connection = keep-alive
concurrent = 25 #Number of concurrent requests
reps = 1000 #Number of repetitions
file = ./urls.txt
delay = 0 #If you are using for benchmarking value should be 0
benchmark = true
Note the file options above. It indicate to use the urls.txt file in the current folder as the url file. An entry in the urls.txt is as follows.
This entry is for a http POST request
http://localhost:80/axis2/services/Benchmark POST < data/echoDoubles-10.xml
Here is a typical output from siege.
Transactions: 25000 hits
Availability: 100.00 %
Elapsed time: 20.18 secs
Data transferred: 18.33 MB
Response time: 0.01 secs
Transaction rate: 1238.85 trans/sec
Throughput: 0.91 MB/sec
Successful transactions: 25000
Failed transactions: 0
Longest transaction: 0.49
Shortest transaction: 0.00
siege is very good tool but to me it seems to be somewhat less capable than apache bench in its sieging capabilities.
hammerhead uses a configuration file and a scenario folder to execute.
hammerhead –conffile=/home/damitha/workspace/apache2_performance/hammerhead/test.conf -s20 -oresults
This tells hammerhead to use test.conf configuration file and run for 20 seconds.
In the test.conf you have an entry like following
scn scenario directory can contain any number of .scn files.
An example .scn file is
HContent-type: application/soap+xml; charset=UTF-8
B<soapenv:Envelope xmlns:soapenv=”http://www.w3.org/2003/05/soap-envelope”><soapenv:Header /><soapenv:Body><ns1:echoDoubles xmlns:ns1=”http://www.extreme.indiana.edu/wsdl/Benchmark1″><ns1:input>0.0</ns1:input><ns1:input>1.0</ns1:input></ns1:echoDoubles></soapenv:Body></soapenv:Envelope>
You can think the entries of the .scn file as key value pairs. The first letter in the each line(in block letter) represent the key. rest of the line represent the value. For example B reperesent the http POST message.
Here is sample output file for hammerhead.
Config File : /home/damitha/workspace/apache2_performance/hammerhead/test.conf
Machine : 127.0.0.1:80
Time : Fri Jul 11 11:11:52 2008
SEED = : 1215754872
Parent Process PID = : 12090
Total Run Time (sec) : 20
Sessions : 2
Session Sleep Time (msec) : 0
Startup Lag Time (seconds) : 1
Failures : 0
NoVerify : 0
Total Requests Served : 10
Total Turnaround Time (msec) : 49990
Average Request Time (msec) : 4999
Total Responses : 10
Average Responses / sec : 0.250000
Total Response Time (msec) : 6
Average Response Time (msec) : 0
Scenario Throughput : 10
Sequence Throughput : 12
Run Time (sec) : 40
Scenarios / sec : 0.250000
Sequences / sec : 0.300000
Content Length bytes : 7690
Content Length bytes / sec : 192
Read Length bytes : 7690
Read Length bytes / sec : 192
javabench is basically apache bench written in java. You can use it in the same way you use apache bench. However it is not as capable sending many requests per second.
You can download a complementary upload of javabench from
After downloading you can run the tool by using the following script(modify it according to requirement)
Here is the script
for f in ./*.jar
java -classpath $JAVABENCH_CLASSPATH org.apache.http.contrib.benchmark.HttpBenchmark -k -T “application/soap+xml; charset=UTF-8″ -p data/data.xml -n 1000 -c 32 http://localhost:80/axis2/services/Benchmark > results/result.txt