These days I am working on some benchmark testing of Axis2/C. While looking for good tools for the purpose I came across several good candidates. I would like to discuss about some of these tools.
Tools that impressed me and disscussing here are
ab(apache workbench), siege, hammerhead and javabench and http_load. I also tried tsung and curl-loader but could not get it work for me. Here it again prove that however good a software be it does not make appeal to average user if it does not provide a good impression on first try. What most developers expects from a good software is to get them immediately familier with it with some intuitive and working samples. To me it should provide a good <toolname> -h help menu, easy and upto the point samples first. A comprehensive documentations should follow next. tsung seems to be a good performance testing tool that provide a good distributed testing enviroment.
Since my perticular interest is on testing web services one of my main requirement is supporting http POST method. http_load seems to be a good tool but it does not support POST. So I am not discussing it further here.
The tool most impressed me is ab(apache bench) that ships with apache2. So naturally it is the tool I selected for my purpose.
ab -k -T “application/soap+xml; charset=UTF-8″ -p data/data.xml -n 1000 -c 32 http://localhost:80/axis2/services/Benchmark > results/result.txt
Here -k indicates whether to use keep-alive feature of http 1.1. You can use -H to add any custom http header you want.
-T options is for giving the content type. Using -p option you can give your data file which contain basically the soap envelope you need to send. Use ab -h to see the meaning of all options.
Following is a result I got after executing the tool
Concurrency Level: 10
Time taken for tests: 0.084 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100
Total transferred: 97910 bytes
Total POSTed: 77550
HTML transferred: 76900 bytes
Requests per second: 1195.69 [#/sec] (mean)
Time per request: 8.363 [ms] (mean)
Time per request: 0.836 [ms] (mean, across all concurrent requests)
Transfer rate: 1143.26 [Kbytes/sec] received
905.52 kb/s sent
2048.78 kb/s total
There is a configuration file called siegerc.
siege –rc=./siegerc -H”Content-type: application/soap+xml; charset=UTF-8″ -H”Connection: Keep-Alive”
Almost all ouf your configurations can be put into this configuration file.
Following are some of the options available there
verbose = false
logging = false
protocol = HTTP/1.1
connection = keep-alive
concurrent = 25 #Number of concurrent requests
reps = 1000 #Number of repetitions
file = ./urls.txt
delay = 0 #If you are using for benchmarking value should be 0
benchmark = true
Note the file options above. It indicate to use the urls.txt file in the current folder as the url file. An entry in the urls.txt is as follows.
This entry is for a http POST request
http://localhost:80/axis2/services/Benchmark POST < data/echoDoubles-10.xml
Here is a typical output from siege.
Transactions: 25000 hits
Availability: 100.00 %
Elapsed time: 20.18 secs
Data transferred: 18.33 MB
Response time: 0.01 secs
Transaction rate: 1238.85 trans/sec
Throughput: 0.91 MB/sec
Successful transactions: 25000
Failed transactions: 0
Longest transaction: 0.49
Shortest transaction: 0.00
siege is very good tool but to me it seems to be somewhat less capable than apache bench in its sieging capabilities.
hammerhead uses a configuration file and a scenario folder to execute.
hammerhead –conffile=/home/damitha/workspace/apache2_performance/hammerhead/test.conf -s20 -oresults
This tells hammerhead to use test.conf configuration file and run for 20 seconds.
In the test.conf you have an entry like following
scn scenario directory can contain any number of .scn files.
An example .scn file is
HContent-type: application/soap+xml; charset=UTF-8
B<soapenv:Envelope xmlns:soapenv=”http://www.w3.org/2003/05/soap-envelope”><soapenv:Header /><soapenv:Body><ns1:echoDoubles xmlns:ns1=”http://www.extreme.indiana.edu/wsdl/Benchmark1″><ns1:input>0.0</ns1:input><ns1:input>1.0</ns1:input></ns1:echoDoubles></soapenv:Body></soapenv:Envelope>
You can think the entries of the .scn file as key value pairs. The first letter in the each line(in block letter) represent the key. rest of the line represent the value. For example B reperesent the http POST message.
Here is sample output file for hammerhead.
Config File : /home/damitha/workspace/apache2_performance/hammerhead/test.conf
Machine : 127.0.0.1:80
Time : Fri Jul 11 11:11:52 2008
SEED = : 1215754872
Parent Process PID = : 12090
Total Run Time (sec) : 20
Sessions : 2
Session Sleep Time (msec) : 0
Startup Lag Time (seconds) : 1
Failures : 0
NoVerify : 0
Total Requests Served : 10
Total Turnaround Time (msec) : 49990
Average Request Time (msec) : 4999
Total Responses : 10
Average Responses / sec : 0.250000
Total Response Time (msec) : 6
Average Response Time (msec) : 0
Scenario Throughput : 10
Sequence Throughput : 12
Run Time (sec) : 40
Scenarios / sec : 0.250000
Sequences / sec : 0.300000
Content Length bytes : 7690
Content Length bytes / sec : 192
Read Length bytes : 7690
Read Length bytes / sec : 192
javabench is basically apache bench written in java. You can use it in the same way you use apache bench. However it is not as capable sending many requests per second.
You can download a complementary upload of javabench from
After downloading you can run the tool by using the following script(modify it according to requirement)
Here is the script
for f in ./*.jar
java -classpath $JAVABENCH_CLASSPATH org.apache.http.contrib.benchmark.HttpBenchmark -k -T “application/soap+xml; charset=UTF-8″ -p data/data.xml -n 1000 -c 32 http://localhost:80/axis2/services/Benchmark > results/result.txt
Read Full Post »