Benchmark testing tools for Web Services

These days I am working on some benchmark testing of Axis2/C. While looking for good tools for the purpose I came across several good candidates. I would like to discuss about some of these tools.

Tools that impressed me and disscussing here are

ab(apache workbench), siege, hammerhead and javabench and http_load. I also tried tsung and curl-loader but could not get it work for me. Here it again prove that however good a software be it does not make appeal to average user if it does not provide a good impression on first try. What most developers expects from a good software is to get them immediately familier with it with some intuitive and working samples. To me it should provide a good <toolname> -h help menu, easy and upto the point samples first. A comprehensive documentations should follow next. tsung seems to be a good performance testing tool that provide a good distributed testing enviroment.

Since my perticular interest is on testing web services one of my main requirement is supporting http POST method. http_load seems to be a good tool but  it does not support POST. So I am not discussing it further here.

apache bench

The tool most impressed me is ab(apache bench) that ships with apache2. So naturally it is the tool I selected for my purpose.

ab -k -T “application/soap+xml; charset=UTF-8″ -p data/data.xml -n 1000 -c 32 http://localhost:80/axis2/services/Benchmark > results/result.txt

Here -k indicates whether to use keep-alive feature of http 1.1. You can use -H to add any custom http header you want.

-T options is for giving the content type.  Using -p option you can give your data file which contain basically the soap envelope you need to send. Use ab -h to see the meaning of all options.

Following is a result I got after executing the tool

Concurrency Level:      10
Time taken for tests:   0.084 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Keep-Alive requests:    100
Total transferred:      97910 bytes
Total POSTed:           77550
HTML transferred:       76900 bytes
Requests per second:    1195.69 [#/sec] (mean)
Time per request:       8.363 [ms] (mean)
Time per request:       0.836 [ms] (mean, across all concurrent requests)
Transfer rate:          1143.26 [Kbytes/sec] received
905.52 kb/s sent
2048.78 kb/s total

siege

There is a configuration file called siegerc.

siege –rc=./siegerc -H”Content-type: application/soap+xml; charset=UTF-8″ -H”Connection: Keep-Alive”

Almost all ouf your configurations can be put into this configuration file.

Following are some of the options available there

verbose = false

logging = false

protocol = HTTP/1.1

connection = keep-alive

concurrent = 25 #Number of concurrent requests

reps = 1000 #Number of repetitions

file = ./urls.txt

delay = 0 #If you are using for benchmarking value should be 0

benchmark = true

Note the file options above. It indicate to use the urls.txt file in the current folder as the url file. An entry in the urls.txt is as follows.

This entry is for a http POST request

http://localhost:80/axis2/services/Benchmark POST < data/echoDoubles-10.xml

Here is a typical output from siege.

Transactions:               25000 hits
Availability:              100.00 %
Elapsed time:               20.18 secs
Data transferred:           18.33 MB
Response time:                0.01 secs
Transaction rate:         1238.85 trans/sec
Throughput:                0.91 MB/sec
Concurrency:               10.68
Successful transactions:       25000
Failed transactions:               0
Longest transaction:            0.49
Shortest transaction:            0.00

siege is very good tool but to me it seems to be somewhat less capable than apache bench in its sieging capabilities.

hammerhead

hammerhead uses a configuration file and a scenario folder to execute.

hammerhead –conffile=/home/damitha/workspace/apache2_performance/hammerhead/test.conf -s20 -oresults

This tells hammerhead to use test.conf configuration file and run for 20 seconds.

In the test.conf you have an entry like following

Scenario_Directory /home/damitha/workspace/apache2_performance/hammerhead/scn

scn scenario directory can contain any number of .scn files.

An example .scn file is

NSample POST
HContent-type: application/soap+xml; charset=UTF-8
HConnection: Keep-Alive
RPOST /axis2/services/Benchmark

B<soapenv:Envelope xmlns:soapenv=”http://www.w3.org/2003/05/soap-envelope”><soapenv:Header /><soapenv:Body><ns1:echoDoubles xmlns:ns1=”http://www.extreme.indiana.edu/wsdl/Benchmark1″><ns1:input>0.0</ns1:input><ns1:input>1.0</ns1:input></ns1:echoDoubles></soapenv:Body></soapenv:Envelope>
T0

You can think the entries of the .scn file as key value pairs. The first letter in the each line(in block letter) represent the key. rest of the line represent the value. For example B reperesent the http POST message.

Here is sample output file for hammerhead.

Config File                  : /home/damitha/workspace/apache2_performance/hammerhead/test.conf
Machine                      : 127.0.0.1:80
: 127.0.0.1:80
: 127.0.0.1:80
: 127.0.0.1:80
: 127.0.0.1:80
: 127.0.0.1:80
: 127.0.0.1:80
Time                         : Fri Jul 11 11:11:52 2008
SEED =                       : 1215754872
Parent Process PID =         : 12090
Total Run Time (sec)         : 20
Sessions                     : 2
Session Sleep Time (msec)    : 0
Startup Lag Time (seconds)   : 1
Failures                     : 0
NoVerify                     : 0
Total Requests Served        : 10
Total Turnaround Time (msec) : 49990
Average Request Time  (msec) : 4999
Total Responses              : 10
Average Responses / sec      : 0.250000
Total Response Time (msec)   : 6
Average Response Time (msec) : 0
Scenario Throughput          : 10
Sequence Throughput          : 12
Run Time (sec)               : 40
Scenarios / sec              : 0.250000
Sequences / sec              : 0.300000
Content Length bytes         : 7690
Content Length bytes / sec   : 192
Read Length bytes            : 7690
Read Length bytes / sec      : 192

javabench

javabench is basically apache bench written in java. You can use it in the same way you use apache bench. However it is not as capable sending many requests per second.

You can download a complementary upload of javabench  from

http://wso2.org/files/java-bench-0.1.zip

After downloading you can run the tool by using the following script(modify it according to requirement)

Here is the script

for f  in ./*.jar
do
JAVABENCH_CLASSPATH=$JAVABENCH_CLASSPATH:$f
done

export JAVABENCH_CLASSPATH

java -classpath $JAVABENCH_CLASSPATH org.apache.http.contrib.benchmark.HttpBenchmark -k -T “application/soap+xml; charset=UTF-8″ -p data/data.xml -n 1000 -c 32 http://localhost:80/axis2/services/Benchmark > results/result.txt

---

Possibly related posts: (automatically generated)

• Five things I did not do this weekend
• Office Watch: Road Trip!
• Obamamania is Passed – And That’s a Good Thing