Feeds:
Posts
Comments

Archive for the ‘Software that impressed me’ Category

Try Eclim

I have started using eclim as my Java, C and C++ development environment.

eclim is a project aimed for vim developers to incorporate the power of Eclipse into vim editor.

eclim provide host of features for java development that can be used from within your vim environment.
Some of them are

– Context sensitive code completion
– Code correction suggestions with option to apply a suggestion
– Java source and java doc searching capabilities.

Above features together with the most of the essential features one would expect from a java development environment make eclim a strong development environment for vim users.

The only catch is that you need to start a daemon called eclimd in order to use eclipse features from within vim. But that’s OK with me.

There is another way of using eclim as a vi editor from within Eclipose IDE. However I don’t like to play with Monsters like Eclipse.

My way of doing things is using the most familiar tool for me to get a job done. In that respect Unix pipe concept, which emphasize “use the best tool for the task at hand and if need to do more things plumb it with other tools. One tool do just the thing it is intended, nothing more, nothing less”.

See here for a set of eclim features. This getting started guide is the only document I followed to get my eclim environment running.

Read Full Post »

In some of my previous blogs [1], [2], [3] I mentioned about some of the open source performance testing tools I have been using. Among them were httperf, Autobench, Apache Bench and Java Bench.

I used these tools heavily to performance test WSO2 open source products. While doing so I understood that the performance test tool, usually the load generator is not alone adequate to fulfill my performance test requirements. Load generation is only a part of my requirement. The bigger picture required a systematic approach to performance testing which, none of these tools alone could fulfill. So what are my requirements?

– Provide easy to use standard configuration system so that tests could be configured easily.

– A standard way to provide my input for the tests

– A standard way to collect output from my tests.

– Way to collect system statistics while running the tests

– Automate the whole process. Just provide input data, required configuration and run the test. That’s it. Then collect the results

– A standard way to organize tests and the results

Commercially avaiable performance testing tools provide some or all of these requirements but for a very high cost. The name of the framework I implemented with Eranda to fulfill above requirements, is Ravana. Ravana will be released during next couple of weeks licensed with GPL. Ravana will be released with helpful documentation that will get you familiar with Ravana immediately.

Using Ravana is simple. All your input, configuration, output for a particular test scenario will come  under one test scenario you define. You just need to define your input, configuration under that scenario and then run the test.

With Ravana you can use one of httperf, java bench or apache bench as the load generator. It also support a JMS load generator.

If you are too curious and need immediately try  Ravana, check it out from https://svn.wso2.org/repos/wso2/branches/commons/performance-tools/ravana2. The code in svn is pretty stable now.

[1] https://damithakumarage.wordpress.com/2009/03/15/benchmark-testing-with-httperf/

[2] https://damithakumarage.wordpress.com/2008/07/20/profiling-tools-that-i-used-to-profile-axis2c/

[3] https://damithakumarage.wordpress.com/2008/07/11/benchmark-testing-tools-for-web-services/

Read Full Post »

It is a long time I’m back into Mathematics. Time has changed a lot. There are abundance of resources available for students of Mathematics out there in Internet which is almost unthinkable before a decade. Life is so easy. No need to run for the Library just to get familier with that Definition or Theorem.
I remember loading dozens of diskettes just to run Mathematica or Maple and with absolutely no resources to know how to get that thing or this thing done.
I’m greatly indebted to open source comunity to see a vast amount of software and learning resources available today just in a click away.
Following are some software I found very useful and at the same time seems very advanced. I’m yet to explore them more but the first impression dive me into writing this.

R – A statistical package that helps you do all that satistics computing and graphics. apt-get install r-base would do all the installing. It is said that this can do anything provided by it’s commercial counterpart S-Plus.

maxima – A CAS(Computer Algebra System) tool which has great support for numerical and symbolic representations. It has several GUI frontend tools like wxmaxima and mascyma. apt-get install maxima will do all the installation. The closest proprietary counterpart is Maple.

scilab – This is also a great numerical tool for doing any complex computation. The closest proprietary software is Matlab.

There are so many more great tools like Octave, Euler, Sage, Freemat etc etc.
I particularly found that GNU Octave scripts are compatible with Matlab for most of my purposes. Where it is not compatible I could find workarounds pointed by other users.

I also came across a great OpenOffice spreadsheet macro plugin called OOStat which is a very useful basic statistical package.
More on these tools as I experiment with these

Read Full Post »

I needed a good tool to easily map the results from my performance tests into human readable graphs. After much R&D I finally settled with the tool called gnuplot. Earlier I used openoffice chart tool for drawing simple charts. But it has limited capabilities and not so easy to customize. With gnuplot I could easily draw my histgrams just by giving all data in a file and giving all commands in a separate file. Following I explain how.
First install gnuplot on your working machine. In ubuntu

apt-get install gnuplot

Then I created my command file called data.p

# set terminal png transparent nocrop enhanced font arial 8 size 420,320
set output 'mygraph.png'
set xlabel "Message Type"
set ylabel "Requests per second"
set boxwidth 0.9 absolute
set style fill solid 1.00 border -1
set style histogram clustered gap 1 title offset character 0, 0, 0
set datafile missing '-'
set style data histograms
set xtics border in scale 1,0.5 nomirror rotate by -45 offset character 0, 0, 0
set xtics ("echoDoubles" 0.00000, "echoInts" 1.00000, "echoMeshInterfaceObjects" 2.00000, "echoSimpleEvents" 3.00000, "echoStrings" 4.00000)
set title "Large Data Sets(500 elements)\nResults from Apache Bench"
set yrange [ 0.00000 : 700. ] noreverse nowriteback
plot 'mydata.dat' using 2:xtic(1) ti col, '' u 3 ti col

It easy to understand the commands by following gnuplot tutorial

Then I created my data file called mydata.dat

# Large Data Sets(500 elements)
#
Region Axis2/C Waspc
echoDoubles 340.07 273.04
echoInts 452.17 318.69
echoMeshInterfaceObjects 162.90 79.40
echoSimpleEvents 101.16 68.78
echoStrings 542.96 270.54

Note that commented lines are interpreted as comments by gnuplot. Data columns are separated by tabs or spaces but not by colons or any other delimiter.
Now execute command gnuplot and go into the gnuplot command line interface. Now from within gnuplot execute

gnuplot> load ‘./mydata.p’

which will draw the following histogram

mygraph

This approach is very easy because each time I update my performance test results I just need to change the data in my .dat file and after that just execute gnuplot to redraw the graphs.

Read Full Post »

In a previous blog I discussed benchmark testing tools I used for some benchmark perposes. At that time I really used Apache Bench(ab) for the purpose. However when I am again doing some bench mark testing I stumbled upon a new tool(at least for me) httperf which seems to be more advanced and reliable than ab. Although there were ample guidance on how to use this for http GET requests I could not find any easy to read guide on how to use httperf with http POST requests. Following I try to explain how httperf could be used in testing my services.

httperf --server localhost --port 9090 --uri /axis2/services/weather
     --num-conn 4000 --num-call 16 --rate 64 --timeout 5 > report

Above command is a simple scenario where I request a HTTP get request on a servic running on context /axis2/services/weather on localhost.

It is important to understand the  options for the command.

1. –server – ip of the machine service is running

2. –port – port the service is running

3. –uri – The context path of the service on the server

4. –num-con – Number of test calls made to the service

5. –num-call – Number of calls per tcp connection.

6. –rate – Number of connections created per second to make requests to the service.

7. –time-out – This is the maximum time httperf wait for a successful response.

It is important to understand the correct balance between num-conn, num-call and rate options.

The maximum number of requests the httperf would generate = num-call * rate

Clearly your num-conn could be reasonably higher than this number. You could experiment between num-call and rate to produce a good load to the server. The actual number of requests to the server you will see in the generated report could be less than this number depending on the saturation of the server, client or network bandwidth. If such thing happen you need to reduce your num-call or rate options and find out at what point exactly saturation happen. If your num-conn is very large your server may exaust all servicing processes after some time.

Now let’s look at a more advance scenario. Here I am sending a http post request with some payload defined in the file name inputfile.

httperf --server localhost --port 80 --uri /axis2/services/weather --hog --method POST
    --add-header="Content-type:application/soap+xml;charset=UTF-8``\n``"
    --wsesslog=4000,0,inputfile --max-piped-calls 16 --rate 64 --timeout 5  > result

In adition to the payload you can also specify the http method and the server context uri in this input file. For more descriptions on writing this input file read [1]. Following is the content of the inputfile I used.

/axis2/services/weather method=POST contents="<soapenv:Envelope xmlns:soapenv=
\"http://www.w3.org/2003/05/soap-envelope\"><soapenv:Header xmlns:wsa=
\"http://www.w3.org/2005/08/addressing\"><wse:Identifier xmlns:wse=
\"http://schemas.xmlsoap.org/ws/2004/08/eventing\">
urn:uuid:57becf58-0fc3-1de1-24ad-0012f0b857ac</wse:Identifier>
<wsa:To>http://localhost:9090/axis2/services/weather</wsa:To>
<wsa:Action>http://schemas.xmlsoap.org/ws/2004/08/eventing/GetStatus</wsa:Action>
<wsa:MessageID>urn:uuid:14d256a2-0f95-1de1-2b1a-0012f0b857ac</wsa:MessageID>
</soapenv:Header><soapenv:Body><wse:GetStatus xmlns:wse=
\"http://schemas.xmlsoap.org/ws/2004/08/eventing\"></wse:GetStatus>
</soapenv:Body></soapenv:Envelope>"

The important additional options of the above command is
1.–wsesslog – This value for this option is a three comma separated parameters. First one is the number of request sessions you need to make to the server. Second is used to emulate a real user delay between sessions. Third is the inputfile.

2. –max-piped-calls – Number of calls need in one tcp connection(in http 1.1). This is used instead of num-call here.

[1] http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/node6.html

Read Full Post »

Anjuta is now wonderful. Those of you seeking a modern C/C++ ide environment Anjuta could be the best choice. Although vi/gdb/valgrind trio is still  my favourite development environment, now and then I look at ide’s whether they are matured enough for my taste. About 3 years back when I had a look at Anjuta I thought that it is promising but it still did not invited me to adopt. At that time I tried with Kdeveloper and Eclipse cdt plugin environment as well which were lagging far behind as robust ide environments at that time.

Today I tried to build Axis2/C in cdt environment. It seemed improved and just as I wanted to try it on my machine but it crashed while doing very simple task. That’s end of cdt with me at least for some time to come.

But this marvelous Anjuta. It is unbelievable how beautiful and matured she has become. It has whole set of features that I could not imagine three years back. If you are really interested, do some playing with her.

Using it for Axis2/C is very simple.

You can build anjuta from apt repositories in dabian environments. Make sure that you also build autogen project. Then after starting Anjuta just create a new project for Axis2/C. What you need to do is, in new menu select ‘Project from existing sources’ sub menu. That’s it. Rest is very intuitive.

But I found that still Unjuta is lacking with good context sensitive menu popup.

Read Full Post »

For Axis2/C developers in linux platform valgrind is a indispensable tool. It helps you debug and profile your services and modules.

The gdb/valgrind is very powerful combination for debugging your applications in linux.

Before you use these tools make sure that you have built your applications and Axis2/C with debugging enabled. This is a simple check on seeing whether the configure.ac in Axis2/C root source folder has -g entry in CFLAGS as in

CFLAGS=”$CFLAGS -ansi -Wall -Werror -Wno-implicit-function-declaration -g -D_GNU_SOURCE”

You can debug Axis2/C clients by

valgrind ./echo 2> result

This will show any context errors associated with your client code. Context errors simply says that your code is broken at the specified places. If you need to further investigate these points you can use gdb with the specified source files line numbers. If you find zero context errors in your code then you are lucky. Then you can further investigate by

valgrind –leak-check=full ./echo 2> result

which show you all the memory leaks you have in your code. However this will hide the repetitions. To see them also you can use.

valgrind –leak-check=full –leak-resolution=high 2> result

You can debug server side similarly.

valgrind –leak-check=full ./axis2_http_server 2> result

or if you want to debug with Axis2/C apache module

valgrind –leak-check=full ./httpd -X 2> result

You can use result file to investigate on context errors and leaks as in the client side.

However there is a caveat when it comes debugging Axis2/C modules and Axis2/C services. valgrind could not show leaks in modules and services. I suspect this is because of dll loading libraries. The solution is to build your service/module statically with Axis2/C engine. Currently Axis2/C does not provide any facility to do this(I suggest this for future releases). So we need to do a small hack to get this done. I will explain here how I did the hack in order to debug Sandesha2/C module.

Say you need to debug your in flow path. I created the in handler header as following and put it in axis2c/src/core/engine folder

#ifndef SANDESHA2_IN_HANDLER_H
#define SANDESHA2_IN_HANDLER_H

/**
* @file sandesha2_sender.h
* @brief Sandesha Sender Interface
*/

#include <axutil_allocator.h>
#include <axutil_env.h>
#include <axutil_error.h>
#include <axutil_string.h>
#include <axutil_utils.h>
#include <axis2_conf_ctx.h>

#ifdef __cplusplus
extern “C”
{
#endif

axis2_status_t AXIS2_CALL
sandesha2_in_handler_invoke(
struct axis2_handler *handler,
const axutil_env_t *env,
struct axis2_msg_ctx *msg_ctx);

/** @} */
#ifdef __cplusplus
}
#endif
#endif /* SANDESHA2_IN_HANDLER_H */

Now go and include that file in phase.c file. Also add the following code into phase_invoke function

if(!axutil_strcmp(handler_name, “SandeshaGlobalInHandler”) || !axutil_strcmp(
handler_name, “SandeshaInHandler”))
{
AXIS2_LOG_DEBUG(env->log, AXIS2_LOG_SI,
“dam_handler_name %s. dam_phase_name %s”, handler_name, phase->name);
if(!axutil_strcmp(handler_name, “SandeshaGlobalInHandler”))
{
status = sandesha2_global_in_handler_invoke(phase->first_handler, env, msg_ctx);
}
if(!axutil_strcmp(handler_name, “SandeshaInHandler”))
{
status = sandesha2_in_handler_invoke(phase->first_handler, env, msg_ctx);
}
}
else
status = axis2_handler_invoke(handler, env, msg_ctx);
if (!status)
{
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
“Handler %s invoke failed within phase %s”, handler_name, phase->name);
return status;
}

Also link to sandesha2/C module library in the axis2c/src/core/engine/Makefile.am and don’t forget to include your module include files also there in.

That’s just the trick. Now you can build Axis2/C and your module and use valgrind as explained before.

At this point I would like to add that adding static module build support for Axis2/C is a great thing to do. Adding modules to Axis2/C is not a frequent phenomena. Actually we currently have only few modules like Rampart, Sandesha2/C and Savan/C. Making these statically built with Axis2/C might give us some performance advantage. We can give an Axis2/C configuration option to build certain standard modules statically with Axis2/C.

To do profiling Axis2/C modules/services with valgrind see previous entry here.

Read Full Post »

Older Posts »