Best of New in Kamailio 3.1.0 - #1: Interactive Config Debugger

Version 3.1.0 is just to be released, proposed date being October 6, 2010, that means about one week and a half since today. So it's time for the blog series to show what are the best of the new features to be introduced by this major release. Backed up by the biggest development team in the history of the project, 3.1.0 comes with a large set of new features, along with tighter integration between Kamailio (OpenSER) and SIP Express Router (SER) - the sip-router.org development project.

Let the show begin ...

One of the topics gathering complaints in the past was the complexity of debugging the config file. xlog module did the heavy work so far to help administrators understand how the processing of SIP traffic is going through the config file (I still consider xlog the best friend of config file writers, and it has some new goodies I will talk about them later).

Well, I guess this new features will make happy a lot of people: version 3.1.0 brings a new module, named debugger, providing an interactive configuration file debugger and printing of execution trace.

It allows to execute step by step the configuration file, discover which actions are involved in routing a particular SIP message and evaluate pseudo-variables in the context of that message. For those familiar with gdb tool, you can make a similar association. Tools like sercmd (command line tool, installed by default in the same location as Kamailio binary) or siremis (web management interface available at http://siremis.asipto.com) can be used to control the module with RPC, from same host or remotely.

How it works? You can tell Kamailio to stop before executing first line of config when a SIP message is received (also, you can set breakpoints at certain lines of your config). Then, via RPC commands, you can instruct to execute next action, evaluate a config variable or to execute all next actions. External applications such as sipsak or sipp can be used to inject a SIP message that is problematic, in order to debug it.

Let's look at an example: if calls set to you by a particular peering partner originating traffic from 10.0.0.10 are not routed properly, you can add at beginning for main routing route next lines:

if($si=="10.0.0.10")
dbg_breakpoint("1"); 

Then, when a call is sent from there, kamailio stops to execute the actions after dbg_breackpoint("1"); line. You connect with sercmd and see which kamailio process read it and issue dbg.pb next commands, walking step-by-step through the actions of config file.

You can see an example in the readme of debugger module, some snippets pasted next:

sercmd> dbg.bp show 6402
at bkp [/etc/kamailio/debugger.cfg:369] a=6 n=route

sercmd> dbg.bp next 6402
exec [/etc/kamailio/debugger.cfg:369] a=6 n=route

sercmd> dbg.bp eval 6402 $fu
$fu : t=str v=sip:test@kamailio.org 

You will see each executed action for it and at each step you can print any config variable, helping to understand where is the problem in routing logic or content of SIP message.

Another useful feature of debugger module is printing execution trace. Unlike break-pointing, execution trace functionality does not stop the SIP message processing, but writes log messages with informative details about executed actions, such as config file name, line and action name. Here is an example:

9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=501 a=17 n=if
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=470 a=25 n=has_totag
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=386 a=17 n=if
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=379 a=26 n=is_method
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=386 a=25 n=t_check_trans
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=389 a=6 n=route
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=643 a=3 n=return
9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=393 a=26 n=remove_hf 

This is quick and easy way to see the config actions execution path, from start to end.

Printing execution trace can be controlled at runtime, via RPC commands, without need to restart the SIP server, the admins being able to enable or disable it globally or per process.

Stay tuned for new posts of this series in the next days...