The Corelatus Blog - Entries tagged gth

Erlang example code for GTH

Sun, 1 Feb 2009 16:55:04 GMT

Years ago, I wrote a few bits of example code to help customers get started with controlling a GTH. If you were kind, you'd call them minimal, e.g. the Erlang one was about 20 lines long.

None of our customers that use Erlang were very impressed by that, so I'd hand out part of the code we use to do API testing (a cut-down XML parser and a little library to handle sockets) along with sternly issued instructions that this was just example code and that live systems needed something more robust. Not great, but I had other things to do.

I finally got around to completing and releasing a proper native Erlang API for the GTH. No (visible) XML. It's a gen_server, so it's well-behaved and simple enough so that you can do almost anything from the Erlang shell, e.g. to turn on an E1:


1> {ok, A} = gth:start_link("172.16.2.7").
{ok, <0.44.0>}
2> gth:set(A, "pcm1B", [{"status", "enabled"}]).
ok

and put an audio file on a timeslot:


3> {ok, Bin} = file:read_file("/tmp/resample_q_alaw.raw").
{ok,<<...>>}
4> {ok, _ID, P_socket} = gth:new_tcp_player(A, "1B", 2).
{ok,"strp1",#Port<0.1097>}
5> gen_tcp:send(P_socket, Bin).
ok
6> flush().

There's a direct mapping between GTH commands and function names in the gth.erl module, e.g. the XML <set> command becomes gth:set(). While writing the gth.erl module, I also rewrote our entire API test suite (60 modules, 30kloc) to use the gth.erl interface. It ended up under 20kloc. I'd guess half of that reduction comes from gth.erl being nicer to use and the other half comes from cleaning up "while I was at it".

The code, with examples: gth_erlang_api.zip

While writing gth.erl, I had a go at doing something similar in Python. But that's another story.

SOX parameters for downsampling to 8kHz Alaw

Thu, 19 Feb 2009 16:50:16 GMT

A technician working for an operator mailed me a few days ago wondering why the recorded voice clips they use for their IVR sound so bad, "like they're coming from the bottom of a deep well". It turned out that the clips actually sounded OK on a telephone, just not through his laptop's speaker. He asked if I recommend any specific filter parameters when converting audio from 44.1kHz wav to 8kHz Alaw voice clips.

An example

I took this audio snippet from the introduction to an audio book. It was originally a .ogg file. I converted it to a .wav file with a 44.1kHz sampling rate and 16 bits per sample. For my purposes any artefacts from ogg vorbis are negligible.

1_mono.wav (44.1kHz, 16 bit linear samples)

Next, I converted it to 8kHz Alaw using sox. 8kHz Alaw is what runs on the fixed telephone network in most of the world. (The US uses a minor variant, μlaw):

sox 1_mono.wav -A -r 8000 2_8kHz_alaw.wav

2_8kHz_alaw.wav (8kHz, 8 bit Alaw samples)

That sounds a bit less clear than the original, but it's OK. It's what you'd expect coming out of a telephone. There's some weirdness though. The audible difference between the two files varies from one PC to another and even one playback program to another. Why? Because laptop speakers vary in quality and because playback programs usually quietly convert everything back to 48kHz or 44.1kHz sampling rates, and they do it with different approaches. For fun, I resampled to 44.1kHz:

sox 2_8kHz_alaw.wav -r 44100 -s 3_resampled.wav

3_resampled.wav (44.1kHz, 16 bit linear samples)

2_8kHz_alaw.wav and 3_resampled.wav should sound almost the same. But on some PCs they sound markedly different.

The GTH just plays octets (bytes)

The GTH has a simple approach to playing back audio. It just copies the bytes you give it to the destination timeslot. No format or rate conversion happens, though the GTH does make sure the data is played out at the E1's frame rate (8000Hz). The downside of that is that you have to convert all the files for your IVR system before giving them to a GTH, e.g. using sox. The upside is that it's simple. Nothing happens behind your back.

What are the best SOX options to use?

I don't know. I used to suggest the following as a reasonable starting point:

sox original.wav -r 8000 -c 1 -A -t raw gth.raw resample -q

As of a few years ago, sox improved and the 'resample' effect got deprecated. So now I suggest just letting sox do what it thinks is best:

sox original.wav -r 8000 -c 1 -A -t raw gth.raw

At the time of writing, it uses its 'rate' effect with reasonable default parameters for the bandwidth and filter characteristics. I experimented a bit with the -m, -h, -v and -s switches for the 'rate' effect. I could not reliably hear a difference, let alone decide that one sounded better.

Why does the phone system use 8kHz anyway?

There's a certain sound quality level expected in telephone networks, and part of that is that the network carries everything up to about 3500Hz. Analog local loop specifications mention that, and pretty much all digital telephone systems use an 8kHz sampling rate, which is what you need to be able to carry audio up to 3.5kHz. Even the GSM and AMR codecs start off with the assumption that the incoming audio is limited to 3500Hz.

So the bar is set pretty low. I haven't come across any systems which set out to provide higher quality, e.g. even skype compresses the hell out of the audio to save bandwidth. Even when both parties in a conversation have huge amounts of it. Surprising, why not aim for VOIP to sound much better than a regular telephone?

GTH audio streaming: why stream over TCP?

Fri, 6 Mar 2009 16:30:38 GMT

GTH lets you stream audio from a TCP socket to a timeslot on an E1/T1 line. Some people are surprised by the choice to use TCP. When I added that support back in 2002, my first thought was to use RTP (RFC 1889). RTP is simple: you just dump the audio in a UDP packet with some timestamping information and shoot it out on ethernet at the right rate.

I'd worked with RTP before and I'd been at a couple of SIP interops where most of the attendees had trouble emitting audio at 'the right rate', i.e. 8000 samples/s. One manufacturer's system would emit 8007 samples/s. Another would play it back at 7999 samples/s. What do you do with the extra 8 samples per second? If you do nothing, you get endlessly growing delays and, eventually, a buffer overflow. If you come up with a strategy for throwing away samples, it's bound to interact badly with something, sooner or later.

The thing is, when you're streaming in pre-recorded audio, you don't need it to be at the right rate. You just need to make sure it doesn't overrun or underrun the GTH's internal buffer. I.e. you need flow control, not rate control. TCP has flow control, and everyone knows how to use TCP sockets. In 2002, doing things that way was right at the limit of what our 50MHz embedded CPU could keep up with. Now it's no problem at all.

Python

I'm playing around with python at the moment. Here's how to put some data on an E1 timeslot, straight from the python shell. First, set up a listening TCP socket:


import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.listen(0)
addr, port = s.getsockname()

Next, open another socket to the GTH command port (2089) and tell it we want to stream in audio on the socket we opened above:


a = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
a.connect(("172.16.2.7", 2089))
my_ip, _port = a.getsockname()
command = "" % (my_ip, port)
header = "Content-type: text/xml\r\nContent-length: %d\r\n\r\n" % len(command)
a.sendall(header + command)

Finally, accept() and send the data:


d, _ = s.accept()
d.sendall("hello world")
d.close()

That looks OK to me, though I imagine the style betrays my Erlang mindset. There's a more complete python example at the bottom of the API page.

The timestamp field in signalling headers

Mon, 23 Mar 2009 20:40:32 +0100

When the Corelatus GTH is used to monitor (sniff) signalling, it sends each sniffed packet to your server over a TCP socket, along with a header. For instance, for SS7 MTP-2 the header looks like this:

octet 0x00:  Length (16 bits)
octet 0x02:  Tag (16 bits)
octet 0x04:  Flags (16 bits)
octet 0x06: Timestamp (48 bits)

Every field is big-endian, i.e. the most significant byte comes first. Here's an actual header from a GTH, octet by octet:

00 1c 00 00 00 00 01 20  34 ee fa 61 99 99 99 99 ...

The timestamp is thus 0x012034eefa61, or decimal 1237838658145. For most applications, you just want to know which packet came first, so the interpretation of that number doesn't matter much, though it's useful to know that it's the number of milliseconds since the unix epoch. (wikipedia has a decent article about unix time)

Sometimes, though, you want to represent that as a human-readable time. Unix (and, most likely, Win32) provides functions to do that in the C library, so, after throwing away the last three digits (the milliseconds), this C program does it:


#include <time.h>
#include <stdio.h>

int main() {
  const time_t time_stamp = 1237838658;
  printf("%d corresponds to %s\n", time_stamp, ctime(&time_stamp));

  return 0;
}

The output agrees with what the clock on my wall says:

1237838658 corresponds to Mon Mar 23 21:04:18 2009

Python

Since I've been messing around with python, the same thing in python:


>>> import time
>>> time.ctime(1237838658)
'Mon Mar 23 21:04:18 2009'

Erlang

Erlang doesn't have an interface to the 'ctime' call, but you can use the gregorian calendar functions:


1> Epoch = calendar:datetime_to_gregorian_seconds({{1970, 1, 1}, {0,0,0}}).
62167219200
2> calendar:gregorian_seconds_to_datetime(1237838658 + Epoch).
{{2009,3,23},{20,4,18}}

Why use milliseconds?

Why is the GTH timestamp in milliseconds instead of either seconds or a 'timeval'-like seconds + microseconds?

We chose millisecond resolution for several reasons. Firstly, the shortest possible useful packet in SS7 takes a bit more than a millisecond to transmit at 64kbit/s. Secondly, the practical limit of NTP time synchronisation over the internet is about one millisecond at a typical site.

Decoding MTP-3 and ISUP

Wed, 25 Mar 2009 13:55:20 GMT

Sometimes, you want to look at the signalling on an E1 and use it to figure out when telephone calls start and stop. In SS7 networks, call setup and tear-down is done by the ISUP layer, which fits in to the SS7 stack like this:

Layer 4: ISUP
Layer 3: MTP-3
Layer 2: MTP-2
Layer 1: MTP-1 (typically an 2Mbit/s E1 or a 1.5Mbit/s/T1)

If you have a GTH connected to the E1 you're interested in, either via a DXC or a monitor point, the GTH takes care of layers one and two. That leaves MTP-3 and ISUP to you.

The easiest way to decode MTP-3 and ISUP is to let wireshark do it for you. There's a note about how to do that on Corelatus' official site. But this blog entry is about how to decode MTP-3 and ISUP yourself.

A signal unit (packet)

In SS7, packets are usually called "signal units". Here's what an SS7 signal unit looks like 'on the wire', octet by octet, with MTP-2 and MTP-1 already decoded:

  8d c8 1f 85 02 40 00 00  35 00 01 00 21 00 0a 02
  02 08 06 01 10 12 52 55  21 0a 06 07 01 11 13 53
  55 00 6e 00

MTP-3

The start of the packet is the MTP-2 (ITU-T Q.703) and MTP-3 (ITU-T Q.704) headers. These headers are easy to decode because they are always fixed-length:

Octet(s)	Value	Purpose
00--01	8d c8	MTP-2 sequence numbers, safe to ignore
02	1f	MTP-2 length indicator. Anything less than 3 is reserved for MTP-2 itself and should be discarded.
03	85	MTP-2 SIO. The SIO tells us which 'service' the signal unit is intended for. Q.704 sections14.2.1 and 14.2.2 tell us that anything ending in hex 5 is for ISUP.
04--07	02 40 00 00	MTP-3 Routing label. The routing label is just a "from" and "to" address in the SS7 network. For most applications we can ignore it. Q.704 figure 3 shows what's in the routing label.

Upshot: to see calls start and stop, all we have to do for MTP-3 is:

Look at the length indicator (offset 2) and discard any signal unit where it's less than 3.
Look at the SIO (offset 3). Discard if (SIO & 0x0f != 5)

ISUP

The rest of the signal unit is ISUP. Annex C in ITU-T Q.767 tells us how to decode ISUP. ISUP is fiddly because there are several types of ISUP packets, because several of those types have optional fields and because some of those fields are variable length. Here are the octets we have left after removing MTP-2 and MTP-3:

  35 00 01 00 21 00 0a 02  
  02 08 06 01 10 12 52 55 
  21 0a 06 07 01 11 13 53 
  55 00 6e 00

The first two octets are the CIC. The third octet is the Message type.

The CIC (Q.767 C.1.2) tells us which circuit this call uses. All the signalling for one call has the same CIC. In ITU networks, it's a 12-bit value packed into the field in little-endian byte order. In this case CIC=0x0035. We're sniffing an E1 line, so C.1.2.a tells us that the lower five bits correspond to the timeslot (timeslot 5) and the rest identifies the E1 itself.

The Message Type (Q.767 Table C-3) field tells us what sort of ISUP message this signal unit is. 0x01 is an IAM. 0x10 is RLC. For a minimal "show me what calls are going through the system" hack, we only need to look at the IAM (comes at the start of the call, contains the A and B numbers) and the RLC (sent when the call is finished) messages.

Now we know that the CIC=0x35, that the message is an IAM and we still have about a dozen octets to decode. Q.767 table C-16 tells us how to decode an IAM. There are some uninteresting fixed-length fields followed by the B number and then the A number. Look at the code (or Q.767, section C.3.7) if you're interested in the details. All we really care about is that these octets

  06 01 10 12 52 55 21

represent the B number: 21255512. You can see the number in the raw data if you skip the first three octets and swap every second digit.

Turning those ISUP steps into an algorithm to decode one signal unit:

Save the CIC
Is the message an IAM? Decode it as an IAM, which is fiddly.
Is the message an RLC? Just print the CIC.

That's all you need to do to make a simple system which prints the start and end of each call. To do something useful, you need to maintain a table of in-progress calls and match up the IAM and RLC messages with the same CIC. You also need to handle things like systems restarting.

Erlang code

Everything discussed above is implemented in the ss7_sniffer.erl example on corelatus.com. It makes good use of Erlang's binary syntax, e.g. here's the MTP-3 decoder:


    mtp3(<<_Sub:4, Service_indicator:4>>, <<DPC:14, OPC:14, SLS:4,
						    Rest/binary>>) -> 
	case Service_indicator of 
	0 -> % Management 
        ignore; 
	1 -> % Test/maintenance 
        ignore; 
	3 -> % SCCP 
        ignore; 
	5 -> 
        isup(DPC, OPC, SLS, Rest); 
	9 -> % B-ISUP; similar to ISUP, but not compatible. 
        ignore; 
	X -> 
        io:fwrite("ignoring SU with unexpected service indicator=~p\n", [X]) 
	end.

It looks a lot like one of the examples in the original paper about the binary syntax.

Python Code

The same thing done in Python is fairly straightforward once you discover the Python 'struct' library, which is basically the same thing as PERL's pack/unpack. The code is in sniff_isup.py, inside the GTH python examples zip.

It feels like I haven't discovered whatever it is python people use to unpack bitfields, e.g. something neater than:


    is_even = ((ord(num[1]) &amp; 0x80) == 0)

or, better still, a clean way to decode the MTP-3 routing label.

Comments

Bartosz, 28. January 2010

Hey. Just stumbled on this page while I was googling after Mtp3 procedures. I'm wondering if you have any materials that you could share regarding link setup - I'm working on open source stacks for ss7. Im able to setup mtp2,3 and layer 4(signle link), now I'm working on getting it work with more than one and frankly Qs for mtp3 are not very helpful.

(website)

Matt, 28. January 2010

@Bartosz: I only really work with MTP2. My customers use MTP3, so I know quite a bit about it, but you probably know more. For MTP2, the standard is difficult to read, but comprehensive. MTP3 seems harder. I have a copy of "Signalling in Telecommunications Networks" by van Bosse et al, it's useful for getting started, but I always end up trudging through the standards.

Interesting to see someone working on an open SS7 stack. Up until now, I've only been aware of openss7.org (written in C, some parts seem complete, the project seems to be a one-man effort).

Or: I don't think I can help much, but what you're doing looks interesting.

Matt

Emza, September 15, 2009

I found it very important but still not enough for me. I was looking for ISUP MTP-3 and ISUP decoding in C or in C#(not only start and end singnal unit but general which includes all message types). I would like to thank you but can help in my case please.

Thanks again
Embza

Matt, September 15, 2009

@Emza, decoding the rest of the message types is 'just' a matter of working your way through ITU-T Q.767 (ITU standards are now available for free at http://www.itu.int) and writing code to handle each and every section in the standard. Same basic idea as the messages I did. Doing it all is a few days of drudge work, which is why I haven't done it here.

About doing it in C or C#. I know nothing about C#. I work with C most days, though. Doing this sort of protocol decoding in C is fairly straightforward, but it's inevitably going to be more tedious than doing it in something like python, perl or erlang. In some applications the performance gain might be worth the extra effort.

Meskerem David, September 15, 2009

It is a great job. Keep working...

Can u please me one brief algorithm of decoding ISUP. Especially I am not clear how to decode the variable length parameters and the optional ones.

Thanks
Meskerem

Matt, September 15, 2009

@Meskerem, In an IAM, the A-number and B-number parameters are variable length. The algorithm for finding the start is trivial, it's just a pointer offset. Here's what it looks like in the python example code:

    

      def isup_iam(_, CIC, sif):
      # First 5 octets can be ignored
      bnum_pointer = ord(sif[5])
      anum_pointer = ord(sif[6])
      bnum = sif[5 + bnum_pointer:]
      anum = sif[7 + anum_pointer:]
      print "IAM called party: %s calling party: %s CIC=%d" \
      % (isup_number(bnum), isup_number(anum), CIC)

      # And here's how the example code figures out the length:

      # Decode an ISUP number, as per C 3.7
      def isup_number(num):
      length = ord(num[0]) - 2

the complete code is at http://www.corelatus.com/gth/api/gth_python_examples.zip

(the site also has exactly the same decoding routines in Perl and Erlang)

Perl example code for GTH: SS7 ISUP decoding and playback/record

Thu, 28 May 2009 23:25:45 +0200

To help people get started, www.corelatus.com has some example code for doing useful things with GTH units.

Now it also has Perl example code. It does the same thing as the python examples:

Enable an E1/T1 port
Start MTP-2 monitoring on a timeslot and decode SS7 ISUP (to print out when calls start and stop). I wrote a post a while back about how to decode ISUP.
Dump the contents of a timeslot to a file (for later analysis)
Feed a file into a timeslot (for playback of previously captured files)

It's built on top of a Perl module which provides a Perl API for a subset of the GTH API.

A quick example

Here's a quick example of how it's used. We want to enable (turn on) the first E1/T1 interface on a GTH module:


my $api = gth_control->new($host);
$api->send("<set name='pcm$span'><attribute name='mode' value='E1'/></set>");
defined $api->next_non_event()->{ok} || die("error from GTH (bogus PCM?)");
$api->bye();

It's good for experimenting.

The Perl module the examples are based on, gth_control.pm, is at a level which makes it useful for experiments and prototypes. To build a full-fledged product on top of it, more work is needed.

For a start, you'd probably want to move the XML generation (like the '<set name=...' code above) out of the application code and into the gth_control.pm module, thus making it a pure Perl interface.

Next, you need to come up with a strategy to deal with concurrency, because being limited to recording one timeslot at a time is fine for lab work, but not fine for (say) a voicemail system.

Download

The zipfile of the code is linked from the bottom of the API page.

Generating DTMF using a 'player' on GTH

Tue, 9 Jun 2009 12:31:17 +0200

The GTH can transmit in-band signalling tones on a timeslot. That's useful for testing and for building active in-band signalling systems.

DTMF

The tones transmitted when the subscriber presses a number key on fixed or mobile handset are called DTMF. Wikipedia has an article about it. To generate DTMF, all we really need to know is that there are 16 possible DTMF signals, that each signal is made up of two sine waves of particular frequencies and that sending the signal for 100ms is a reasonable thing to do.

Here's a .zip file with DTMF tones in it. Each file is raw ALAW data, i.e. it's ready for the GTH to play (transmit) on a timeslot.

The GTH has two ways of playing tones. One way is to stream the audio data in over a TCP socket each time we want to play it. I wrote a post about that earlier. The other way is to store the sample data on the GTH and command its playback whenever it's needed. Since there's a small number of different tones (12, or 16 if you want to use the A/B/C/D tones as well) and the tones are short, storing them on the GTH makes sense. To store the tone:


<new><clip name='dtmf5'></new>
(and now send the 800 byte file)

to play the tone later on:


<new><player>
  <clip id='clip dtmf5'/>
  <pcm_sink span='3A' timeslot='19'/>
</player></new>

Sequences of tones

Sometimes you want to transmit a sequence of DTMF tones, for instance to simulate a subscriber dialling a number. The GTH lets you start a player with a sequence of tones like this:


<new><player>
   <clip id='clip dtmf5'/><clip id='clip dtmf6'/><clip id='clip dtmf8'/>
   <pcm_sink span='3A' timeslot='19'/>
</player></new>

But that isn't a valid sequence of DTMF tones. Why not? Because DTMF expects a gap between tones. The cleanest way to handle that is to define another clip consisting of just silence and putting it between each tone. A good 'silence' value on E1 lines is 0x54. 60ms (480 samples) is a reasonable length.

Other in-band tones

DTMF in-band signalling is used in pretty much all handsets (telephones), mostly for dialling, but also to navigate menus in IVR systems. But before SS7 became popular, in-band signalling in the form of CAS and SS5 was even used to communicate call setup information between exchanges. GTH can also generate those tones, but that can be the subject of another post.

How does TCP behave on an interrupted network?

Thu, 27 Aug 2009 23:24:22 GMT

GTH E1/T1 modules are always controlled by a general-purpose server, usually some sort of unix machine. The server and GTH are connected by ethernet and communicate using TCP sockets. Normally, that ethernet connection is chosen to be simple and reliable, for instance by putting the server and the GTH in the same rack, connected to the same ethernet switch.

I experimented a bit to see what happens when that network gets interrupted. I interrupted the network in a reproduceable way by disabling and re-enabling the server's ethernet port for a known length of time while running a <recorder>. (A <recorder> sends all the data, typically someone talking, from an E1 timeslot to the server over a TCP socket, 8000 octets per second.)

Capturing the ethernet packets

Here's what I did to capture traffic and interrupt the ethernet:

  tcpdump -w /tmp/capture.pcap -s 0 not port 22
  sudo ifconfig eth0 down; sleep 5; sudo ifconfig eth0 up

A trace where traffic recovers in time to prevent an overrun

The GTH buffers about two seconds of timeslot traffic. So a 'sleep' of about a second won't result in an overrun. Here's what it looks like in wireshark:

Packet	Time	Direction	Flags	Seq. #

133	7.596	GTH -> server	[PSH, ACK]	59393
134	7.633	server -> GTH	[ACK]	1
135	7.724	GTH -> server	[PSH, ACK]	60417
136	7.761	server -> GTH	[ACK]	1
137	7.852	GTH -> server	[PSH, ACK]	61441
138	7.889	server -> GTH	[ACK]	1
139	7.980	GTH -> server	[PSH, ACK]	62465
140	8.017	server -> GTH	[ACK]	1
141	8.108	GTH -> server	[PSH, ACK]	63489
142	8.145	server -> GTH	[ACK]	1
143	8.236	GTH -> server	[PSH, ACK]	64513
144	8.273	server -> GTH	[ACK]	1
145	8.364	GTH -> server	[PSH, ACK]	65537
146	8.401	server -> GTH	[ACK]	1
147	10.151	GTH -> server	[PSH, ACK]	66561
148	10.151	server -> GTH	[ACK]	1
149	10.151	GTH -> server	[ACK]	67585
150	10.151	server -> GTH	[ACK]	1

Everything up to packet 146 is normal: the GTH (172.16.2.5) sends 8000 octets every second and the server (172.16.2.1) acks them. It happens to be in chunks of 1024 octets about eight times per second. After packet 146, about 8.4 seconds after the capture started, the ethernet interface went down and stayed down for 1s. The TCP stream started up again after about 1.5s and then 'caught up' by sending many packets in quick succession.

A trace where traffic didn't recover

I took a second trace similar to the first one, except this time, I disabled ethernet for about five seconds:

Packet Time     Source IP     Dest IP    SPort   DPort
----------------------------------------------------------------------
 28   1.040083  172.16.2.5 -> 172.16.2.1 54271 > 45195 [PSH, ACK] Seq=7169
 29   1.040095  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1
 30   1.168065  172.16.2.5 -> 172.16.2.1 54271 > 45195 [PSH, ACK] Seq=8193
 31   1.168078  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1 
 32   1.296067  172.16.2.5 -> 172.16.2.1 54271 > 45195 [PSH, ACK] Seq=9217
 33   1.296079  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1 
 34   1.424068  172.16.2.5 -> 172.16.2.1 54271 > 45195 [PSH, ACK] Seq=10241
 35   1.424081  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1
 36   7.782851  172.16.2.5 -> 172.16.2.1 54271 > 45195 [PSH, ACK] Seq=11265
 37   7.782863  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1
 38   7.783406  172.16.2.5 -> 172.16.2.1 54271 > 45195 [ACK] Seq=12289
 39   7.783413  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1
 40   7.783569  172.16.2.5 -> 172.16.2.1 54271 > 45195 [ACK] Seq=13737
...
 50   7.784962  172.16.2.5 -> 172.16.2.1 54271 > 45195 [FIN, PSH, ACK] Seq=23873
 51   7.784972  172.16.2.1 -> 172.16.2.5 45195 > 54271 [ACK] Seq=1
 52   7.785026  172.16.2.1 -> 172.16.2.5 45195 > 54271 [FIN, ACK] Seq=1
 53   7.785348  172.16.2.5 -> 172.16.2.1 54271 > 45195 [ACK] Seq=25322

Everything is normal up to packet 35. Then, ethernet is suspended for five seconds and TCP takes a further second to recover, which causes a buffer overrun on the GTH (172.16.2.5). The GTH closes the socket at packet 50 and also sends an overrun event to the application so that it knows why the socket was closed.

Bottom line

GTH uses IP for control and traffic. It is important that the IP link between the GTH and the server is simple and reliable. Ideally the GTH and server should be in the same rack and be connected by an ethernet switch.

It's possible for a system to survive a short interruption (less than a second) to the ethernet traffic without pre-recorded calls getting interrupted. For longer interruptions, all bets are off.

(Interruptions aren't the only type of network problem, e.g. radio networks such as 802.11 can suffer significant packet loss, which can trigger TCP congestion avoidance. But that's another topic.)

Audio power levels on E1/T1 timeslots: the digital milliwatt

Sat, 10 Oct 2009 21:33:52 +0200

Sometimes, you want to know when the audio on an E1/T1 timeslot has gotten louder than some limit. In a voice mail application, that's useful for catching mistakes such as a subscriber leaving a message but then not hanging up the phone properly---you don't want to record hours and hours of silence. In an IVR application, you might want to keep an eye on the audio level so that a frustrated (shouting!) subscriber can be forwarded to a human operator.

GTH provides a "level detector" to do that sort of thing. You start a level detector on a timeslot, give it a loudness threshold, and it'll notify you whenever the audio on the timeslot goes over that threshold. Here's an example command which notifies you if the power on timeslot 13 of an E1/T1 is louder than -20dBm0:


  <new><level_detector threshold='-20'>
     <pcm_source span='2A' timeslot='13'/>
  </level_detector></new>

The algorithm is:

Take a 100ms block of audio (800 samples)
Square all the samples
Sum the squares
If the sum exceeds the loudness threshold, send an XML event

There are some details to worry about.

The digital milliwatt

The threshold, e.g. -20 in the example above, has to be relative to something. The standard reference power level in telecommunications is the milliwatt. ITU-T G.711, table 5 and 6 defines the sequences which represent a milliwatt:

A-law: 34 21 21 34 b4 a1 a1 b4
μ-law: 1e 0b 0b 1e 9e 8b 8b 9e

Here's what a few periods of the digital milliwatt look like:

In this post, the unit 'dBm0' means power, in dB relative to the digital milliwatt, as defined by the sequences above. If you have no idea what dB means, wikipedia has a decent article.

What's the loudest possible sound on a timeslot?

170 is the highest value possible in A-law encoding. It corresponds to linear 4032. That's about 6dB louder than the digital milliwatt.

85 is the smallest value possible in A-law encoding. It corresponds to linear -1. That's about 66dB softer than the digital milliwatt.

The range -66dBm0...+6dBm0 sets an upper bound on the range of power on a timeslot. Then there are other things which further limit the practical range, so you're unlikely to actually use a +6dBm0 threshold in practice, but it's there if you want it.

Is the G.711 definition the best one?

The sequence given in G.711 is a 1kHz sine wave. The sampling rate on E1/T1 is 8kHz, so the reference sequence can be expressed in just eight values. That's nice, but that also leads to small errors, about 0.13dB, because of quantisation.

ITU-T O.133 discusses that problem in detail and proposes a test signal which specfically is not 1kHz (i.e. not a submultiple of the sampling rate). For most practical purposes, 0.13dB doesn't matter and so the simple and robust thing to do is to use the well-defined and well-known G.711 sequence as a reference.

Here's what a few periods of a 1020Hz signal look like. Notice that the samples, i.e. the red crosses, don't appear in the same spot one period later---that way we don't get the same errors over and over again.

1020Hz signal sampled at 125us intervals

Testing pitfall: the .wav header

GTH players plays raw A-law or μ-law data. If you feed a player a .wav file in 8kHz A-law, or μ-law if your network uses μ-law, there will be a very short bit of noise at the start of the playback because the .wav header gets treated as though it were audio.

When testing level detection, especially at quiet levels, that header noise is enough to trigger a detector. Here's a .wav of a 1000Hz sine wave at about -30dBm0:

00000000  52 49 46 46 42 27 00 00  57 41 56 45 66 6d 74 20  |RIFFB'..WAVEfmt |
00000010  12 00 00 00 06 00 01 00  40 1f 00 00 40 1f 00 00  |........@...@...|
00000020  01 00 08 00 00 00 66 61  63 74 04 00 00 00 10 27  |......fact.....'|
00000030  00 00 64 61 74 61 10 27  00 00 d5 c4 f5 f1 f3 f1  |..data.'........|
00000040  f5 c4 d5 44 75 71 73 71  75 44 d5 c4 f5 f1 f3 f1  |...DuqsquD......|
*
00002740  f5 c4 d5 44 75 71 73 71  75 44                    |...DuqsquD|

The first 58 octets (bytes) are the header. If we turn that header into a periodic signal, it's at about -8dBm0, which is fairly loud. With the default period parameter of 100ms in the level detector, that'll cause a false level of about -11dBm0.

The period parameter

The level_detector has an optional parameter, the period. The period sets the size of the audio block the GTH considers when measuring the power. A short period makes the GTH responsive to sudden changes in power on the timeslot, which would be useful in an application such as figuring out which of the people in a conference call are currently talking. A long period averages out the power over a longer time, which is useful in deciding whether a voicemail recording has finished.

The default is 100ms.

Sample files

This .zip file contains sample recordings with 1kHz sine waves at 0, -10, -20, -30, -40 and -50 dBm0. The .wav versions are useful for listening to or importing into an audio editing program to calibrate the level meter. The .raw versions are just plain A-law samples---you can use them with a GTH player.

GTH 'C' API

Wed, 20 Jan 2010 10:47:37 GMT

This post is for people who want to use C to control a GTH. Other languages (e.g. Erlang, Java, Python and Perl) are easier to work with, but in some applications you want the complete control that C gives you.

Corelatus provides a C API for GTH. The C API lets you control a GTH using plain C function calls---all of the XML wire format is taken care of.

Here's an example of how to use it to record speech on an E1/T1 timeslot, something you'd typically do in a voicemail system:


#include "gth_apilib.h"
...

  GTH_api api;                     // GTH_api represents one GTH API connection
  int result;
  int data_socket;
  char buffer[2000];
  char job_id[MAX_JOB_ID];
  int octet_count;

  result = gth_connect(&api, "172.16.1.10");    // Assuming the default GTH IP 
  assert(result == 0);

  // We want to record audio on the E1/T1 called "1A", on timeslot 3.
  data_socket = gth_new_recorder(api, "1A", 3, job_id);

  while ( (octet_count = recv(data_socket, buffer, sizeof buffer, 0)) ) {
    // do whatever you want with the received data
  }

...

The recording above happens in the 'while' loop. It continues forever (i.e. until you abort it). The audio data is bit-for-bit identical to what was on the E1/T1 timeslot, so this can be used for recording both voice and signalling.

Further examples

The C API code includes further examples to:

Record audio or signalling from a timeslot to a .wav file
Play back previously recorded files
Install (upgrade) the firmware on a GTH
Monitor (sniff) SS7 MTP-2 and save the signalling to a wireshark (PCAP) compatible file.
Monitor CAS R2 signalling.

Standalone Parser

Aside: the C API code includes a standalone parser for the XML responses the GTH emits. You can use the parser without using the rest of the API library, if you want to.

Decoding the Gb Interface with Wireshark

Wed, 31 Mar 2010 22:26:29 GMT

The Gb interface is part of the packet radio data network (GPRS) in GSM, it sits between the BSC and the SGSN and carries subscriber data headed to and from the internet.

Wireshark understands how to decode the Gb interface, so you can use wireshark to look through data sniffed from a Gb interface by a Corelatus GTH. Here's what it looks like:

How to tell the GTH to capture Gb

To look at a GPRS network like this, you need to do a few things:

Connect one of the GTH's E1 interfaces to the E1 (or T1) interface carrying the Gb interfaces. You typically do that at a cross connect panel, using a G.772 monitor point.
Enable the E1 interface you connected.
Tell the GTH to start decoding frame relay on that interface
Convert the captured data to the file format which wireshark understands, libpcap.
Open the captured file in wireshark. (On unix-like operating systems, including OSX, you can also pipe into wireshark to get a live view of the capture.)

Taking those steps one at a time, starting with #2:

Enable the E1 interface


  <set name='pcm3A'><attribute name='monitoring' value='true'/></set>

Tell GTH start decoding frame relay

The Gb interface uses frame relay on E1. Different sites use different configurations of timeslots. One common setup is to use timeslots 1--15. Another common setup is to 1--15 + 17--31. The GTH can handle any setup.


  <new>
    <fr_monitor ip_addr='172.16.2.1' ip_port='1234'>
      <pcm_source span='3A' timeslot='1'/>
      <pcm_source span='3A' timeslot='2'/>
      <pcm_source span='3A' timeslot='3'/>
      ..
      <pcm_source span='3A' timeslot='15'/>
    </fr_monitor>
  </new>

Convert the captured data

GTH sends out data in a format described in the API manual. Wireshark wants the data to be in libpcap format. save_to_pcap.erl, in the sample Erlang code for GTH can do the conversion, like this:


  save_to_pcap:from_file("/tmp/captured.raw", "/tmp/captured.pcap").

A lazier approach is to let save_to_pcap.erl configure the GTH and start the capture:


  save_to_pcap:frame_relay("172.16.2.7", "3A", lists:seq(1,15), "gprs.pcap").

The C version of save_to_pcap can currently only convert MTP-2, not frame relay. If you want it extended, send mail.

Start up wireshark

By default, wireshark decodes frame relay as 'FRF 3.2/CISCO HDLC'. That's not quite what we want. Go to Edit/Preferences/Protocols/FR and change the encapsulation to 'GPRS Network Service'. Now you get full decoding.

Decoding UMTS (3G) Interfaces with Wireshark

Mon, 24 May 2010 16:46:18 GMT

Many 3G networks use ATM on their internal interfaces, e.g. on the Iub and Iu-PS interfaces. Those interfaces carry both control information (radio environment information, attach/detach messages, location updates) and also subscriber data, for instance IP traffic.

Wireshark understands how to decode those ATM interfaces. Here's an example of an interface sniffed by a GTH. The interface was carrying IP traffic over ATM on an E1 line.

How to tell the GTH to capture an ATM link

To look at a 3G network like this, you need to:

Connect one of the GTH's E1 interfaces to the E1 (or T1) interface carrying the ATM interface. You typically do that at a cross connect panel, using a G.772 monitor point.
Enable the E1 interface you connected.
Tell the GTH to start decoding ATM AAL5 (and/or AAL2) on that interface
Convert the captured data to the file format which wireshark understands, libpcap.
Open the captured file in wireshark. (On unix-like operating systems, including OSX, you can also pipe into wireshark to get a live view of the capture.)

Taking those steps one at a time, starting with #2:

Enable the E1 interface


  <set name='pcm3A'><attribute name='monitoring' value='true'/></set>

Tell GTH start decoding ATM AAL5

IP traffic on ATM is always carried in AAL5. The timeslot arrangement is usually 1--15 + 17--31. A few sites share the E1 with other protocols, this is called fractional ATM. The GTH can handle either scheme.


  <new>
    <atm_aal5_monitor ip_addr='172.16.2.1' ip_port='1234' vpi='0' vci='5'>
      <pcm_source span='3A' timeslot='1'/>
      <pcm_source span='3A' timeslot='2'/>
      <pcm_source span='3A' timeslot='3'/>
      ..
      <pcm_source span='3A' timeslot='15'/>
      <pcm_source span='3A' timeslot='17'/>
      ..
      <pcm_source span='3A' timeslot='31'/>
    </fr_monitor>
  </new>

In this example, the VPI/VCI is 0/5. If you know the VPI/VCI in advance, great. If you don't, the GTH can sniff traffic at the AAL0 interface and show you which VPI/VCI are active on the link.

Convert the captured data

GTH sends out data in a format described in the API manual. Wireshark wants the data to be in libpcap format. save_to_pcap.erl, in the sample Erlang code for GTH can do the conversion, like this:


  save_to_pcap:from_file("/tmp/captured.raw", "/tmp/captured.pcap").

A lazier approach is to let save_to_pcap.erl configure the GTH and start the capture:


  save_to_pcap:aal5("172.16.2.7", "3A", lists:seq(1,15) ++ lists:seq(17,31),
  {0,5}, "aal5.pcap").

The C version of save_to_pcap can currently only convert MTP-2, not AAL5. If you want it extended, send mail (address at top right).

Start up wireshark

Recent versions of Wireshark, e.g. 1.2.7, can decode such capture files out of the box, without any configuration. Finished.

Porting C Socket Code to Windows

Wed, 16 Jun 2010 13:48:38 GMT

I just ported the C example code for controlling Corelatus E1/T1 hardware to Visual Studio 2010. Doing that was easier than expected. This first part of this post is about how to use that code. The second part is a bit of history about what I had to do to port the code.

Obtaining Visual Studio

Microsoft have their C/C++ compiler available for free download on microsoft.com

The install process is pretty standard for windows, if you're accustomed to Microsoft products, there are no surprises here.

Compiling the code

Start the "Visual Studio Command Prompt". By default, that's in "Start/Programs/Microsoft Visual Studio 2010 Express".
Navigate to the directory you unpacked the sample code in. Remove any old .exe files with 'del *.exe'.
Run 'nmake /f NMakefile'. After a couple of seconds, you have six brand-new .exe files. Here's what this step looks like:

If you just wanted to see how to compile the code, you can stop reading now. The rest of the post is just about what I had to do to make this work with Microsoft's compiler.

Hurdle #1: MS nmake vs gnumake vs Visual Studio projects

Conclusion: I ended up using Microsoft's nmake and wrote a separate NMakefile for it.

My code gets built by 'make', through a Makefile. Visual Studio doesn't include a make program which can understand normal Makefiles, leaving me a choice of either using the Visual Studio IDE or using Microsoft's nmake.

I spent half an hour trying the IDE route. When you make a new project, you have to choose what sort it is. "CLR Console Application" seemed closest to what I wanted to do, so I chose that. Maybe "Makefile Project" would have worked better. Things got increasingly confusing from there. What's a "solution?" Why are "precompiled headers" enabled? I just want to compile a few hundred lines of C. At that point, I gave up, the IDE is clearly not the easiest way for me to get started.

Next, I tried Microsoft's replacement for make, called nmake. That didn't start flawlessly either, nmake uses its own syntax. Initially, I thought I might be able to write one Makefile which both gnumake and nmake understand. But Microsoft's syntax seems to be too different to allow that. Oh well. The NMakefile is just a dozen lines or so.

Hurdle #2: Missing header files

Conclusion: I used #ifdefs to include different header files when compiling in a win32 environment.

Visual Studio doesn't provide some of the header files the code uses, e.g. there's no 'unistd.h' and no 'sys/socket.h'. Odd. Providing most of unistd.h shouldn't be too hard.

The point of this exercise is to get the code to compile using a Microsoft toolchain, so I didn't investigate using cygwin or mingw.

In the end, it turns out that I wasn't using anything from 'unistd.h' which didn't get provided by something else in win32. 'socket.h' (and friends) was a bit harder, but it turns out that's provided by 'winsock2.h'.

Hurdle #3: 'Mostly compatible' socket API

Conclusion: winsock2 is different to the BSD socket API, but the differences can be worked around with just a few changes.

Sockets were invented on unix, so I assumed that other OSes would just copy the interface. But no, not on win32. There seem to be at least two attempts to make a socket API for windows, called 'winsock' and 'winsock2'. The first seems to be deprecated. The main problems I hit were:

You can't use close() on a socket, you have to use closesocket()
You can't use read() on a socket, you have to use recv()
errno doesn't report socket errors, you have to call a special Windows Socket error reporting function.

None of those problems are hard to get around.

Hurdle #4: GNU and Microsoft can't agree on function names

Both GNU and Microsoft have replacements for some functions which are prone to security problems. If you use plain strcat() or strcpy, visual studio warns about security problems.

Disabling the warnings would be quick but feels like cheating. So I looked at the 'less insecure' alternatives. GNU call them strncat and strncpy whereas Microsoft call them strcat_s and strcpy_s. And the arguments are the same way around. It would have been nice if GNU and MS had solved this the same way, but no. Minor annoyance.

Hurdle #5: Structure packing pragmas

Conclusion: Both MS and GNU understand the #pragma pack(N) syntax, but only GNU understands the __attribute__((__packed__)) syntax.

A couple of the examples (save_to_pcap.c and record.c) use structures to encode or decode data defined by an external format. For instance, here's what the wire format of a 'pcap' protocol capture file looks like:

    
      typedef struct {
        unsigned int magic;
        unsigned short major_version;
        unsigned short minor_version;
        unsigned int GMT_to_localtime;
        unsigned int sigfigs;
        unsigned int snaplen;
        unsigned int network;
      } PCAP_global_header;

We have to tell the compiler that spacing out the fields to get a certain alignment isn't allowed. The recommended GNU way to do that is to add __attribute__((__packed__)) after the structure. The recommended Microsoft way is to add #pragma pack(1) somewhere before the structure, and then remember to change the packing back again before hitting any code which is sensitive to performance.

Since this is just example code, we don't care about that last bit of performance, so we can just leave the packing at 1.

Aside: the structure above is a bit sloppy because just quietly assumes that an int is 32 bits and a short 16.

Hurdle #6: Windows firewall

Some of the example code opens TCP sockets for listening. By default, the windows firewall doesn't allow that. Just click "fine, ok, let me do this".

Unfortunately, that looks ugly for the user---the program will fail the first time you run it, and the pop-up box says "publisher unknown". Does anyone know how to improve on this? It'd be nice if the user could approve the publisher, i.e. me, once, and then every program works.

How long did the porting take?

A couple of days, including getting Visual Studio installed. That's for a few thousand lines of code with a bit of socket IO and file IO.

If I had to do it again, it'd take an afternoon or so.

The Corelatus Blog - Entries tagged gth

Erlang example code for GTH

SOX parameters for downsampling to 8kHz Alaw

An example

The GTH just plays octets (bytes)

What are the best SOX options to use?

Why does the phone system use 8kHz anyway?

GTH audio streaming: why stream over TCP?

Python

The timestamp field in signalling headers

Python

Erlang

Why use milliseconds?

Decoding MTP-3 and ISUP

A signal unit (packet)

MTP-3

ISUP

Further reading

Erlang code

Python Code

Comments

Bartosz, 28. January 2010

Matt, 28. January 2010

Emza, September 15, 2009

Matt, September 15, 2009

Meskerem David, September 15, 2009

Matt, September 15, 2009

Perl example code for GTH: SS7 ISUP decoding and playback/record

A quick example

It's good for experimenting.

Download

Generating DTMF using a 'player' on GTH

DTMF

Sequences of tones

Other in-band tones

How does TCP behave on an interrupted network?

Capturing the ethernet packets

A trace where traffic recovers in time to prevent an overrun

A trace where traffic didn't recover

Bottom line

Audio power levels on E1/T1 timeslots: the digital milliwatt

The digital milliwatt

What's the loudest possible sound on a timeslot?

Is the G.711 definition the best one?

Testing pitfall: the .wav header

The period parameter

Sample files

GTH 'C' API

Further examples

Standalone Parser

Decoding the Gb Interface with Wireshark

How to tell the GTH to capture Gb

Enable the E1 interface

Tell GTH start decoding frame relay

Convert the captured data

Start up wireshark

Decoding UMTS (3G) Interfaces with Wireshark

How to tell the GTH to capture an ATM link

Enable the E1 interface

Tell GTH start decoding ATM AAL5

Convert the captured data

Start up wireshark

Porting C Socket Code to Windows

Obtaining Visual Studio

Compiling the code

Hurdle #1: MS nmake vs gnumake vs Visual Studio projects

Hurdle #2: Missing header files

Hurdle #3: 'Mostly compatible' socket API

Hurdle #4: GNU and Microsoft can't agree on function names

Hurdle #5: Structure packing pragmas

Hurdle #6: Windows firewall

How long did the porting take?