Brain dump – what fascinates me

Brain dump – what fascinates me

A small brain dump of topics that currently fascinate me. These are mostly pointers and maybe it is interesting to follow it.

Books/Reading:

My kobo ebook reader has the Site Reliability Engineering book and I am now mostly done. It is kind of a revelation and explains my interest to write code but also to operate infrastructure (like struggling with ruby, rmagick, nginx…). I am interested in backends since… well ever. The first time I noticed  it when we talked about Kolab at LinuxTag and I was more interested in the backend than the KDE client. At sysmocom we built an IoT product and the backend was quite some fun, especially the scale of one instance and many devices/users, capacity planning and disk commissioning, lossless upgrades.

It can be seen in my non FOSS SS7 map work on traffic masquerading and transparent rewriting. It is also clear to see which part of engineering is needed for scale (instead of just installing and restarting servers).

Lang VM design

One technology that made Java fast (Hotspot) and has seen its way into JavaScript is dynamic optimization. Most Just in Time Compilers start with generating native code per method, either directly or after the first couple of calls when the methods size is significant enough. The VM records which call paths are hot, which types are used and then can generate optimized code (e.g. specialized for integers, remove type checks). A technique pioneered at Sun for the “Self” language (and then implemented for Strongtalk and then brought to Java) was “adaptive optimization and deoptimization” and was the Phd topic of Urs Hoelzle (Google’s VP of Engineering). One of the key aspects is inlining across method boundaries as this removes method look-up, call stack handling and opens the way for code optimization across method boundaries (at the cost of RAM usage).

In OpenJDK, V8 and JavaScriptCore this adaptive optimization is typically implemented in C++ and requires quite some code. The code is complicated as it needs to optimize but also need to return to a basic function (deoptimize, e.g. if a method changed or the types passed don’t match anymore), e.g. in the middle of a for loop with tons of inlined code (think of Array.map being inlined but then need to be de-inlined). A nice and long blog post of JSC can be found here describing the On Stack Replacement (OSR).

Long introduction and now to the new thing. In the OpensmalltalkVM a new approach called Sista has been picked and I find it is genius. Like with many problems the point of view and approach really matters. Instead of writing a lot of code in the VM the optimizer runs next to the application code. The key parts seem to be:

  • Using branches taken/not-taken as indicator how hot a path is. The overhead of counting these seem to be better than counting method calls/instructions/loops.
  • Using the Inline Caches for type information on call sites (is that mono-, poly- or megamorphic?)
  • Optimize from one set of Bytecode to another set of Bytecode.

The revelation is the last part. By just optimizing from bytecode to bytecode the VM remains in charge of creating and managing machine code. The next part is that tooling in the higher language is better or at least the roundtrip is more quick (edit code and just compile the new method instead of running make, c++, ld). The productivity thanks to the abstraction and tooling is likely higher.

As last part the OSR is easier as well. In Smalltalk thisContext (the current stack frame, activation record) is an object as well. At the right point (when the JIT has either written back variables from register to the stack or at least knows where the value is) one can just manipulate thisContext, create and link news ones and then resume execution without all the magic in other VMs.

Go, Go and escape analysis

Ken Thompson and Robert Pike are well known persons and their Go programming language is a very interesting system programming language. Like with all new languages I try to get real world experience with the language, the tooling and which kind of problems can be solved with it. I have debugged and patched some bigger projects and written two small applications with it.

There is plenty I like. The escape analysis of the compiler is fun (especially now that I know it was translated from the Plan9 C compiler from C to Go), the concurrency model is good (though allowing shared state), the module system makes sense (but makes forking harder than necessary), being able to cross compile to any target from any system.

Knowing a bit of Erlang (and continuing to read the Phd Thesis of Joe Armstrong) and being a heavy Smalltalk user there are plenty of things missing. It starts with vague runtime error messages (e.g. panicslice not having parameters) and goes to runtime and post-runtime inspection. In Smalltalk thanks to the abstraction a lot of hard things are easy and I would have wished for some of them to be in Go. Serialize all unrecovered panics? Debugging someone else’s code seems like pre 1980…

So for many developers Go is a big improvement but for some people with a wider view it might look like a lost opportunity. But that can only be felt by developers that have experienced higher abstraction and productivity.

 

Unsupervised machine learning

but that is for another dump…

Starting with a Diameter stack

Starting with a Diameter stack

Going from 2G/3G requires to learn a new set of abbreviations. The network is referred to as IP Multimedia Subsystem (IMS) and the HLR becomes Home subscriber server (HSS). ITU ASN1 to define the RPCs (request, response, potential errors), message structure and encoding in 2G/3G is replaced with a set of IETF RFCs. From my point of view names of messages, names of attributes change but the basic broken trust model remains.

Having worked on probably the best ASN1/TCAP/MAP stack in Free Software it is time to move to the future and apply the good parts and lessons learned to Diameter. The first RFC is to look at is RFC 6733 – Diameter Base Protocol. This defines the basic encoding of messages, the requests, responses and errors, a BNF grammar to define these messages, when and how to connect to remote systems, etc.

The core part of our ASN1/TCAP/MAP stack is that the 3GPP ASN1 files are parsed and instead of just generating structs for the types (like done with asn1c and many other compilers) we have a model that contains the complete relationship between application-context, contract, package, argument, result and errors. From what I know this is quite unique (at least in the FOSS world) and it has allowed rapid development of a HLR, SMSC, SCF, security research and more.

So getting a complete model is the first step. This will allow us to generate encoders/decoders for languages like C/C++, be the base of a stack in Smalltalk, allow to browse the model graphically, generate fancy pictures, …. The RFC defines a grammar of how messages and grouped Attribute-Value-Pairs (AVP) are formatted and then a list of base messages. The Erlang/OTP framework has then extended this grammar to define a module and relationships between modules.petitparser_diameter

I started by converting the BNF into a PetitParser grammar. Which means each rule of the grammar becomes a method in the parser class, then one can create a unit test for this method and test the rule. To build a complete parser the rules are being combined (and, or, min, max, star, plus, etc.) with each other. One nice tool to help with debugging and testing the parser is the PetitParser Browser. It is pictured above and it can visualize the rule, show how rules are combined with each other, generate an example based on the grammar and can partially parse a message and provide debug hints (e.g. ‘::=’ was expected as next token).

After having written the grammar I tried to parse the RFC example and it didn’t work. The sad truth is that while the issue was known in RFC 3588, it has not been fixed. I created another errata item and let’s see when and if it is being picked up in future revisions of the base protocol.

The next step is to convert the grammar into a module. I will progress as time permits and contributions are more than welcome.

Build or buy a GSM HLR? Is there an alternative?

Build or buy a GSM HLR? Is there an alternative?

The classic question in IT is to buy something existing or to build it from scratch. When wanting to buy an off the shelves HLR (that actually works) in most cases the customer will end up in a vendor lock-in:

  • The vendor might enforce to run on a hardware sold by your vendor. This might just be a dell box with a custom front, or really custom hardware in a custom chasis or even requiring you to put an entire rack. Either way you are trapped to a single supplier.
  • It might come with a yearly license (or support fee) and on top of that might be dongled so after a reboot, the service might not start as the new license key has not been copied.
  • The system might not export a configuration interface for what you want. Specially small MVNOs might have specific needs for roaming steering, multi IMSI and you will be sure to pay a premium for these features (even if they are off the shelves extensions).
  • There might be a design flaw in the protocol and you would like to mitigate but the vendor will try to charge a premium from you because the vendor can.
The alternative is to build a component from scratch and the initial progress will be great as the technology is more manageable than many years ago. You will test against the live SS7 network, maybe even encode messages by hand and things appear to work but only then the fun will start. How big is your test suite? Do you have tests for ITU Q787? How will you do load-balancing, database failover? How do you track failures and performance? For many engineering companies this is a bit over their head (one needs to know GSM MAP, need to know ITU SCCP, SIGTRAN, ASN1, TCAP).
But there is a third way and it is available today. Look for a Free Software HLR and give it a try. Check which features are missing and which you want and develop them yourself or ask a company like sysmocom to implement them for you. Once you move the system into production maybe find a support agreement that allows the company to continuously improve the software and responds to you quickly. The benefits for anyone looking for a HLR are obvious:
  • You can run the component on any Linux/FreeBSD system. On physical hardware, on virtualized hardware, together with other services, not with other services. You decide.
  • The software will always be yours. Once you have a running system, there will be nothing (besides time_t overflowing) that has been designed to fail (no license key expires)
  • Independence of a single supplier. You can build a local team to maintain the software, you can find another supplier to maintain it.
  • Built for change. Having access to the source code enables you to modify it, with a Free Software license you are allowed to run your modified versions as well.
The only danger is to make sure to not fall in the OpenCore trap surrounded by many OpenSource projects. Make sure that all you need is available in source and allows you to run modified copies.
Speeding up my SIP/MGCP Smalltalk Parsers

Speeding up my SIP/MGCP Smalltalk Parsers

When creating the MGCP and SIP implementation I didn’t want to do string splitting/scanning myself but follow the grammar of the two RFCs. I decided to use the PetitParser framework. PetitParser is a parsing
combinator. Which mostly mean you create small parsers and combine them with things like a sequence (parse this, than that and expect the input to be fully consumed).

For a long time this approach was just okay but I recently started to use the code on a under powered ARM system and the performance started to hurt. The MGCP/SIP Parser is created at runtime and then the result will be used.

Measuring

Besides things being too slow it is important to know how slow it is and to see if a change has made a difference or not. GNU Smalltalk has a Time class that provides a monotonic nanosecond clock. The easiest way to benchmark is to use an approach like:
 [ | start | start := Time nanosecondClock. benchmarkCode. start – Time nanosecondClock] value.

This will give me a number. Now with a language like Smalltalk there will be extra runtime code so there will be a noticeable variance.

Improving the PetitParser GNU Smalltalk port

PetitParser needs to use backtracking to go back in the stream when one element of a choice could not be parsed. This means PetitParser will heavily exercise >>#position, >>#position: and >>#atEnd. In Pharo the implementation has a position variable but in GNU Smalltalk the implementation has a pointer that points to the next character. When a method is called in Smalltalk an activation record (stack frame, context) needs to be created. GNU Smalltalk has the optimization to detect methods that simply return a value or instance variable. In the case of >>#position and >>#position: an arithmetic operation will be executed and can not be optimized. I have decided to change the PetitParser code to use >>#pointer and >>#pointer: to avoid the extra activation cost.

Local PetitParser hacks

In my older port of PetitParser.209 the PPRepeatingParser class will store the parsed elements in an OrderedCollection and at the end convert it to an Array. My client code will simply iterate over the
result and I don’t need to pay the price of the extra memory collection and copying.
With Smalltalk and open classes I can simply load my own/improved/reduced version of the >>#parseOn:
and benefit for the speed-gain in my implementation.

Speeding up parser construction

In PetitParser one defines the parser in selectors and on creating everything from the >>#start will be turned into the parser. In case some basic parsers are re-used one can create instance variables and the PetitParser baseclass will then cache the result in that variable. In the past I haven’t used this mode too much but I had a lot of parts that are used more than once. This was a significant speed-up in parser construction. At least in PetitParser.209 (I think later versions had some improvements) not every caching will make things more quick. It is good to benchmark both number of created parsers and number construction speed.

Simplifying the Grammar

In both MGCP and SIP I have a SIPGrammar/MGCPGrammar class and then a subclass called SIPParser/MGCPParser that creates a structure my client code can work with. When creating the SIPGrammar/MGCPGrammar I followed the RFC BNF but e.g. for matching the possible parameters I had to create a choice parser with many many choices but all of them are in the form of “Key: Value”. So instead of checking the valid keys with the Grammar, I should check the structure in the grammar and do the key validation inside the parser class.

Using PetitParser instead of following the Grammar

In the case of SIP there a lot of rules that specify where whitespace can be. This means I created PPSequenceParsers where some elements consume the space that nobody cares about it. The PPParser class already knows the >>#trim selector to deal with such things. It will automatically take away leading and trailing whitespace.
In SIP (e.g. with the challenge BNF) there is one mandatory parameter followed by optional elements separated by comas. PetitParser has a built-in way to express such things. The >>#separatedBy: selector will take a parser (e.g. one that can parse a comma) as parameter and then parse one or more occurrences.

Using PPObjectPredicateParser

In PetitParser one can write #digit asParser / #word asParser and this will either parse a single digit or a single character. In both cases a PPObjectPredicateParser with a PPCharsetPredicate (with an internal look-up table) will be created. I had many of such occurrences and could simplify the code.

Creating a custom parser

In SIP some parameters can be of the nature of key=value or key=”VALUE”. The later is a quoted-string that permits certain characters and certain escape sequences. The rule to parse this were nested character parsers of choices of choices. The resulting parser was very slow. A simple string to parse a nonce with a quoted-string could take 40ms. I decided to write my own SIPQuotedStringParser.

Summary

The MGCPParser construction time was in the range of 20 seconds, after the change we are in the ballpark of a second or such and the situation was similar for the SIPParser. A testcase to parse a 401 SIP message went from 200ms to 70ms. This is on a slow ARM with the plain interpreter and there should still be some room for improvements.

Outlook

I think PetitParser could have some further optimizations. Instead of the PPCharSetPredicate and the PPObjectPredicateParser there could be a PPCharSetPredicateParser that avoids the call to >>#value:. and one creating two PPCharSetPredicateParsers and joining them with a PPChoiceParser one could simply join the two look-up tables. This would optimize #digit asParser / #blank asParser and save one LookupTable. The next thing would be to create sparse PPCharSetPredicate when one knows that only a subset will be filled.
In terms of GNU Smalltalk we need to port to GNU lightning 2.0 and we would gain JIT support for ARM and can take it from there as well. Another option would be to start having ByteCode to ByteCode optimizations like inlining often called methods (with and without OSR).
Last but not least we could use MrGwen’s GNU lightning bindings and JIT a parser from the PetitParser representation.
Ported DBI and DBD-PostgreSQL from GNU Smalltalk to Pharo

Ported DBI and DBD-PostgreSQL from GNU Smalltalk to Pharo

We are building a gsmSCF in Pharo. Normally I would use MongoDB and Voyage to store my objects but as a gsmSCF deals with the balance of users I wanted to use something that supports transactions.

Looking at the SQL support in Pharo. I found SqueakDBX, DBXTalk, native Postgres drivers but all of them lacked prepared statement support and one really doesn’t want to format queries by hand.
This is when I decided to port GNU Smalltalks DBI framework and the DBD-PostgreSQL driver. It is not the greatest database framework (QtSql is really close to that) but it does support prepared statements and I have running systems that use this framework.
I have used the gst-convert utility to convert the syntax from GST to FileOut syntax as understood by Pharo and converted the use of DateTime to DateAndTime, Dictionary>>#from: to Dictionary>>#newFrom: and some other incompatibilities. I have fixed some other incompatibilities by hand, e.g. >>#subString: aChar changed to >>#subString: aString and >>#nl to >>#cr. This gave me a port of ROE, DBI and DBD-PostgreSQL relatively quickly. I had some issues with Monticello not properly detecting certain extensions and I had to re-save the methodSource to fix that.
I spent the most time porting the DBD-PostgreSQL FFI code from GNU Smalltalk to NativeBoost of Pharo. The easy part could be converted easily. The literal array in Pharo is interesting and the usage in the >>#nbCallOut: is quite funny. The most difficult part was to create a char** for one call-out and type conversation. I had a de-tour using the NBExternalArrayType and then went to use NativeBoost class>>#allocate: and >>#asNBExternalString. It didn’t help that when NativeBoost can’t convert a parameter and there will be use-less error message. This is specially annoying when a function takes like 6 arguments or more. Anyway, I had some success yesterday night.
In GNU Smalltalk we are using namespaces so we have DBI.Connection as the entry point. In Pharo this is Connection right now but I should call it DBIConnection in the future. A small example is below and with simple tests it looks like there are no memory leaks in the code. I need to make the code work reliable across image save/restore.

 | connection statement result |
connection := Connection connect: ‘dbi:PostgreSQL:dbname=aDb’ user: nil password: nil.
statement := connection prepare: ‘SELECT * from table WHERE col > $1 AND type = $3’.
result := statement executeWithAll: #(1 ‘abc’).
[result atEnd] whileFalse: [
   | row |
   row := result next.
   row at: ‘columnName’.
].

GNU Smalltalk deployment with images and image resumption

GNU Smalltalk deployment with images and image resumption

Once up on a time I was sitting in a cold hall at the Barcelona exhibition ground, a power outage has taken down several DVB-H platforms (racks consisting of servers, streamers, RF equipment…) and once power was restored red LEDs were blinking, systems not coming up automatically, hordes of engineers trying to boot the right kernel, trying to remember the multicast routes they had typed in by hand, chaos, hectic. It was interesting to witness that as we could lay back, continue with our work, as our platform was configured to come up automatically and it did.

One has to assume that stuff breaks, software, disk, RAM, CPU, power supply, anything. The only answer to that is be prepared, build your software to deal with failure (or nicely try to restart), make sure all systems start automatically, have backups, have a plan.

For my GSM (telephony) in Smalltalk components I didn’t do that yet, mostly because my code was not very mature, I was still building up the trust relationship and so my deployment consisted of manually running GNU screen, then GNU Smalltalk, manually loading my code, starting things up. It was about time to correct that.

The tool of choice is the gst-remote application, it will run a GNU Smalltalk image, it can daemonize and it will listen for input from other systems. While moving to this deployment my fellow Smalltalkers were kind enough to fix some bugs on the way. There was an issue of gst-remote exiting when saving an image, the Delay class not functioning properly on resume, the wrong FileDescriptors being closed on image resumption. It is very nice to get help that fast!

My deployment now consists of building GNU Smalltalk packages, having a Smalltalk script to load the package and call start method of that package followed by taking a snapshot and exiting. I can then resume the image and if the software is prepared to deal with image resumption it will work.

Eval [
| name image |
name := Smalltalk arguments first.
image := Smalltalk arguments second.

(PackageLoader packageAt: name)
fileIn;
start.
ObjectMemory
snapshot: image;
quit.
]

Going down the Sangoma rabbit hole

Going down the Sangoma rabbit hole

I ended up needing to configure FreeSWITCH to use a Sangoma ISDN card. At first we were foolish and used the setup script to install libraries into our system.

Cleaning up

It took dpkg -S and manual filtering to clean this mess. On a debian system there are files in /usr/ that do not belong to packages, these can be links to /etc/alternatives, or in case of python be bytecode files (*.pyc).

Second attempt with building packages

The next attempt was to use checkinstall -D make install but the source distribution of wanpipe is such a mess that checkinstall refuses to work, I have created a handmade debian package with some split up. The most annoying issues are the installation of .svn, .deps, .libs directories, using /etc/wanpipe for examples and firmware (they belong somewhere else), also installing the package requires to overwrite files of other packages (Linux comes with a wanrouter.ko too). There also appears to be some fishy licensing part, the source distribution is shipping with some .o files, the module claims to be GPL though.

First attempt with FreeSWITCH

FreeSWITCH allows to build debian packages out of the box, and it worked, it also built the freetdm module. But my module was missing support for wanpipe, adding the wanpipe-dev, rebuilding fixed the problem.

Second attempt with FreeSWITCH

After configuring the ISDN card using the wancfg_fs utility (and not allowing it to mess with the free switch config, as it is breaking it), I saw the cards but I was missing a library to actually use the card for Voice Calls.

ISDN libraries

I needed to get the libsng-isdn library, of course without sourcecode, also without a LICENSE file at all. Who may use this library? Under which circumstances? I started with checkinstall but this does not handle shlib providers very well, so I had to package the library myself

Third attempt with FreeSWITCH

I was able to recompile the FreeTDM module, and I could configure the ISDN trunk and use the the fs_cli to orginate a call.

Summary

FreeSWITCH and FreeTDM are quite powerful, also having support for creating debian packages is great too. The Sangoma experience is less exciting.

OsmoST SIP (SIP in Smalltalk)

OsmoST SIP (SIP in Smalltalk)

Intro

The last couple of days, to some degree weeks I was implementing a SIP stack for GNU Smalltalk to be used in my Smalltalk GSM project. The first time I encountered SIP was around 2001 when we were excited to run a SIP Phone on our Linux powered iPAQs (Linphone on Opie).

For Smalltalk I started with looking at SipStack by Frank Shearar (who was very responsive to questions regarding his code and SIP in general). The main problem was that his stack was incomplete and as usual it is difficult to pick things up without knowing SIP and not knowing where the code was heading.

Grammar

I began with mapping the BNF SIP Grammar to rules for the PetitParser parsing framework. I have shortened some rules (e.g. because the generic rule is a catch-all one and I don’t need the specialized form yet).

Complexity

The next big task was to deal with the concepts of a SIP Dialog, a SIP Session, a SIP Transaction and getting the relationship of these entities right. E.g. to place a call one creates an unconfirmed Dialog, prepares the INVITE Transaction and passes it to the transport layer. During this transaction one can get a reply that leads to a confirmation of the dialog. In fact one can end up with multiple confirmed dialogs (called forking, the target phones start ringing in multiple places and can be picked up multiple times as well). To make matters worse some answers need to be sent sent within the same transaction, some open a new one. My code seems to work now but there are some things I don’t do properly (e.g. routing). I have documented this and if I have a need for those I will address them.

Processes

In the last couple of years I was mostly dealing with a single select IO model, coming to Smalltalk makes me use Processes to read and write from sockets. My basic model is to have a RX Process to use Socket>>#next and a TX process that will do SharedQueue>>#next and then Socket>>#nextPutAll. But with having multiple processes one is required to do proper locking, e.g. have a simple locking hierarchy (first pick the lock of the CallAgent, then the lock of a Dialog/Session). This all works nicely until I reached the point that I have two sub systems talking to each other.

For setting up a call I might have one leg coming from GSM and use the MGCP Protocol, the other leg might be SIP. To hang-up I will need to inform the other leg of the call about it, but this means I will need to have a locking hierarchy that first takes all locks of the left leg of a call, then all locks of the right one which in turns means me I need to dispatch messages without any locks being held. Now besides having RX from maybe a couple of different Processes, I also deal with timeouts that will come from different Processes as well.

Central dispatch

To put an end to this (and the complications in the code) I created a simple queue that will BlockClosure>>#value a block from a central dispatch process, this way I have serialized everything and can mostly ignore locking. I hope that Smalltalk VMs will have a nice M:N Process model in the future and that my code will evolve into having different dispatchers running on these N native processes.

Using the code

st> PackageLoader fileInPackage: #OsmoSIP.
st> transport :=SIPUdpTransport startOn: ‘172.16.254.55’ port: 5061.
st> useragent := SIPUserAgent createOn: transport.
st> transport start. “Starts the RX/TX process”

st> call := SIPCall fromUser: ‘sip:1000@on-waves.com’
host: ‘172.16.1.72’ port: 5060 to: ‘sip:9198@172.16.1.72’ on: useragent.
st> call createCall: ‘SomeSDP file’.
st> call cancel. “E.g. before it is connected”
st> call hangup. “E.g. when the call is connected.”
st> call terminate. “End the call somehow..”

GSM in Smalltalk – a GSM Toolkit

GSM in Smalltalk – a GSM Toolkit

I started to play with smalltalk somewhere in February, more specific with the GNU Smalltalk implementation. Like it is with any new language and class library it takes a while to get productive and it took me until somewhere the last month where I finally started to do GSM handling in Smalltalk and thanks to laf0rge the code is now in a public repository and hosted along the other Osmocom projects. You can see all the subprojects over here.

So far I can create and parse the ipaccess IPA protocol, SCCP, BSSAP, BSSMAP and messages for Call Control and Mobility Management of GSM48. I also have a small Web-UI to connect to a MSC and then do a Location Updating Request and a Mobile Originated Call for a configurable IMSI (you will need to know your AuKey for that). It is very easy to parse and create the whole stack of messages, adding new GSM48 messages is easy as well (i still consider writing a program to get this directly from the GSM48 PDF).

This code is probably very well suited to do load tests of a MSC, e.g. find out how many LUs it can do a second, also to play easily with various quirks with it. Thanks to the nature of smalltalk one can easily change the creation of messages at runtime without needing to reconnect to the MSC. The next thing would probably to use it for fuzzing a MSC…

One note of caution, it is my first Smalltalk project and there are probably plenty of things I do wrong, I already know some of them and will work on refactoring it to a better structure. The next stop is nice ASN1 support, I will base it on the LDAP work.