Erlang is a language designed for building highly concurrent, highly available systems. And, based on our experience using it to build a large-scale VoIP soft-switch, it does a great job of this.

However…every language has a collection of “gotchas” which developers often don’t discover until they’re neck-deep in building a big system. If they’re really lucky, they’ll discover them before the users do.

I love Erlang as a language, but in this respect it’s no different from any others. This talk aims to briefly cover the things it would have been nice to have known before we ran head-long into them such as:

• How to crash your VM (and how not to)
• Message queues: they’re not magic
• Why you can’t just run Erlang as a Unix-style service (and how you can)
• Hot Code Loading: not as easy as you might hope
• The OTP: Forget everything they taught you in the first Erlang lesson
• System monitoring is even more important than you think

This workshop will examine some of the challenges posed by having to work with a large legacy codebase while attempting to leverage more modern, powerful and appropriate languages and ways to overcome them.

The problem:

• Your existing C/C++ code is not Erlang (or Haskell or any of the good languages).
• Your existing code assumes all the other code it directly talks to is in the same un-good language.
• You really want to use a good language, but your manager won't let you because of the above two.

At a minimum, you'll need marshalling / de-marshalling to be able to communicate between old and new code. But creating this often results in a maintenance nightmare where any time you want to change a structure, you need to change its description in each language specific declaration as well as the (de)marshalling code.

This workshop will build up, from the basics, a system that will automatically generate language specific structure declarations as well as marshalling code, all generated from a single canonical source, leaving you only one place to edit when you need to change the structure.

Everyone loves to start a project with a clean slate, but the reality that very often we end up being asked to maintain and extend existing systems that were written by people who hadn't realised that Erlang is the correct solution for every problem we're ever presented with. Our company's VoIP softswitch was no exception.

Having realised the mistake of the previous designers, the clear solution was to rewrite everything in Erlang. This was both self-evidently sensible and, for some reason, also utterly unacceptable to management who couldn't understand why we thought there was any problem with the current system.

So instead we decided that we'd just write all the new parts in Erlang, and rewrite other parts in a piecemeal manner as opportunities arose.

Of course, it turns out that that's easier said than done. The C++ code was written with the (initially well-founded) assumption that it would all be talking to other C++ code in a homogeneous environment, from the compilers all the way down to the hardware. This allowed the developers to take a number of “clever” shortcuts, all of which made plugging in the new Erlang parts several orders of magnitude more difficult than it would have been if some language-agnostic IPC system had been used.

The solution we developed, named Yarn, is a code generator implemented in Erlang using erlydtl and yecc. It allows us to specify, in a single canonical location, data schemas and structures that are used across multiple languages. All the declaration and marshalling code is then created automatically as part of the build. After success with Erlang and C++, it has now extended to include generation of Python, Java and SQL.

This talk will examine the technical problems that lead us down the road to Yarn, the details of its implementation, and the many “gotchas” we hit in the process of building it.

Few technical metrics are more closely watched in the world of online service delivery than a system's “uptime”. Response speed, number of concurrent users and data throughput are all forgotten about very quickly when users can't even get onto your system.

The Erlang language was designed with fault tolerance as a core principle. Fault tolerance describes the ability of a system to keep running in the event of some kind of unexpected problem, without external assistance (in other words, without waking up your sysadmins) and with minimal impact to users. Consider a single monolithic C++ program, handling 10,000 concurrent users all with persistent TCP connections. If one user does something unusual and hits a poorly tested code path, causing a segfault, then 10,000 people see their connection drop. That's (obviously) not fault tolerant. If just that one user's connection dies and everyone else's carries on, that's fault tolerance (and also far preferable!).

While Erlang's basic features give a degree of fault tolerance, they're not a silver bullet. The share-nothing memory model and ability to separate your system into thousands of isolated threads are extremely useful when building robust systems, but they're a foundation rather than the whole solution.

The other killer for uptime is rolling out upgrades and fixes. Without some clever infrastructure, sooner or later your customers will see “We're down for scheduled maintenance – come back in 30 minutes”. One solution to this is “hot upgrades” whereby code can be upgraded on the fly with literally no downtime or interruption to ongoing services.

This tutorial covers Erlang's Open Telephony Platform (which has almost nothing to do with telephony per se) and how it delivers on Erlang's promise of robust, fault tolerant, hot-upgradable software.