I was suggested to write a blog post about my experience here at the European Parliament, inside the WebComm unit and especially about the development of the Facebook Chat Application we just finished.

I don’t feel terribly confident about writing non-scientific content, especially with all those talented editors around me, but I like challenges and I thought it might be an interesting experiment.

So where to start? First of all let’s break some myths here. The European institutions and especially the European Parliament are not only composed of old-lazy-boring-suit-wearing guys. I mean I only discovered a small part of the European Parliament, which is the WebComm unit, part of the DG Comm. and all I could see where young, motivated, dynamic people. This is quite the opposite of my preconceived ideas and I really like to be surprised. Of course we have to face a little bit of administrative slowness for some things but in general I’m able to do my job in a smart, agile and constructive way. So that’s done.

About the Facebook chat application. I can say it has been quite a challenge for mostly 3 reasons :

1. Real-time web application
2. Scalability
3. Agility needed

1) Real-time web application

Maybe it’s time to do a short (and incomplete) history of web development

Traditional web applications are not designed to be “real-time”. By “real-time” I mean that all the users connected to the application must be signaled all the changes as soon as they are made (in real-time) and not only when they refresh the page. Maybe it’s time to do a short (and incomplete) history of web development :

1) Static pages: at first they were static html pages. A guy would write HTML* code in some text files, save those in some directory. Other people could then connect to www.someserver.com, their browser would then start from the index.html file located at the root of the web directory, parse it and display it. The user would typically click a link and would then load another html file eg. contacts.html and see the contacts page of the site.

This had many disadvantages :

1) HTML is not easy to write for anybody, so you need specialists for any page produced

2) You had to write html for each and every thing, if you wanted a gallery with 200 items, you would have 200 html files or 1 gigantic one.

3) Content was really static, meaning static text, borders and pictures**

Trying to ameliorate, server (scripting) technologies were developed. In fact you don’t have to write every line of html of what the user sees. You can write code that generates HTML files. Let say you have a e-shop, you don’t have to create an HTML page for every article, you create an article template, an administration site where men can enter information about the products, you then associate urls to different products like www.eshop.com/products/1 www.eshop.com/products/2 would return an HTML file generated “on the fly” by the server using the template and filing in data from the administration site.

This illustrates Mathieu's point (or so we are told)

But server technologies couldn’t solve entirely the third point (really static content), Content could be generated by the server and customized for the user, it wouldn’t make the user’s experience much richer as the user was still seeing a static HTML file and clicking to the next static text/image page. So “they” came up with javascript which was a way to modify what the user see with scripts. It changed web programming forever because you could now write logic that would execute on users actions to the page.

For example you could have a button that would become red when the mouse was over it, or a panel that would open when clicked. Well I’m digressing a little bit, let’s hurry up. After that they invented “ajax”*** so that the browser could contact the server using javascript and fetch new content to refresh your page, so you didn’t have to “navigate” to another page on every action that required interaction with the server (like viewing other items or submitting a form).

Nevertheless all these evolutions lead to the webpages you see now, well that’s not entirely true, because javascript could make the pages more dynamic but it couldn’t make them more “multimedia”, and the users wanted videos and sounds, that’s were macromedia jumped in and came up with flash, that was later bought by adobe and that we still use to watch videos and sounds on internet, recently this changed because we see a trend to come back to pure html/javascript with the coming of html5 and it’s video and audio tags as well as css3 for the animations.

2°) Scability

To explain my point, all these evolutions didn’t solve my problem, because as I said, a Chat must be a “realtime” application and all this javascript and servers stuff cannot do real time applications, why would you ask me ? Because in this standard model content is send to the clients/users only when the user click somewhere or do something, at this precise moment a connection is made to the server, which responds with required content. But in a multiuser chat, when somebody sends a message, all the other users of the chat must receive this message ASAP.

A solution could be to have each client connect the server every second or so to ask if there is a new message. The problem with this and that’s where we get to the second difficulty, is that “it doesn’t scale”! First let’s explain the term : “It doesn’t scale” is a shortcut for something like : it works for 5 users but it doesn’t scale up to 10000 users.

Why this method of having each client ping the server every second doesn’t scale ? Because if you have 10000 clients connected, that makes you 10000 connections by second to the server just for the “realtime” features, meaning that it doesn’t event include the users loading the page, interacting, liking and sending message and you get those 10000 connections per second even if nothing happen during this time.

After a lot of experimentation and researching I chose the following setup: nginx + uwsgi + python/django + APE =)

The solution needs to be able to send messages to the clients as soon as (and only when) a new message arrives. That’s were “server push” arrived, it’s actually an umbrella term to describe the possibility of sending data from the server to the client when new data arrives and not when the user refresh. Real push is called “http streaming” and is not supported by older browsers aka IE. So tweakers came up with a solution called “long-polling” the browser opens a connection to the server, the server wait to answer, the connections is kept open, when fresh data arrives, the server answer like he should have directly when the browser receives data, it handle it and then reopen a “long-polling” connection to the server. You can visualize all this in the Figure 1.

After a lot of experimentation and researching I chose the following setup:

nginx + uwsgi + python/django + APE =)

What are all these geeky terms ?

Nginx is a next gen webserver that’s capable of handling a ridiculous huge amount of connection on ridiculously small hardware. It is in my opinion a huge concurrent to the fat apache out there.

Uwsgi is a “fast, self-healing and developer/sysadmin-friendly application container server coded in pure C” and it executes python code that runs the django framework.

Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. What this all means is that you can be a lot more productive with python than with other low-level compiled languages (because high level means you don’t care about the low level stupid stuff, and interpreted means that you don’t have to recompile the program after every change, it just runs directly)

Django is “The web framework for perfectionists with deadlines” or a high-level Python Web framework that encourages rapid development and clean, pragmatic design.

APE is Ajax Push Engine, one of the more mature libraries out there for doing server push, it can be configured to user different transport and is also written in C (read : it’s really fast)

So all the business logic and serverside was developed in django.

Real-times updates are sent from django to APE which send them to all connected users and then there is a lot of HTML/CSS/Javascript/Jquery to hold it together in the browser :)

So now we have enough “nerderies” I think. Feel free to contact me if you have some technical questions.

3°- Agility

That’s were I get to my last point : Agile development. We didn’t want to come up with the “wonderfullest chat on paper” that would turn unusable in practice. So we progressively developed features and tested it among us to feel and touch the thing, changing it multiple times. And that’s what agile is all about : small, realistic iterations that are validated, or in other words : if you do a lot of small steps you can only do some small steps in the wrong direction, but if you try to do a huge step, you can end-up doing a huge step in the wrong direction and lose a lot of time and money.

Agile thus equals programmers not programming wrong, but also not getting crazy during long periods alone in their desks. And small iterations, means frequent tests and meetings, and that’s where we get to the interesting part of all:

Humans working together to share with other humans their dream: democracy and freedom!

As a bonus, here is a video showing the coding process of our facebook’s chat application.

Visualization of the EP-FB chat source code evolution. from Mathieu VDH on Vimeo.

Cheers,

Your fellow Facebook developer.

* imagine HTML as language to layout text, images and frames on any screen

** picture were introducted in 1994

*** Asynchronous Javascript And XML

I’ve known for a long time that our video streams don’t comply with Macs. It’s a subject we discuss about almost every two months since I started working here. But today, I wanted to understand why. So, like Magnum P.I., I investigated. In order to explain the situation, I’ll have to enter the technical world, hence the caveats below.

CAVEAT: I don’t understand half of what I am writing below. I am NOT an IT guy. Please, please, please don’t write me to bash me on mistakes: better propose a correction in the comments area, that’d be nice.

CAVEAT #02 : The following has been proofread and corrected by one of the genius geek downstairs who prefers to remain anonymous to avoid spam from the other geeks around him (they live in a kind of tribe or something). Still, if something sounds wrong, blame me, not him.

I’ll use the Toyota Five Why method, it sounds cool.

1°- Why Mac users can’t play the hearings’ video streams properly?

Well, they can but they’ll get all audio streams at once. It sounds like the Tower of Babel after its fall. The reason is our video streams are encoded in Windows Media Video format (.wmv) while Macs better work with MPEG4 format. In order to correctly select the video with their language of choice, all users (including PC users) need the latest version of Explorer or Firefox and the latest version of Windows Media Player.

That’s right: to play a video on Internet, you need two things. A browser (Safari, Explorer etc.) and a player (Quicktime, Flash player, Windows Media…). Macs can play .wmv files all right (which is why Macs users can still benefit from our Hearings Video on demands) but they are less efficient in coping with .wmv live streams which include multiple audio files.

I am telling you it works on my PC

2°- Why don’t we encode in MPEG4?

In order to turn an event into a digital movie, you need hell of a lot of stuff (cameras, micros, robotic cameras etc.). To produce the actual file you want to stream, you need video encoding cards. All our audiovisual gear encode live streams in .wmv. We encode the Video on Demand files of the Plenary session in both: in .wmv and in .mpeg4. We don’t have the technical capacity to encode the live streams in both formats at the same time because we don’t have enough encoding video cards. The choice has been made to encode live streams in .wmv.

3°- Why did we chose to encode in .wmv in the first place ?

When we started to broadcast the Plenary session live in video, the .wmv format was the only format that allowed multilanguage url. Multilanguage url allows you to associate one video stream with different audio streams. You don’t have to duplicate the video file, you associate it with a selected audio stream from many available. MPEG4, I am told, was not good at that but it became better lately. At that time, PC users were the vast majority. They still are. Except for all those creative people and journalists who insist on following the Hearings, dammit ;-)

Linux people are extremely good at finding things out by themselves. They know how to play any kind of video streams. They’re super-geeks, you know

4°- Why don’t you use a Flash player like YouTube and all p0rn websites?

This is a good one. Flash video player have become extremely popular on Internet lately and they can work with all video streams source (.wmv or .mpeg4). Users just need the latest version of Flash on their computers and everyone is happy. Flash players are used by europarltv (except for the live streaming).

But. In order to stream a video, you need a transport protocol. We use the protocol rtsp. This protocol doesnt go well in Flash player, mind you, since Adobe (the owner of Flash technology) prefers one uses the rtmp protocol, which belongs to them. If we would move to rtmp protocol, we’d have to buy a lot of licenses. The solution is currently scrutinized. Also, we would use a closed transport protocol, “closed” opposing here the “open source” philosophy.

5- Why don’t you adress Linux users?

My internal IT experts said: “Linux people are extremely good at finding things out by themselves. They know how to play any kind of video streams. They’re super-geeks, you know”. But, to be fair, he also said: “Non-geek Linux users would be lost if we were to chose complicated to set up players, codex, plud-ins and so on.” Hence his taste for a Flash+MPEG4 solution.

Bonus question: will you adress this problem before the next century?

Yes, we will. So far, four solutions are possible.

- We invest in more machines and encoding video cards so we can simultaneously encode our live events in as many video formats as possible, or abandon the .wmv. In all cases, we should be able to propose MPEG4 for live streamings and video on demand. MPEG4 is a pre-condition for almost all possible solutions. Of course, the best amongst you will suggest we use SilverLight (Flash à la Microsoft) which can use a .wmv stream in a SilverLight player. But few have tried. It would cost a lot in research and analysis. And it would be also a proprietary format.

- We buy enough rtmp licenses and we develop Flash players embedded on our website ;

- We develop Flash players embedded on our website able to read directly our MPEG-4 stream via rtsp protocol ;

- We all move to html 5 wich proposes a new “player video” tag which transfers the video playing’s responsibility to the the user browser and not to the video player anymore.

I hope the last sentence is correct because you lost me somewhere around the protocol of transport thing.

The different EP IT teams are working on it. And we will certainly impose easy live streaming for all as a pre-condition of our future new website.