XING is fully available 99.85% of the time.

It happens pretty rarely, but sometimes our platform is affected too by operational faults and malfunctions. Of course we try to keep these hiccups to an absolute minimum. Our content is stored and supplied from two different data processing centers for instance, which means that our users can be re-routed from one to the other dependently on demand and performance.  All systems have a failover cluster operation guaranteeing high performance and stability, as well as having sufficient reserves to cope better with peak loads. Making sure our platform is both available and fast is our number one priority.

Sometimes though, like last Wednesday’s morning for instance, we do encounter unforeseen problems. There isn’t an IT system anywhere that is 100% fault-tolerant – there will always be systematic single points of failure that will send part or all of the entire system crashing with them. In this latest case a defect network component corrupted a mechanism, which was specifically required to safeguard against failure (Spanning Tree Protocol). We will be analyzing and intercepting this error to make sure it doesn’t crop up again.

A look at the figures for the past twelve months shows that we meet our high targets, with an average uptime ratio of 99.95%. In other words, this means that XING can’t have been offline for more than four hours and 2 minutes over the entire year. Or to put it yet another way: This is equivalent to around 44 seconds a day. Crashes affecting just certain parts of the platform have also been included in this statistic.

Here are the availability figures for the past twelve months:

To form a meaningful graph from these figures the vertical axis needs to be scaled up dramatically in order to see any fluctuations at all:

Chart: XING availability during the last twelve months.

Despite this record, every minute of downtime is a blow to us – as we know how important it is that our members can send messages, make contacts and network at any time. That’s why we’ll keep doing everything in our power to maximize stability and availability of our platform.

Susanne Reppin (left) and Dr. Johannes Mainusch on the wjax stage.

Just how agile is XING? How agilely do we feel? At this year’s w-jax 09, Johannes Mainusch and myself talked openly about the XING culture, what kind of people we are, what we’re aiming for and what’s important to us. XING releases weekly updates on the platform, innovating therefore all the time during ongoing operation of the platform. We don’t want to have to put a halt to anything, and not when it comes to agile software development either.

XING already develops projects effectively using Scrum (see also Scrum @ XING – a case study). Alongside this, we’ve been working for nine months to put Kanban in place for one our maintenance teams – as both a methodology and philosophy. The initial idea for this came from Ralf Wirdemann, who’s an agile coach. Selecting Kanban fits in with our belief in agile and lean software development and to drive this forward in the future.

People took a great interest in our speech (here’s the slideshare and our PDF with notes, both in German). We explained why Scrum is so well-suited to our project work, as it protects the team from disruptions during the sprints – but that precisely this protection is contra-productive when applied to a team operating in close interplay with live operation of the platform. Kanban is considerably better suited to integrating disruptions – which bugs are after all – in an ad hoc way, and is therefore ideal for a maintenance team.

One of the major success factors here is of course the team itself. All team members were already experienced in agile development and wanted to learn more in a professional context. After about six weeks we were all surprised at how much more effectively and fast we could now go about our daily work.

Please note: The full talk held at the w-jax 2009 in Munich will be published in the next few weeks by w-jax.

The reason: A module we have developed ourselves that serves to load files in a browser once the page itself has already been loaded. This is useful because it means we don’t have to send all functionalities and graphics that might possibly be used through the network at once for every page accessed, which saves you a lot of time waiting for pages to load. If you then decide to use a feature or function that isn’t yet loaded – such as the photo upload in your profile, this small additional feature can then be loaded separately.

The browser feature that our module accesses posed a security loopholes in earlier versions of Internet Explorer. You can read more about this here: Microsoft Internet Explorer onreadystatechange Memory Corruption Vulnerability.

The good news is that Microsoft solved this problem in their own browser some time ago. (It might be worth you visiting windowsupdate.com to make sure your operating system and Internet Explorer are both fully up-to-date.)

If you use an outdated version of the virus scanner F-Secure though, which has not received an update since the removal of the security loophole in Internet Explorer, it will show you a false alarm if you use the Microsoft browser.

Björn Kaiser, one of our software experts, had this to say about the matter: “We did everything in our power here to make sure we avoided any major difficulties here. Even though it wasn’t a security breach at all, we still didn’t want to unsettle our users with nonsensical warning messages from a virus scanner that wasn’t updated. He spent three days trying to pinpoint the exact source of the problem, combing through 60,000 lines of Javascript code to locate the one line that triggers the warning message. One harmless line, lots of work! Thanks to my colleague Ingo as well for all the great debugging he did …”

Doing without onreadystatechange was out of the question. This would have meant that certain modules wouldn’t work at all anymore. The solution our developers came up with was relatively simple in comparison: Thanks to

var handler = ["onr", "eadystatec", "hange"].join(”");

the virus scanner no longer recognizes the supposedly bad “onreadystatechange“. The string function join enables the components to be brought together and implemented though – without triggering the false alarm.

There was of course no danger to your data or computer at any point in time here. We contacted the manufacturer too to make sure that this was in fact a case of a false alarm.

Having said this you should make it a top priority, in your own interests, to update your virus scanner to the latest standards. At the same time we’ll continue to do everything to ensure the most secure networking possible on our platform, without any unnecessary warning messages.

This works well – you define the requirements for the specific field (such as <input>) and Rails returns the page with an error message, also highlighting the corresponding field in red. Sounds good, but that’s actually where the problem is. Rails surrounds the field with a <div> that is styled via CSS.

This additional <div> may lead to CSS problems especially in pixel-precise layouts such as the XING html forms, it may break the HTML structure – and it will add completely unneeded DOM complexity.

Changing how Rails handles fields with errors

The technical details of how we solved it follow. We were not the first ones to encounter this problem, inquirylabs also talked about (and solved) it in their blog post Highlight background of fields with errors. The basic idea is rather simple: We rewrote the method Rails uses to handle form field errors to create cleaner html. The best solution we can think of is to add a class directly to the error elements, and that class has styling to border the error fields in red. No added DOM elements, no more broken structure, just clean code.

Here is the default rails html output

<div class="fieldWithErrors">
<input />Test</div>

Here is what our changes output

<input class="input-error" />Test

Luckily, Rails already provides a simple entry point for cleaning things up. In active_record_helper.rb in the actionpack gem, there is a class variable defined @@field_error_proc, which can be redefined to apply custom error handling on the html input fields. We’ve uploaded the code to github: http://github.com/xing/classy_field_errors.

It’s a plugin, so just:

• script/plugin install git://github.com/xing/classy_field_errors.git
• Put the css contained in field_error_proc.css in your base css file

The method we call inside the block, does exactly what we want. You pass it an element and a css class. It then applies that css class to input, textarea, or select elements, either adding the class to the existing classes or adding a class attribute to the element.

The method itself just calls the HtmlElement class method add_css_class which does what you’d think it does: It instantiates an HtmlElement object, adds the class passed into the elements and returns the changed html_element (which is really a string).

So, with the added class, we can do the same error highlighting, but without any added DOM complexity and with the intended semantic separation of HTML for content and CSS for styling.

Our solution produces the same basic results as inquirylabs, but also took care of 2 edge cases:

• it only adds the CSS class if the element doesn’t have it already, so in the rare case that an input field has two errors, the code remains clean
• the code also works with self-closing <input/> tags

Thanks to Ladislav Martincik, Sebastian Roebke, Mark Schuette and Hendrik Mans who all contributed to making this code highly efficient and clean.

To give you a very simplified example, imagine a form where the user can check which languages he or she speaks:

Let’s assume that every language option should be stored as a boolean value in a “users” database table, being either “true” or “false”, depending on which of the boxes are checked.

Technically, modern relational database systems such as MySQL can handle many columns per table just fine. However, adding new columns to a table while the database is on load is not that easy if the table already has millions of rows. This is a change that could take hours to complete, depending on the database system and data size. Additionally, there are limits of how many columns can be stored in a single table.

For quite some time, we have been using a nice technique to store boolean values of some large tables more efficiently as a single bit field. Using this approach, we can add new boolean attributes whenever we need to, without needing to alter the database tables, which would cause a downtime. For the part of our system that is based on the Ruby on Rails framework, we have solved this via our own plugin, that we are happy to share:

FlagShihTzu – A rails plugin to store a collection of boolean attributes in a single ActiveRecord column as a bit field.

We have uploaded the plugin with documentation on how it works. It was already covered in episode #5 of the Ruby5 podcast!

As some of our other XING open source projects, the code is hosted on GitHub, a great website to share and collaborate on open source software. So if you have ideas to improve the plugin, please contribute!

The logo from "IE6 must die!"

Oh, how we would love to shout: “IE6 must die!”. (Did you hear about the twitter movement? Or about this action from more than 70 websites?) Oh, how we would like to make our platform available in black and white to all Internet Explorer users (for the small proportion of the online world who remember the changeover from black and white TV to color…). But alas this is a helpless battle of the honorable Web Don Quixotes against the electronic windmills of the great multinationals of this world.

Why all this fluster you may ask? Here’s the reason:

Over the past few weeks we have been monitoring which browsers our members use to surf our site. I had to laugh a little when I read recently in a Web gazette that 11.7% of Germans still used IE6. I didn’t realize the figure could be pinpointed so precisely!

So here’s our precise data for the last seven days:

Overview of browsers used for XING.

This graph shows usage of the different major browsers against time shown on the horizontal axis. The green area at the bottom – representing usage of IE6 – is particularly interesting here. Its structure reveals three things immediately:

1. IE6 is not used very much at the weekend.
2. Chinese users, a high proportion of which use IE6, surf the platform during the European night.
3. During working hours on weekdays, members using IE6 exceeds the 20% mark.

So why is this? Well, because IE6 is so prevalent still in so many large corporations. I know from previous companies I have worked in how reluctant firms can be to switch browsers when the current one is connected up to anything from 50 to several hundred Internet applications, all of which have to work every single day. At tens of thousands of work stations throughout an organization.

It isn’t worth pestering users to switch either, as they very often don’t have administrative access to their work computer. And the administrators in large companies often don’t have the remit or the budget to carry out a changeover on all computers in a company. And if I was CEO of a large firm I think I’d probably be hesitant about initiating such an infrastructural project with uncertain risks in a completely heterogeneous environment.

To sum up: It seems that Internet Explorer 6 is (regrettably) here to stay, for as long as it still operates on current Windows operating systems at least.

… Come to think of it, XING in black & white might well have a certain retro, futuristic charm about it.

So, at the moment, you might have several iFrames on your start page. An elegant solution, except for the heavy load they put on your browser – the extreme case, Internet Explorer, requires up to 8 Megabyte of RAM for each rendered iFrame. Furthermore, as the front and profile page often extends vertically way beyond the viewport. But still we have to render the content in iFrames you might not even see.

Wait. Do we have to do that?

The answer is obviously no. This was one of the things that seriously bugged me, and I really wanted to come up with a solution. So, in the weekly four hours that each developer has for his own independent (of course: XING-related) research, I tried to develop a mechanism that:

(1)    Detects whether an application is in the viewport

(2)    Only renders and loads the iFrame if an empty place-holding element is visible

(3)    Performs the check (1) regularly to detect whether rendering is necessary now

My first solution was to initially render the iFrame and only set its “src” attribute when the element is visible.
Well, that caused problems in a particular browser – hint: Youtube will soon be phasing out support for it – who throws a mixed content warning on SSL encrypted pages if the “src” attribute is empty, “about:blank” or simply missing.
Besides that the iFrame is the most expensive html tag across all browsers in terms of rendering speed.

Therefore I decided to initially put an empty html element (eg. <div>) into the page, which gets replaced by the application iFrame as soon as it appears in the user’s viewport.

The last step was not that high on the challenge scale again:  The script checks whether the iFrame is visible every 200ms, and also on resizing the window and scrolling within it. Easy stuff.

The really yummy part: Not only do our customers and we profit enormously from this – we reduced the OpenSocial-induced traffic by 50%, but also for coders using the JavaScript framework Prototype. We’ve released the function as a plugin for Prototype on GitHub for you to download and embed in your projects. It adds a custom event, named “within:viewport”, which can be used like this:

loadIframe = function() {
  var placeholder, iframe;
 
  placeholder = this;
  iframe = new Element("iframe", {
    width: 315,
    height: 315,
    src: "http://iframe-url"
  });
  placeholder.replace(iframe);
};
 
// This is where the magic happens...
$("placeholder").observe("within:viewport", loadIframe);

The plugin can also be used for other expensive operations like for instance to load large graphics only when they are visible to the user.

Happy Hacking!

Why we chose Ferret initally

Ferret was introduced when we needed a search service for XING Jobs, which was the first part of our application using the Ruby on Rails framework. At that time, Ferret was just right. As it is written in super fast C code, it became  first choice when it comes to performance. But it is also partly written in Ruby, which allowed us to write the analyzer, query parser and tokenizer in our favorite language (oh the joy!). Ferret has served us very well until now.

But things have changed. Over the years, we have extended the Ferret-search to XING Events and Company Profiles, increasing our initial search volume by magnitudes. Furthermore, the search queries created by our application became far more complex, for example using several languages in just one indexed field. This lead to scaling problems, and our experiences using a DRb-based (think RMI for Ruby) remote interface for the search service turned out to be brittle and crash prone under certain circumstances, especially when multiple clients caused high load on the search servers. Another problem we faced was that, as a new developer it takes quite some time to become acquainted with Ferret and our implementations around it.

So on one of those occasions where we had to go into debugging mode deep within the DRb world, we really wished to have a simpler remote interface, maybe something like HTTP that could easily be scaled by standard means (think load balancers). On another occasion we encountered some serious memory leaks that had gone unnoticed under moderate load with simple queries. They had to be fixed by ourselves because the few official maintainers were busy with other projects or generally unavailable when we tried to reach them. Patches in their bug tracker were left untouched for months, and the official website is frequently unavailable.

We were left with a couple of choices: We could basically maintain ferret ourselves, find someone who would do it for us or find an alternative and switch. As you can imagine this ensured a lot of heated discussions within our engineering department, since in the end ferret was “kind of working”. As the frequency of problems increased, we chose to investigate other possibilities and technologies.

Ferret-ing out alternatives

One of the main lessons we learned while staying with ferret, and we learned it the hard way, was that a good search engine needs a good community of users and developers contributing. There was no such thing for Ferret. Of all the named arguments, this is the most important one. In general we think that when using an open source product, you better choose one that has a lively community around it. This guarantees that errors that were fixed by third parties will flow back into the main product, that you find experts to consult you around the world and that you can exchange knowledge on forums, wikis and conferences. So we started looking for a search service that:

1. Was easy to scale
2. Had a “developer-friendly” interface we could hook into
3. Was actively developed and supported

So we evaluated SOLR, which is based on the Lucene information retrieval library, just like Ferret. This means the syntax for our analyzers, query parsers and tokenizers do not differ so much from what we are used to. But unlike Ferret, SOLR is a real industry standard. There are lots of maintainers, bug reporters, wiki authors, consulting companies and conference panels, so in short: There is a profound ecosystem surrounding the SOLR search engine.

We carefully watched how long it takes until a SOLR maintainer reacts to an issue created in their JIRA bugtracker, and these guys are really fast and willing to solve all incoming bugs. They include patches written by third parties in the official sources, and release new versions of their software regularly. Furthermore, as SOLR is an industry standard, we already had developers in our team that have worked with it before. And, last but not least, SOLR works over a standard HTTP interface. On the one hand, this liberates us from awkward DRb code, on the other hand this allows painless scaling with standard HTTP load balancers. Furthermore, the acts_as_solr Rails plugin makes it feel like any other Ruby library.

While Ferret works with Ruby only, SOLR can be used with a plethora of programming languages. So it is no wonder that our fellow perl programmers have already made the switch to SOLR about a year ago. Employing only one search engine for all parts of our application will reduce the maintenance effort in the future. The shorter training period will ensure that all of our developers (even the ones trying to avoid DRb) will be familiar with our search technology. And, beyond that, SOLR offers far more features than Ferret, like faceted search, allowing us to implement a more powerful and user friendly search, for example through freely filtering and ordering search results by all available dimensions.

The upcoming migration

The introduction of SOLR follows a very smooth migration path. First, we cleaned up our own search interface for the Rails application, which is search engine agnostic now. Then we configured SOLR to mimic the behavior of our proven and tested Ferret search engine, to have a drop in replacement. Next, we will pick a beta group out of our 7 million members. This is what happens next, so you might be among them.

Search queries of this beta group will be run by SOLR instead of Ferret to compare the quality of the results. As soon as we are sure everything works as expected, we will extend the beta group to all users gradually. At this point in time, we are ready to develop new and awesome search features with SOLR technology under the hood.

Our start page after logging in got faster. It takes significantly less time until the user can interact with the application.

Before, the alignment of the start page boxes (e.g. ‘What’s new in my network’, ‘Visitors to my profile’) was calculated via JavaScript, a time-consuming task. Since drag and drop of the boxes always causes a reflow,  any new combination of boxes implied a re-calculation.

This was acceptable until the new feature boxes required the ability to self-adjust in dimension. The idle time while waiting for the content to render plus the calculation of the alignment could, depending on the browser’s JavaScript engine, easily have taken a few seconds or more, especially in Internet Explorer.

We now use the CSS 2.1 property display: inline-block for these boxes (and some hacks for Firefox 2 and Internet Explorer < 8 that do not support this property). Inline-block is a CSS-concept of blocks that form ‘lines’ (in CSS-terms: line-boxes), similar to what single words in a text do. Even if a word is taller, the line contains all the words.

The browser flows the resulting lines of boxes into a vertical stack. The start page, even if a reflow occurs, self-aligns by pure CSS. In fact the old JavaScript solution tried to re-engineer the behavior of inline-blocks. Still, drag and drop requires Scriptaculous’ JavaScript methods, but not the alignment, which is time-critical for frontend performance and user experience.