Announcing Conduit Beta

I'm incredibly excited to announce that the instant messaging server Conduit is finally reliable enough to enter Beta. It can be used as a proper Matrix homeserver to talk with other clients and servers of the Matrix network. Let's see what that means:

The current state of instant messaging

WhatsApp, Facebook Messenger, iMessage - these are some of the most used chat applications. But that does not imply it has to be this way. Each of these programs was developed by a huge corporation with the goal of producing as much profit as possible. It's no surprise they sacrifice safety and privacy to sell user data. Even if programs like Signal promise to be better, they still centralize control in the USA.

Matrix is a chat protocol where data is only shared where they should be shared to. There is no official, central place where all data is stored. Instead there are many small servers that can all talk to each other. Additionally, Matrix has all properties you would expect from a modern messenger, for example end-to-end encryption.

What is Conduit?

With Matrix everyone can host their own server and use them to chat with any other user of the network, but if you look more closely you will see that many users don't do that and instead join one of the few big servers. This will centralize all user data again. One reason for that is that it can be difficult and expensive to host your own Matrix server. The Matrix server program "Synapse" which is used most of the time started off as a prototype and ended up focusing on big server deployments with more powerful hardware.

The Conduit project is an alternative to Synapse which is more efficient for smaller servers. It is open-source under the Apache-2.0 license and you can set up your own within minutes.

Git and Documentation: https://gitlab.com/famedly/conduit\

Conduit was developed in Rust, which is known for performance and reliability. Rust gives us a lot more confidence in our code. We know exactly when we have to do error handling, indicated by Option or Result types, and at what points the program could crash, indicated by .unwrap() or .expect(). This allowed us to quickly implement new features and handle all edge cases afterwards. Libraries like serde make our serialization and deserialization code very simple and readable, so we can be sure they follow the Matrix spec. In fact our code was following the spec more rigorously than the rest of the ecosystem, allowing us to uncover bugs in other programs and fix them. For example the Synapse server always sends some json fields, even if they are not required. Some clients became too dependent on that field. We could find these problems when testing these clients with Conduit.

Conduit is a single process with an embedded database, so we get rid of a lot of latencies. We started by using the Rust library sled as our database. It is an easy to use embedded key-value database and we structured Conduit around this key-value approach, which is similar to a binary tree map, but in permanent storage. Unfortunately, we ran into high RAM usage problems and decided to abstract our backend behind a trait and added support for SQLite. This turned out to be both efficient and performant enough. We now use SQLite as a key-value database in Conduit Beta. In the future we might move to a proper key-value database.

Why is developing a Matrix homeserver complicated?

One of the most difficult problems in developing a Matrix homeserver is sending and receiving messages from other servers. This looks trivial at first, but servers have to be extremely careful with data from other servers. Here's an example:

There are three servers: A, B and C. Server A wants to cause confusion and tells server B a user was banned. At the same time it tells server C the person is still in the room but just changed their name. To avoid a conflict, Matrix uses a complicated algorithm called "State Resolution". When server B and C talk to each other, they will notice the conflict. Now all "versions" of the room are compared and combined into a single "correct" version. Soon all servers will agree that the user was banned.

A lot of time went into implementing this algorithm in Conduit and getting rid of all bugs.

Conduit is now in the Beta phase. You can already use it as a Synapse replacement, but there still are some smaller problems.

Try it out to see if it is something for you!

Links

Website: https://conduit.rs
Git and Documentation: https://gitlab.com/famedly/conduit
Chat with us: #conduit:fachschaften.org

Donate

Liberapay: https://liberapay.com/timokoesters/
Bitcoin: bc1qnnykf986tw49ur7wx9rpw2tevpsztvar5x8w4n