Fediversity/matrix/nginx/workers

Table of Contents

• Reverse proxy for Synapse with workers
• Optimizations
• Maps
• Upstreams
• Locations

Reverse proxy for Synapse with workers

Changing nginx's configuration from a reverse proxy for a normal, monolithic Synapse to one for a Synapse that uses workers, is a big thing: quite a lot has to be changed.

As mentioned in Synapse with workers, we're changing the "backend" from network sockets to UNIX sockets.

Because we're going to have to forward a lot of specific requests to all kinds of workers, we'll split the configuration into a few bits:

• all proxy_forward settings
• all location definitions
• maps that define variables
• upstreams that point to the correct socket(s) with the correct settings
• settings for private access
• connection optimizations

Some of these go into /etc/nginx/conf.d because they are part of the configuration of nginx itself, others go into /etc/nginx/snippets because we need to include them several times in different places.

Important consideration

This part isn't a quick "put these files in place and you're done": a worker-based Synapse is tailor-made, there's no one-size-fits-all. This documentation gives hints and examples, but in the end it's you who has to decide what types of workers to use and how many, all depending on your specific use case and the available hardware.

Optimizations

In the quest for speed, we are going to tweak several settings in nginx. To keep things manageable, most of those tweaks go into separate configuration files that are either automatically included (those under /etc/nginx/conf.d) or explicitly where we need them (those under /etc/nginx/snippets).

Let's start with a few settings that affect nginx as a whole. Edit these options in /etc/nginx/nginx.conf:

pcre_jit on;
worker_rlimit_nofile 8192;
worker_connections 4096;
multi_accept off;
gzip_comp_level 2;
gzip_types application/javascript application/json application/x-javascript application/xml application/xml+rss image/svg+xml text/css text/javascript text/plain text/xml;
gzip_min_length 1000;
gzip_disable "MSIE [1-6]\.";

We're going to use lots of regular expressions in our config, pcre_jit on speeds those up considerably. Workers get 8K open files, and we want 4096 workers instead of the default 768. Workers can only accept one connection, which is (in almost every case) proxy_forwarded, so we set multi_accept off.

We change gzip_comp_level from 6 to 2, we expand the list of content that is to be gzipped, and don't zip anything shorter than 1000 characters, instead of the default 20. MSIE can take a hike...

These are tweaks for the connection, save this in /etc/ngnix/conf.d/conn_optimize.conf.

client_body_buffer_size 32m;
client_header_buffer_size 32k;
client_max_body_size 1g;
http2_max_concurrent_streams 128;
keepalive_timeout 65;
keepalive_requests 100;
large_client_header_buffers 4 16k;
server_names_hash_bucket_size 128;
tcp_nodelay on;
server_tokens off;

We set a few proxy settings that we use in proxy_forwards other than to our workers, save this to conf.d/proxy_optimize.conf:

proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

For every proxy_forward to our workers, we want to configure several settings, and because we don't want to include the same list of settings every time, we put all of them in one snippet of code, that we can include every time we need it.

Create /etc/nginx/snippets/proxy.conf and put this in it:

proxy_connect_timeout 2s;
proxy_buffering off;
proxy_http_version 1.1;
proxy_read_timeout 3600s;
proxy_redirect off;
proxy_send_timeout 120s;
proxy_socket_keepalive on;
proxy_ssl_verify off;

proxy_set_header Accept-Encoding "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;

client_max_body_size 50M;

Every time we use a proxy_forward, we include this snippet. There are 2 more things we might set: trusted locations that can use the admin endpoints, and a dedicated DNS-recursor. We include the snippets/private.conf in the forwards to admin endpoints, so that not the entire Internet can play with it. The dedicated nameserver is something you really want, because synchronising a large room can easily result in 100.000+ DNS requests. You'll hit flood protection on most servers if you do that.

List the addresses from which you want to allow admin access in snippets/private.conf:

allow 127.0.0.1;
allow ::1;
allow 12.23.45.78;
allow 87.65.43.21;
allow dead:beef::/48;
allow 2a10:1234:abcd::1;
deny all;
satisfy all;

Of course, subsitute these random addresses for the ones you trust. The dedicated nameserver (if you have one, which is strongly recommended) should be configured in conf.d/resolver.conf:

resolver [::1] 127.0.0.1 valid=60;
resolver_timeout 10s;

Maps

A map sets a variable based on, usually, another variable. One case we use this is in determining the type of sync a client is doing. A normal sync, simply updating an existing session, is a rather lightweight operation. An initial sync, meaning a full sync because the session is brand new, is not so lightweight.

A normal sync can be recognised by the since bit in the request: it tells the server when its last sync was. If there is no since, we're dealing with an initial sync.

We want to forward requests for normal syncs to the normal_sync workers, and the initial syncs to the initial_sync workers.

We decide to which type of worker to forward the sync request to by looking at the presence or absence of since: if it's there, it's a normal sync and we set the variable $sync to normal_sync. If it's not there, we set $sync to initial_sync. The content of since is irrelevant for nginx.

This is what the map looks like:

map $arg_since $sync {
    default normal_sync;
    '' initial_sync;
}

We evaluate $arg_since to set $sync: $arg_since is nginx's variable $arg_ followed by since, the argument we want. See the index of variables in nginx for more variables we can use in nginx.

By default we set $sync to normal_sync, unless the argument since is empty (absent); then we set it to initial_sync.

After this mapping, we forward the request to the correct worker like this:

proxy_pass http://$sync;

See a complete example of maps in the file maps.conf.

Upstreams

In our configuration, nginx is not only a reverse proxy, it's also a load balancer. Just like what haproxy does, it can forward requests to "servers" behind it. Such a server is the inbound UNIX socket of a worker, and there can be several of them in one group.

Let's start with a simple one, the login worker, that handles the login process for clients. There's only one worker, so only one socket:

upstream login {
    server unix:/run/matrix-synapse/inbound_login.sock max_fails=0;
    keepalive 10;
}

Ater this definition, we can forward traffic to login. What traffic to forward is decided in the location statements, see further.

A more complex example are the sync workers. Under Maps we split sync requests into two different types; those different types are handled by different worker pools. In our case we have 2 workers for the initial_sync requests, and 3 for the normal ones:

upstream initial_sync {
    hash $mxid_localpart consistent;
    server unix:/run/matrix-synapse/inbound_initial_sync1.sock max_fails=0;
    server unix:/run/matrix-synapse/inbound_initial_sync2.sock max_fails=0;
    keepalive 10;
}

upstream normal_sync {
    hash $mxid_localpart consistent;
    server unix:/run/matrix-synapse/inbound_normal_sync1.sock max_fails=0;
    server unix:/run/matrix-synapse/inbound_normal_sync2.sock max_fails=0;
    server unix:/run/matrix-synapse/inbound_normal_sync3.sock max_fails=0;
    keepalive 10;
}

The hash bit is to make sure that request from one user are consistently forwarded to the same worker. We filled the variable $mxid_localpart in the maps.

Locations

Now that we have defined the workers and/or worker pools, we have to forward the right traffic to the right workers. The Synapse documentation about available worker types lists which endpoints a specific worker type can handle.

Let's forward login requests to our login worker. The documentation for the generic_worker says these endpoints are for registration and login:

# Registration/login requests
^/_matrix/client/(api/v1|r0|v3|unstable)/login$
^/_matrix/client/(r0|v3|unstable)/register$
^/_matrix/client/(r0|v3|unstable)/register/available$
^/_matrix/client/v1/register/m.login.registration_token/validity$
^/_matrix/client/(r0|v3|unstable)/password_policy$

We forward that to our worker with this location definition, using the proxy_forward settings we defined earlier:

location ~ ^(/_matrix/client/(api/v1|r0|v3|unstable)/login|/_matrix/client/(r0|v3|unstable)/register|/_matrix/client/(r0|v3|unstable)/register/available|/_matrix/client/v1/register/m.login.registration_token/validity|/_matrix/client/(r0|v3|unstable)/password_policy)$ {
    include snippets/proxy.conf;
    proxy_pass http://login;
}

The docs say that the generic_worker can handle these requests for synchronisation requests:

# Sync requests
^/_matrix/client/(r0|v3)/sync$
^/_matrix/client/(api/v1|r0|v3)/events$
^/_matrix/client/(api/v1|r0|v3)/initialSync$
^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$

We forward those to our 2 worker pools, normal_sync and initial_sync, like this, using the variable $sync we defined in maps.conf:

# Normal/initial sync
location ~ ^/_matrix/client/(r0|v3)/sync$ {
    include snippets/proxy.conf;
    proxy_pass http://$sync;
}

# Normal sync
location ~ ^/_matrix/client/(api/v1|r0|v3)/events$ {
    include snippets/proxy.conf;
    proxy_pass http://normal_sync;
}

# Initial sync
location ~ ^(/_matrix/client/(api/v1|r0|v3)/initialSync|/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync)$ {
    include snippets/proxy.conf;
    proxy_pass http://initial_sync;
}

The media worker is slightly different: some parts are public, but a few bits are admin stuff. We split those, and limit the admin endpoints to the trusted addresses we defined earlier:

# Media, public
location ~* ^(/_matrix/((client|federation)/[^/]+/)media/|/_matrix/media/v3/upload/) {
    include snippets/proxy.conf;
    proxy_pass http://media;
}

# Media, admin
location ~ ^/_synapse/admin/v1/(purge_)?(media(_cache)?|room|user|quarantine_media|users)/[\s\S]+|media$ {
    include snippets/private.conf;
    include snippets/proxy.conf;
    proxy_pass http://media;
}