Fediversity/matrix/nginx/workers/README.md

---
gitea: none
include_toc: true
---

# Reverse proxy for Synapse with workers

Changing nginx's configuration from a reverse proxy for a normal, monolithic
Synapse to one for a Synapse that uses workers, is a big thing: quite a lot has to
be changed.

As mentioned in [Synapse with workers](../../synapse/workers/README.md#synapse),
we're changing the "backend" from network sockets to UNIX sockets.

Because we're going to have to forward a lot of specific requests to all kinds
of workers, we'll split the configuration into a few bits:

* all `proxy_forward` settings
* all `location` definitions
* maps that define variables
* upstreams that point to the correct socket(s) with the correct settings
* settings for private access
* connection optimizations

Some of these go into `/etc/nginx/conf.d` because they are part of the
configuration of nginx itself, others go into `/etc/nginx/snippets` because we
need to include them several times in different places.

**Important consideration**

This part isn't a quick "put these files in place and you're done": a
worker-based Synapse is tailor-made, there's no one-size-fits-all. This
documentation gives hints and examples, but in the end it's you who has to
decide what types of workers to use and how many, all depending on your
specific use case and the available hardware.




# Optimizations

In the quest for speed, we are going to tweak several settings in nginx. To
keep things manageable, most of those tweaks go into separate configuration
files that are either automatically included (those under `/etc/nginx/conf.d`)
or explicitly where we need them (those under `/etc/nginx/snippets`).

Let's start with a few settings that affect nginx as a whole. Edit these
options in `/etc/nginx/nginx.conf`:

```
pcre_jit on;
worker_rlimit_nofile 8192;
worker_connections 4096;
multi_accept off;
gzip_comp_level 2;
gzip_types application/javascript application/json application/x-javascript application/xml application/xml+rss image/svg+xml text/css text/javascript text/plain text/xml;
gzip_min_length 1000;
gzip_disable "MSIE [1-6]\.";
```

We're going to use lots of regular expressions in our config, `pcre_jit on`
speeds those up considerably. Workers get 8K open files, and we want 4096
workers instead of the default 768. Workers can only accept one connection,
which is (in almost every case) proxy_forwarded, so we set `multi_accept off`.

We change `gzip_comp_level` from 6 to 2, we expand the list of content that is
to be gzipped, and don't zip anything shorter than 1000 characters, instead of
the default 20. MSIE can take a hike...

These are tweaks for the connection, save this in `/etc/ngnix/conf.d/conn_optimize.conf`.

```
client_body_buffer_size 32m;
client_header_buffer_size 32k;
client_max_body_size 1g;
http2_max_concurrent_streams 128;
keepalive_timeout 65;
keepalive_requests 100;
large_client_header_buffers 4 16k;
server_names_hash_bucket_size 128;
tcp_nodelay on;
server_tokens off;
```

We set a few proxy settings that we use in proxy_forwards other than to our
workers, save this to `conf.d/proxy_optimize.conf`:

```
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
```

For every `proxy_forward` to our workers, we want to configure several settings,
and because we don't want to include the same list of settings every time, we put
all of them in one snippet of code, that we can include every time we need it.

Create `/etc/nginx/snippets/proxy.conf` and put this in it:

```
proxy_connect_timeout 2s;
proxy_buffering off;
proxy_http_version 1.1;
proxy_read_timeout 3600s;
proxy_redirect off;
proxy_send_timeout 120s;
proxy_socket_keepalive on;
proxy_ssl_verify off;

proxy_set_header Accept-Encoding "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;

client_max_body_size 50M;
```

Every time we use a `proxy_forward`, we include this snippet. There are 2 more
things we might set: trusted locations that can use the admin endpoints, and a
dedicated DNS-recursor. We include the `snippets/private.conf` in the
forwards to admin endpoints, so that not the entire Internet can play with it.
The dedicated nameserver is something you really want, because synchronising a
large room can easily result in 100.000+ DNS requests. You'll hit flood
protection on most servers if you do that.

List the addresses from which you want to allow admin access in
`snippets/private.conf`:

```
allow 127.0.0.1;
allow ::1;
allow 12.23.45.78;
allow 87.65.43.21;
allow dead:beef::/48;
allow 2a10:1234:abcd::1;
deny all;
satisfy all;
```

Of course, subsitute these random addresses for the ones you trust. The
dedicated nameserver (if you have one, which is strongly recommended) should
be configured in `conf.d/resolver.conf`:

```
resolver [::1] 127.0.0.1 valid=60;
resolver_timeout 10s;
```


# Maps {#maps}

A map sets a variable based on, usually, another variable. One case we use this
is in determining the type of sync a client is doing. A normal sync, simply
updating an existing session, is a rather lightweight operation. An initial sync,
meaning a full sync because the session is brand new, is not so lightweight.

A normal sync can be recognised by the `since` bit in the request: it tells
the server when its last sync was. If there is no `since`, we're dealing with
an initial sync.

We want to forward requests for normal syncs to the `normal_sync` workers, and
the initial syncs to the `initial_sync` workers.

We decide to which type of worker to forward the sync request to by looking at
the presence or absence of `since`: if it's there, it's a normal sync and we
set the variable `$sync` to `normal_sync`. If it's not there, we set `$sync` to
`initial_sync`. The content of `since` is irrelevant for nginx.

This is what the map looks like:

```
map $arg_since $sync {
    default normal_sync;
    '' initial_sync;
}
```

We evaluate `$arg_since` to set `$sync`: `$arg_since` is nginx's variable `$arg_`
followed by `since`, the argument we want. See [the index of
variables in nginx](https://nginx.org/en/docs/varindex.html) for more
variables we can use in nginx.

By default we set `$sync` to `normal_sync`, unless the argument `since` is
empty (absent); then we set it to `initial_sync`.

After this mapping, we forward the request to the correct worker like this:

```
proxy_pass http://$sync;
```

See a complete example of maps in the file [maps.conf](maps.conf).


# Upstreams

In our configuration, nginx is not only a reverse proxy, it's also a load balancer.
Just like what `haproxy` does, it can forward requests to "servers" behind it.
Such a server is the inbound UNIX socket of a worker, and there can be several
of them in one group.

Let's start with a simple one, the `login` worker, that handles the login
process for clients. There's only one worker, so only one socket:

```
upstream login {
    server unix:/run/matrix-synapse/inbound_login.sock max_fails=0;
    keepalive 10;
}
```

Ater this definition, we can forward traffic to `login`. What traffic to
forward is decided in the `location` statements, see further.

A more complex example are the sync workers. Under [Maps](#Maps) we split sync
requests into two different types; those different types are handled by
different worker pools. In our case we have 2 workers for the initial_sync
requests, and 3 for the normal ones:

```
upstream initial_sync {
    hash $mxid_localpart consistent;
    server unix:/run/matrix-synapse/inbound_initial_sync1.sock max_fails=0;
    server unix:/run/matrix-synapse/inbound_initial_sync2.sock max_fails=0;
    keepalive 10;
}

upstream normal_sync {
    hash $mxid_localpart consistent;
    server unix:/run/matrix-synapse/inbound_normal_sync1.sock max_fails=0;
    server unix:/run/matrix-synapse/inbound_normal_sync2.sock max_fails=0;
    server unix:/run/matrix-synapse/inbound_normal_sync3.sock max_fails=0;
    keepalive 10;
}
```

The `hash` bit is to make sure that request from one user are consistently
forwarded to the same worker. We filled the variable `$mxid_localpart` in the
maps.


# Locations

Now that we have defined the workers and/or worker pools, we have to forward
the right traffic to the right workers. The Synapse documentation about
[available worker
types](https://element-hq.github.io/synapse/latest/workers.html#available-worker-applications)
lists which endpoints a specific worker type can handle. 

Let's forward login requests to our login worker. The [documentation for the
generic_worker](https://element-hq.github.io/synapse/latest/workers.html#synapseappgeneric_worker)
says these endpoints are for registration and login:

```
# Registration/login requests
^/_matrix/client/(api/v1|r0|v3|unstable)/login$
^/_matrix/client/(r0|v3|unstable)/register$
^/_matrix/client/(r0|v3|unstable)/register/available$
^/_matrix/client/v1/register/m.login.registration_token/validity$
^/_matrix/client/(r0|v3|unstable)/password_policy$
```

We forward that to our worker with this `location` definition, using the
`proxy_forward` settings we defined earlier:

```
location ~ ^(/_matrix/client/(api/v1|r0|v3|unstable)/login|/_matrix/client/(r0|v3|unstable)/register|/_matrix/client/(r0|v3|unstable)/register/available|/_matrix/client/v1/register/m.login.registration_token/validity|/_matrix/client/(r0|v3|unstable)/password_policy)$ {
    include snippets/proxy.conf;
    proxy_pass http://login;
}
```

The docs say that the `generic_worker` can handle these requests for synchronisation
requests:

```
# Sync requests
^/_matrix/client/(r0|v3)/sync$
^/_matrix/client/(api/v1|r0|v3)/events$
^/_matrix/client/(api/v1|r0|v3)/initialSync$
^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$
```

We forward those to our 2 worker pools, `normal_sync` and `initial_sync`, like
this, using the variable `$sync` we defined in maps.conf:

```
# Normal/initial sync
location ~ ^/_matrix/client/(r0|v3)/sync$ {
    include snippets/proxy.conf;
    proxy_pass http://$sync;
}

# Normal sync
location ~ ^/_matrix/client/(api/v1|r0|v3)/events$ {
    include snippets/proxy.conf;
    proxy_pass http://normal_sync;
}

# Initial sync
location ~ ^(/_matrix/client/(api/v1|r0|v3)/initialSync|/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync)$ {
    include snippets/proxy.conf;
    proxy_pass http://initial_sync;
}
```

The media worker is slightly different: some parts are public, but a few bits
are admin stuff. We split those, and limit the admin endpoints to the trusted
addresses we defined earlier:

```
# Media, public
location ~* ^(/_matrix/((client|federation)/[^/]+/)media/|/_matrix/media/v3/upload/) {
    include snippets/proxy.conf;
    proxy_pass http://media;
}

# Media, admin
location ~ ^/_synapse/admin/v1/(purge_)?(media(_cache)?|room|user|quarantine_media|users)/[\s\S]+|media$ {
    include snippets/private.conf;
    include snippets/proxy.conf;
    proxy_pass http://media;
}
```