Fediversity/matrix/synapse/workers.md

---
gitea: none
include_toc: true
---

# Worker-based setup

Very busy servers are brought down because a single thread can't keep up with
the load. So you want to create several threads for different types of work.

See this [Matrix blog](https://matrix.org/blog/2020/11/03/how-we-fixed-synapse-s-scalability/)
for some background information.


# Redis

First step is to install Redis.

```
apt install redis-server
```

For less overhead we use a UNIX socket instead of a network connection to
localhost. Disable the TCP listener and enable the socket in
`/etc/redis/redis.conf`:

```
port 0

unixsocket /run/redis/redis-server.sock
unixsocketperm 770
```

Our matrix user (`matrix-synapse`) has to be able to read from and write to
that socket, which is created by Redis and owned by `redis:redis`, so we add
user `matrix-synapse` to the group `redis`.

```
adduser matrix-synapse redis
```

Restart Redis for these changes to take effect. Check if port 6379 is no
longer active, and if the socketfile `/run/redis/redis-server.sock` exists.


# Synapse

First, create the directory where all the socket files for workers will come,
and give it the correct user, group and permission:

```
mkdir /run/matrix-synapse
dpkg-statoverride --add --update matrix-synapse matrix-synapse 0770 /run/matrix-synapse
```

Add a replication listener:

```
listeners:

  ...

  - path: /run/matrix-synapse/replication.sock
    mode: 0660
    type: http
    resources:
      - names:
        - replication 
```

Check if the socket is created and has the correct permissions. Now point
Synapse at Redis in `conf.d/redis.yaml`:

```
redis:
  enabled: true
  path: /run/redis/redis-server.sock
```

Check if Synapse can connect to Redis via the socket, you should find log
entries like this:

```
synapse.replication.tcp.redis - 292 - INFO - sentinel - Connecting to redis server UNIXAddress('/run/redis/redis-server.sock')
synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.replication.http.ReplicationRestResource object at 0x7f95f850d150> to path b'/_synapse/replication'
synapse.replication.tcp.redis - 126 - INFO - sentinel - Connected to redis
synapse.replication.tcp.redis - 138 - INFO - subscribe-replication-0 - Sending redis SUBSCRIBE for ['matrix.example.com/USER_IP', 'matrix.example.com']
synapse.replication.tcp.redis - 141 - INFO - subscribe-replication-0 - Successfully subscribed to redis stream, sending REPLICATE command
synapse.replication.tcp.redis - 146 - INFO - subscribe-replication-0 - REPLICATE successfully sent
```


# Workers

Workers are Synapse instances that perform a single job (or a set of jobs).
Their configuration goes into `/etc/matrix-synapse/workers`, which we have to
create first.

Create a worker for 

# Preparing for sockets

Because we use sockets for speed, we should make sure only those programs that
need access to those sockets get it.

Create a new group and add the users that need access to it:

```
addgroup --system clubmatrix
addgroup matrix-synapse clubmatrix
addgroup www-data clubmatrix
```

Create one systemd service file for all workers:

```
[Unit]
Description=Synapse %i
AssertPathExists=/etc/matrix-synapse/workers/%i.yaml

# This service should be restarted when the synapse target is restarted.
PartOf=matrix-synapse.target
ReloadPropagatedFrom=matrix-synapse.target

# if this is started at the same time as the main, let the main process start
# first, to initialise the database schema.
After=matrix-synapse.service

[Service]
Type=notify
NotifyAccess=main
User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=-/etc/default/matrix-synapse
ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
SyslogIdentifier=matrix-synapse-%i

[Install]
WantedBy=matrix-synapse.target
```

And create the `matrix-synapse.target`, which combines all Synapse parts into
one systemd target:

```
[Unit]
Description=Matrix Synapse with all its workers
After=network.target

[Install]
WantedBy=multi-user.target
```