7 KiB
Table of Contents
Worker-based setup
Very busy servers are brought down because a single thread can't keep up with the load. So you want to create several threads for different types of work.
See this Matrix blog for some background information.
The traditional Synapse setup is one monolithic piece of software that does everything. Joining a very busy room makes a bottleneck, as the server will spend all its cycles on synchronizing that room.
You can split the server into workers, that are basically Synapse servers themselves. Redirect specific tasks to them and you have several different servers doing all kinds of tasks at the same time. A busy room will no longer freeze the rest.
Workers communicate with each other via socket files and Redis.
Redis
First step is to install Redis.
apt install redis-server
For less overhead we use a UNIX socket instead of a network connection to
localhost. Disable the TCP listener and enable the socket in
/etc/redis/redis.conf:
port 0
unixsocket /run/redis/redis-server.sock
unixsocketperm 770
Our matrix user (matrix-synapse) has to be able to read from and write to
that socket, which is created by Redis and owned by redis:redis, so we add
user matrix-synapse to the group redis.
adduser matrix-synapse redis
Restart Redis for these changes to take effect. Check if port 6379 is no
longer active, and if the socketfile /run/redis/redis-server.sock exists.
Synapse
Workers communicate with each other over sockets, that are all placed in one directory. To make sure only the users that need access will have it, we create a new group and add the users to it.
Then, create the directory where all the socket files for workers will come, and give it the correct user, group and permission:
groupadd --system clubmatrix
useradd matrix-synapse clubmatrix
useradd www-data clubmatrix
mkdir /run/matrix-synapse
dpkg-statoverride --add --update matrix-synapse clubmatrix 2770 /run/matrix-synapse
Add a replication listener:
listeners:
...
- path: /run/matrix-synapse/replication.sock
mode: 0660
type: http
resources:
- names:
- replication
Check if the socket is created and has the correct permissions. Now point
Synapse at Redis in conf.d/redis.yaml:
redis:
enabled: true
path: /run/redis/redis-server.sock
Check if Synapse can connect to Redis via the socket, you should find log entries like this:
synapse.replication.tcp.redis - 292 - INFO - sentinel - Connecting to redis server UNIXAddress('/run/redis/redis-server.sock')
synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.replication.http.ReplicationRestResource object at 0x7f95f850d150> to path b'/_synapse/replication'
synapse.replication.tcp.redis - 126 - INFO - sentinel - Connected to redis
synapse.replication.tcp.redis - 138 - INFO - subscribe-replication-0 - Sending redis SUBSCRIBE for ['matrix.example.com/USER_IP', 'matrix.example.com']
synapse.replication.tcp.redis - 141 - INFO - subscribe-replication-0 - Successfully subscribed to redis stream, sending REPLICATE command
synapse.replication.tcp.redis - 146 - INFO - subscribe-replication-0 - REPLICATE successfully sent
Every worker has its own configuration file, we'll put those under
/etc/matrix-synapse/workers. Create it, and then one systemd service file for
all workers:
[Unit]
Description=Synapse %i
AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
# This service should be restarted when the synapse target is restarted.
PartOf=matrix-synapse.target
ReloadPropagatedFrom=matrix-synapse.target
# if this is started at the same time as the main, let the main process start
# first, to initialise the database schema.
After=matrix-synapse.service
[Service]
Type=notify
NotifyAccess=main
User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=-/etc/default/matrix-synapse
ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
SyslogIdentifier=matrix-synapse-%i
[Install]
WantedBy=matrix-synapse.target
And create the matrix-synapse.target, which combines all Synapse parts into
one systemd target:
[Unit]
Description=Matrix Synapse with all its workers
After=network.target
[Install]
WantedBy=multi-user.target
Create workers
We need a configuration file for each worker, and the main process needs to know which workers there are and how to contact them.
The latter is done in the ...
Temporary block
We're going to configure a few different workers:
- client-sync
- roomworker
- federation-sender
- mediaworker
Client-sync
This type needs both an inbound socket to receive stuff from nginx, and a replication socket to communicate with the rest. We probably want a few of these workers. The configuration should look like this:
worker_app: "synapse.app.generic_worker" # Always this unless "synapse.app.media_repository"
worker_name: "clientsync1" # Name of worker specified in instance map
worker_log_config: "/etc/matrix-synapse/logconf.d/clientsync.yaml" # Log config file
worker_listeners:
# Include for any worker in the instance map above:
- path: "/run/matrix-synapse/replication_clientsync1.sock"
type: http
resources:
- names: [replication]
compress: false
# Include for any worker that receives requests in Nginx:
- path: "/run/matrix-synapse/synapse_inbound_client_sync1.sock"
type: http
x_forwarded: true # Trust the X-Forwarded-For header from Nginx
resources:
- names:
- client
- consent
Roomworker
These don't need a replication socket as they're not in the instance map, but they do need an inboud socket for nginx to pass stuff to them. We want a few of these workers, we may even configure a worker for one specific busy room...
Configuration should look like this:
worker_app: "synapse.app.generic_worker"
worker_name: "roomworker1"
worker_log_config: "/etc/matrix-synapse/logconf.d/roomworker.yaml"
worker_listeners:
- path: "/run/matrix-synapse/inbound_roomworker1.sock"
type: http
x_forwarded: true
resources:
- names:
- client
- consent
- federation
compress: false
Mediaworker
To make sure the worker takes care of handling media, and not the main process. You need to tell the main process to to keep its hands off media, and which worker will take care of it:
enable_media_repo: false
media_instance_running_background_jobs: "mediaworker1"
Then define the worker, like this:
worker_app: "synapse.app.media_repository"
worker_name: "mediaworker1"
worker_log_config: "/etc/matrix-synapse/logconf.d/mediaworker.yaml"
worker_listeners:
- path: "/run/matrix-synapse/inbound_mediaworker1.sock"
type: http
x_forwarded: true
resources:
- names: [media]