Added more workers documentation and added worker compatibility for Draupnir.

matrix/draupnir/README.md

@@ -53,9 +53,10 @@ Copy it to `production.yaml` and change what you must.
 
 | Option | Value | Meaning |
 | :---- | :---- | :---- |
-| `homeserverUrl` | `http://localhost:8008` | Where to communicate with Synapse |
+| `homeserverUrl` | `http://localhost:8008` | Where to communicate with Synapse when using network port|
+| `homeserverUrl` | `http://unix://run/matrix-synapse/incoming_main.sock` | Where to communicate with Synapse when using UNIX sockets (see [Workers](../synapse/workers.md)) |
 | `rawHomeserverUrl` | `https://matrix.example.com` | Same as `server_name` |
-| `accessToken` | access token | Copy from login session |
+| `accessToken` | access token | Copy from login session or create in [Synapse Admin](../synapse-admin)) |
 | `password` | password | Password for the account |
 | `dataPath` | `/opt/Draupnir/datastorage` | Storage |
 | `managementRoom` | room ID | Room where moderators command Draupnir |

matrix/synapse/workers.md

@@ -230,11 +230,11 @@ instance_map:
     path: "/run/matrix-synapse/replication_mediaworker.sock"
 ...
   normal_sync1:
-    path: "unix:/run/matrix-synapse/inbound_normal_sync1.sock"
+    path: "unix:/run/matrix-synapse/replication_normal_sync1.sock"
   normal_sync2:
-    path: "unix:/run/matrix-synapse/inbound_normal_sync2.sock"
+    path: "unix:/run/matrix-synapse/replication_normal_sync2.sock"
   normal_sync3:
-    path: "unix:/run/matrix-synapse/inbound_normal_sync3.sock"
+    path: "unix:/run/matrix-synapse/replication_normal_sync3.sock"
 ```
 
 
@@ -320,8 +320,7 @@ worker the same layout. Here's the configuration for the `login` worker:
 version: 1
 formatters:
   precise:
-    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s
+    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
-- %(message)s'
 handlers:
   file:
     class: logging.handlers.TimedRotatingFileHandler
@@ -370,8 +369,43 @@ documentation](https://docs.python.org/3/library/logging.config.html#configurati
 for all the ins and outs of logging.
 
 
+# Systemd
 
-# The rest
+You want Synapse and its workers managed by systemd. First of all we define a
+`target`: a group of services that belong together.
+
+```
+systemctl edit --force --full matrix-synapse.target
+```
+
+Feed it with this bit:
+
+```
+[Unit]
+Description=Matrix Synapse with all its workers
+After=network.target
+
+[Install]
+WantedBy=multi-user.target
+```
+
+First add `matrix-synapse.service` to this target by overriding the `WantedBy`
+in the unit file (`systemctl edit matrix-synapse.service`):
+
+```
+[Install]
+WantedBy=matrix.target
+```
+
+The same `WantedBy` need to go in the unit files for every worker. For the
+workers we're using a template instead of separate unit files for every single
+one. Create the template:
+
+```
+systemctl edit --full --force matrix-synapse-worker@
+```
+
+Fill it with this content:
 
 ```
 [Unit]
@@ -402,115 +436,12 @@ SyslogIdentifier=matrix-synapse-%i
 WantedBy=matrix-synapse.target
 ```
 
-And create the `matrix-synapse.target`, which combines all Synapse parts into
+Every worker needs to be enabled and started individually. Quickest way to do
-one systemd target:
+that, is to run a loop in the directory:
 
 ```
-[Unit]
+cd /etc/matrix-synapse/workers
-Description=Matrix Synapse with all its workers
+for worker in `ls *yaml`; do systemctl enable --now matrix-synapse-worker@$worker; done
-After=network.target
-
-[Install]
-WantedBy=multi-user.target
 ```
 
-
+After a reboot, Synapse and all its workers should be started.
-# Create workers
-
-We need a configuration file for each worker, and the main process needs to
-know which workers there are and how to contact them.
-
-The latter is done in the ...
-
-
-## Temporary block
-
-We're going to configure a few different workers:
-
-* client-sync
-* roomworker
-* federation-sender
-* mediaworker
-
-
-### Client-sync
-
-This type needs both an inbound socket to receive stuff from nginx, and a
-replication socket to communicate with the rest. We probably want a few of
-these workers. The configuration should look like this:
-
-```
-worker_app: "synapse.app.generic_worker" # Always this unless "synapse.app.media_repository"
-worker_name: "clientsync1" # Name of worker specified in instance map
-worker_log_config: "/etc/matrix-synapse/logconf.d/clientsync.yaml" # Log config file
-
-worker_listeners:
-  # Include for any worker in the instance map above:
-  - path: "/run/matrix-synapse/replication_clientsync1.sock"
-    type: http
-    resources:
-      - names: [replication]
-        compress: false
-  # Include for any worker that receives requests in Nginx:
-  - path: "/run/matrix-synapse/synapse_inbound_client_sync1.sock"
-    type: http
-    x_forwarded: true # Trust the X-Forwarded-For header from Nginx
-    resources:
-      - names:
-        - client
-        - consent
-```
-
-### Roomworker
-
-These don't need a replication socket as they're not in the instance map, but
-they do need an inboud socket for nginx to pass stuff to them. We want a few
-of these workers, we may even configure a worker for one specific busy room...
-
-Configuration should look like this:
-
-```
-worker_app: "synapse.app.generic_worker"
-worker_name: "roomworker1"
-worker_log_config: "/etc/matrix-synapse/logconf.d/roomworker.yaml"
-
-worker_listeners:
-  - path: "/run/matrix-synapse/inbound_roomworker1.sock"
-    type: http
-    x_forwarded: true
-    resources:
-      - names:
-        - client
-        - consent
-        - federation
-        compress: false
-```
-
-### Mediaworker
-
-To make sure the worker takes care of handling media, and not the main
-process. You need to tell the main process to to keep its hands off media, and
-which worker will take care of it:
-
-```
-enable_media_repo: false
-media_instance_running_background_jobs: "mediaworker1"
-```
-
-Then define the worker, like this:
-
-
-```
-worker_app: "synapse.app.media_repository"
-worker_name: "mediaworker1"
-worker_log_config: "/etc/matrix-synapse/logconf.d/mediaworker.yaml"
-
-worker_listeners:
-  - path: "/run/matrix-synapse/inbound_mediaworker1.sock"
-    type: http
-    x_forwarded: true
-    resources:
-      - names: [media]
-```
-
-