Fediversity/simple-nixos-fediverse
Archived
7
1
Fork 2
Code Issues 23 Pull requests 1 Actions Projects Wiki Activity

use garage s3 storage #7

New issue
Open
opened 2024-05-24 21:22:50 +02:00 by taeer · 12 comments
taeer commented 2024-05-24 21:22:50 +02:00
Collaborator
Copy link

Most fediversity apps have options to use s3 storage as a backend.

  • data replication
  • scales better (?)
  • easier to migrate (?)
Most fediversity apps have options to use s3 storage as a backend. - data replication - scales better (?) - easier to migrate (?)
taeer commented 2024-05-24 22:26:31 +02:00
Author
Collaborator
Copy link

Resources:

Resources: - [garage docs for integrating with apps](https://garagehq.deuxfleurs.fr/documentation/connect/apps/) - [mastodon docs for s3 storage](https://docs.joinmastodon.org/admin/optional/object-storage/) - [peertube docs for remote storage](https://docs.joinpeertube.org/maintain/remote-storage)
taeer commented 2024-05-25 01:00:55 +02:00
Author
Collaborator
Copy link

Current status:

Mastodon

Successfully uploads images into s3 storage, and tries to display that image from the correct URL. If I copy-paste the URL from the web inspector, the image shows up.

However, on mastodon itself the image does not show up. This might be a CORS issue, though changing those settings didn't help.

PeerTube

Works!

PixelFed

It seems to be storing the uploaded images correctly in s3, but then when it tries to display them it's using a different URL. Going to that url does not display the image. A couple possibilities

  • one part of the app understand it's supposed to be using s3, and another doesn't
  • that URL should be redirecting to the proper place, and we're running into the same issue as Mastodon of not being able to display the image on a different page / at a different URL.
  • the URL should be redirecting to the proper place, but the redirection itself is broken.
# Current status: ## Mastodon Successfully uploads images into s3 storage, and tries to display that image from the correct URL. If I copy-paste the URL from the web inspector, the image shows up. However, on mastodon itself the image does not show up. This might be a CORS issue, though changing those settings didn't help. ## PeerTube Works! ## PixelFed It seems to be storing the uploaded images correctly in s3, but then when it tries to display them it's using a different URL. Going to that url does not display the image. A couple possibilities - one part of the app understand it's supposed to be using s3, and another doesn't - that URL should be redirecting to the proper place, and we're running into the same issue as Mastodon of not being able to display the image on a different page / at a different URL. - the URL should be redirecting to the proper place, but the redirection itself is broken.
taeer commented 2024-06-06 15:55:28 +02:00
Author
Collaborator
Copy link

Mastodon update

It's not CORS. I solved the CORS problem already successfuly. Instead, it's security from the other end. Rather than garage refusing to serve the image, mastodon is refusing to display it because of Content-Security-Policy rules.

If I doctor the response headers for the page in the chromium devtools window to remove Content-Security-Policy, it works. The images load.

However, this doesn't immediately suggest a way to solve this in the NixOS configuration. Mastodon is supposed to set the CSP correctly automatically. The code for that is in content_security_policy.rb (among other places). I found a bunch of mastodon issues that had to do with CSP, but none seemed to correspond to this case.

I also tried doctoring the header in the nginx reverse proxy config, but I couldn't get that working. It's not a great long-term solution anyways, so I don't want to put more effort into it.

## Mastodon update It's **not** CORS. I solved the CORS problem already successfuly. Instead, it's security from the other end. Rather than garage refusing to serve the image, mastodon is refusing to display it because of Content-Security-Policy rules. If I doctor the response headers for the page in the chromium devtools window to remove `Content-Security-Policy`, it works. The images load. However, this doesn't immediately suggest a way to solve this in the NixOS configuration. Mastodon is **supposed** to set the CSP correctly automatically. The code for that is in [content_security_policy.rb](https://github.com/mastodon/mastodon/blob/main/app/lib/content_security_policy.rb) (among other places). I found a bunch of mastodon issues that had to do with CSP, but none seemed to correspond to this case. I also tried doctoring the header in the `nginx` reverse proxy config, but I couldn't get that working. It's not a great long-term solution anyways, so I don't want to put more effort into it.
taeer commented 2024-06-25 12:57:28 +02:00
Author
Collaborator
Copy link

I'm working towards a minimal reproduction of the mastodon/garage issues using Nixos Tests taeer/simple-nixos-fediverse#9 (comment)

I'm working towards a minimal reproduction of the mastodon/garage issues using Nixos Tests https://git.fediversity.eu/taeer/simple-nixos-fediverse/issues/9#issuecomment-97
taeer commented 2024-07-25 14:04:41 +02:00
Author
Collaborator
Copy link

Minimal reproduction is done, and I've submitted an issue upstream with mastodon
https://github.com/mastodon/mastodon/issues/31145

Minimal reproduction is done, and I've submitted an issue upstream with mastodon https://github.com/mastodon/mastodon/issues/31145
taeer commented 2024-09-02 17:29:58 +02:00
Author
Collaborator
Copy link

Pixefled update:
There are two issues preventing us from seeing images posted.

  1. Has nothing to do with garage. Even with garage disabled, the same behaviour persists. See this nixpkgs issues: https://github.com/NixOS/nixpkgs/issues/235147
  2. Why are we accessing the filesystem to begin with, rather than garage? Because nginx is configured to do so in the pixelfed module in nixpkgs.
Pixefled update: There are two issues preventing us from seeing images posted. 1. Has nothing to do with garage. Even with garage disabled, the same behaviour persists. See this nixpkgs issues: https://github.com/NixOS/nixpkgs/issues/235147 2. Why are we accessing the filesystem to begin with, rather than garage? Because nginx is configured to do so in the pixelfed module in nixpkgs.
Niols commented 2024-09-05 17:22:22 +02:00
Owner
Copy link

Pixelfed update

Commit 553a03b971 seems to feature a working Pixelfed+Garage setup. Here are some noteworthy comments:

  • There exists a DANGEROUSLY_SET_FILESYSTEM_DRIVER flag (formerly just FILESYSTEM_DRIVER) that one can put to s3 and that makes all the storage rely on the S3 backend (Garage, in our case). It is not clear whether we actually want this.

  • This flag differs from the FILESYSTEM_CLOUD flag that specifies that medias (but not avatars, for instance) should be uploaded to the cloud (Garage, in our case). It seems to not have much effect when it is s3 and so is DANGEROUSLY_SET_FILESYSTEM_DRIVER. If the later is not set, then the medias are first uploaded on the local storage and then copied to Garage. It is unclear if and when they are removed from the local storage.

  • Pixelfed uses the AWS_URL to which it appends the media's path and shows that on the client path. If requesting this file does not work, a piece of code on the client replaces the src by a “no preview” image. One should look in the developer tools to check the actual request that's being made.

  • In the commit mentioned, we set AWS_URL to "", meaning the src will just be the media's path. This works in the presence of a proxy configuration for Nginx routing the request to the Garage web address internally.

## Pixelfed update Commit 553a03b971d18e5a69e217552c973587b7a0d315 seems to feature a working Pixelfed+Garage setup. Here are some noteworthy comments: - There exists a `DANGEROUSLY_SET_FILESYSTEM_DRIVER` flag (formerly just `FILESYSTEM_DRIVER`) that one can put to `s3` and that makes _all_ the storage rely on the S3 backend (Garage, in our case). It is not clear whether we actually want this. - This flag differs from the `FILESYSTEM_CLOUD` flag that specifies that medias (but not avatars, for instance) should be uploaded to the cloud (Garage, in our case). It seems to not have much effect when it is `s3` and so is `DANGEROUSLY_SET_FILESYSTEM_DRIVER`. If the later is not set, then the medias are first uploaded on the local storage and then copied to Garage. It is unclear if and when they are removed from the local storage. - Pixelfed uses the `AWS_URL` to which it appends the media's path and shows that on the client path. If requesting this file does not work, a piece of code on the client replaces the `src` by a “no preview” image. One should look in the developer tools to check the actual request that's being made. - In the commit mentioned, we set `AWS_URL` to `""`, meaning the `src` will just be the media's path. This works in the presence of a proxy configuration for Nginx routing the request to the Garage web address internally.
taeer commented 2024-09-05 18:06:29 +02:00
Author
Collaborator
Copy link

Images not showing up without DANGEROUSLY_SET_FILESYSTEM_DRIVER seems to be a generic issue with pixelfed on nixpkgs. See https://github.com/NixOS/nixpkgs/issues/235147 for discussion.

If we apply the patch someone there came up with, it works. See 1d8f514240 for a working config without DANGEROUSLY_SET_ETC_ETC

Images not showing up without `DANGEROUSLY_SET_FILESYSTEM_DRIVER` seems to be a generic issue with pixelfed on nixpkgs. See https://github.com/NixOS/nixpkgs/issues/235147 for discussion. If we apply the [patch someone there came up with](https://git.hubrecht.ovh/hubrecht/nixos/src/commit/411ce0386c475eb569381b6a9f719ec49e4fc04e/machines/dell-r730/pixelfed/01-local-filesystem.patch), it works. See 1d8f514240cb8982683da9d6bed5af91fe041360 for a working config without `DANGEROUSLY_SET_ETC_ETC`
taeer commented 2024-09-05 18:11:00 +02:00
Author
Collaborator
Copy link

In the commit mentioned, we set AWS_URL to "", meaning the src will just be the media's path. This works in the presence of a proxy configuration for Nginx routing the request to the Garage web address internally.

we can also set AWS_URL to the public web endpoint of garage http://pixelfed.web.garage.localhost:3902/. In this case we don't need to set any nginx proxy_pass. This is done in the next commit e7ffd94c5e.

> In the commit mentioned, we set `AWS_URL` to `""`, meaning the `src` will just be the media's path. This works in the presence of a proxy configuration for Nginx routing the request to the Garage web address internally. we can also set `AWS_URL` to the public web endpoint of garage `http://pixelfed.web.garage.localhost:3902/`. In this case we don't need to set any nginx proxy_pass. This is done in the next commit e7ffd94c5e0169dd71afc3e882c335f3abafae1b.
Niols commented 2024-09-05 18:20:12 +02:00
Owner
Copy link

Found this discussion which links to these two issues. Lots of confusing things, but in particular:

image

Found [this discussion](https://github.com/pixelfed/pixelfed/discussions/5150) which links to [these](https://github.com/pixelfed/pixelfed/issues/4348) [two](https://github.com/pixelfed/pixelfed/issues/4348) issues. Lots of confusing things, but in particular: ![image](/attachments/1637c0c8-152e-470f-a099-cce6fd99792a)
image.png
75 KiB
Niols commented 2024-10-29 18:03:42 +01:00
Owner
Copy link

https://garagehq.deuxfleurs.fr/documentation/connect/apps/#mastodon

https://garagehq.deuxfleurs.fr/documentation/connect/apps/#mastodon
Niols commented 2024-10-30 19:35:17 +01:00
Owner
Copy link

Mastodon update

We thought today that we managed to reproduce the content-security-policy (CSP) issue as we did manage to observe images being marked as (blocked:csp) in the Chrome dev tools. However, the content-security-policy (CSP) of the main page did contain garage URLs for img-src, media-src and connect-src, but for port 80 as opposed to port 8080 which we were using for local testing. When actually using port 80 locally, everything seems to work fine.

Our theory is that the original issue was resolved by introducing Nginx as a reverse proxy. What we observed today is in fact a different issue having to do with port forwarding in testing environments.

## Mastodon update We thought today that we managed to reproduce the content-security-policy (CSP) issue as we did manage to observe images being marked as `(blocked:csp)` in the Chrome dev tools. However, the content-security-policy (CSP) of the main page did contain garage URLs for `img-src`, `media-src` and `connect-src`, but for port 80 as opposed to port 8080 which we were using for local testing. When actually using port 80 locally, everything seems to work fine. Our theory is that the original issue was resolved by introducing Nginx as a reverse proxy. What we observed today is in fact a different issue having to do with port forwarding in testing environments.
This repo is archived. You cannot comment on issues.
No Branch/Tag specified
Branches Tags
main
ci
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Fediversity/simple-nixos-fediverse#7
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?