Pawb.Social Announcements crashdoom • Now • 96%

[Maintenance Update] Lemmy v0.19.4 / v0.19.5 Updates

We've successfully upgraded Lemmy from v0.19.3 to v0.19.5 and have a bunch of changes to share! ### Bug Fixes * Previously users were reporting that the web client kept logging them out. This was suspected to be due to a misconfiguration of the cookie storing the session information and should be fixed. * Some communities were slow to load. Federation changes in v0.19.4 and v0.19.5 should rectify some federation issues. Additionally, we increased the number of database connections for Lemmy and will gradually increase this if we identify any further slowdowns. (Longer term, we're planning to upgrade the database hardware which should improve the performance for all services) ### Local Only Communities When creating or editing your communities, you'll now see a "Visibility" option that allows you to choose between "Public" and "Local Only". The "Public" setting will continue to operate as normal and federate your community to all other instances we federate with. "Local Only" will disable federation for your community and restrict the community to being viewed by local logged in users only; Logged out users may not be able to see your community until they log in. ### Image Proxying Once we've established that the database is no longer under substantial load and the performance issues previously experienced are fixed, we'll be enabling an image proxying mode which will allow us to locally cache images from remote instances rather than serving them directly. This will allow us to improve the performance of loading images without applying additional load to other instances, and prevent remote bad actors from trying to scrape IP addresses. ### Other changes There are a handful of other miscellaneous changes that you can view in the full change logs: - https://join-lemmy.org/news/2024-06-07_-_Lemmy_Release_v0.19.4_-_Image_Proxying_and_Federation_improvements - https://join-lemmy.org/news/2024-06-19_-_Lemmy_Release_v0.19.5_-_A_Few_Bugfixes

Pawb.Social Announcements crashdoom • Now • 91%

[Maintenance] Sept 22 - Login Issues on Pawb.Social

We’re aware of an issue affecting our Lemmy service (pawb.social) that seems to be causing users to be randomly logged out. We believe we’ve identified the issue thanks to a bug fix in a newer version of the UI code. We’re needing to verify there are no other breaking changes for the upgrade process, and expect to perform maintenance this Sunday, Sept 22nd at Noon US Mountain Time (https://everytimezone.com/s/7abd4955). We’ll provide an update tomorrow to confirm the maintenance window. Thank you to everyone who brought this to our attention and we apologize for the inconvenience.

Pawb.Social Announcements crashdoom • Now • 100%

[Maintenance] Feb 7 - Mastodon Data Migration

**tl;dr summary** furry.engineer and pawb.fun will be down for several hours this evening (5 PM Mountain Time onward) as we migrate data from the cloud to local storage. We'll post updates via our announcements channel at https://t.me/pawbsocial. ---- In order to reduce costs and expand our storage pool, we'll be migrating data from our existing Cloudflare R2 buckets to local replicated network storage, and from Proxmox-based LXC containers to Kubernetes pods. Currently, according to Mastodon, we're using about 1 TB of media storage, but according to Cloudflare, we're using near 6 TB. This appears to be due to Cloudflare R2's implementation of the underlying S3 protocol that Mastodon uses for cloud-based media storage, which is preventing Mastodon from properly cleaning up no longer used files. As part of the move, we'll be creating / using new Docker-based images for Glitch-SOC (the fork of Mastodon we use) and hooking that up to a dedicated set of database nodes and replicated storage through Longhorn. This should allow us to seamlessly move the instances from one Kubernetes node to another for performing routine hardware and system maintenance without taking the instances offline. We're planning to roll out the changes in several stages: 1. Taking furry.engineer and pawb.fun down for maintenance to prevent additional media being created. 2. Initiating a transfer from R2 to the new local replicated network storage for **locally generated** user content first, then remote media. (This will happen in parallel to the other stages, so **some media may be unavailable** until the transfer fully completes). 3. Exporting and re-importing the databases from their LXC containers to the new dedicated database servers. 4. Creating and deploying the new Kubernetes pods, and bringing one of the two instances back online, pointing at the new database and storage. 5. Monitoring for any media-related issues, and bringing the second instance back online. We'll be beginning the maintenance window at **5 PM Mountain Time** (**4 PM Pacific Time**) and have no ETA at this time. We'll provide updates through our existing Telegram announcements channel at https://t.me/pawbsocial. **During this maintenance window, furry.engineer and pawb.fun will be unavailable until the maintenance concluded.** Our Lemmy instance at pawb.social will remain online, though you may experience longer than normal load times due to high network traffic. ---- Finally and most importantly, I want to thank those who have been donating through our **[Ko-Fi page](https://ko-fi.com/pawbsocial)** as this has allowed us to build up a small war chest to make this transfer possible through both new hardware and the inevitable data export fees we'll face bringing content down from Cloudflare R2. Going forward, we're looking into providing additional fediverse services (such as Pixelfed) and extending our data retention length to allow us to maintain more content for longer, but none of this would be possible if it weren't for your generous donations.

Pawb.Social Announcements crashdoom • Now • 93%

Lemmy v0.19.3

We've updated to Lemmy v0.19.3! For a full change log, see the updates below: - [v0.19.0](https://join-lemmy.org/news/2023-12-15_-_Lemmy_Release_v0.19.0_-_Instance_blocking,_Scaled_sort,_and_Federation_Queue) - [v0.19.1](https://join-lemmy.org/news/2023-12-20_-_Lemmy_Release_v0.19.1_-_Outgoing_Federation_fix) - [v0.19.2](https://join-lemmy.org/news/2024-01-10_-_Lemmy_Release_v0.19.2_-_More_Federation_Fixes) - [v0.19.3](https://join-lemmy.org/news/2024-01-22_-_Lemmy_Release_v0.19.3_-_A_Few_Bugfixes) ## Major changes ### Improved Post Ranking There is a new scaled sort which takes into account the number of active users in a community, and boosts posts from less-active communities to the top. Additionally there is a new controversial sort which brings posts and comments to the top that have similar amounts of upvotes and downvotes. Lemmy’s sorts are detailed [here](https://join-lemmy.org/docs/users/03-votes-and-ranking.html). ### Instance Blocks for Users Users can now [block instances](https://github.com/LemmyNet/lemmy/pull/3869). Similar to community blocks, it means that any posts from communities which are hosted on that instance are hidden. However the block doesn’t affect users from the blocked instance, their posts and comments can still be seen normally in other communities. ### Two-Factor Auth Rework Previously 2FA was enabled in a single step which made it easy to lock yourself out. This is now fixed by using a [two-step process](https://github.com/LemmyNet/lemmy/pull/3959), where the secret is generated first, and then 2FA is enabled by entering a valid 2FA token. It also fixes the problem where 2FA can be disabled without passing any 2FA token. As part of this change, 2FA is disabled for all users. This allows users who are locked out to get into their account again. ### New Federation Queue Outgoing federation actions are processed through a [new persistent queue](https://github.com/LemmyNet/lemmy/pull/3605). This means that actions don’t get lost if Lemmy is restarted. It is also much more performant, with separate senders for each target instance. This avoids problems when instances are unreachable. Additionally it supports horizontal scaling across different servers. The endpoint /api/v3/federated_instances contains [details about federation state](https://github.com/LemmyNet/lemmy/pull/4104) of each remote instance ### Remote Follow Another new feature is [support for remote follow](https://github.com/LemmyNet/lemmy-ui/pull/1875). When browsing another instance where you don’t have an account, you can click the subscribe button and enter the domain of your home instance in the popup dialog. It will automatically redirect you to your home instance where it fetches the community and presents a subscribe button. [Here is a video showing how it works](https://github.com/LemmyNet/lemmy-ui/pull/1875#issuecomment-1727790414). ### Moderation Reports are now [resolved automatically](https://github.com/LemmyNet/lemmy/pull/3871) when the associated post/comment is marked as deleted. This reduces the amount of work for moderators. There is a new [log for image uploads](https://github.com/LemmyNet/lemmy/pull/3927) which stores uploader. For now it is used to delete all user uploads when an account is purged. Later the list can be used for other purposes and made available through the API.

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] pisskey.io

- **Instance**: pisskey.io - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Nazi imagery, affiliated with poa.st and other known abusive instances - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/pisskey.io)

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] the.asbestos.cafe

- **Instance**: the.asbestos.cafe - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Homophobia, harassment, trolling, admin involved abuse - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/the.asbestos.cafe) Block has been applied to the entire domain. ::: spoiler Evidence - https://the.asbestos.cafe/objects/14decba5-b6fb-4c20-b8d7-8cf68374ec58 - https://the.asbestos.cafe/objects/094cfc21-7182-471a-90cc-a493fb126635 - https://the.asbestos.cafe/objects/764af9ee-288c-4151-aa41-b2e295dc95cc - https://the.asbestos.cafe/objects/3f5960bb-8fc1-41b0-b180-7748cd0e15b3 - https://the.asbestos.cafe/objects/bee81dc2-85ee-45ae-b1b6-39f2d2d4dd6f :::

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] nyanide.com

- **Instance**: lab.nyanide.com - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Trolling, harassment, homophobia, nazi imagery, admin / mod engaged abuse - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/lab.nyanide.com) Block has been applied to the entire domain. ::: spoiler Evidence    :::

cross-posted from: https://pawb.social/post/3393854 > - **Instance**: cunnyborea.space > - **Type**: Defederation > - **Affects**: Pawb.Social, furry.engineer, pawb.fun > - **Reason**: Racism, antisemitism, homophobia, abusive admin, nazi imagery > - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/cunnyborea.space) > > ::: spoiler Evidence > - https://i.imgur.com/HgvMVOh.png > - https://union.place/@inquiline/111247251661793670 > :::

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] cunnyborea.space

- **Instance**: cunnyborea.space - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Racism, antisemitism, homophobia, abusive admin, nazi imagery - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/cunnyborea.space) ::: spoiler Evidence - https://i.imgur.com/HgvMVOh.png - https://union.place/@inquiline/111247251661793670 :::

cross-posted from: https://pawb.social/post/3337642 > - **Instance**: bv.umbrellix.org > - **Type**: Defederation > - **Affects**: Pawb.Social, furry.engineer, pawb.fun > - **Reason**: Toxicity, abusive admin, death threats, emotional abuse > - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/bv.umbrellix.org) > > ::: spoiler Evidence > - (CW: Abuse) https://bv.umbrellix.org/notice/AZwi9Is6mxT9vo5TSi > - https://sunbeam.city/@kayttaja@toot.site/111142509643025163 > ::: > > **Admin Note**: This block applies to the root domain (*umbrellix.org*) and all associated sub-domains.

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] bv.umbrellix.org

- **Instance**: bv.umbrellix.org - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Toxicity, abusive admin, death threats, emotional abuse - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/bv.umbrellix.org) ::: spoiler Evidence - (CW: Abuse) https://bv.umbrellix.org/notice/AZwi9Is6mxT9vo5TSi - https://sunbeam.city/@kayttaja@toot.site/111142509643025163 ::: **Admin Note**: This block applies to the root domain (*umbrellix.org*) and all associated sub-domains.

Pawb.Social Announcements crashdoom • Now • 100%

[Maintenance] Oct 8th Uplink infrastructure change

Currently, we’re running the Ubiquiti Dream Machine directly as the modem via PPPoE, but there appears to be an intermittent issue with the software implementation that results in periodic downtimes of a few minutes while it reconnects. We’re looking at switching this back out with the ISP provided router in pass through mode to negate the PPPoE connectivity drop. We don’t expect this to take longer than 1 hour to switch over and test for reliability before bringing the services back up. We’ll be performing this maintenance around 11 AM US Mountain Time, and will provide updates via the Telegram channel at https://t.me/pawbsocial.

cross-posted from: https://pawb.social/post/3077174 > - **Instance**: liberdon.com > - **Type**: Defederation > - **Affects**: Pawb.Social, furry.engineer, pawb.fun > - **Reason**: Harassment, trolling, misinformation, sexism, homophobia > - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/liberdon.com) > > ::: spoiler Evidence > - https://liberdon.com/@ghast > - https://liberdon.com/@ghast/111178732016951128 > - https://liberdon.com/@rimugu/111178647646581863 > - https://liberdon.com/@aristoh4ck8r/111177188127718387 > - https://liberdon.com/@gunkslinger/111175171054509579 > - https://liberdon.com/@TylerAbeoJordan/111174105820298526 > - https://liberdon.com/@TylerAbeoJordan/111174078685171464 > - https://liberdon.com/@TylerAbeoJordan/111173743376687580 > - https://liberdon.com/@medicineman9/111173062104385765 > - https://liberdon.com/@spinneria/111169619274938220 > - https://liberdon.com/@nosat/111165212240469156 > ::: >

Pawb.Social Announcements crashdoom • Now • 100%

[Unexpected Downtime Postmortem] Oct 6th

One of the data storage systems (CEPH) encountered a critical failure when Proxmox lost connection to several of its nodes, ultimately resulting in the CEPH configuration being cleared by the Proxmox cluster consensus mechanism. **No data, except ElasticSearch, was stored on CEPH.** When the connection was lost to the other nodes, a split-brain occurred (when nodes disagree on which changes are authoritative and which should be dropped). As we tried to recluster all of the nodes, a resolution occured that resulted in the ceph.conf file being wiped and the data on CEPH being unrecoverable. Thankfully, we’ve suffered no significant data loss, with the exception of having to rebuild the Mastodon ElasticSearch indexes from 6 AM this morning to present. I’d like to profusely apologize for the inconvenience, but we felt it necessary at the time to offline all services as part of our disaster recovery plan to ensure no damage occurred to the containers while we investigated.

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] liberdon.com

- **Instance**: liberdon.com - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Harassment, trolling, misinformation, sexism, homophobia - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/liberdon.com) ::: spoiler Evidence - https://liberdon.com/@ghast - https://liberdon.com/@ghast/111178732016951128 - https://liberdon.com/@rimugu/111178647646581863 - https://liberdon.com/@aristoh4ck8r/111177188127718387 - https://liberdon.com/@gunkslinger/111175171054509579 - https://liberdon.com/@TylerAbeoJordan/111174105820298526 - https://liberdon.com/@TylerAbeoJordan/111174078685171464 - https://liberdon.com/@TylerAbeoJordan/111173743376687580 - https://liberdon.com/@medicineman9/111173062104385765 - https://liberdon.com/@spinneria/111169619274938220 - https://liberdon.com/@nosat/111165212240469156 :::

Pawb.Social Announcements crashdoom • Now • 100%

Migration of Pawb.Social media

We've migrated all of the Pawb.Social (Lemmy) media from within the container to the new local NAS. We'll be performing the same updates for pawb.fun and furry.engineer over the course of the next week, minimizing downtime where possible. We will need to take the services offline once the initial transfer is completed to perform a final check for any new data, and transition to the local NAS, but we'll post an announcement prior to the final maintenance for each instance. These changes are to allow us longer and cheaper retention of remote and local media across all of our services, and to allow us to begin backing up media in case of a catastrophic failure.

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] asimon.org

- **Instance**: asimon.org - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Racism, homophobia, transphobia, trolling, abusive admin - **Fediseer Action**: [Censured](https://gui.fediseer.com/instances/detail/asimon.org) ::: spoiler Evidence https://asimon.org/@coin/posts/Aa8tebnT4kYwTVwrwW https://asimon.org/@coin/posts/AZxmCGq8YxrIx5B6UC https://asimon.org/objects/eda7faae-454b-46f7-a309-94066c261ce6 https://asimon.org/@coin/posts/AZKA8C9YIF3zLb2WLA :::

Pawb.Social Announcements crashdoom • Now • 100%

Changes to Mastodon search and you

With the upgrade to Mastodon 4.2.0, the way the search system works is changing to allow more broad searches across posts made across the fediverse. Cutting to the chase: **Full-text search** across opted-in posts will be possible in the next few days. ---- In more depth, the system will allow you to search across your own posts and posts you've interacted with from others irrespective of their opt-in status for full-text search, but will additionally allow you to search **ANY** posts made by users that opt-in to allow their posts to be indexed. To that effect, a new setting was added to the **Public Profile** > **Privacy and Reach** page called "Include public posts in search results" which is **disabled** by default. Turning this on will allow any user across the Fediverse to search and find your posts using full-text search. Shortcuts: - [furry.engineer users click here](https://furry.engineer/settings/privacy) - [pawb.fun users click here](https://pawb.fun/settings/privacy) - For everyone else: There's Mastercard ~~(Sorry not sorry xD)~~ But, just take your instance's domain and add `/settings/privacy` to the end (e.g. `https://INSTANCE.TLD/settings/privacy`)  To allow our users to decide if this is a feature they want to use on their posts going forward, we won't be enabling ElasticSearch until *Friday Sept. 22*. ---- Additionally, new keywords will be added to the search system: - `from:me` - `before:2022-11-01`, `after:2022-11-01`, `during:2022-11-01` - `language:fr` - `has:poll` - `in:library` for searching only in posts you have written or interacted with These will be enabled on Friday along with the rest of the changes to the search system. ---- If you've got any questions, please let us know!

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] eveningzoo.club

- **Instance**: eveningzoo.club - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Trolling, racism, harassment, no moderation policy ::: spoiler Evidence - https://eveningzoo.club/notice/AZQrHEzRzNTbuXl2VU - https://eveningzoo.club/notice/AZRLrlgctoMuMoHpYW :::

Pawb.Social Administrative Actions crashdoom • Now • 100%

[Defederation] shitpost.cloud

- **Instance**: shitpost.cloud - **Type**: Defederation - **Affects**: Pawb.Social, furry.engineer, pawb.fun - **Reason**: Trolling, racism, harassment, no moderation policy ::: spoiler Evidence - https://shitpost.cloud/@Hoss/posts/AZQgtK4WdzKlw6SGiO - https://shitpost.cloud/@Hoss/posts/Aa5wKz5h08jGFC2fw0 - https://shitpost.cloud/@Hoss/posts/AZQgtK4WdzKlw6SGiO :::