This is the home of the social.coop tech group! If you are new to the group, please see the new user guide.

Services

The tech group is responsible for operating and maintaining the following services:

Primary services
Social.coop mastodon instance admin panel
wiki.social.coop public wiki https://wiki.social.coop
Supporting services
registrar gandi
registrant organization: Xarxa integral de professionals i usuaries
DNS / DDOS protection cloudflare
Mailgun for emails sent by mastodon mailgun
@social.coop email aliases webarch.mail
Object store for backups and digital assets digital ocean spaces
Monitoring / metrics datadog
Code repos git.coop/social.coop/tech

Our git.coop repositories

Repo purpose
tech gitlab group list of all repos
sauce docker config and some systemd services (to be migrated)
ansible server configuration
pass encrypted password store for shared passwords
wiki code for the public metalsmith wiki

Mastodon runbook

Our fediverse instance is the raison d`etre of the social.coop coop. This is what the community signs up for and our primary responsibility. The primary points of administration are:

If you need server access please see server access and How to grant shell access

Mastodon is running under Docker-compose.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. We are using Systemd as the init system for docker-compose.

systemd services

service purpose
social.coop-mastodon a service to control the mastodon installation via docker-compose
social.coop-remove-media runs the media cleanup command to remove remote media >7 days old via a .timer
certbot runs the renewals via .timer

logs

command purpose
systemctl list-timers lists timers!
journalctl -f tail ALL system logs
docker-compose logs -f web view and tail web logs (when in /opt/social.coop/sauce/docker/)
docker-compose logs -f db view and tail db logs (you get the pattern?)
journalctl -f -u certbot see when certbot was run
journalctl -f -u social.coop-mastodon see the output from the docker-compose commands ran with systemctl, but not the docker container logs themselves
journalctl -f -u social.coop-remove-media see what the remove media command is up to

Service management

All of these commands must be run on runko.social.coop in the /opt/social.coop/sauce/docker/ directory.

command purpose
docker-compose ps List all Docker containers
docker-compose stop redis Stop a service
docker-compose start redis Start a service
sudo docker-compose scale sidekiq-default-q=5 scale a service, in this case create 5 sidekiq default queue workers
docker-compose up -d (or systemd refresh social.coop-mastodon - does same thing) Redeploy (only changed things)

If you want to run commands make sure to use the --rm argument, or the containers will hang around.

e.g. docker-compose run --rm web rails console (to get a rails console)

Backups

Configure by the pg-dump-to-s3 ansible role.

Location of Postgres database files: /opt/social.coop/var/lib/postgresql/data/

Mastodon upgrade notes

  • it is recommended to run upgrades inside of tmux in case you are disconnected from the server.
  • get current version from social.coop
  • find next version from mastodon github releases
  • check upgrade notes
  • check whether there are DB migrations
  • make backup? systemctl start pg-dump-to-s3.service
    • takes 15 mins or so?
  • separate command to see backup progress
  • make merge request on git.coop sauce repo to bump version in a couple of places in docker-compose.yaml
  • git diff v3.1.2..v3.1.3 -- docker-compose.yml in mastodon repo after pulling to check whether there were any changes we should consider mirroring to our docker-compose file
  • could be cool to make these merge requests in advance
  • write a toot announcing upgrade and boost on admin account
  • touch file on server to activate maintenance mode
  • actually do the upgrade
    • migration command creates a fresh web container and runs the migration command and then deletes that new container
  • turn maintenance mode off
  • we copy static assets outside of the container so they can be served by nginx
    • there's a command for this which moves stuff into a temporary dir in nginx and pulls assets out of docker container into that folder in docker container
  • ssh forwarding is nice, then with sudo -E -s you have ssh access to stuff you do from host machine(?)

Hardware

Mastodon and Wiki run on runko.social.coop, a dedicated server at Hetzner:

  • 32GB RAM (4x RAM 8192 MB DDR3)
  • i7-4770 CPU @ 3.40GHz).
  • 2x 250 GB disks (SSD)
  • Ubuntu 18.04

Storage

We are using LVM.

NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
	  sda             8:0    0 223.6G  0 disk 
	  ├─sda1          8:1    0     1G  0 part /boot
	  └─sda2          8:2    0 222.6G  0 part 
	    ├─vg0-root1 253:0    0    25G  0 lvm  /
	    ├─vg0-root2 253:1    0    25G  0 lvm  
	    └─vg0-opt   253:2    0 396.1G  0 lvm  /opt
	  sdb             8:16   0 223.6G  0 disk 
	  └─sdb1          8:17   0 223.6G  0 part 
	    └─vg0-opt   253:2    0 396.1G  0 lvm  /opt
  LV    VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
	    opt   vg0 -wi-ao---- 396.13g                                                    
	    root1 vg0 -wi-ao----  25.00g                                                    
	    root2 vg0 -wi-a-----  25.00g

opt is mounted at /opt.

wiki.social.coop

wiki.social.coop has two main purposes:

  1. a public-facing site with information about social.coop
  2. the registration system for new users

The code repo for the project is tech/wiki.social.coop and the content comes from the general wiki.

It's configured/deployed via ansible using the wiki.social.coop role and the wiki tag, so ansible-playbook server.playbook.yml --tags wiki will set it up.

The configuration secrets are stored in the pass repo at:

deployment/wiki/gitlab_token
	  deployment/wiki/gitlab_username
	  deployment/wiki/mailgun_password
	  deployment/wiki/webhook_secret

On the server it lives at: /opt/social.coop/wiki.social.coop/ and is running as the systemd service wiki.social.coop.service.