Personal Drive

TL;DR

This blog post will cover how to setup a monitored 📊 & secure personal cloud storage ⏏️, behind a proxy ⚙️, to let you expose other nice services on your home server 🏠

Runaway from Google Drive and Google Photos 📷, host your cherry picked bytes at home 🚀 !

Intro

My girlfriend recently ran into a stressfull issue, let me explain briefly the root of the problem 😬

She enjoy, as I do, taking pictures of our trips and daily life. Fortunately, Google has exposed the incredible feature of saving and sharing Photos from mobile in a glimpse, and also make use of shared albums. Let’s say it, it is incredible !

However, as time goes, the amount of photos reached the free tier limit, and we found unacceptable to pay (far more when you are aware that you are giving “freely” to Google a huge sample of photos, that could be used as dataset for ML engineering stuff) for an extra storage amount.

Furthermore, as there is no proper isolation between Drive, Photos and Gmail, Google was sending my girlfriend alerts saying that she couldn’t receive more mail, pretty uncomfortable when you are using this email as your primary source of digital contact (train & airplane ticket, bank account, cellphone billing..)

POV of a new email 📧 arriving on almost full Gmail inbox 🤣 :

The craftsmanship which resides in me came up with an idea to leverage the Raspberry Pi I use at home to stream medias ( Plex Subscription, well deserved money), but there to host photos and drive storage.

Here is the system context :

And here is the detailled overview of the container “RaspberryPI” ( C4 model terminology)

It puts a little bit more pressure because it has to be more backed up than delegating it to Google, but I felt confident about setting this up (and Uptobox back me up in case my HDD at home burn or get in trouble).

This is the architecture I’ve come up to, I wanted it to be isolated with some dockers containers for the ease of deploy/redeploy, secure during connections and ultimately I wanted it to be totally free (big thanks to Newtcloud team for the incredible work). The backup will be made by sync local hard drives and store it on an Uptobox account (only incurring cost + DNS name, but it’s a few euros per month and I would have used theses services anyway).

To monitor the whole stack, a Grafana dashboard will detail a little bit the system characteristics (CPU, RAM usage, filesystem usage of the Linux OS partition, network traffic, docker uptimes)..

Here is a little overview of components used :

• NextCloudPi
• Grafana
• Docker
• Let’s encrypt
• Traefik

All theses graphs have been generated using Structurizr, here is the architecture expressed in the DSL of C4 model :

workspace "NextCloudPi" "Home Personnal Storage System" {

    model {
        u = person "Managed Users"
        monitoring = softwareSystem "Grafana"
        webcloudprovider = softwareSystem "Cloud Storage external Provider (Uptobox)"
        personnalcloudstorage = softwareSystem "RaspberryPI" {
            webapp = container "NextCloudPi Web Application"
            database = container "Database"
            physicalstorage = container "HDD1"
            physicalbackupstorage = container "HDD2"

            u -> webapp "Manage personal data throught computer"
            webapp -> database "Store elements"
            database -> physicalstorage "is mounted on"
            physicalstorage -> physicalbackupstorage "is replicated (CRON)"
            physicalstorage -> webcloudprovider "is replicated (CRON)"

        }
        client = softwareSystem "Smartphone"
        monitoring -> personnalcloudstorage "Display and monitor system info"
        personnalcloudstorage -> client "Display and share data"
        client -> personnalcloudstorage "Upload to backup data"
        u -> client "Manage personal data throught smartphone"

    }

    views {

    }

}

Now the base roots of the architecture have been written down, we can move on into deploying the solution.

Deploying the solution

Firewall

First we need to secure your exposed raspberry pi with ufw and fail2ban. As the pi will be exposed to the internet, if you wish to access it in SSH, it is important to setup a good firewall to avoid brute force attacks, SSH fuzzers and all kind of scary stuff like that (do not worry, with a proper long password and good banning of failed IPs, it is pretty decent and you will not be bothered).

Let’s start by installing fail2ban paquet

sudo apt-get update && apt-get install fail2ban

Launch and verify status of fail2ban

sudo systemctl start fail2ban && sudo systemctl enable fail2ban && sudo systemctl status fail2ban

Create a default conf for the jail ban by modifying this file /etc/fail2ban/jail.d/custom.conf.

Be sure to add some IPs you will not include in the jail, in order to do not block yourself in case you mess up passwd.

[DEFAULT]
ignoreip = 127.0.0.1 124.32.5.48
findtime = 10m
bantime = 24h
maxretry = 3

Quick explanation of settings If an IP fail over 3 times to connect in SSH during the last 10 minutes, it goes to jail for a ban time of 24h ☠️

Monitor the activity of ssh by activating the watch of ssh daemon, sshd, modifying this file /etc/fail2ban/jail.d/custom.conf

[sshd]
enabled = true

Reload fail2ban configuration to apply new jails configs

sudo systemctl restart fail2ban

Breaking into the Pi is now far more difficult, good job 😈

Here is a small snapshot at the moment when I was writing theses lines. You can get yours (wait a few minute that some bot try to connect to your server).

sudo fail2ban-client get sshd banip --with-time

Look at all theses IPs that have been jailed !!

45.136.153.217  2023-07-28 19:27:25 + 86400 = 2023-07-29 19:27:25
134.122.88.190  2023-07-29 02:07:52 + 86400 = 2023-07-30 02:07:52
193.35.18.169   2023-07-29 02:26:39 + 86400 = 2023-07-30 02:26:39
170.64.187.84   2023-07-29 04:06:54 + 86400 = 2023-07-30 04:06:54
159.203.46.152  2023-07-29 04:20:42 + 86400 = 2023-07-30 04:20:42
45.129.14.64    2023-07-29 04:27:53 + 86400 = 2023-07-30 04:27:53
5.187.112.81    2023-07-29 07:05:13 + 86400 = 2023-07-30 07:05:13
116.110.84.51   2023-07-29 14:19:34 + 86400 = 2023-07-30 14:19:34
45.128.232.71   2023-07-29 15:03:22 + 86400 = 2023-07-30 15:03:22

Add NAT rules with port 22 on your router, so that you can ssh from the internet.

I also activate ufw on the pi exposed to avoid traffic coming in on unexpected ports, even if I do not open them in the router, but it’s a security.

Install ufw with :

sudo apt install ufw

Enable ssh before applying the firewall, otherwise you will make your Pi unreachable remotely 😂:

sudo ufw allow ssh

Now enable the firewall :

sudo ufw enable

Reverse Proxy with Docker

The use of traefik is very simple, all you have to do is to add proper lables on docker services, and traefik will watch upon the docker socket to know what to do. Very powerful and evolutive !

Let’s first create the docker network that will be used to communicate with other services :

docker network create pi

To enable Traefik, we first have to enable the docker service of traefik. To do so, instanciate the following docker-compose file :

version: '3.4'
services:
  traefik:
      image: 'traefik:2.3'
          container_name: 'traefik'
    restart: 'unless-stopped'
    ports:
      - '80:80'
      - '443:443'
      - '8080:8080'
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
      - './traefik/traefik.toml:/etc/traefik/traefik.toml:ro'
      - './traefik/traefik_dynamic.toml:/traefik_dynamic.toml:ro'
      - './ssl:/etc/ssl:ro'
    networks:
      - pi

  whoami:
    image: 'traefik/whoami'
    restart: 'unless-stopped'
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.whoami.rule=PathPrefix(`/whoami{regex:$$|/.*}`)'
      - 'traefik.http.services.whoami.loadbalancer.server.port=80'
      - 'traefik.http.routers.whoami.tls'
    networks:
      - pi

networks:
  pi:
    external: true

Make use of 2 files, the static traefik configuration (including the enabling of logs, of dashboard and docker socket watching)

Here is the content of the file to place under traefik/traefik.toml :

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.websecure]
    address = ":443"

[api]
  dashboard = true

[providers.docker]
  watch = true
  network = "web"
  exposedByDefault = false

[providers.file]
  filename = "traefik_dynamic.toml"

[accessLog]

We define :

• 2 entrypoints, upon the port 80 for simple HTTP protocol, and a HTTPS access on port 443
• The activation of the cool traefik dashboard, reachable at http://yoururl.com/dashboard/, (NB: Be sure to add the last trailing slash after dashboard, otherwise the dashboard is not accessible.)
• The activation of docker socket watching, to add new services on the fly and to disable docker exposure if not configured by lables (better for security considerations)
• A dynamic configuration file traefik_dynamic.toml
• The display of access log on the reverse proxy

Great, the traefik serve now serve proxied services throught websecure and web entrypoints. Let’s now secure the vizualization dashboard by generating and adding an htpass middleware in front of dashboard entrypoint :

sudo apt-get install apache2-utils

Generate a hash for your admin password, the result is displayed on stdout:

htpasswd -Bnb admin superpassword123

Here is the dynamic configuration referencing the basic auth middleware :

[http.middlewares.simpleAuth.basicAuth]
  users = [
    "admin:$2y$05$fJ.OOvXhMrXSC4s6uDgtp.q320RmTyHQ7iuya90TKS./LXa5QoJue"
  ]

[http.routers]
  [http.routers.unsecurerouter-api]
    rule = "Host(`192.168.X.X`) || Host(`yoururl.fr`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
    entrypoints = ["web"]
    middlewares = ["simpleAuth"]
    service = "api@internal"

  [http.routers.securerouter-api]
    rule = "Host(`192.168.X.X`) || Host(`yoururl.fr`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
    entrypoints = ["websecure"]
    middlewares = ["simpleAuth"]
    service = "api@internal"
    [http.routers.securerouter-api.tls]

[[tls.certificates]]
    certFile = "/etc/ssl/ssl-cert-snakeoil.pem"
    keyFile = "/etc/ssl/ssl-cert-snakeoil.key"

We defined :

• A middleware using the previously generated password for admin user
• Some routers to redirect paths beginning by /dashboard to the traefik dashboard
• A secure and an unsecure router, the secure ones are using certificates stored under /etc/ssl (We will review review later on how to generate yours with Let’s Encrypt)

Let’s fire up the traefik service, navigate to the docker-compose file, which reside alongside dynamic and static toml configuration files :

docker-compose up -di

Now navigate to your dashboard, at http://yoururl/dashboard/ and you should see this :

Well done, your proxy is working and ready to serve all your services!!

Setup Monitoring with Grafana

Now let’s deploy a Grafana dashboard with a docker compose, add some labels to configure routing and proxying with traefik, and you are good to go to monitor your server 24/24 7/7 without a pain.

To do so, instanciate a docker-compose.yml file with a few services in a monitoring folder.

Prepare some data folders (for persisting data accross reboots of your PI).

mkdir ~/monitoring && cd ~/monitoring && mkdir -p prometheus/data grafana/data && \
sudo chown -R 472:472 grafana/ && \
sudo chown -R 65534:65534 prometheus/

Here is the docker-compose file :

version: "3"
services:
  grafana:
    container_name: monitoring-grafana
    image: grafana/grafana:latest
    hostname: rpi-grafana
    restart: unless-stopped
    user: "472"
    networks:
      - pi
    expose:
      - 3000
    env_file:
      - ./grafana/.env
    volumes:
      # /!\ To be modified depending on your needs /!\
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    depends_on:
      - prometheus
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=PathPrefix(`/grafana{regex:$$|/.*}`)"
      - "traefik.http.routers.grafana.tls"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"
      - "traefik.frontend.headers.customRequestHeaders=Authorization:-"
      - "traefik.http.routers.grafana.middlewares=simpleAuth@file"

  cadvisor:
    container_name: monitoring-cadvisor
    image: zcube/cadvisor:latest
    hostname: rpi-cadvisor
    restart: unless-stopped
    privileged: true
    networks:
      - pi
    expose:
      - 8080
    devices:
      - /dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /etc/machine-id:/etc/machine-id:ro

  node-exporter:
    container_name: monitoring-node-exporter
    image: prom/node-exporter:latest
    hostname: rpi-exporter
    restart: unless-stopped
    networks:
      - pi
    expose:
      - 9100
    command:
      - --path.procfs=/host/proc
      - --path.sysfs=/host/sys
      - --path.rootfs=/host
      - --collector.filesystem.ignored-mount-points
      - ^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
      - /:/host:ro,rslave

  prometheus:
    container_name: monitoring-prometheus
    image: prom/prometheus:latest
    hostname: rpi-prometheus
    restart: unless-stopped
    user: "nobody"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
    networks:
      - pi
    expose:
      - 9090
    volumes:
      # /!\ To be modified depending on your needs /!\
      - ./prometheus/data:/prometheus
      - ./prometheus:/etc/prometheus/
    depends_on:
      - cadvisor
      - node-exporter
    links:
      - cadvisor:cadvisor
      - node-exporter:node-exporter

networks:
  pi:
    external: true

• Prometheus expose some system metrics from the RaspberryPI (CPU, RAM…)
• Node-exporter send the data to monitoring system (grafana)
• Grafana expose the data behind the traefik proxy
• cAdvisor monitor containers exposed on the RaspberryPI

NB: Be sure to install a 64 bit version of Ubuntu otherwise some containers won’t be available !

Fire up the stack :

docker-compose up -d

Now head to the grafana dashboard at http://yoururl/grafana/ and

• Open another web browser tab and follow this url Grafana.com Shared Dashboard
• Click on COPY ID to clipboard or Download JSON
• Head to your own dashboard at http://yoururl/grafana/ and click on Import under Dashboard tab
• Paste the ID and click on Load or Upload JSON file with the downloaded file
• Click on Load

Et voilà 🔥

Now you should see this :

Pretty cool, isn’t it ? All your dockers are monitored and also your RaspberryPI !

Deploy NextcloudPi

Now everything to host Nextcloud on your RaspberryPI is up and running, let’s prepare your hard drive you are going to setup in order to use an additionnal external storage rather than the root volume of your pi.

It’s better to isolate storage purposes but it’s not mandatory, you can store everything on the root volume if you prefer to do so.

If you plan to use an external storage, follow along with me !

First we need to identify the drive id we will format (do not mess up with other connected drive and be sure to be on your target system before applying anything!!)

lsblk

Here is an example of my setup with 2 hard drive (a 2 To one and a 500 Gb one) attached on my RaspberryPI (using this powered USB hub ).

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 465.8G  0 disk
├─sda1   8:1    0   256M  0 part /boot/firmware
└─sda2   8:2    0 465.5G  0 part /
sdb      8:16   0   1.8T  0 disk
└─sdb1   8:17   0   1.8T  0 part
sdc      8:32   0 465.7G  0 disk
└─sdc1   8:33   0 465.7G  0 part

Let’s now format the hard drive with a file system that allow users permissions and so on. Be aware that everything on the hard drive will wiped out during the process !

An example here by formatting the partition on the third hard drive with ext4 (GNU/Linux filesystem)

sudo mkfs -t ext4 /dev/sdc1

Now we will edit the fstab, e.g. we will indicate to the OS where each drive should be mounted.

Identify the UUID of your formatted disk, by running this command :

lsblk -f

Edit the file to enter theses values, replacing the UUID and the target location to something explicit :

UUID=THE-ADDRESS-GOES-HERE /mnt/directory/location ext4 defaults 0

Now you can mount the HDD (it will be done automatically at each startup and boot) :

fstab mount -a

Change the ownership of the target location to the user that will be used by Nextcloud in the docker image :

sudo chown www-data:www-data -R /mnt/directory/location

Now, as one doesn’t change a team that wins, let’s setup another docker-compose file to fire up the Nextcloud stack.

version: "3.4"
services:
  nextcloud:
        container_name: nextcloudpi
            image: ownyourbits/nextcloudpi
    ports:
      - 8880:80
      - 8443:443
      - 4443:4443
    volumes:
      - '/media/backup2to/ncp/:/data'
      - 'ssl/ssl-cert-snakeoil.pem:/etc/ssl/certs/ssl-cert-snakeoil.pem:ro'
      - 'ssl/ssl-cert-snakeoil.key:/etc/ssl/private/ssl-cert-snakeoil.key:ro'
    restart: unless-stopped
    command: 127.0.0.1

Where is traefik here ? It is not used as you can see because we are binding the port directly to the host, but not on the ports 80 and 443 so your webserver is available to host a ton of other services !

Be sure to open theses ports in your NAT to let internet traffic go along the way. No need for filtering as everything is secure on your Owncloud stack.

Retrieve admin password by running ncp-config inside the docker after you had docker-compose up -d the stack. Take the opportunity also to allow your DNS name or local IP to the dashboard, otherwise you will be unable to reach it !

Download data from Google Photos with Google Takeout.

Dump the zip content in your Nextcloud data folder :

unzip 'takeout-yourzips-0*.zip' -d /media/backup2to/ncp/ncdata/data/yourusername/files/Photos/Takeout

Change again the ownership of the dumped photos :

sudo chown www-data:www-data -R /mnt/directory/location/ncdata/data/

Rescan the content of your files :

docker exec -u 33 -it sh -c /var/www/nextcloud/occ files:scan --all

If your are using the excellent Nextcloud app Memories, regenerate EXIF metadata from your newly updated photos

docker exec -u 33 -it sh -c /var/www/nextcloud/occ memories:migrate-google-takeout

Here is a preview (blurred to keep a little bit of privacy) , so cool isn’t it ?

Congratulations, you own your data from now, hosted at home !! 🎊

Everything is almost done, all we have left to do, is to make a DNS challenge in order to generate a wildcard SSL certificate. You wouldn’t transfer all your data, password, notes, agenda in plain unciphered connections ? Neither do I 😄

Let’s go for enabling HTTPS on traefik and nextcloud !

First, install certbot on your machine

sudo apt update
sudo apt install certbot

Download the acme challenge script and make it executable :

wget https://github.com/joohoi/acme-dns-certbot-joohoi/raw/master/acme-dns-auth.py
chmod +x acme-dns-auth.py

Edit the informations of the script to make it launchable using python3 Move the script to certbot folder :

sudo mv acme-dns-auth.py /etc/letsencrypt/

Set up the acme-dns-certbot :

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d \*.your-domain -d your-domain

Be sure to substitue domain names and escaping the the asterix with the backslash !

Look at the output, and add the proper CNAME entry in your DNS provider. Example output :

Output
...
Output from acme-dns-auth.py:
Please add the following CNAME record to your main DNS zone:
_acme-challenge.your-domain CNAME 12345678-9abc-def0-1234-56789abcdef0.auth.acme-dns.io.

Waiting for verification...
...

You should add CNAME 12345678-9abc-def0-1234-56789abcdef0.auth.acme-dns.io entry.

Once ok the certbot will detect the dns change and generate the wildcard certificates. Copy the generated SSL private and public certificates along traefik (look at the docker-compose file) and nextcloud one.

Make a folder ssl :

ssl
├── ssl-cert-snakeoil.key
└── ssl-cert-snakeoil.pem

Copy it next to docker-compose of traefik and nextcloud.

Add this router to traefik dynamic configuration, in order to tell to traefik to serve this service under SSL. Traefik will also rewrite the URL in order to link to the good port, if you try to go to yoururl/nextcloud.

[http.middlewares]
  [http.middlewares.redirectnextcloud.redirectRegex]
    regex = "^http(s?)://yoururl.fr/nextcloud(.*)"
    replacement = "https://yoururl.fr:8443"

[http.routers]
  [http.routers.nextcloud]
    rule = "PathPrefix(`/nextcloud`)"
    entrypoints = ["websecure"]
    middlewares = ["redirectnextcloud"]
    service = "nextcloud"
    [http.routers.nextcloud.tls]

[http.services]
  [http.services.nextcloud.loadBalancer]
    [http.services.nextcloud.loadBalancer.healthCheck]
      path = "/"
      scheme = "https"
      hostname = "yoururl.fr"
      port = "8443"

Your are now backed up by a secure encrypted SSL connection 😎

Install Sync clients on Smartphones

For convenient usage, you can install nextcloud client on Android devices to upload directly files and take notes (Drive and Keep replacement).

here is the link for Android apps :

• Nextcloud
• Nextcloud Notes

To sync your calendar, tasks and contacts, you can use Davx5

• DAVX5 on Play Store
• DAVX5 on FStore

To use and view photos as efficiently than on Google Photos App:

• Photos

You can move around will all your data in your pocket and keep all your documents centralized at home.

To setup clients, please follow Davx5 and Nextcloud documentation. Very efficient !

Setup a 3-2-1 backup strategy

Now everything is at home be sure to backup you data efficiently, otherwise you would loose everything. Let’s follow the 3-2-1 rule :

• A backup on 3 differents locations
• On 2 separate physical volumes
• With at least 1 copy stored remotely (in case everything burns at home)

Keep in mind that a real backup solution is a backup that is recoverable. Be sure to test your solution by trying to import backup time to time and check eveything is ok (very easy with docker compose, just instanciate a new one and change source volume).

A simple solution is to use the ncp-config backup tool and to rsync the zipped content on the hard drive on another hard drive. It can be restored with ncp-config too.

 sudo rsync -vlr --info=progress2 /media/principalstorage/ncp/backup.zip /media/storagebackup/ncp/backup.zip

In case of hard drive failure, the other drive would contain your precious data.

Once a month, upload a copy to Uptobox, in order to save data in another safe place.

Once the zip produced, it is ready to be shipped, using uptobox-cli NB: Be sure to update nameserver for Cloudflare ones, as sometimes Uptobox is kicked out by french FAI

Setup the cli, install with npm install and fill config.js with proper credentials and set premium to 1.

Once done, upload your file

node /home/pi/uptobox-cli/index.js uploadFiles backupnextcloud.zip

Conclusion

Congratulations, you have now a fully backed up system with your photos, documents, calendar, contacts stored at home.

Furthermore, with traefik you can plug another services and expose them very easily on your server, just using some labels on your docker-compose services.

To conclude, Grafana dashboard help you monitor all your stack and ensure you are filling up your disks or getting low on RAM for instance.

Be sure to regularly backup your data (give a go to CRON jobs, they will save some much time). As it is automatically done, just ensure that the job still work times to times, and try to do the recover exercice in case something goes wrong. As a free advice, I encourage you to save your docker-compose files in a VCS (private to do not expose hashs) system, like Github or Gitlab.

Feel free to share with other of your household for them to create an account, they will be able to store their data and you can share files with them.

Give a go to “Notes” app and to “memories”, I find them amazing. You can install easily them from the admin dashboard in the admin account directly in Nextcloud.

Feel free to share with me in comments your tips about backup or some other system you have found that are as cool as Nextcloud, I heard about CasaOS but I don’t know if it’s as good as Nextcloud or not.

See you soon, and thanks for your time.

Some sources and refering links that helped me setting up this whole system.

Thank you to all developers that have shared advices and recommendations. ❤️ to Nextcloud team

• Nextcloud Guide 2020

• Let’s Encrypt and SSL certificates

• Wildcard certificate

• Official Guide NextcloudPI

• Rsync guide

• 3-2-1 Model