I recently decided to poke around and see how difficult it was to install Ceph on a Raspberry Pi. While it has been done a bunch of times before, I wanted to see how the latest and greatest (Spring 2019-ish) code worked, and how much effort was required. The great news is that it’s almost turn-key at this point using an arm64 debian distribution. The luminous ceph packages are available and they work!
nnnnIt’s not fast by any stretch, primarily because of the network-to-USB-attached-disk contention within the RPi . But it does work. Longer term, memory consumption is an issue . But I’ve let it idle for over a week and heartbeats/scrubs seem to run ok.
nnnnThis whole setup becomes far more interesting when you consider it applied to one of the RPi alternatives, some of which have PCIe NVMe slots, and much faster networking and CPU. Especially when you consider the price trajectory of NVMe TLC and QLC storage devices!
nnnnI’m mostly just pasting the commands from my shell history here so I can use them another time.
nnnnCookbook-level instructions
nnnnFirst, get the “Buster” beta for the raspberry pi and write it to an SD card. More info: https://itsfoss.com/debian-raspberry-pi/
nnnnAs Root:
nnnnn# Initial installation of packages not installed by default with Debian arm64 bustern apt-get upgraden apt-get updaten apt-get updaten apt-get install lvm2 sudo gnupg gnupg2 gnupg1 lsb-base lsb-releasenn# Setup the ceph repo and add some more packagesn wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -n echo deb https://download.ceph.com/debian/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.listn echo $?n apt cleann apt-get updaten apt-get install ceph-deployn adduser ceph-deployn echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploynn# /dev/sda is a USB attached SSD... I set it up with 4 partitions to do 4 OSDsn fdisk /dev/sdan# had to reboot, kernel wouldn't see new partition table otherwisen shutdown -r nownn# For some reason, the distribution installed by ceph-deploy with the armhf repo setup above didn't install the ceph-volume systemd service.n cd /lib/systemd/system/n wget https://raw.githubusercontent.com/ceph/ceph/master/systemd/ceph-volume%40.serviceAs user ceph-deploy:
nnnnn# Setup public key ssh access (to ourselves)n# Server is called 3pio in DNS.n ssh-keygenn ssh 3pion ssh-copy-id ceph-deploy@3pion ssh 3pionnn# Use ceph-deploy to create a cluster (and install ceph software)n mkdir testclustern cd tesclustern ceph-deploy new 3pion vim ceph.confn ceph-deploy install --release luminous --no-adjust-repos 3pion ceph-deploy mon create-initialnn# For some reason ceph-deploy doesn't copy the keyrings to /etc/ceph so I did this manually.n sudo cp *keyring* /etc/ceph sudo chmod o+rx /etc/ceph/*n ceph-deploy osd create 3pio --data /dev/sda1n ceph-deploy osd create 3pio --data /dev/sda2n ceph-deploy osd create 3pio --data /dev/sda3n ceph-deploy osd create 3pio --data /dev/sda4n ceph-deploy admin 3pion ceph-deploy mgr create 3pionn# Ceph should be up, check it out and do a few thingsn ceph -sn ceph osd treen ceph osd pool create testpool 512 512 replicated 0n ceph -sn rados bench 15 write -p testpool -b 524288n ceph -sn ceph dfn ceph osd dfAll done!
nnnnroot@3pio:/home/pi# ceph -sn cluster:n id: b6623e8d-a8f8-455f-83ca-1b49752c2a1en health: HEALTH_WARNn application not enabled on 1 pool(s)nn services:n mon: 1 daemons, quorum 3pion mgr: 3pio(active)n osd: 4 osds: 4 up, 4 innn data:n pools: 1 pools, 128 pgsn objects: 1.77k objects, 885MiBn usage: 5.77GiB used, 194GiB / 200GiB availn pgs: 128 active+clean
Leave a Reply