Stephen Blinick's Blog – SteSoft.com

I recently decided to poke around and see how difficult it was to install Ceph on a Raspberry Pi. While it has been done a bunch of times before, I wanted to see how the latest and greatest (Spring 2019-ish) code worked, and how much effort was required. The great news is that it’s almost turn-key at this point using an arm64 debian distribution. The luminous ceph packages are available and they work!

It’s not fast by any stretch, primarily because of the network-to-USB-attached-disk contention within the RPi . But it does work. Longer term, memory consumption is an issue . But I’ve let it idle for over a week and heartbeats/scrubs seem to run ok.

This whole setup becomes far more interesting when you consider it applied to one of the RPi alternatives, some of which have PCIe NVMe slots, and much faster networking and CPU. Especially when you consider the price trajectory of NVMe TLC and QLC storage devices!

I’m mostly just pasting the commands from my shell history here so I can use them another time.

Cookbook-level instructions

First, get the “Buster” beta for the raspberry pi and write it to an SD card. More info: https://itsfoss.com/debian-raspberry-pi/

As Root:

# Initial installation of packages not installed by default with Debian arm64 buster
 apt-get upgrade
 apt-get update
 apt-get update
 apt-get install lvm2 sudo gnupg gnupg2 gnupg1 lsb-base lsb-release

# Setup the ceph repo and add some more packages
 wget -q -O- 'https://download.ceph.com/keys/release.asc' |  apt-key add -
 echo deb https://download.ceph.com/debian/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
 echo $?
 apt clean
 apt-get  update
 apt-get install ceph-deploy
 adduser ceph-deploy
 echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy

# /dev/sda is a USB attached SSD... I set it up with 4 partitions to do 4 OSDs
 fdisk /dev/sda
# had to reboot, kernel wouldn't see new partition table otherwise
 shutdown -r now

# For some reason, the distribution installed by ceph-deploy with the armhf repo setup above didn't install the ceph-volume systemd service.
 cd /lib/systemd/system/
 wget https://raw.githubusercontent.com/ceph/ceph/master/systemd/ceph-volume%40.service

As user ceph-deploy:

# Setup public key ssh access (to ourselves)
# Server is called 3pio in DNS.
 ssh-keygen
 ssh 3pio
 ssh-copy-id ceph-deploy@3pio
 ssh 3pio


# Use ceph-deploy to create a cluster (and install ceph software)
 mkdir testcluster
 cd tescluster
 ceph-deploy new 3pio
 vim ceph.conf
 ceph-deploy install --release luminous --no-adjust-repos 3pio
 ceph-deploy mon create-initial

# For some reason ceph-deploy doesn't copy the keyrings to /etc/ceph so I did this manually.
 sudo cp *keyring* /etc/ceph     sudo chmod o+rx /etc/ceph/*
 ceph-deploy osd create 3pio --data /dev/sda1
 ceph-deploy osd create 3pio --data /dev/sda2
 ceph-deploy osd create 3pio --data /dev/sda3
 ceph-deploy osd create 3pio --data /dev/sda4
 ceph-deploy admin 3pio
 ceph-deploy mgr create 3pio

# Ceph should be up, check it out and do a few things
 ceph -s
 ceph osd tree
 ceph osd pool create testpool 512  512 replicated 0
 ceph -s
 rados bench 15 write -p testpool -b 524288
 ceph -s
 ceph df
 ceph osd df

All done!

root@3pio:/home/pi# ceph -s
  cluster:
    id:     b6623e8d-a8f8-455f-83ca-1b49752c2a1e
    health: HEALTH_WARN
            application not enabled on 1 pool(s)

  services:
    mon: 1 daemons, quorum 3pio
    mgr: 3pio(active)
    osd: 4 osds: 4 up, 4 in

  data:
    pools:   1 pools, 128 pgs
    objects: 1.77k objects, 885MiB
    usage:   5.77GiB used, 194GiB / 200GiB avail
    pgs:     128 active+clean

In my previous post, I described how I stumbled into RTL-SDR and connecting it to a Raspberry Pi.  It wasn’t working well for me due to network lag, hiccups, and sample rate limitations.  I didn’t want to give up, and the code looked like it could be improved.   This post describes some of the changes and the results.

The goal

The goals I had were to:

• Move the RTL-SDR dongle to a raspberry pi connection in a closet, far away from my desktop with the GQRX software & GUI.
• Utilize as much of the bandwidth of the RTL-SDR dongle as possible.
• Hopefully enable all of this over wifi, because an ethernet cable limits the placement of the antenna/pi, and the PiZeroW doesn’t even have an ethernet interface.

What I changed

From the source, rtl_tcp.c uses two threads.  One receives a call-back from the RTL-SDR library any time there is new data from the USB dongle.  The other thread is supposed to take any new data and write it to the TCP socket so the kernel can transmit this data over the network to GQRX (or whatever client).

I decided to rip out the semaphore between the TCP and USB threads — remove all locking.  I also removed the linked list structure and allocation / deallocation entirely, so that none of this code would have to call malloc() or free() in the performance path. .  I replaced these with a single statically allocated circular buffer.  Both threads would interact with the data through a non-blocking polled-mode interface.   The USB-reader thread writes data into the circular buffer at head, and the TCP worker reads/transmits data from the same circular buffer at tail.   Neither thread touches the same variables, they only each update head or tail themselves.

The USB Worker:

 if ((ringbuf_head+len) < (unsigned int)ringbuf_sz)
 {
      memcpy(((unsigned char*)(ringbuf+ringbuf_head)), buf, len);
 }
 else
 {
      memcpy(((unsigned char*)ringbuf+ringbuf_head), buf, ringbuf_sz-ringbuf_head);
      memcpy((unsigned char*)ringbuf, buf+(ringbuf_sz-ringbuf_head), len-(ringbuf_sz-ringbuf_head));
 }
 ringbuf_head = (ringbuf_head + len) % ringbuf_sz;

The TCP Worker:

 unsigned int sendchunk;
 if (ringbuf_tail < ringbuf_head)
     sendchunk = ringbuf_head - ringbuf_tail;
 else
     sendchunk = ringbuf_sz - ringbuf_tail;
 bytessent = send(s, (unsigned char*)(ringbuf+ringbuf_tail), sendchunk, 0);
 bytesleft -= bytessent;
 ringbuf_tail = (ringbuf_tail + bytessent) % ringbuf_sz;

With these changes, both threads are free to run as fast as they can.  Ideally, the network can keep up with the USB data stream coming in.  So the TCP worker thread should be reading data just behind the USB thread for the lowest latency.   If there is a ‘hiccup’ and the TCP thread doesn’t get to run, or blocks for some reason, then the amount of data not yet transmitted in the buffer will grow.

In the worst case, so much data is in the buffer that the USB worker thread runs into the “tail” and cannot write any new data.  I decided in this case to trim the buffer, and move the tail ahead.  This would result in the oldest data being lost.  But this is real-time information and I figured it was better to stay current than introduce a lag that could never go away.

The results

The performance improvement was tremendous!   Ditching ethernet, the Raspberry Pi 3B+ was able to do the maximum bandwidth of the RTL-SDR at 3.2Million samples per second, even over wifi.  This is a sustained 53Mbps over wifi, not bad.   Even at this maximum value allowed by GQRX, the lag is never more than one data chunk from the USB.  If there’s an occasional blip on raspberry pi that pre-empts the TCP sender, it quickly recovers and never gets more than 2-3 chunks behind.   Changing channels is now instantaneous even at the highest bitrate.  And switching to Wifi means that I can simply stick the Raspberry pi to the antenna assembly and place it anywhere for the best reception!

I also tried it with a Raspberry Pi Zero W.  I was immediately able to jump around frequencies with very little (1/4 second) lag at 1.9Million samples per second.  It was very stable, with about 4-6 data chunks in the circular buffer at any time.   I was not however able to extend beyond 1.9Msps.  When I did this, the Raspberry Pi Zero W CPU was overloaded and couldn’t keep up with the data rate.  But at a reasonable data rate, the lag is always zero.

Also, the “ll+ now (##)” messages are gone :). I replaced those with a 30-second update on the overall bitrate and the number of bytes that are inflight between the USB dongle and the TCP socket.

I did do some “stress testing” and reduced wifi reception or overloaded the Raspberry Pi Zero W.  Listening to FM radio was still “OK”, but every 10 seconds or so there was a “jump” when it would truncate the buffer.  The lag was at worst about 3 seconds with this design.  In the previous design, the lag would sometimes extend to 15+ seconds, which was so long you weren’t sure any change took effect or not.

Summary:

• Raspberry Pi 3B+
  • Before: Could only support up to 1.92Msps over wifi with 1-2 seconds of lag.  Ethernet could do 2.4Msps.
  • After: Able to set the sample rate to the dongle maximum of 3.2Msps over wifi with instant channel changes.
• Raspberry Pi Zero W
  • Before: Couldn’t get above 0.92 Msps, and the lag was often 5-10 seconds.  Not stable.
  • After: Able to achieve 1.92Msps with very good lag (feels like about a quarter second).

I really enjoyed jumping into “software defined radio on the cheap” and can’t wait to try out other extensions or applications that can use this data.  I’ve heard SDR# is a really good application, albeit closed source.  I just need to find a windows machine.  Given the improvement and the other people and blogs citing these issues, I will probably submit a PR soon with the changes in case it helps anyone else.

Update: I submitted a PR.  https://github.com/osmocom/rtl-sdr/pull/6

I’d been looking for a reason to use another Raspberry Pi for some time.  For years I’ve had an old Oregon Scientific weather station that had 4 really nice 433mhz weather sensors, but I no longer used the weather station base.     I wanted to setup a raspberry pi that would somehow retrieve the temperature readings these old sensors transmitted, and have a script that would push them to an influxdb container for graphing.

First, I went cheap.  I bought a 433 ASK/OOK GPIO interfaced board  from amazon for 7 bucks. Of course this required another $25 in jumper wiring and connectors, and once I had it hooked up I immediately found that it couldn’t demodulate whatever protocol these sensors used.  It was primarily used for very very cheap temperature sensors, or short range signal communication between two embedded boards.

RTL-SDR

A while later, a friend of mine pointed me at an RTL-SDR USB dongle, and for $30 you can get a pretty nice kit which has antennas, the dongle, and a few mounts.  I purchased it and immediately plugged it into my Macbook and installed gqrx.

This resulted in instant gratification: a very cool graphical interface with a waterfall of the radio spectrum. I quickly forgot about the temperature sensors as I realized I could visually poke around anywhere in the spectrum. I could browse anything, from ATC traffic control frequencies, to DPS/FD VHF, Weather forecasts, and even listen to stereo FM radio.  Even better, a VERY robust software package, rtl_433,  is available that decodes the 433 temperature sensors, from demodulating the transmissions all the way through decoding the proprietary packets and displaying them neatly or formatting them in a useful json blob.  I’ll get back to doing that at some point.

Enter the Raspberry Pi

I quickly realized that the computers and other junk on my desk weren’t helping my signal reception very well. I also didn’t like having the antennas everywhere.  I found that there’s a part of the rtl-sdr package called rtl_tcp which is perfect:  It simply provides the raw data output from the RTL-SDR USB dongle over a TCP connection.  GQRX already supports this as well.  You can put your antenna and dongle far away, preferably somewhere with the best reception, hook it up to a cheap raspberry pi, and still listen to everything on your desktop!  I was excited to try this with a raspberry pi.  I immediately imagined zip-tieing  a Raspberry Pi Zero W to the antenna mast and sticking it in a closet on the 2nd floor.

RTL-SDR is already an upstream package for Raspbian, so it’s as simple as sudo apt-get install rtl-sdr on your raspberry pi and off you go.

Not so fast! (literally)

I connected the dongle to an rpizerow I already had, installed RTL-SDR, and ran rtl_tcp.  I pointed gqrx to the IP of the raspberry pi (following an excellent guide such as: http://www.hagensieker.com/blog/page/?post_id=70&title=make-a-raspberry-pi-3-rtl-sdr-server ) and I was in business quickly.

Except, it wasn’t so usable. I couldn’t use the full bandwidth of the RTL-SDR dongle.  The lower the samples-per-second, the less of the spectrum you can see at once.  Even at one of the lowest bandwidths (0.92Msps) there was a few seconds lag between changing the frequency and seeing it in GQRX.  This was quite annoying, as GQRX doesn’t realize this, and so you get a few seconds of LOUD static if you’re changing modulation at the same time.  And it’s no fun to jump around frequencies.

And there wasn’t really any way to go above 1.15M samples per second.  When I had it directly connected to my laptop’s USB, GQRX was able to do 3.2M samples per second, and I could see a huge range of the radio spectrum, jumping around instantly.  But here, I had to limit the sample rate, reducing the range I could see at once and the quality of the signal.   Also, it wasn’t that stable. I saw a lot of these messages from rtl_tcp:

ll+, now 3
ll-, now 0
ll+, now 1
ll+, now 2
ll+, now 3
ll-, now 0
ll+, now 1
ll+, now 2
ll+, now 3
ll-, now 0
ll+, now 1
ll+, now 2
ll+, now 3
ll-, now 0
ll+, now 1
ll+, now 2

Sometimes this number grew and grew into the hundreds.  A quick google showed this meant the rtl_tcp program wasn’t able to send the data it had, and had to allocate more buffers.    This post outlines the problem, and explains that the data is stored in an ever growing linked list of data blobs read from the USB.

I immediately switched to a Raspberry Pi 3B+ and a wired ethernet connection.  I figured the Ethernet connection and 4 cores would solve it.  But it did not, even with a CPU usage of ~17% there was still significant lag above 1.92Msps.  This indicated that the threads of the program were not being scheduled and likely blocking on something.  I next experimented by increasing the MTU — which has the effect of decreasing the number of ACKs and TCP packets coming back to the Raspberry pi.  This actually helped reduce lag and the “ll+ spiral”, somewhat.  But I couldn’t increase the bandwidth to what I had with direct USB, and the raspberry pi wasn’t overloaded in network utilization or CPU utilization.

I decided to check out the source and found a few potential issues:

• Each data chunk read via the USB was dropped into a newly allocated buffer
• Every buffer only lived until it was transmitted over the TCP socket
• All buffers were inserted and removed from a linked list.
• There was a single lock between the thread reading data from the USB and transmitting data over the TCP socket.  Therefore, in some cases, the TCP thread could prevent the USB thread from executing and vice-versa.
• Both threads could block/sleep and wait on a signal if there was lock contention.

The net result was a large amount of malloc/free heap overhead, as well as the bookkeeping required for the linked list element insertion and removal, and even worse,  threads would halt or go to sleep if the other thread had a lock.  This meant that when the TCP worker and USB-reader worker got “in phase” they could continually step on each other.   Also most importantly,  the design of this code was to simply grow the linked list of data blobs if it wasn’t able to push the data through from the USB to the network.  This meant that every “hiccup” put the data stream further behind “real-time”, increasing the “lag” I felt when changing frequencies.   This is the reason the author in the blog about raspberry pi as a rtl_tcp host remarks: “Note that tuning is a bit slow. But it works.”

I’m sure this code performs quite well on a multi-core Intel Core i5 or other mainstream desktop CPU.  But the raspberry pi needs more finesse.  Having a raspberry pi in a closet with a well-placed antenna seemed like such a compelling use case for rtl_tcp.  And given the issues in the code, I felt there was a chance to improve it.

To be continued in part 2.

While my day job often involves creating highly available enterprise storage products, I am more interested in building cheap solutions at home that can accomplish things only expensive products did a few years ago.  In this case, I need to backup all my personal files: financial, documents, family photos/videos, etc.   I’ve had few variants of this setup in the past, usually built on a PC or a laptop.  This time, I’m going to do it as tiny and low-cost as possible.

In general, if you want to backup some data, it’s probably important to you.  If it’s important, you want to make sure this backup copy doesn’t get lost also, if your main copy is lost.  Obviously, putting a copy on a USB drive and storing it in a shelf above your computer is not going to cut it.   A house break-in, fire, etc will knock out both copies.   At the same time, if your data is far away from you, it’s hard to control it.  A fire, or break-in could occur, or nefarious people could get to your data.

So I wanted something that is small, unassuming, can be stashed at another location such as another house, and was completely encrypted.   I want to make sure if I lose track of it, there’s basically no way to access the data, and it would be very difficult to even figure out what it had on it.

The hardware

I hit up amazon for the parts:  A Raspberry Pi 3, 32GB SDCard, Canakit power supply, a 2TB Laptop drive in a shiny aluminum case, and a very cool FLIRC matching aluminum case.   At the time, total cost was about $150.   The HDD is easily powered from the Raspberry Pi’s USB port, as long as you have a decent power-supply.   Stability is important, because if this thing locks up it’ll be hard to reboot it since it’ll be off site somewhere.

The Pi: https://www.amazon.com/gp/product/B01CMC50S0

A pretty backup drive: https://www.amazon.com/gp/product/B00FRHTTIA

A nice FLIRC Case: https://www.amazon.com/gp/product/B07349HT26

AC Adapter: https://www.amazon.com/gp/product/B00MARDJZ4

32GB SD Card: https://www.amazon.com/gp/product/B010Q57T02

The Software

For the OS, I went with the standard: Raspbian (Stretch).      As far as the rest of the software goes, the core components I used are:

RSnapshot: http://rsnapshot.org/

Tinc (VPN):  https://www.tinc-vpn.org/

CryptSetup/dm-crypt: https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt

Theory of operation

OK, so I didn’t write this as a cookbook or HowTo.  I also don’t want to be too specific about how it’s setup for security’s sake.  Essentially, the raspberry pi will boot, and phone home over VPN (tinc).  It doesn’t know how to unlock or mount the drive it has.   Once it has phoned home, I can then manually enter the encryption keys and mount the drive, which contains a set of scripts and keys for the backup to work.   It also has a set of scripts on the encrypted disk that can be used to setup and initialize a blank raspberry pi that is attached.  I’m assuming here the SDCard it runs can fail easily, or that the drive itself may be the only thing I can retrieve in the future.

In normal operation, everything is run on the raspberry pi itself.  If everything is mounted,  the backup process can run periodically and automatically, pulling all the data from various servers I have and storing it on the attached drive.  I could even have it backup data from various cloud providers. I don’t need to “push” data to the remote backup, it “pulls”.

This works great with rsnapshot, which as it pulls data uses rsync and hard links to keep multiple point-in-time copies of data.  This means I have a view of ALL my data today, yesterday, and every day for 14 days, as well as weeks and month snapshots.  The overall space utilization for my own uses is very low even with dozens of daily, weekly, monthly copies.   The reason is that my data doesn’t change much over time, I mostly add pictures and documents but I don’t rewrite things very often.

Summary

The end result is a tiny, low power, $150 2-Terabyte encrypted backup solution that can be placed just about anywhere there is internet connectivity.  It automatically backs up data on a set schedule, and can protect against either loss of my main computer, theft of the equipment, mistake in deleting the data, or other malicious attacks.  Alternatively, if it gets lost due to a similar situation at the remote site, no personal data is left out of my control.  The data itself, as well as the knowledge of how to run the backups, is entirely encrypted.  If the whole thing gets destroyed, it isn’t a huge loss either.  I’m assuming it’s far enough away that a house fire or theft won’t affect both sites at once.

This kind of feature set would have been very expensive less than ten years ago!   When you consider the cost of even the cheapest amazon backups (where you don’t have as much control over security and you pay monthly forever), it’s a great value.