Low Cost Operations Metrics Display Setup

One of the most important roles of any operations team is metrics gathering and display.  Without metrics you are flying blind in Ops, and without proper display those metrics are rarely consumed by your team. After Cozy’s move into our new office we decided to set up a wall of displays for Ops and Dev metrics.  As a startup we’re not made of money so we needed to do this on a bit of a budget.  Here’s the simple setup we used and the script we used to setup our hosts.

We had initially setup two borrowed Raspberry Pi’s connected to small monitors we had leftover in the office.  While this worked the displays were too small to be seen from across the room and the Pi is just too slow for modern dashboards applications.  We decided to purchase two small TVs to augment our existing displays, and four new low end PCs.  For the low end PCs we settled on the Radxa Rock Pro and for the TVs we used Seiki 32” TVs.

Seiki TVs

seiki_tv

We purchased 32″ Seiki TVs, which run about $170 each on Amazon. These are without a doubt the cheapest TVs you could possibly buy and it shows.  Poor contrast ratio, bad color preproduction, and they’re a bit on the thick side.  Why did we get them then?  Well we’re not watching Blu-Rays on these, we just need them to show metrics so everything that makes the a poor choice as a TV makes them perfect for metrics monitors.

Radxa Rock Pro

radxa_pro

Radxa is a small (2 person?) Chinese based company that sells what is effectively a Android phone in a small plastic case.  The systems have a quad core ARM processor, 2GB RAM, 8GB NAND storage (Pro model), HDMI, and ethernet/wifi.  At $99 including the case they’re a great alternative to the Pi since they offer significantly more CPU power and have the built in storage. They can run either Android, which ships on the system, or Linaro Linux which is an Ubuntu derivative that you need to flash on to the system.  We chose to flash Linaro onto the hosts since we wanted a more full featured PC.

Flashing the Radxa

Originally we thought it made sense to buy the Radxa Rock Pro model, which includes the onboard NAND storage, 2GB ram (vs 1GB), and a plastic case. After realizing that flashing the NAND requires a special image file and an application that only runs on Windows we ended up just buying micro SD cards instead. Most images in the Radxa community are in the SD image format and flashing can be done via a PC or Mac without any special utilities.  I’d recommend buying a name brand Class 10 16GB micro SD card for plenty of storage.  They’re about $10 shipped on eBay. If you’re on a budget might actually make more sense to just buy the Radxa Rock Lite and a case vs. the Pro also.

Setting up Linux on the Radxa

The Radxa ships with Linaro Linux, which is a light version of Ubuntu for ARM processors.  It used the LDXE desktop and a different kernel version, but otherwise it’s basically Ubuntu.  Since we had four of these hosts we created a small script to automate the setup.  The script performs the following.

  1. Sets the hostname
  2. Removes a bogus entry left in the resolveconf file
  3. Sets the password for the rock user (make sure to change this in the script)
  4. Disables bluetooth, cups, saned, and ppp-dns ervices
  5. Turns off the front LEDs which are horribly bright
  6. Remove NetworkManager and sets up eth0 / wlan0 using /etc/network/interfaces ( ake sure to change wlan0 auth in the script)
  7. Sets a new unique MAC address for eth0 using the last digit of the hostname.  The Rock Pros do not have unique MAC addresses so this will cause lots of issues when you have more than one unless you set a new address in software
  8. Resizes the root drive.  Oddly enough the Rock ships with an image where the root filesystem is next to nothing is size so installing a few packages will fill up the filesystem even though you have gigs of space on the SD card.
  9. Sets a boot script to sync the time since the Rock has no battery to save the data on reboot
  10. Upgrades all packages to the latest
  11. Installs VIM
  12. Installs Firefox and sets it to open at boot
  13. Installs X11VNC and sets a password (make sure to change in the script)
  14. Installs OpenVPN and configures resolvconf to work better with VPN tunnels

Download the setup script here:

Github Repo

Mounting the InfoRads

To mount the TVs and PCs to our wall we chose a simple fixed VESA mount from Monoprice ($7.50). We then bought a 1/4 inch masonry bit ($4) and 25 pack of 1.5 inch hammer set wall anchors ($14).  We carefully determined where we wanted to monitors to rest and then taped the backing of the wall mount to our concrete walls.  With the backings taped we drilled the 24 mounting holes and hammered in the anchors.  Keep in mind that once the anchors are in they’re pretty much never coming out so get it right the first time.

With the TVs mounted to the walls we bought 12 foot extension cords ($20) and simple cable channels ($20).  Each TV has it’s own extension cord which powers the PC and the TV allowing us to easily power reset the devices by unplugging the proper extension cord.  The Radxas were mounted to  to the back of each TV using a large patch of velcro ($4), which hides the PC and also allows access to the reset button and the SD card if needed.

End Result

I won’t lie the Radxas were a lot more than we had bargained for.  There’s a lot of quirks with these systems that we had to workaround in the setup script.  Once we worked out all the issues those the inforads have been wildly useful.  We rotate Librato dashboards on 2 screens and show build / test status screens out of Jenkins on two others.  Using these screens has led us to notice odd behavior on our site and increased our awareness of test status.

Inforads Mounted on Wall

One comment

  1. Russell Senior

    What radio do those Radxa Rock Pro devices have? Their website isn’t very informative. “Wifi” isn’t very specific, driver wise. 😉

Leave a Reply

Your email address will not be published. Required fields are marked *