DStrickler

Author: Dave Strickler

  • Consider Benevolent Dictatorships

    I’ve started, run, and sold a few successful companies. Each one was different from the last one because I was constantly learning. But early on, I took on a management method that I used throughout my career – I ran each as a Benevolent Dictatorship. Now before you go looking it up on Wikipedia, it’s a term I used with my own definition.

    A Benevolent Dictatorship, in my terms, means I was very interested in other opinions on problems as they often brought me to see new ideas. Opinions of others added a perspective that fleshed out details I didn’t previously see., and with rare exceptions, I relished each one. But regardless of how many I listened to, the proverbial buck stopped with me. I made the final decisions and was responsible for their outcome – good or, sometimes, unfortunately, bad.

    If you’re going to run any organization, there’s a temptation to run it “By Committee”, pushing away decisions, so it makes it easier to wag a finger when they don’t produce great results. But it’s harder to take responsibility. It’s harder to run a Benevolent Dictatorship. It’s harder to shoulder the responsibility and be ready to take on the consequences. But if you don’t, who will?

     

  • Why Use Zabbix Templates?

    There’s nothing as tedious as setting up the same type of server over and over, adding the same Items and Triggers. In any environment, we tend to use the same types of servers because we’ve standardized on them. It could be in your house (“My family is all on Apple products”) or at work (“We’re a Linux and Windows shop”).  So if you’re setting up 20 laptops, it’s the same Items and Triggers on each one.

    To handle this headache, Zabbix uses Templates to make the configuration easier. At the Host level, you can add Items and Triggers one by one or save yourself a lot of time an use the Templates field. Choosing a Template for the server allows the addition of pre-defined Items and Triggers so you don’t have to type them all in, but there an extra bonus. If you change the Template, the changes ripple out and affect all your Items and Triggers using that Template. Imaging configuring 20 Hosts, each with 30 Items, each with 60 Triggers, and realizing you made an error and need to update most of them – but if you use a Template you only have to make a few changes.

    My Zabbix server monitors hundreds of Hosts, but even if you only have a few, try a few pre-fined Templates to save yourself a lot of time. In a future post, I’ll show you how you can create your own Templates, and you’ll be stunned at their power.

  • Why Use Zabbix Global Macros?

    Once your Zabbix configuration starts to monitor for that a few hundred Items, you’re going to be relying heavily on Templates. Along with using Templates to replicate the monitoring of more than one server, you probably need to start using Global Macros.

    The concept of a Template is “write once, use many”, and Global Macros are very similar. An example might be a license key for an external service or the credentials for an internal service (there are better ways of storing secrets), perhaps a SQL database that is used just for Zabbix. Using these credentials and values scattered throughout your code is just going to give you headaches if they ever change, and we all know they will.

    The idea here is that the values are placed at a global level in your Zabbix configuration and then used anywhere you’d like. Need to change credential values? Just change it once in the Global Macros and the change ripples through Zabbix. An example is below of Global Macros is below.

    To use them, just put values in (mine are fuzzied out) and then use them throughout as you would any Macro. But instead of at the Macro value being attached to the Host, it’s defined everywhere. An example is below:

    When you save an Item like this, each time it runs, it will look for the value of the Macro called {$HOST_DNS} – saving you lots of configuration headaches, now and especially in the future as your configuration changes.

  • Why is Rebooting Linux Pointless?

    Spoiler alert: It’s not pointless.

    I’ve taken a lot of flack over the years in saying that it’s OK to reboot Linux. While you can read on the ‘net how it’s pointless to do so, a statement like that has too many nuances to be considered fact. A better statement would be to say that, in many cases, Linux doesn’t need to be rebooted (it’s not MS Windows, after all). Linux does an amazing job of recovering from runaway RAM issues and even out-of-disk issues. I’ve even seen CPU usage bring a Linux server to a standstill, only to have it recover after I’ve left it for dead.

    Linux, and all its flavors, is built on an amazing set of technologies, but it’s not perfect. Sometimes a good old-fashioned, quick reboot helps.

    Reasons for a reboot might be an upgrade to its installed software, security patch(s), or Linux itself. When logging into a Linux box that needs a reboot, it should notify you, so it’s pretty easy to know when to do it. And it’s also pretty quick.

    So the next time someone insists you don’t need to ever to reboot Linux, just smile and do it while they’re not looking.

  • Getting your Unifi Router Failover working

    One of my Unifi Gateways, a USG-Pro-4, is located in a house where the internet can go out in a storm. Since the internet is provided by Comcast, and it’s just “household grade”, there’s not much I can complain about. But we’ve all experienced an outage just when you need it the most.

    My traditional way of dealing with this is yelling around the house, “Internet is out – everyone go on your hot spots”. Typically it’s back online in an hour or so, and then I have to make sure everyone has switched back over to the house WiFi. What I’d really love is to get the failover port working on my gateway, but that seemed to be expensive and difficult to set up until now.

    What I’ve rigged up will cost you about $50 in parts and takes about 30 minutes to set up. Once it’s set up, the gateway will failover if the main connection fails, and it will failback (the opposite of failover) when your main connection is back online. In my setup with the connector linked to my iPhone, this failover only works if I’m in the house and my phone is fairly near this connector, but that’s about the only downside.

    I’ve accomplished this by buying a BrosTrend AC1200 WiFi to Ethernet Adapter  (about $50) connector, which consumes a WiFi signal, probably from your phone via your hot-spot, and turns it into a cabled ethernet connection. With the cabled ethernet plugged into the WAN2 port on your Unifi, and a simple configuration,  the Unifi gateway now sees the WAN2 port as having a connection to the internet if the WAN1 connection goes down.

    Basic Steps

    1. Configure your connector BrosTrend AC1200 to connect to your cell phone’s hotspot.
    2. With an ethernet cable, connect your connector to the WAN2 port of your Unifi Router.
    3. Configure the WAN2 port as a failover for the WAN1 port.
    4. With everything cabled and configured, simulate the failover by unplugging your main internet connection from the Unifi gateway.
    5. Wait about 30-60 seconds for it to failover to WAN2, and you can test your internet connection as you normally would.
    6. When you’re satisfied it’s working, plug back in the main internet connection to your Unifu gateway and watch ut failback.

     

    To configure the BrosTrend, just follow the instructions in the box. It’s pretty easy and consists of plugging in the unit, waiting for it to boot in a few seconds, switching your WiFi on your laptop to use the SSID that the BrosTrend just created, and following a few prompts to connect it to your phone’s hotspot. When it’s working, you’ll see a blue “Signal” LED on the BrosTrend remain solid, indicating a good signal to your cellphone.

    Then you can use any ethernet cable, including the one it comes with, to connect the BrosTrend to your Unifi Router’s WAN2 port. Remember, the BrosTrend will need power as well, but I’m assuming you have that close to your Unifi Router.

    Configuring your WAN2 port as a failover is so easy in the Unifi Controller console; you can miss it if you’re not watching for it. Go into your Console, click on the gear icon in the lower left, and then on “Internet: above. Then choose “Add Secondary Internet Connection”.

     

    Once you get the screen to add in a secondary connection, fill it in with a name that makes sense to you, fill in your guess at connection speeds, and choose “Auto” for the rest of the settings. If you’re using a Dynamic DNS, you can switch to manual and configure it, but in either “Auto” or “Manual” mode, it’s important that the “Load Balancing” be set to “Failover Only”. This needs to be done as there are only two modes, and the other, called “Balanced” will use your main ‘net connection half of the time and your iPhone for the other half. While that may be fun to experiment with, you’re going to max out your hotspot for no good reason. Setting it to “Failover Only” is the setting we want, which means it will only use this setting configuration if the main ‘net connection fails.

     

    Once you have this configuration saved, on the next screen, you will need to specify the port for this new BosTrend device to use. Port #4 is common as it’s the “WAN2” port on USG-Pro-4 routers. And again, make sure “Load Balancing” is set to “Failover Only”.

     

    After that, you’re all set. You should be able to simulate a failure with your main ‘net connection, and in a few seconds, the BosTrend should take over, using your phone’s hotspot for your entire network. Just reconnect your main ‘net connection, and the fallback should happen.

    If you get an error saying it can’t save your WAN2 settings, check the settings on your UGS-Pro-4 to make sure that Port #4 is enabled. If it’s disabled, it can’t be assigned to anything.

    Now you have a failover for your Unifi Router for about $50 and 30 minutes of your time. I hope you never have to use it, but you’ll thank me if you do.

     

     

     

     

     

  • Unifi Console sees existing Devices as offline

    With seemingly no changes, my network had a bad start to the day.

    Devices just started dropping off the Console, but the network was fully functioning. While not an emergency, as there wasn’t a network outage, I was “blind” as the Console wasn’t showing Devices, except for the USG4Pro itself and a few Clients.

    I waited for a good time (is there ever a good time), and I rebooted my USG4Pro, but this only had the situation worse as now nothing was showing up in the Console, and the connection to the ‘net stopped. Now this was an emergency.

    I then SSHed into my USG4Pro and, with the “info” command, found that it’s the connection to the Controller, which lives on the ‘net, was using a DNS address (I did this a few years ago). Using the “set-inform” command, I set it to an IP address, so it didn’t have to do the DNS lookup. Perhaps our DNS was down? Unlikely, but it was worth simplifying the network. Without a connection from the USG4Pro into my Controller, this could cause the issue.

    Unable to resolve the DNS address, so I set it to the Console’s IP address.

     

    With the IP address in place, I was getting an “Unknown[11]” error.
    With the new error of “Unknown[11]”, I was getting worried. When a vendor reports “unknown” in anything, it’s never a good sign.

    Since the IP address change didn’t seem to help, I powered-cycled the USG4Pro, hoping a full power cycle would pick up the changes as opposed to a soft reboot. That brought the USG4Pro back online as well as ‘net connectivity, but not the other devices (switches and APs). I then power cycled each of the other Devices, and they started showing up on the Console. Within a few minutes, the Devices started showing up in the Console.

    I have no idea what caused the initial problem or if the DNS-to-IP change solved it, but the emergency was over quickly.

  • Why choose Zabbix?

    If your network is complex enough to manage, you want to choose a solid monitoring tool. I like to look at these criteria when evaluating monitoring solutions:

    1. It’s not difficult to set up. It doesn’t have to be easy, but I don’t want a setup that’s so complex it makes me dread the task.
    2. Consistently presents me with easy to consume data that’s normally a bother to get on my own.
    3. Flexible when my needs change, and if there’s anything I’ve learned as an Admin, get ready for change.
    4. Has an active development cycle with a robust user community.

    Before I chose Zabbix, I looked at a lot of solutions. I lost track of the Docker containers I configured with packages with crazier names than Zabbix. And while there are some good ones out there, I kept coming back to Zabbix for a number of reasons.

    Setting Up Zabbix – The Inevitable Learning Curve
    I don’t care how much of a computer savant you are, setting up a new piece of software is a chore. You’ve got a learning curve you didn’t think would be that bad, clashing with what you think might be a bug, but you’re really not sure, so you forge ahead and inevitably configure things the wrong way and have to start again. I mean, who reads manuals – right? Well, with Zabbix, the learning curve isn’t that bad, especially if you embrace Templates. Sure, it’s new to you, and the concepts are foreign, but within a few hours, you should be able to get the gist of it if you’ve used monitoring software before.

    Presenting the Data – Don’t Tell Me What I Already Know
    Once Zabbix is up and running, the data streaming into it gets massive. Even if you start out just monitoring your home network, it’s easy to have upwards of 50 or more devices (called Hosts in Zabbix) returning data. Since each of the Hosts may be dozens of data points collected on it, you’ll soon have a dizzying amount of data. Normally, this could get quickly overwhelming, but Zabbix serves up the data in styles that fit your needs.

    Zabbix comes pre-configured with the single Pane-of-Glass display, found on the menu as Problems. It’s a timeline of all the current issues on your network, clearly laid out, easy to read,  and “glanceable”, meaning if you left it running on a monitor that you just glanced at, you’d know how your network was performing, even if it had a thousand Hosts on it.

    If you want to customize your view, Zabbix offers Dashboards. While it comes with a few pre-configured, it allows you to create your own and paint the screen your way – in a way that makes sense to you. You can fill a Dashboard with charts, maps, network diagrams, etc., all tailored to your needs. Pages on the Dashboard can even cycle if you want. And you can have unlimited Dashboards. I have some that are very specific to networks I’m monitoring for customers and a few that are overviews of all the networks I’m watching over – too many to mention.

    Change – It’s What’s For Dinner
    Just when you think you’ve got everything under control, change happens. We’ve all been through it, and we know that the tools you were using that weren’t flexible enough to deal with the change got left behind. Zabbix shines in changing environments due to easy configuration, and here’s that term again, Templates. They allow you to perform complex reconfigurations in moments. Server relocation, reconfiguration, new networking equipment, etc. – all handed with a few clicks. Without Templates, even Zabbix would be a pain to reconfigure. Templates simply give you more time to focus on other issues, and everyone in IT has plenty of issues to deal with.

    It’s Not Going To Fade Away
    Zabbix is Open Source, but don’t let that fool you. There’s a real company behind it that’s doing all the development for free, but if you get stuck, they’re there to help. While they do charge for support, it’s reassuring to know it’s out there. As for a community of users, it’s very large and active. Simply google Zabbix, and you’ll find an incredible array of resources. From user forums, to how-to videos, to yearly conferences, Zabbix is alive and well.

    I’ve been using Zabbix to monitor a dizzying amount of Hosts, both on my home network, a WAN into additional sites, and servers in the Cloud. All of this runs with very little upkeep. Zabbix is a keeper.

  • Monitor a Mac’s TimeMachine backups

    If your supporting Mac’s in your environment, you probably have come across TimeMachine, Apple’s answer to backups to a local disk or NAS.

    While configuring TimeMachine on a Mac is very easy, and good Admin wants to keep track that these backups are actually being run on a regular basis. The problem with this is that Apple doesn’t give you a way to easily monitor this. But I went digging, and found a way.

    TimeMachine does have a utility installed on the Mac, and you can get some data from it into Zabbix, but it’s not enough to be really useful. If you’re curious, here are the Items I configured. Note I do a little dancing to parse the output of these system.run[] commands, but they should work well on any Mac.

    Note that these commands are being run on the Mac itself, and that Zabbix is running them via the Agent. This works well, but our problem still remains as these functions tell us basic info about the backup, but not the last time that one was really run.

    In order to sense last time TimeMachine was run, we exploit the drive needing attached to the Mac, as when TimeMachine makes a backup, it has to attach to a disk, even if it’s a on the network. All we need to do is check a date stamp on a directory, and you can get an approximate time of the last backup. In Zabbix this means we can use a few variables.

    • {$TIMEMACHINE_BACKUP_PATH} — The path off the root of the disk which should be the same for all your Macs on the network, like /Volumes/TimeMachine_Disk
    • {$MY_BACKUP_PREFIX} — Often times the name if the Mac, like “TOMSMAC”

    … and then use the zbx_mac_timemachine_backups_templates.json template that you can import into Zabbix and then change variable names. Note that while {$TIMEMACHINE_BACKUP_PATH} is probably fine for all your Macs who use the same TimeMachine directory to back up their files, the {$MY_BACKUP_PREFIX} should be configured at the Host level and changed to match the user’s mac’s name.

  • AWS Throttles Its Customers

    I’ve been working with large files for a few years now. These zip files are normally over a gigabyte and can easily go into the 50-100 gigabyte range. They contain data that can only be compressed to about 50%, so there’s no way to make them much smaller – they are what they are.

    These files need to be uploaded from the field and processed in the cloud, and I’ve chosen AWS to do the storage via S3 and the processing via EC2. While we’ve been generally pleased with each, we have seen that AWS throttles data to and from S3 and also the creation of large instances in EC2.

    This means that if I have a 10 gigabyte zip file I need uploaded to EC2, even if I’m using a network-optimized AWS image that supposedly can handle over a gigabit ethernet, my upload speeds are in the 30-70 megabit range. And yes, I’m uploading from a 1 gigabit connection with a cable – not WiFi, and yes, I’ve tested uploading to various resources, and yes, they all upload at about 800 megabits/sec, the reasonable throughput of a 1 gigabit line. The same goes for downloading from S3. My speeds can be in the range of 14-80 megabits/sec.

    You’ll also find that when spinning up a large instance, such as a 64 cores and 192 gigabytes of RAM, AWS often reports back that they are out of capacity in a particular zone. Trying other zones will have the same results. AWS is simply under-provisioned for handling customer needs that are “out of the norm”.

    I do understand that handling multi-gigabyte files is unusual, and so is spinning up an EC2 instance that costs about $100/day, but no where does AWS say “don’t tax our systems because we can’t handle it, and will throttle you”.

    If you have these needs, you have now been warned about AWS.

  • Tuning MySQL’s Innodb for Smaller Sites

    Scour the ‘net for how to reduce your MySQL server footprint, and you won’t find much. Everyone seems to want to beef up its settings, and I’ve been one of them. When you have a server cluster taking thousands of requests per second, the battle cry is often for more RAM to aid in caching data. But for smaller sites, like a neighborhood hiking club, you will have a very different server setup.

    Smaller sites will be run on smaller servers with smaller resources. With a server that’s only got 4GB of RAM, every megabyte matters.

    Like any good household budget trimming exercise, the first thing you want to look for is caching or buffers that may be too large. If you’re running the MySQL Innodb engine, the default cache is 128MB of RAM. While that doesn’t seem like much, if you’re data set size is 32MB, that’s a lot of wasted RAM that can’t be used for anything else. 32MB of data will never fill a 128MB cache.

    While you’ll need to figure out how which you’ll need to allocate, find your config file (usually in /etc/mysql/my.cf) and make sure you have a line like this:

    innodb_buffer_pool_size=32M # Overall size of pool. The default is 128MB
    innodb_buffer_pool_chunk_size = 8M # Default is 128MB

    Then restart MySQL and see how your system performs. Still fast enough? If you’ve removed unneeded cache RAM you shouldn’t see any difference in the speed, and on small sites, caching may not be needed at all.

    Another thing to look for is excessive disk usage. If you’re not running MySQL in a cluster, and in a smaller site I doubt you are, you can also remove the files used for the Binary Log, which is used to replicate MySQL to different servers, such as a redundant MySQL server. To do this, and something like this into your /etc/mysql/my.cf file. This should save you a lot of disk space as these log files can get huge over time.

    binlog_expire_logs_seconds = 68400 # 1 day in seconds. Choose a value that fits your usage.

    Can you think of any other tips for a smaller site? Just let me know in the comments.