Dave Strickler

Author: Dave Strickler

  • Why Use Zabbix Global Macros?

    Once your Zabbix configuration starts to monitor for that a few hundred Items, you’re going to be relying heavily on Templates. Along with using Templates to replicate the monitoring of more than one server, you probably need to start using Global Macros.

    The concept of a Template is “write once, use many”, and Global Macros are very similar. An example might be a license key for an external service or the credentials for an internal service (there are better ways of storing secrets), perhaps a SQL database that is used just for Zabbix. Using these credentials and values scattered throughout your code is just going to give you headaches if they ever change, and we all know they will.

    The idea here is that the values are placed at a global level in your Zabbix configuration and then used anywhere you’d like. Need to change credential values? Just change it once in the Global Macros and the change ripples through Zabbix. An example is below of Global Macros is below.

    To use them, just put values in (mine are fuzzied out) and then use them throughout as you would any Macro. But instead of at the Macro value being attached to the Host, it’s defined everywhere. An example is below:

    When you save an Item like this, each time it runs, it will look for the value of the Macro called {$HOST_DNS} – saving you lots of configuration headaches, now and especially in the future as your configuration changes.

  • Why is Rebooting Linux Pointless?

    Spoiler alert: It’s not pointless.

    I’ve taken a lot of flack over the years in saying that it’s OK to reboot Linux. While you can read on the ‘net how it’s pointless to do so, a statement like that has too many nuances to be considered fact. A better statement would be to say that, in many cases, Linux doesn’t need to be rebooted (it’s not MS Windows, after all). Linux does an amazing job of recovering from runaway RAM issues and even out-of-disk issues. I’ve even seen CPU usage bring a Linux server to a standstill, only to have it recover after I’ve left it for dead.

    Linux, and all its flavors, is built on an amazing set of technologies, but it’s not perfect. Sometimes a good old-fashioned, quick reboot helps.

    Reasons for a reboot might be an upgrade to its installed software, security patch(s), or Linux itself. When logging into a Linux box that needs a reboot, it should notify you, so it’s pretty easy to know when to do it. And it’s also pretty quick.

    So the next time someone insists you don’t need to ever to reboot Linux, just smile and do it while they’re not looking.

  • Getting your Unifi Router Failover working

    One of my Unifi Gateways, a USG-Pro-4, is located in a house where the internet can go out in a storm. Since the internet is provided by Comcast, and it’s just “household grade”, there’s not much I can complain about. But we’ve all experienced an outage just when you need it the most.

    My traditional way of dealing with this is yelling around the house, “Internet is out – everyone go on your hot spots”. Typically it’s back online in an hour or so, and then I have to make sure everyone has switched back over to the house WiFi. What I’d really love is to get the failover port working on my gateway, but that seemed to be expensive and difficult to set up until now.

    What I’ve rigged up will cost you about $50 in parts and takes about 30 minutes to set up. Once it’s set up, the gateway will failover if the main connection fails, and it will failback (the opposite of failover) when your main connection is back online. In my setup with the connector linked to my iPhone, this failover only works if I’m in the house and my phone is fairly near this connector, but that’s about the only downside.

    I’ve accomplished this by buying a BrosTrend AC1200 WiFi to Ethernet Adapter  (about $50) connector, which consumes a WiFi signal, probably from your phone via your hot-spot, and turns it into a cabled ethernet connection. With the cabled ethernet plugged into the WAN2 port on your Unifi, and a simple configuration,  the Unifi gateway now sees the WAN2 port as having a connection to the internet if the WAN1 connection goes down.

    Basic Steps

    1. Configure your connector BrosTrend AC1200 to connect to your cell phone’s hotspot.
    2. With an ethernet cable, connect your connector to the WAN2 port of your Unifi Router.
    3. Configure the WAN2 port as a failover for the WAN1 port.
    4. With everything cabled and configured, simulate the failover by unplugging your main internet connection from the Unifi gateway.
    5. Wait about 30-60 seconds for it to failover to WAN2, and you can test your internet connection as you normally would.
    6. When you’re satisfied it’s working, plug back in the main internet connection to your Unifu gateway and watch ut failback.

     

    To configure the BrosTrend, just follow the instructions in the box. It’s pretty easy and consists of plugging in the unit, waiting for it to boot in a few seconds, switching your WiFi on your laptop to use the SSID that the BrosTrend just created, and following a few prompts to connect it to your phone’s hotspot. When it’s working, you’ll see a blue “Signal” LED on the BrosTrend remain solid, indicating a good signal to your cellphone.

    Then you can use any ethernet cable, including the one it comes with, to connect the BrosTrend to your Unifi Router’s WAN2 port. Remember, the BrosTrend will need power as well, but I’m assuming you have that close to your Unifi Router.

    Configuring your WAN2 port as a failover is so easy in the Unifi Controller console; you can miss it if you’re not watching for it. Go into your Console, click on the gear icon in the lower left, and then on “Internet: above. Then choose “Add Secondary Internet Connection”.

     

    Once you get the screen to add in a secondary connection, fill it in with a name that makes sense to you, fill in your guess at connection speeds, and choose “Auto” for the rest of the settings. If you’re using a Dynamic DNS, you can switch to manual and configure it, but in either “Auto” or “Manual” mode, it’s important that the “Load Balancing” be set to “Failover Only”. This needs to be done as there are only two modes, and the other, called “Balanced” will use your main ‘net connection half of the time and your iPhone for the other half. While that may be fun to experiment with, you’re going to max out your hotspot for no good reason. Setting it to “Failover Only” is the setting we want, which means it will only use this setting configuration if the main ‘net connection fails.

     

    Once you have this configuration saved, on the next screen, you will need to specify the port for this new BosTrend device to use. Port #4 is common as it’s the “WAN2” port on USG-Pro-4 routers. And again, make sure “Load Balancing” is set to “Failover Only”.

     

    After that, you’re all set. You should be able to simulate a failure with your main ‘net connection, and in a few seconds, the BosTrend should take over, using your phone’s hotspot for your entire network. Just reconnect your main ‘net connection, and the fallback should happen.

    If you get an error saying it can’t save your WAN2 settings, check the settings on your UGS-Pro-4 to make sure that Port #4 is enabled. If it’s disabled, it can’t be assigned to anything.

    Now you have a failover for your Unifi Router for about $50 and 30 minutes of your time. I hope you never have to use it, but you’ll thank me if you do.

     

     

     

     

     

  • Unifi Console sees existing Devices as offline

    With seemingly no changes, my network had a bad start to the day.

    Devices just started dropping off the Console, but the network was fully functioning. While not an emergency, as there wasn’t a network outage, I was “blind” as the Console wasn’t showing Devices, except for the USG4Pro itself and a few Clients.

    I waited for a good time (is there ever a good time), and I rebooted my USG4Pro, but this only had the situation worse as now nothing was showing up in the Console, and the connection to the ‘net stopped. Now this was an emergency.

    I then SSHed into my USG4Pro and, with the “info” command, found that it’s the connection to the Controller, which lives on the ‘net, was using a DNS address (I did this a few years ago). Using the “set-inform” command, I set it to an IP address, so it didn’t have to do the DNS lookup. Perhaps our DNS was down? Unlikely, but it was worth simplifying the network. Without a connection from the USG4Pro into my Controller, this could cause the issue.

    Unable to resolve the DNS address, so I set it to the Console’s IP address.

     

    With the IP address in place, I was getting an “Unknown[11]” error.
    With the new error of “Unknown[11]”, I was getting worried. When a vendor reports “unknown” in anything, it’s never a good sign.

    Since the IP address change didn’t seem to help, I powered-cycled the USG4Pro, hoping a full power cycle would pick up the changes as opposed to a soft reboot. That brought the USG4Pro back online as well as ‘net connectivity, but not the other devices (switches and APs). I then power cycled each of the other Devices, and they started showing up on the Console. Within a few minutes, the Devices started showing up in the Console.

    I have no idea what caused the initial problem or if the DNS-to-IP change solved it, but the emergency was over quickly.

  • Why choose Zabbix?

    If your network is complex enough to manage, you want to choose a solid monitoring tool. I like to look at these criteria when evaluating monitoring solutions:

    1. It’s not difficult to set up. It doesn’t have to be easy, but I don’t want a setup that’s so complex it makes me dread the task.
    2. Consistently presents me with easy to consume data that’s normally a bother to get on my own.
    3. Flexible when my needs change, and if there’s anything I’ve learned as an Admin, get ready for change.
    4. Has an active development cycle with a robust user community.

    Before I chose Zabbix, I looked at a lot of solutions. I lost track of the Docker containers I configured with packages with crazier names than Zabbix. And while there are some good ones out there, I kept coming back to Zabbix for a number of reasons.

    Setting Up Zabbix – The Inevitable Learning Curve
    I don’t care how much of a computer savant you are, setting up a new piece of software is a chore. You’ve got a learning curve you didn’t think would be that bad, clashing with what you think might be a bug, but you’re really not sure, so you forge ahead and inevitably configure things the wrong way and have to start again. I mean, who reads manuals – right? Well, with Zabbix, the learning curve isn’t that bad, especially if you embrace Templates. Sure, it’s new to you, and the concepts are foreign, but within a few hours, you should be able to get the gist of it if you’ve used monitoring software before.

    Presenting the Data – Don’t Tell Me What I Already Know
    Once Zabbix is up and running, the data streaming into it gets massive. Even if you start out just monitoring your home network, it’s easy to have upwards of 50 or more devices (called Hosts in Zabbix) returning data. Since each of the Hosts may be dozens of data points collected on it, you’ll soon have a dizzying amount of data. Normally, this could get quickly overwhelming, but Zabbix serves up the data in styles that fit your needs.

    Zabbix comes pre-configured with the single Pane-of-Glass display, found on the menu as Problems. It’s a timeline of all the current issues on your network, clearly laid out, easy to read,  and “glanceable”, meaning if you left it running on a monitor that you just glanced at, you’d know how your network was performing, even if it had a thousand Hosts on it.

    If you want to customize your view, Zabbix offers Dashboards. While it comes with a few pre-configured, it allows you to create your own and paint the screen your way – in a way that makes sense to you. You can fill a Dashboard with charts, maps, network diagrams, etc., all tailored to your needs. Pages on the Dashboard can even cycle if you want. And you can have unlimited Dashboards. I have some that are very specific to networks I’m monitoring for customers and a few that are overviews of all the networks I’m watching over – too many to mention.

    Change – It’s What’s For Dinner
    Just when you think you’ve got everything under control, change happens. We’ve all been through it, and we know that the tools you were using that weren’t flexible enough to deal with the change got left behind. Zabbix shines in changing environments due to easy configuration, and here’s that term again, Templates. They allow you to perform complex reconfigurations in moments. Server relocation, reconfiguration, new networking equipment, etc. – all handed with a few clicks. Without Templates, even Zabbix would be a pain to reconfigure. Templates simply give you more time to focus on other issues, and everyone in IT has plenty of issues to deal with.

    It’s Not Going To Fade Away
    Zabbix is Open Source, but don’t let that fool you. There’s a real company behind it that’s doing all the development for free, but if you get stuck, they’re there to help. While they do charge for support, it’s reassuring to know it’s out there. As for a community of users, it’s very large and active. Simply google Zabbix, and you’ll find an incredible array of resources. From user forums, to how-to videos, to yearly conferences, Zabbix is alive and well.

    I’ve been using Zabbix to monitor a dizzying amount of Hosts, both on my home network, a WAN into additional sites, and servers in the Cloud. All of this runs with very little upkeep. Zabbix is a keeper.

  • Monitor a Mac’s TimeMachine backups

    If your supporting Mac’s in your environment, you probably have come across TimeMachine, Apple’s answer to backups to a local disk or NAS.

    While configuring TimeMachine on a Mac is very easy, and good Admin wants to keep track that these backups are actually being run on a regular basis. The problem with this is that Apple doesn’t give you a way to easily monitor this. But I went digging, and found a way.

    TimeMachine does have a utility installed on the Mac, and you can get some data from it into Zabbix, but it’s not enough to be really useful. If you’re curious, here are the Items I configured. Note I do a little dancing to parse the output of these system.run[] commands, but they should work well on any Mac.

    Note that these commands are being run on the Mac itself, and that Zabbix is running them via the Agent. This works well, but our problem still remains as these functions tell us basic info about the backup, but not the last time that one was really run.

    In order to sense last time TimeMachine was run, we exploit the drive needing attached to the Mac, as when TimeMachine makes a backup, it has to attach to a disk, even if it’s a on the network. All we need to do is check a date stamp on a directory, and you can get an approximate time of the last backup. In Zabbix this means we can use a few variables.

    • {$TIMEMACHINE_BACKUP_PATH} — The path off the root of the disk which should be the same for all your Macs on the network, like /Volumes/TimeMachine_Disk
    • {$MY_BACKUP_PREFIX} — Often times the name if the Mac, like “TOMSMAC”

    … and then use the zbx_mac_timemachine_backups_templates.json template that you can import into Zabbix and then change variable names. Note that while {$TIMEMACHINE_BACKUP_PATH} is probably fine for all your Macs who use the same TimeMachine directory to back up their files, the {$MY_BACKUP_PREFIX} should be configured at the Host level and changed to match the user’s mac’s name.

  • AWS Throttles Its Customers

    I’ve been working with large files for a few years now. These zip files are normally over a gigabyte and can easily go into the 50-100 gigabyte range. They contain data that can only be compressed to about 50%, so there’s no way to make them much smaller – they are what they are.

    These files need to be uploaded from the field and processed in the cloud, and I’ve chosen AWS to do the storage via S3 and the processing via EC2. While we’ve been generally pleased with each, we have seen that AWS throttles data to and from S3 and also the creation of large instances in EC2.

    This means that if I have a 10 gigabyte zip file I need uploaded to EC2, even if I’m using a network-optimized AWS image that supposedly can handle over a gigabit ethernet, my upload speeds are in the 30-70 megabit range. And yes, I’m uploading from a 1 gigabit connection with a cable – not WiFi, and yes, I’ve tested uploading to various resources, and yes, they all upload at about 800 megabits/sec, the reasonable throughput of a 1 gigabit line. The same goes for downloading from S3. My speeds can be in the range of 14-80 megabits/sec.

    You’ll also find that when spinning up a large instance, such as a 64 cores and 192 gigabytes of RAM, AWS often reports back that they are out of capacity in a particular zone. Trying other zones will have the same results. AWS is simply under-provisioned for handling customer needs that are “out of the norm”.

    I do understand that handling multi-gigabyte files is unusual, and so is spinning up an EC2 instance that costs about $100/day, but no where does AWS say “don’t tax our systems because we can’t handle it, and will throttle you”.

    If you have these needs, you have now been warned about AWS.

  • Tuning MySQL’s Innodb for Smaller Sites

    Scour the ‘net for how to reduce your MySQL server footprint, and you won’t find much. Everyone seems to want to beef up its settings, and I’ve been one of them. When you have a server cluster taking thousands of requests per second, the battle cry is often for more RAM to aid in caching data. But for smaller sites, like a neighborhood hiking club, you will have a very different server setup.

    Smaller sites will be run on smaller servers with smaller resources. With a server that’s only got 4GB of RAM, every megabyte matters.

    Like any good household budget trimming exercise, the first thing you want to look for is caching or buffers that may be too large. If you’re running the MySQL Innodb engine, the default cache is 128MB of RAM. While that doesn’t seem like much, if you’re data set size is 32MB, that’s a lot of wasted RAM that can’t be used for anything else. 32MB of data will never fill a 128MB cache.

    While you’ll need to figure out how which you’ll need to allocate, find your config file (usually in /etc/mysql/my.cf) and make sure you have a line like this:

    innodb_buffer_pool_size=32M # Overall size of pool. The default is 128MB
    innodb_buffer_pool_chunk_size = 8M # Default is 128MB

    Then restart MySQL and see how your system performs. Still fast enough? If you’ve removed unneeded cache RAM you shouldn’t see any difference in the speed, and on small sites, caching may not be needed at all.

    Another thing to look for is excessive disk usage. If you’re not running MySQL in a cluster, and in a smaller site I doubt you are, you can also remove the files used for the Binary Log, which is used to replicate MySQL to different servers, such as a redundant MySQL server. To do this, and something like this into your /etc/mysql/my.cf file. This should save you a lot of disk space as these log files can get huge over time.

    binlog_expire_logs_seconds = 68400 # 1 day in seconds. Choose a value that fits your usage.

    Can you think of any other tips for a smaller site? Just let me know in the comments.

  • How to monitor Unifi equipment

    There’s been a lot of talk on the ‘net about monitoring Unifi equipment. Unifi is a brand of prosumer networking equipment from a company called Ubiquiti. If you’ve ever wanted “the good stuff” for your home or office network, make sure to have a look at them, but I digress.

    One of the drawbacks of Unifi equipment is that they don’t publish their SNMP MIBs, so if you’re trying to add a Unifi router, switch, access point, etc. into your monitoring system, you’re in for a lot of work as you poke about with “snmpwalk” and see what you find. But, if you run Zabbix as your monitoring system, you’re in luck. Zabbix makes this easy, as they do with many things.

    Let’s take the example of a Unifi switch, but this concept works for all of their networking devices I’ve tested.

    The trick in all of this is the Zabbix Template which does not come pre-installed. To install it you can download it here (for credit, visit the author’s GitHub repository) and unzip it. You’ll find an xml file in the zip, and you’ll want to import it into your Templates. To do this, go into your Templates under the Configuration menu, click on the Import button and follow the directions.

     

    Once you’ve imported the Template, it will be available to add to a Host in Zabbix.

    Finally, to monitor the switch, create a Host which looks something like this in the image below. Note I’ve added the “UbiquityAirOS SNMP” Template. I’ve also added an Interface that points to the switch’s network address as SNMP. Since a Zabbix Agent can’t run on a switch, I’ve just used the network address as an SNMP device, so it doesn’t show any errors when looking for a missing Agent. Indeed, Unifi switches are SNMP enabled, so talking on that port won’t produce an error.

    Once you’ve created the Host, assigned the correct networking address as SNMP, and added the Template, just wait for the Template to do its magic. Be patient, as it may take a few minutes to sniff out the switch and its settings. When it does, you’ll see lots of Items created and Triggers to go with those Items, meaning you should have to do very little from here on out.

    Once the Items start to appear, you can look in the Latest Data menu item under Monitoring and select your switch’s Host. You’ll see lots of data appear that looks a little like the image below. Of course, depending on the model of your switch, different things will be monitored and thus shown in your Zabbix as Latest Data.

    Now that it’s up and running a few things to note.

    1. You may get notices of links (often called a “port”) in the switch going offline sometimes. These are not active ports, but perhaps a port that’s not currently in use, and Zabbix discovered this and alerted you. The first time this is a little alarming, but you soon realize that these ports are not in use, and you can disable them if you’d like.
    2. If you upgrade the Unifi device and use the same network address, you’ll get a whole new set of data in Zabbix that will mix with the old data. This mixing is probably confusing, so I’d recommend clicking on the “Unlink and Clear” link near the Template, and then adding the Template back into Zabbix. This will clear out the old data and only have the new data showing up.

  • How to get the “up(1)” to show for a Item’s status value.

    In Zabbix, when you poll an Item for a Host, you sometimes get back a numeric “1” (one) or “0” (zero), but you’d like to display it in English. as an Up or Down value. You’ll typically see this when you poll a Service on a Host, such as SSH or HTTPS. 

    When defining the Item to check, look for the “Value Mapping” field noted by the red arrow in this Template. Simply select the entry Service State, and then update the Item. When the Item next gets data, often every minute, the new value will look like “up (1)” or “down (0)”. 

    Remember, this is just aesthetics. The value stored is still “1” or “0” as a numeric, but now it’s presented in a more human-readable format.

    If you don’t have the Value Map defined, do it under your Template.
     

    Once the Value Mapping is done, add this to the rest of the Template, and then assign the Template to an Item. Note the selection of the Value Mapping below.