Raspberry Pi rebooting itself when it becomes unreachable from outside networks

RPI Shooting Linux Penguin

Introduction

If you have a Raspberry Pi in a remote location, far away from where you are, then it’s important to be able to reach the Pi from the distance (from outside networks) all the time. Unfortunately things can go wrong. For example it might happen that the modem or the router to which the remote Pi is attached temporarily looses the Internet connection and sometimes the Pi’s network interface is not able to automatically recover from this problem and reconnect. Or there could be some trouble on the Pi itself which could cause the connection to be lost and not recovered. No matter how well things are set up, it could always happen that the Raspberry Pi goes offline and thus becomes unavailable from the distance. This is a problem, because the only way to get the machine back online is to go to the remote location and reboot it. That means wasted time and possibly also unnecessary fuel consumption (for traveling to the Pi’s location). If the Pi is used in headless mode (no screen, keyboard or mouse attached), then the problem is even more serious because you cannot communicate with the Pi, so the only way to reboot it is to pull the power plug, which is not a healthy thing to do. You should always shut down the OS from software before removing power:

sudo shutdown -h now

Failure to do this may result in damage to the SD card (it happens rarely, but it does happen sometimes).

So if we need our remote Raspberry Pi (or maybe even a local Pi) to recover the lost connection, it must be able to reboot itself whenever the problem of connection loss occurs. In some cases it might be enough to just restart the network interface service instead of rebooting:

sudo service network restart

However, in other cases something else might be stuck, so the safest solution (from the point of view of recovering online availability) is the brute method of the Raspberry Pi rebooting itself (unless of course some software runs on it which should not be restarted if it can be avoided).

The most important questions that need to be answered before we can implement the solution are:

    • How long is the Pi allowed to be offline before it decides to reboot itself?
    • How can the Pi detect that it’s unreachable from an outside network?
    • How can the Pi reboot itself?

 

Executing tasks periodically and automatically

To execute a task periodically and automatically on the Raspberry Pi (and on Linux in general), you have to add a cron job. cron is the Linux system scheduler, which executes the tasks specified in its list periodically. To edit the list of scheduled cron tasks, open it with the following command:

sudo crontab -e

If you need to add a new scheduled task, add a line to the bottom of the file, in the following form:

Minute Hour DayOfMonth Month DayOfWeek Task

Every line contains 6 words. The first 5 words form a pattern which defines when to run the task and the last word is the path to the task (script, program) to be executed.
The time pattern formed by the first 5 words is quite flexible. You can use wildcards (*) instead of each word, lists of values, intervals of values or single values. A few examples:

    • 0 17 10 3 1 example.sh – executes the example script at 17:00 on the 10th of March if it falls on Monday
    • 0 17 * * 1 example.sh – executes the example script at 17:00 every Monday
    • 0 17 * * * example.sh – executes the example script every day at 17:00
    • 15 * * * * example.sh – executes the example script every hour, when the minute equals 15
    • 0,15,30,45 example.sh – executes the example script every hour, when the minute equals 0,15,30 or 45
    • 0 17-20 * * * example sh – executes the example script at 17:00, 18:00, 19:00 and 20:00, every day
    • */15 * * * * example.sh – executes the example script every 15 minutes
    • * * * * * example.sh – executes the example script every minute

As you can see from these examples, the smallest scheduling interval allowed by cron is 1 minute. When you open the cron tab with sudo (sudo crontab -e), you are actually opening the root user‘s cron tab, so the tasks added to the list will be executed as root. For more details about cron, see the related Wikipedia page.

 

Attempt to guarantee online availability of the Raspberry Pi

The simplest approach to getting around the loss of online availability is scheduling a periodic reboot. This can be done by adding a cron job to the root’s cron tab:

sudo crontab -e

Add the following line to the bottom of the cron file:

0 0 * * * /sbin/shutdown -r

Finally save and close the cron file (in case of nano CTRL+o, CTRL+x)

The above line schedules a system reboot once a day, every day, at midnight (minutes=0, hours=0).

The main problem with this approach is that it just executes a blind reboot periodically, without checking if it’s really needed. In most cases it won’t be needed, the reboot will be unnecessary. It’s not the best of ideas to do this. It might not be a tragedy to execute an unnecessary reboot once a day, but what if the Internet connection is lost soon after the reboot? Can we afford to not have online availability for the next almost 24 hours? Probably not. But that means that we would need to schedule a more frequent reboot, perhaps every 4 hours. The Pi would keep rebooting itself (mostly unnecessarily) quite frequently. So this solution is just not good enough.

A better approach would be for the Pi to check if it can access the outside world and only reboot itself if it cannot. For example it could periodically try to download the main Google search page (http://www.google.com), or some other page that is almost for sure available online all the time. In case it could not download the given page, it would reboot itself. Unfortunately this solution isn’t good enough either because what we’re interested in is being able to access the Pi from outside, not the Pi accessing the outside world (the other direction of communication).

 

A real solution

And so we arrive to the final solution, which is for the Raspberry Pi to periodically check if it has been successfully accessed from outside and reboot itself if it hasn’t been accessed from outside for more than a given period of time. Unfortunately this means that somebody has to access the Pi from time to time, otherwise it will think that it’s unreachable and reboot itself. If the Pi only reboots itself when it detects that it has not been accessed for a fairly long period of time (perhaps 24 hours), then a human user could log onto it every day to prevent the rebooting when it’s accessible. This can work, but it’s a bit inconvenient. It’s a lot better if a second Raspberry Pi, located in an outside network, periodically logs into the first Pi and executes a command which leaves some trace behind to let the first Pi know that it has been accessed from the outside and that it should not reboot itself. Unfortunately for this solution a second Raspberry Pi is needed, but the good thing is that the solution can be implemented both ways, meaning that both Pis can keep each other in check. If the possibility of communication is lost in ether direction, the Pi that becomes inaccessible will see that the other one did not log into it for more than the allowed time frame and it will reboot itself.

 

The actual implementation

The first thing that we need is a shell script that resides on the Pi which must be accessible from outside. This script will check its own last modification time and reboot the Pi if more time has elapsed since the last modifiation than allowed. Using the modification time of the script to decide whether a reboot is needed or not is convenient because no other file is needed. Below is a shell script to achieve this (also available for download here):

# How many minutes to allow to pass before rebooting the machine?
MAX_MINUTES_SINCE_LAST_TOUCH=240 #4 hours

# How many minutes of warning (broadcast to logged in users) before actual reboot?
MINUTES_OF_WARNING_BEFORE_REBOOT=2

# Obtain the time when this script was last touched
t_last_touch=`stat -c %Y $0`

# Calculate the difference in seconds between
# the current time and the time of the last touch
t_diff_sec=$(( $(date +”%s”) – $t_last_touch ))

# Calculate the difference in minutes
t_diff_min=$(( $t_diff_sec / 60 ))

# Display the elpased time
echo “Time elapsed since the last modification of ${0##*/}: $t_diff_min min ($t_diff_sec sec).”

# If more than the given number of minutes have elapsed
# since the last login, then reboot the machine
# Give logged in users 5 minutes to save their stuff
# (5 minutes warning before the actual reboot)
# Touch this script file before reboot to reset the counter
if (( $t_diff_min >= $MAX_MINUTES_SINCE_LAST_TOUCH ))
then
touch $0
/sbin/shutdown -r +$MINUTES_OF_WARNING_BEFORE_REBOOT
fi

Let’s assume that this script is saved as /soft/scripts/rebooter.sh
Make sure that the script’s access rights are set properly, so that the users who need to run it can do so. For example you may allow it’s owner and the users who are in the owner’s group read/write/execute the script and everybody else just read it:

sudo chmod 774 /soft/scripts/rebooter.sh

The second thing that we need is a cron job that executes this script periodically. For example the below line executes the script every 15 minutes:

0,15,30,45 * * * * /soft/scripts/rebooter.sh

Note that it’s safer to execute it when the system time minute value equals 0, 15, 30 or 45 than to execute it simply every 15 minutes (*/15 for the minute pattern is less safe than 0,15,30,45) because if something goes wrong and the reboot timer is not reset to 0 after a reboot, then you will still have a maximum of 15 minutes to log in and fix the problem. If the cron entry’s minute pattern would be */15 instead, then the script would run again after every reboot, and if the timer were not reset, it would keep rebooting the machine in an infinite loop.

The third thing needed on the Pi for which we are trying to guarantee online availability is resetting the reboot counter to 0 after each reboot, so that after a reboot the checking script does not think that it needs to reboot the Pi again. To achieve this, open the /etc/rc.local script file (which is executed by the operating system automatically after every reboot) and add a line to it which modifies the rebooter script’s last modification time:

open the file: sudo nano /etc/rc.local
add the line: touch /soft/scripts/rebooter.sh
save the file: CTRL+o
and exit the editor: CTRL+x

Finally, one last thing is needed: on the second Raspberry Pi (which keeps the first one in check) a periodic task must be scheduled to log into the first Pi and update the modification time of the rebooter script. In other words, we must add a cron job to the second Pi to execute a remote command over SSH on the first Pi, to update the modification time of the script. If the first Pi becomes inaccessible, the second Pi will fail executing this remote command and the modification time of the rebooter script residing on the first Pi will not be updated, which will cause the periodically executed rebooter script to reboot the fist Pi after a while. Log into the second Pi and add the needed line to the crontab:

sudo crontab -e
add the line: */30 * * * * ssh [email protected] ‘touch /soft/scripts/rebooter.sh’
save it and close the editor (in case of nano CTRL+o, CTRL+x)

This line will cause the second Pi to log in every 30 minutes as user1 into the first Pi (reachable at the address pi1.com) and update the modification time of the rebooter script.

For this to work, the user from the second Pi must be able to log in without a password, as a trusted user, onto the first Pi as user user1, via SSH. The cron job is executed as root, so both the root user and user1 must be granted passwordless SSH access to the first Pi, when logging in automatically as user1 onto the first Pi. To understand why this is needed, how it works and for a guide to help you set up the passwordless, automatic SSH login, read this article.

 

Notes

In the form presented above the script reboots the Pi on which it exists if more then 4 hours have passed since it was last modified, but it detects this only when it is run by cron. In our example we scheduled the script to run every 15 minutes. This means that the maximum time for which the Pi can be offline (in the worst case) is a little less than 4 hours and 15 minutes. The second Pi tries to update the rebooter script’s modification time every 30 minutes, which gives it about 8 attempts to successfully update it before a reboot. SSH connections may sometime fail, but 8 opportunities should be more than plenty. The good thing about this setup is that if the connection between the two machines fails temporarily (let’s say for 2 hours), but then it recovers, the Pi won’t be rebooted unnecessarily because the second Pi will still have time to update the modification time of the rebooter script. Of course, these values (4 hours, 15 minutes, 30 minutes) can be tailored to you own needs, but keep in mind that if the time allowed by the rebooter script to pass before it reboots the machine is very short, then there might not be enough buffer for temporary SSH connection failures. Also, it’s not very healthy for the Pi to reboot itself very frequently. The second Pi should probably not touch the rebooter script’s modification time very frequently because the Raspberry Pi uses and SD card as long term storage medium and SD cards allow a limited amount of writes along their lifetime, so the number of writes should be minimized when possible. Don’t take this too seriously, though. The number of writes supported by modern SD cards is quite huge.

 

Testing

To test that your setup works, you can manually run the rebooter script from the terminal on the first Pi (which hosts the rebooter script). When not enough time has elapsed for a reboot, it displays the number of seconds/minutes elapsed since the last modification. If everything works well, you should see this counter reset after each 30 minutes (when the other Pi logs in and updates the script’s modification time). You should also see the counter reset after a reboot.

 


Comments

Raspberry Pi rebooting itself when it becomes unreachable from outside networks — 13 Comments

  1. hi
    good solution, But is it possible to check raspberry pi with the windows PC connected in a intranet connection before reboot instead of checking 2 pi in the network

  2. Unfortunately shutdown -r now , reboot will conduct lots of secondary software calls (HUPs to all processes, closing I/O and the like). Though this is desired to assure safe shutdown without hazard to the file system it may – and not only rarely – be impossible to start these secondary calls by the same (deeper) reason and so shutdown will fail as the bunch of calls to perform shutdown is not executed.
    A considerable but not optimal alternative is reboot -f which circumvents shutdown calls but may end up in same hassle like pulling the plug. Best alternative is using both the suggesstd solution and in a second escalation raspies hardware watchdog as it is independent of the internal memory, i/o and priority struggle. Because if raspi does not react to shutdown the chance to perform safe shutdown is lost anyway. The hardware watchdog therefore does kind of digital plug disconnection if the shutdown -r … approach failed.

    • Good point! Although I’ve never had a case over the last year (since I’ve been using this) when shutdown -r now failed, I agree that using both approaches, one after the other, is more likely to achieve the desired result of restart.

  3. Hi. I have similar problem – it seems, that my Pi stops running cron jobs after some time (randomly, after several days).

    Unfortunately, when it happens, I’m not able to connect to it over ssh and therefore I’m not able found out what is happening inside. It is not completely frozen (some pages of web server still running etc.), but so far I haven’t realized where is core of trouble… I just tried to log actual date into cron.log and it really seems, that cron stops working.

    What can I do? Some long-term running script which sleeps all the time and once per minute check if cron is running – and if not, then reboot? Or something else?

    Thanks for your advises…#JK

  4. Hi guys!
    Just a quick question. I am very noob with linux commands. Would not be possible to do this without a second pi? For instance not by logging in from a second pi but instead the pi would try to access an external ip google.com for instance every hour. if it can ping it then fine, if not then reboot.
    Thanks
    Peter

    • Good question. Depends on what exactly it mean in your case that the Pi becomes unreachable. For me the problem was that other Pis could not reach it trough SSH. So for me the second Pi is needed. If in your case the problem is that it can’t even ping an external machine, then in your case the suggested solution should be better.

  5. Hello,

    I am running radio studio audio into the USB of a pi2 using BUTT (broadcast using this tool) application.

    It runs perfectly sometimes for weeks at a time and suddenly we lose the stream (usually a weekend, or no staff around to access the engineering room).

    If I am at the site and look at the monitor, the GUI for BUTT is gone, but TOP still shows BUTT running.

    Only clue so far is ‘lost connection to server’ in the log, so maybe it is a network dropping issue. Don’t know why the BUTT GUI would go away though.

    Anyway, I would like to have the pi reset itself if there is no audio streaming.

    Like a ‘dead air’ alarm on the radio transmitter, I’d like to find a spot on the board to check for the stream and use external hardware to reset the pi safely rather than hard power cycle it.

    I just got a couple of these boards: http://www.switchdoc.com/dual-watchdog-timer and was thinking of tying another PI in to monitor the stream. But if that one fails to receive the stream, it would keep resetting the stream pi for nothing.

    Do you think I should go the dual pi route and have two to check on each other? Preferably over the Internet from the manager’s or my place.

    • I’d try the ping solution first if I were you and if it does not help, go for the dual Pi solution.
      Resetting the Pi with external hardware sounds scary to me, I don’t know how it can be safer than a reboot.

  6. Thanks for the reply.

    Recently, in Broadcasting magazine, or maybe Radio, there was an article on using PLCs for station equipment over custom built devices for a station.

    I didn’t like how they skimmed over Pis and the like because of ‘reliability not proven yet’.

    For one PLC, I could have a bunch or programmed Pis around to be swapped into place for our stream.

    Probably could even walk a DJ through swapping in a new one over the phone.

    During weekdays the station manager is very tech savey for that.

    I just try to keep things reliable and redundant as PBS or something. Just showing my age.

    We are non commercial, and donors will put up with some glitches in our stream, especially now as the archive of a show can be put on our page for a few weeks if the stream was down.

    Our stream archive service records off our stream, and as said above we can patch a bad recording from our local archive drive.

  7. I have made a system where arduino (having wifi module) continuously pinging my rpi ip address locally avery 30s and if the rpi not reply back after a minute the relay attached to the arduino will make the rpi reset using rpi reset pin.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.