in Raspberry Pi

Stopping SD Card Corruption on Raspberry Pi’s Raspbian

The following are instructions for minimizing SD card writes for Raspberry Pi’s “Raspbian” Distribution.

If you’re like me, you’ve run into a corrupted SD card too many times to not become hell-bent on making it never happen again. I have the following setup, and it seems to be working well for me.

The biggest offender for Filesystem writes on any linux system is logging. If you are like me, you don’t really look at /var/log after a recycle anyways. This area, and /var/run, a location where lock files, pid files and other “stuff” shows up, are the most common areas for mess-ups. Take a look at your blinking FS light on the board. Our goal is to make that light stay off as long as possible.

Set up tmpfs mounts for worst offenders. Do other tweaks.

Linux has with it the concept of an in-memory filesystem. If you write files to an in-memory filesystem, they will only exist in memory, and never be written to disk. There are two common mount types you can use here: ramfs, which will continue to eat memory until your system locks up (bad), and tmpfs, which sets a hard upper limit on mount size, but will swap things out if memory gets low (bad for raspberry pi, you will probably be hard stopping your device if it is low on memory).

We will first solve the usual corruption culprit and then move on to making sure we are covered when our programs decide to blow up.

The following two lines should be added to /etc/fstab:

none        /var/run        tmpfs   size=1M,noatime         00
none        /var/log        tmpfs   size=1M,noatime         00

There’s more, however. By default, linux also records when a file was last accessed. That means that every time you read a file, the SD card is written to. That is no good! Luckily, you can specify the “noatime” option to disable this filesystem feature. I use this flag generously.

Also, for good measure, i set /boot to read-only. There’s really no need to regularly update this, and you can come back here and change it to “defaults” and reboot when you need to do something.

After this, /etc/fstab should look as follows:

proc            /proc               proc    defaults                    0   0
/dev/mmcblk0p1  /boot               vfat    ro,noatime                  0   2
/dev/mmcblk0p2  /                   ext4    defaults,noatime            0   1
none            /var/run        tmpfs   size=1M,noatime             0   0
none            /var/log        tmpfs   size=1M,noatime             0   0

Go ahead and reboot now to see things come up. Check the Filesystem light on your raspberry pi after it’s fully booted. You should see no blinking at all.

Disable swapping

As a note, since i have done the changes above, i have not corrupted an SD card. I’m not saying I’ve tried very hard, but it is much better, even with power plug pulls, which i tried a few of after doing these changes.

One protection against SD card corruption is an optional, but potentially “I’m glad i did that” change to disable swapping.

The raspberry pi uses dphys-swapfile to control swapping. It dynamically creates a swap partition based on the available RAM. This tool needs to be used to turn off swap, and then needs to be removed from startup.

Run the following commands to disable swapping forever on your system:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

After doing this, call free -m in order to see your memory usage:

pi@raspberrypi ~ $ free -m
             total       used       free     shared    buffers     cached
Mem:           438         59        378          0          9         27
-/+ buffers/cache:         22        416
Swap:            0          0          0

If you reboot, and run a free -m again, you should still see swap at 0. Now we don’t have to worry about tmpfs filesystems swapping out to hard disk!

Housekeeping!

As you go on and install other tools and frameworks on your raspberry pi (like ROS), you need to be aware of where caches and logfiles are written to. If, for example, you use ROS, the nodes will, by default, log to ~/.ros/log/. For these sorts of things, and for ROS in particular, point these logs to a folder on /dev/shm. This mount is created by default on linux boxes, and is a tmpfs filesystem that is globally writable by default. Mine has 88 MB.
For example, pointing ROS logs to this file could be done by adding the following to your .bashrc:

export ROS_LOG_DIR=/dev/shm/rosLogs

If you feel like creating more mounts, feel free, but I have run into ownership issues that required having another script on startup that had to be run as root to chown directories to their proper owners.

Getting to 100%

The only way to fully protect against SD card corruption is to mount your root filesystem as readonly. For me, this was too much of a usability issue. I am editing files and installing new packages too often for this to be feasible. The steps listed above should, however, cover you 98% of the time. Try not to pull power while you’re editing files or installing new packages on your device, and you should be fine. Still make backup images of your SD card every once in a while! This is a best practice no matter what!

Happy Hacking!

Write a Comment

Comment

12 Comments

  1. Can it be done some like readonly fs as default mode, but some script lauching enables write mode, then another script back fs to read only? or maybe adding user which have write permissions?

  2. Thank you verry much for posting this.
    I have a lot of trouble with my Raspberry, trying to make it not ruin my card once a few days, so i came across this post; it will surely prove useful.

    However, there’s something i discovered while playing with my Rasp, trying to see how to make it stable. Maybe it will prove useful to others. It seems that Arch Linux is far more stable than Raspbian, regarding SD card corruption. Without your settings made, i pulled the power plug several times with no problems. I started to read directory tree “dir -Rlh /” and pulled the plug while still listing, still no problems. I started to write a file “cat /dev/random > /root/file” and then pulled the plug and still no problems found. After restart, Arch Linux would still give me SSH and let me login.

    So i would say that this distribution of Linux is much better, at least from this point of vue, than Debian.

    Cheers

  3. Thanks for the info.

    As suggested, i moved /var/log to a 1M ramdrive. However, I found that the /var/log gets used up in about a week (on my system) due to various messages. Auth.log was the largest due to a 5 min cron using sudo to reports some stats to a central server.

    Is there any recommended maintenance of /var/log directory to stop it from getting to 100% used?

    I guess I could have a monitor script run every hour to check the usage and zero a few files (auth.log, syslog, messages) when nearing the limit. But that seems problematic.

    Thanks.

    • I handle this sort of problem by either configuring it, if possible, or by throwing a cron job together that truncates the file after a period of time. “echo > /var/log/auth.log” would work.

      In your case, the “logrotate” tool is probably going to be the right solution. “man logrotate” for more info.

  4. Thankx for the tips. Just one problem. With this line:

    none /var/run tmpfs size=1M,noatime 00

    My wireless keyboard and mouse stop working. If I comment that line in fstab they will work again. Do you have any clue how to solve this?

    • Same issue here.
      none /var/run tmpfs size=1M,native 00
      This line deactivates my wireless mouse and keyboard.
      Any solution/idea how to solve this?

  5. This article is very, very bad advice.
    Disabling swapping: Just don’t do it. The device is swapping for a reason.
    Disabling swapping makes the device slow as hell.
    It is already slow as hell, but after disabling swapping it is nearly impossible to work with it.

    9 out 10 SSH attempts result in ‘Connection timed out’ or ‘timed out while waiting to read’ just because it is completely stuck.

    Why would you store log files in memory any way?
    The device is already short in memory.

    Just buy a decent SD card and you will likely never experience problems.
    I use the cheapest SD card I could find and I pull the power cord regularly and it has never caused any problem for at least 3 months now.

    B.T.W.: most corruption problems arise from overclocking.
    Don’t do that either.

    • Hello,

      1: Swapping only occurs of you run out of RAM.
      2: By Disabling it, I am saying “don’t even try to save me if I’m out of ram”. If you’re swapping, you’re already going to be “slow as hell”.
      3: Your failed SSH attempts are likely linked to this complete exhaustion of device memory. When linux runs out of memory, it is usually configured to start killing processes, so I would suspect that your new SSH connection is being killed over and over again. check /var/log/errors.log to see if you can see that after you successfully log in.

      4: Our definitions of “Short on memory” may be different. By default, your device already has an 80 MB in-memory mount in /dev/shm, by the way. It won’t actually “lock in” that memory until you actually use it. If i tie up all my write-heavy operations using 10M of memory, that is a price i’m willing to pay, as it improves my os ruggedness, and, if you care, the lifetime of the sd card.

      5: I’m sure you can get lucky or that certain sd cards won’t have this issue so much, but mine did. If you would rather not buy a new SD card, this is an option. I am not overclocking my pi, either.

      • Linux not only swapping if runs out of ram, linux uses swap in thousand other cases, check the memory management algorithms in the kernel.

        • “Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space.” – CentOS. You will need to provide proof by contradiction in this case.

  6. In the past i was using voyager linux. This system was working as os for APs, default in ro mode. There were to scripts rw and ro to remount filesystem. Maybe its worth to investigate.