there is another option using write-behind caching on EXT4 so it caches all the writes to ram and then using the COMMIT=xx seconds mount option it will write the changes from ram to disk every so often, the best of both worlds

I was able to reduce my sd card consumption drastically by logging to my remote homeserver. (almost) No local logs anymore. There is another advantage in remote logs: you can find suspects, even after the device in question died. Since there is a homeserver up and running 24/7, it was just a matter of some simple configuration.

This came just as I needed it! Releasing 24 dashboards driven by raspberry pi's next week and was worried about configs being changed and other (python) code config that needs to write to the memory! Much appreciated Andreas!

YES! Finally I can cut the power to my rasppi whenever I want without the fear of destroying the SD-Card!

great features Thanks for sharing your experience with all of us 👍😀

With the newer Raspberry Pi 4 with the higher RAM this overlay makes a lot more sense now. I guess you could use read/write Boot drive and periodically copy the overlay to the SD or to an external network drive for data that may need to be kept longer term. Thanks for the great information.

I also tend to install ZRAM on every pi I own. It removes the need for swap space on the sdcard and it allows you to fit about 50% more stuff into the physical available RAM.

Thank you! Extremely useful, extremely inexpensive, reversable and can be done in seconds. Love it! The overlay wins for me.

Thanks Andreas, I was worrying about my sd card in. my headless Raspberry Pi 4 recoding NOAA weather images 24/7 so I made images of the card if it fails, I haven't had any card fail yet but you saved the day and me worrying, thanks for this tip will implement it : )

I've been using raspberry pi for quite some time It didn't realize that was there. I really enjoyed this video. I really enjoyed listening to you on the ham radio workbench podcast as well. Thanks

It is also very useful to mount the filesystems with noatime,nodiratime options in order to avoid updating access time in the file and directory attributes each time the file or directory is accessed, no matter it was just read.

IIRC, OpenWRT uses OverlayFS, but their implementation is pretty static without lots of hackery. Really neat the Pi Foundation was able to simplify it down to a toggle switch! 👍

excellent... i wondered what that setting was for but never looked into it!...thanks Andreas

Great hint, Andreas! Thank you!

Thanks Andreas. I watch your videos and like them every time but this is the first time I comment. I did not know about this feature and it is very useful for me because my children use Pi 400 for school. They love to mess with the OS and I have to reimage their systems often. Well, not any more. Thanks again

Thank you Adreas for the easy to understand ex[;amation of the Overlay File System. It's just what I needed for my weather station which uses an ePaper display controled by a Pi Zero W.

Thank you this was a simple but effective trick!

Very useful video, Andreas👍- this setup is indeed ideal for 24/7 applications that do "something" and send the results to a remote server or such. Or another idea: Run Minikube on the Pi4 and push containers to it from remote. That way the Pi would become application-agnostic?

Useful AND interesting! I was unaware of the read only setting.

Thank you for these useful tips. Much easier that I had imagined Great comments on this video too. :)

I use a different approach, and I find it much simpler and scalable: I use a NAS, so shared directories to save logs and data. So I get automatically backups (I need to set up only once for the base directory in the NAS), fast disk (but it depends also on network), cheap disks. And I'm able to split or merge services and rpi much easier.

Great video, as always. Also, a great tip I will try both ways. Overlay and with a small SSD drive.

I use RAMFS and mount it over the top of the log directories. Then, when a program writes to the log files in the log directory they are intercepted by RAMFS instead. When the max size is reached they are discarded. They are never written to the SD card. My SD card is in its 4th year of operation (as a LoRa gateway) with no problems so far.

Great. I just needed this.

Very useful and simple, many thanks.

SD cards are not by definition unreliable. They come in different flavors for different use cases. If you pick the right one, an SD card can be more reliable than a big cheap SSD. The most important criteria are the cell technology used and the Bill of materials. Cell technology: * Most consumer cards use TLC or QLC (3D) cell technology. A single cell can typically be rewritten approx 200 to 300 times. * Some consumer cards advertise with 'long endurance' or 'industrial' terms. Often such cards use MLC cell technology (but this is not guaranteed!). In this case a single cell van typically be rewritten approx 2000 to 3000 times. This type is also often used for EMMC memory and modules. * For more demanding environments (automotive, medical, extreme conditions,..), often (p)SLC cards are used. For these type of cards, a single cell can be typically rewritten up to 30000 times. That's >100 times more than a typical consumer card!. You will not find these cards in a regular (online) shop. Cards utilizing this technology often have more advanced flash controllers onboard providing additional security and features compared to consumer grade cards (e.g. power failure protection, better wear leveling...). Bill Of Materials: * Most consumer cards do not have a fixed Bill Of Materials (BOM). This means the internal components can change without notification and the manufacturer will choose whatever is available and the cheapest that complies more or less with the specifications. So for one batch of cards, the cells can be TLC, and a few months later, QLC becomes cheaper and the manufacturer switches to QLC components, or uses another flash controller... This is also the reason why almost no manufacturer will give the complete specifications of the used components for these cards. * Most manufacturers also offer fixed BOM cards. These are cards where the used components are fixed and will not change without notice. This guaranties that if you buy the same card next year, it will behave exactly the same as the ones you already have. An example of a 'cheap' (p)SLC card with fixed BOM is the SDSDQED-016G-XI model by Western Digial/Sandisk. It is a 16GB SLC card with an extended temperature range (-40+85C) and an endurance of 480TBW. It costs about 30 euro's at Mouser. This is typically cheaper than a usb->sata/M2 adapter + decent SSD. For comparison: a Kingston A2000 SSD of 500GB has an endurance of 360TBW. And if you only look at smaller very low cost ssd's, then the endurance is often 40 TBW) So for projects that do not depend on very high speed disk access or require a lot of storage, such SD card is unrivaled. Both for price and reliability. (The explanation above is a bit simplified... there are other factors playing also a role, but this comment is already long enough for now :-)

Thank you very much! It was useful and interesting 🙂

thanks for the knowledge

Useful and interesting as usual.

Great tips, nice video! To be a bit pedantic, overlayFS itself doesn't mean writes are put in RAM. Rather, overlayFS is one piece of the puzzle - it allows redirecting writes to somewhere else, and when combined with tmpfs (RAM-based filesystem) it achieves the reboot-to-discard-temporary-changes feature.

Indeed interesting. Guess I should make work of adapting my RPi asap.

thanks! this helps me a lot!

Nice info! Thanks!

Nice Video, learned something new. Now it would be great if specific log operations could be made to go to the SD card and the majority just to ram...

great video. happy christmas

Very very good video and here I was wondering why my card would die after a few months of usage. Thanks a lot for this video. Now to the terminal window!

nice and useful trick, thanks for the info

Thanks for the info

In addition to these tips I would strongly recommend getting an SD card that's more suitable for this kind of usage. I've personally had good results with the WD purple SD cards, I've got 3 pi's that have been running 24/7 for about 3 years now on an unmodified raspbian installation with full logging enabled (and one of them is constantly writing due to a pihole installation). So far I haven't had any issues. But... I would recommend partitioning only a small part of the SD card (i.e. 25% or so) so the wear leveling algorithms can do their thing and make sure your SD card won't run out of working blocks.

Great find! I'm going to have to check to see if it works on my OctoPi server 😃

I saw this option before in dietPI, I think it's a cool feature, glad the ported it on raspbian too

Wow..I didn't know about this feature..?...thanks!

Instead of a SSD, you can also use a network filesystem and redirect things there through symbolic links. Handy if you have a NAS on the same network. Beware that SAMBA/CIFS does not work well for that and some services will randomly fail if their data are put on a SAMBA mount. NFS on the other hand works fine with everything. You can also use the iotop command for instance 'iotop -o -b -a' to log the cumulative writes of the different processes so you can see who is writing and how much.

Nice!, this will also improve the performance. I am thinking about to create a minimum raspbian version to improbe the boot speed and memory usage on a Raspberry Pi Zero.This will help me for sure.

Thanks Andreas for the tip. What I do is to use a high endurance SD card, specially designed for high number of writing cycles ( they are used in security cameras). Is a very lazy solution though.

thank you for the job

Excellent information, I already put UPS boards on all my Pi’s but this will is a great trick.

thanks to the guy with the Swiss accent, another great tip and information

After 2 SD cards busted, I put an SSD for my Home Assistant RPi4 and it is running since then very reliable, hope I don't jinx it. But very good tip to make the SD card last longer.

I've usually mitigate writes to the sdcard by making sure the noatime and nodiratime option is on the sdcard in the fstab and mounting a small flash drive to the /var directory, and using a larger flashdrive for the home directory (you could put both partitions on one flashdrive). This should eliminate all writes to the sdcard except when installing and updating software. Not on PIs, but on PCs, I frequently use a virtual disk and keep my work on flashdrives, and have scripts with clickable icons to setup, load, and when my session is done, save and destroy the virtual disk.

This was a great video.. I had been wondering why there wasn't an easy way to do this.. Next step would be to map a network drive and use it for logging and other data you dont want to lose, and it would be the best of both worlds! Thanks!

Thanks Andreas, I believe a short lifecycle is typical for low-cost SD. I use several SD cards in raspberries for 3 years already, incl my Majordomo home automation (decades of mqtt sensors, etc.) and had issues while using sandisk-like SDs). After changing to less cheap I've forgotten about it.

Clever ways to save on writes :) Phones uses F2FS file system to save on writes. F2FS supports ZSTD compression. Image with bear minimum of applications would write less compared to stock Rasbian. There are MLC SD cards that last ~10x at same capacity compared to most micro SD cards in the wild. Jet MLC cards cost ~4x (on SSDs the price difference is more reasonable ~2x). Higher capacity cards will last longer (grows lineary with the size). As for disconnecting the drive unsafely - I had luck with usb sata bridge and microSD readers, they usually go before the storage.

Add commit=300 as a mount option in /etc/fstab. EXT4 default is to commit all writes every 5 seconds. If you mount with commit=300 it will cache writes in memory then write them to disk once every 5 minutes. You can reduce your SD wear by 60x by adding that line.

thats a neat trick, plus is more secure, becasue if you loose your rapsberry pi whilst transporting it, you wont have to worry about still being logged into google etc and saved passwords. just enable it on a new install after you have setup wifi and customized the dektop

Does this also help in the event of a sudden power loss? I imagine as long as there's no writes occuring at the time of loss it could be okay?

Just a quick question about your description, why the anti-docker statement? Is it against docker corp (understandable) or the concept of containerization itself? Thanks for your wonderful work towards better IoT and wireless communication!

brilliant!

Very useful for making changes and checking they work correctly. One question, once you make some changes can you then commit the changes to SD so the changes take effect on reboot?

So, why isn't there an option to write the ram files back to sd on close? Thus dramatically reducing the writes

The medium article instructions, while time consuming are better when it comes to usability With overlay fs, you need to reboot every time you need to make any changes and then reboot again to return

Are any of these options available on the old Raspberry Pi model B boards (not Pi 3B but the original one)? Would it for instance be possible to boot from SD card and then re-mount to an SSD on USB? I'm not a unix expert, so commenters forgive me if my question is very stupid.

Very educational content. Just wanted to point out that I'm personally using High Endurance SD by SandDisk which meant for CCTV application (run 24/7 + consistent read & write application). So far works well (more over 8 month with multiple time of trips by thunder storm - in turn will be a sudden power loss) for my simple LAMP project without overlay file which is running 24/7 when ever possible. Maybe can give those a shoot for extensive test?

Hi. Can I do this with twister OS? This OS is so beautiful. I ve seen a review but I want to follow your advice. If Twister OS has this set up for overlay, would be great.

Thank You

I have the same SD-card running since my Raspi 3B+ was new. 2018? 19? Never got corrupted (headlesss lite raspian doing DNS and Cups). I still did that, thanks.

Thank you very much!!!!!!!!!!!!!!

Good video, no mention of emmc SD card adapters?

Does the SSD have the same corruption issue as the SD cards resulting from power outages? I have corrupted Octoprint a few times from someone (me) cutting the power before shutting down. I am definitely going to try the Raspian Function. Thanks!

What would be the difference when leaving the boot partition on read/write while having overlay enabled?

True, I've lost a few SD cards due to unexpected power off

a pxe boot with a tftp server hosting the image of the operating system ? or even , clients on a vnc environment that runs elsewhere and maps the hardware of the pi to that user profile .

I have a ton of raspberry pi computers (mostly 3b & zero W models) and I've done the 'barefoot' method before. I haven't tried the built-in raspbian feature yet but I wonder, does the feature disable write to other storage? If it doesn't, then you could redirect or save 'persistent' files and such to another device like a simple thumbdrive.

I'm testing the overlay file system on 11 Raspberry Pi, running camera applications. Users were instructed to just unplug the power without powering off the system, so I could see if this was a viable solution. So far no issues with sd card corruption. 2 of the Raspberry Pi were running without interruption all day (9-18) for 3 month and they still work perfectly fine. I don't know if i'm lucky or if it works well, so take it with a grain of salt.

I would like this, but is there some way to make certain write process still work, like apt-get upgrade and for my moonraker to keep Klipper and mainsail up to date? Perhaps this is the case, but I am afraid to turn it on and have issues.

Not Pi related but Linux related. In the early days of SATA SSDs I had one at 16GB from Imation. I had the most 'write intensive' volumes mounted on an LSI RAID volume (e.g. /tmp /var/ and SWAP). It lasted over 6 years in 24/7 use before it was replaced by a new computer. I did pull the drive before sending the computer to freegeek. Those are the 'high write cycle' mounts for a system drive. Data drives are not addressed.

Years ago, my daughter had been using my core 2 duo laptop running ubuntu 12LTS booting(with pre-loaded apps) from the dvd drive loading it on ram for several years till the internal cooling fan conked out. You could still save on the hard drive but everything else resets after shutting down.

It was interesting. I do have an original Pi 1 that has the operating system on an attached powered Hard Drive. Just uses SD card to boot.

I have few application that move files from 1 folder to another in my Pi. I assume that the overlay file system will revert everything back on restart? I getting a SSD for my 3B+ soon, which is a better solution for my situation (i think)

Hi Andreas, what are your thoughts about eMMC storage like the RasPiKey as a halfway house between SD cards and SSD storage?

Nice hint I was not aware of! It's there a known command to write down all memory changes to disk aka SD? That would be great if you are experimenting and at the end think, ah I should have saved this. As I understood it, writing or not ALWAYS requires a reboot first, correct?

Hi Andreas, Last week I wanted to update a 2017 installed Raspbian Jessie on its 8 GB µSD card. Of course, I dumped the card beforehand ... or at least I planned to. Because dd reported many defective blocks. The dump ran for 48 hours because of the many retrys. Out of interest, I checked the file systems in the dump: were both okay. Obviously the mechanism that hides defective cells in the µSD card works very well. But I'm also sure that the card was on the verge of total failure ;-) vy 73 de Micha, DD0UL

Have you tried using high endurance SDCards sold for devices like dashcams that repeatedly overwrite the storage?

I feel you should look at using a network drive so you do not need to use so much RAM. One simple way to make a network drive is to use an old desktop as a Linux box and have it share the hard drive to the raspberry pi.

For logs and things I want to keep I would create separate partition. This way reducing system writes and keeping the data I chose.

Perfect; I'll set the SD to RO, and write data to a NAS.

Have you tried using USB sticks instead of the SD cards? I am mostly using small (memory and physical size) USB sticks now, because they are easier to handle and faster than SD cards, if you use USB3. I haven't had any corruption of SD cards or USB sticks in a long while, but i may use the overlay option for my pan and tilt Pi cameras in the future. For a more expensive SSD solution, depending on the CM4 module, one could always use a CM4 board with M.2 SSD connector like the CM4-IO-BASE-A, which starts at about 25€ for the board and 30€ for the CM4.

Add autofs against an NFS box with your homedir, we did diskless workstations that way for years back in the 90’s.

after the loss of 2 SDcards, ive switched to using a USB Stick in my Pi, worked sofar without issues.

Another thing to do is to enable periodic fstrim operations to discard deleted blocks. This applies to all flash based storage: SSDs as well as SD cards.

Awesome, so I guess no UPS will be needed if uptime is not desired? am I correct?

Just to be clear when I want to update the OS, I need to turn the overlay feature and RO off and then reboot, and update the system and then turn it back on and reboot again correct?

I have installed "remote-iot" to my pi3B/3B+ and am monitoring total writes -the only tool I found that would read microSD stats. I added "/dev/root / ext4 rw,noatime,commit=60 0 0" to mtab" to minimize writes. I have computed writes over many days using the "total writes" count, and it is about 10/sec. I am trying to figure how "many" this actually is...... Doesn't seem like much. Hard to translate this. Thanks for your video! Wish stretch/jessie had overlay FS.

Write files to USB Flash & update as needed if data logging with minimal data loss. ...There exists a "HAT" that goes under the Pi and is screwed on, supports an M.2 SSD via a small USB 3.0 adapter. The bonus: you get SSD speeds with low power draw.

Do you have any data or experience on the reliability advantage of an ssd over an sd card? After all, both have a limited number of write cycles.

The overlay system looks like a cool feature but I almost never use raspbian. Tbh, I've never had a problem with tweaking fstab but this is handy to know to recommend to others as it's simpler to explain.

The overlay technique is actually pretty old. They used it for bootable Linux CD-ROMs back in the day.

How does Overlay File System in Read-Write mode works? As in RO mode but writes on SD Card before shutdown or restart? Or is just a temporary solution when updating and just passes the data through RAM to SD Card? Thank you

WD line Purple of microsd cards for video security has a write cycle of up to 16tbw for a capacity of 32gb.

Are SD cards that fail in this mode of operation covered by the warranty?