Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
15 years ago

e2fsck slows boot

Hi,

I'm working with the Cyclone II Dev board and a Compact Flash (ext3 formatted - journalling). We use ext3 because I am writing data to a file on the CF and when we lose power and we reboot we can do an "e2fsck.ext3" and 'fix the file and disk.'

The board is a data recorder and we want it to boot fast and start recording data to the Compact Flash. I've gotten the boot time down to 8-9 seconds without the CF being written to.

I've found that if the power cycle comes shortly after I've been recording the 'e2fsck' doesn't take too long (10-15 seconds). BUT if I've been recording for a minute or two and power cycle IT CAN TAKE OVER 30 SECONDS to recover the journal.

I've tried differen setting with "tune2fs" with no luck. So is there any exper out there that can HELP ME !! Or suggest things to try.

I've found I have to perform the "e2fsck" or I won't boot.

Is it possible to "mark as fix later" so I can boot faster? Eventually the disk will come back to the lab where we can run a full disk fix and we retrieve the data, but while its "out recording in the system" it needs to not lose data.

I saw ext4 in the "make menuconfig - filesystems" but it appears that the "e2fsck tools" don't support that. Would going to ext4 help me and if so how do I get the tools updated?

Thanks in advance

12 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    In most cases, Ext3 (other than ext2) should not do a long-winding check when mounting a not clearly unmounted volume, but use it's journaling tables to do a fast recovering to the latest clean transaction.

    OTOH, a CF is a great problem when unexpected power fail is possible.

    The card needs to do internal wear leveling and thus it maintains tables which sector is stored where in the flash memory, as each flash page (which can be erased only completely) contains thousands of sectors. When a page wear out is detected, data needs to be moved to other flash pages and these tables need to be modified (possible creating additional wear-out conditions). This can happen with any sector write, and unfortunately the time needed for this until all data has been written to the Flash area, usually is not specified by the manufacturer. I suppose it can be some seconds worse case. (Please tell us if you found a CF manufacturer that publishes appropriate numbers !)

    I was told about cases showing that this effect destroys very old data and sometimes even destroys the card in a way that all data is lost and it not even can be reformatted. The manufacture can revive it by rewriting the sector tables, though.

    IMHO you only can use CF devices for writing data in devices permanently enough powered by a battery.

    Using high level means like alternative file systems obviously does not help here.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    @mschnell You can use industrial compact flash cards that don't have this low level integrity loss problem. As an example, we use the cards from STEC that are guaranteed not to become in an unusable state after a power loss. Of course they are a lot more expensive than the ones you can find in supermarkets ;)

    @jpe1313 Could it help if you 'sync' often from the software, forcing a write on the compact flash more regularly? It could reduce the amount of work necessary to recover from the journal (and loose less data). I don't know if you can reduce the buffer size in the kernel to make that a bit more automatic...
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    --- Quote Start ---

    @mschnell You can use industrial compact flash cards that don't have this low level integrity loss problem. As an example, we use the cards from STEC that are guaranteed not to become in an unusable state after a power loss. Of course they are a lot more expensive than the ones you can find in supermarkets

    --- Quote End ---

    Can you post the specs that cover the said problem ?

    -Michael
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Hmmm.

    Do you really believe their wording means that you can switch off power directly after writing a sector and you can be sure that this sector and all other data is safe ?

    It obviously is not possible to do internal wear leveling without consuming any power and time. So if you switch off power shortly after a write, the device will need to react in some way, provided it can live for a short time on internal capacitators. Maybe it can abort the wear leveling in some decent state.

    -Michael
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    All,

    Thanks for all the comments.

    @Daixiwen and @mschnell -

    I do use the Industrial CF that @Daixiwen mentions. I have used it sucessfully in an embedded flight data recorder (multiple copies) with a Cirrus Processor. I used uClinux and did a e2fsck on every boot and the sytem came up in say, 90 seconds. VERY RELIABLE. WE HAVE NEVER HAD A PROBLEM WITH THESE CFs losing data.

    I also do the fsync on this and we have found that we only lose 200 msecs worth of data (we can compare with other recorders on the A/C).

    Now they want data at boot time too. So, we are trying to tackle this with the Nios II on a Cyclone II (we are trying to stay away from the Stratix because of price). Our prototype PCB that should be here soon has enough memory for a 32 second boot (we thought that would do it - we did not think we'd get such BIG numbers for checking the CF).

    With the Cyclone II Dev Board we've got a simulated data rate and I really think we have the VHDL, my driver and the user space code optimized. I've looked at "top" while we are running and 52% of the time is spent doing i/o.

    BACK TO THE "journalling" :

    At boot I'm doing a: e2fsck -pv /dev/sda1

    If the CF is clean I get a 9 second boot.

    If I run it for a short period of time I get 16-20 second boots.

    If I run it for longer periods - my boot time once was 2 minutes 45 seconds !!

    SHOULD I TRY ANY tune2fs settings ??

    SHOULD I check the disk differently at boot ?

    I tried not checking and the system hangs.

    THANKS IN ADVANCE
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    Another issue I'd like to ask about. We are using the:

    Altera CF IDE interface which is PIO 0

    So, is this the bottleneck ?
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    @mschnell there is no guarantee that the last written sector before the power loss will actually be written, there is just a guarantee that the table is still recoverable in a stable state when the power is restored, either the state before the last write or the one after that. There are several ways of doing this, one that comes to mind is two keep two copies of the table in different pages and have a way to detect which one is valid at boot up.

    @jpe1313 a journal recovery can cause a lot of read/write accesses, so using a faster mode would definitely make the mount time faster. We use the Evatronix Compact Flash controller which works well in all PIO and DMA modes. It's not free and I don't know if they have drivers for Linux, though. I don't think it's a cood idea to try to write on the filesystem before it is fully checked, you could overwrite important data and could make it unrecoverable. Did you have a look at other filesystems? Logfs seems to have fast mount times, but I don't know how easy it is to compile a kernel with it.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    @Daixiwen I hadn't considered others (I guess because ext3 worked so well before). I looked at the "make menuconfig - filesystems" for uClinux and it has JFS support, but not logfs.

    I found a good paper from someone who experimented with JSF - www.linux.com/archive/feed/119025

    And it seems it could do the trick. I recompiled and booted the Cyclone II Eval Board with JFS support and I can see that the jfs support is there when I do a "ps -a"

    16 S 0K 0K 0.0 jfsIO

    17 S 0K 0K 0.0 jfsCommit

    18 S 0K 0K 0.0 jfsSync

    I've got to get the jfs support Utilities onto my CentOS 5.5 development laptop so I can format a disk. (I just pulled something off the web on how to use the CentOSPlus Repository (I'm a big novice at that kind of stuff).

    I at least need "jfs_fsck" utility for my uClinux (so I need to figure that one out).

    ANYONE Interested in this or have any Instructions or Advice let me know.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    15 years ago

    BTW, did you consider to switch off the "last access time" management to reduce wear out and risk of corruption ? AFAIK, otherwise any read will result in a write.

    -Michael