Re: [yocto] lost busybox mysteriously

[JH <jupiter.hce@xxxxxxxxx>]

Hi Laurent,

On 1/28/20, Laurent Gauthier <laurent.gauthier@xxxxxxxxxxxx> wrote:
> Hi JH,
>
> As you are not providing enough details in your request we have to
> guess the nature of
> the issue you are seeing.

Let me know what kind of details I haven't provided.

> I will just point out a few things that could help in your situation.
>
> If this is a failure that happens almost every time after you flash a
> new image it could be
> due to NAND blocks being bad.

There was one image, booted kernel fine for the first and second
cycle, then it got shell commands error due to lost busybox. I did see
NAND bad blocks list during booting, but that should be managed by
MTD, won't be a problem to case kernel crash or missing busybox,
right?


> As you mentioned that you have been working with this system for a
> year it could be that repeated
> flashing of UBI images has ended up causing some NAND blocks to go bad.

No, I mean I have been using Yocto to build image and to flash to the
NAND for a year, the devices I am currently using actually a new
revision hardware device, it was only used for the first time when the
new image was installed, the first and second booting was good, the
third got bad missing busybox and shell command, crashed kernel
booting.

> When using UBI it is a good idea to use a flashing procedure that will
> preserve UBI wear-levelling
> information. An example of such a procedure can be found in section
> 4.4 of the following page:
>
> http://variwiki.com/index.php?title=U-Boot_features#Flashing_UBIFS_to_NAND

Than you for the link, yes we have already been planing to work on a
new partition scheme to move MTD partitions to UBIFS volume
partitions. But I don't think the current problem is related to NAND
bad blocks.

> These instructions (ubi part/ubi remove/ubi create/ubi write) need to
> be adjusted for your specific
> hardware layout, but this can help. In short you should update
> individual UBIFS volumes in your UBI
> partition rather then reflash the whole UBI partition.
>
> Another option is to use ubiformat from the u-boot command line. For
> this you would need to research
> the subject by yourself.
>
> Updating your UBIFS volumes this way you should be able to extend
> greatly the life of your NAND if
> you are flashing your development board multiple times a day.
>
> I would like to also comment on a remark made by someone else about
> UBIFS being safe across
> power cycles:
>
> While the assertion that "UBIFS is safe across power-cycles" is true
> in theory in practice you probably
> should avoid to rely on it too much.

There is not practical we can avoid it, the device is an IoT device
installed and operated automatically, there is not practical we can
run command to turn it on and off.

> My recommendations to improve resistance to power-cycles would be the
> following:
>
> 1. If possible mount your UBIFS root-filesystem as read-only, and will
> avoid most issues. This means
>    you should use tmpfs for temporary/volatile filesystems.
> 2. If your root-filesystem cannot be read-only then remount it as
> read-only just before the final shutdown
>    (using for example "mount -o remount,ro /" followed by a "sync") as
> this will limit the possibility of a
>    corruption of the UBIFS occurring on the next reboot.
>
> I hope that you will find what your issue is.

> Kind Regards, Laurent.
>
> On Mon, Jan 27, 2020 at 1:53 PM Quentin Schulz
> <quentin.schulz@xxxxxxxxxxxxxxxxxxx> wrote:
>>
>> Hi JH,
>>
>> On Mon, Jan 27, 2020 at 10:13:37PM +1100, JH wrote:
>> > Hi Andy,
>> >
>> > Thanks for the response.
>> >
>> > On 1/27/20, Andy Pont <andy.pont@xxxxxxxxxxxxxx> wrote:
>> > > JH wrote...
>> > >
>> > >>That the same problem of missing busybox was not just occurred during
>> > >>the device running in the middle of operation, it was also occurred
>> > >>during booting image from NAND, I saw several times that the first
>> > >> and
>> > >>second cycles of booting image from NAND were working well, then some
>> > >>following booting process would be crashed by missing busybox, then
>> > >>could not run whole shell commands. I have been pondering if it could
>> > >>be caused by NAND issue or network virus / fishy? Appreciate any
>> > >>clues.
>> > > The first step is for us to understand what “missing” means?  Have
>> > > you
>> > > got any mechanism (U-Boot, SD card boot, etc.) that will allow you to
>> > > mount and look at the contents of the NAND file system?
>> >
>> > Means that busybox was not there anymore, it mysteriously lost, all
>> > shell commands would no longer available. It cannot to run mount or
>> > any shell commands. There was two scenarios when that happened:
>> >
>> > - In the middle of running, the device all of certain could not run
>> > shell commands and failed mysteriously
>> >
>> > - During the u-boot booting kernel process, there were full errors of
>> > failing shell commands. Let me make it clear,  that booting error did
>> > not occur in the first or second kernel booting after the new image
>> > installation, it happened in the following kernel booting, but there
>> > was nothing to delete busybox accidentally, busybox was just
>> > mysteriously disappeared. Because I could not run ls, I did not know
>> > if there are other things missing. If you ask how I could know the
>> > busybox was missing, I ran the zImage-initramfs to boot the linux in
>> > RAM, then mount the ubi0 to find  out busybox was gone.
>> >
>> >
>> > > If you look at the /bin directory (ls -la /bin/busy*) what do you
>> > > see?
>> > > Have the files been deleted? Truncated? Zero length?
>> >
>> > Could not run ls or any shell commands when the busybox was missing.
>> >
>>
>> /bin/ls -la /bin/busy* ?
>>
>> Maybe something is messing with the PATH environment variable. Or
>> something is removing the symlinks from some binaries to busybox.
>>
>> > > What file system are you using on the NAND flash?  How are the
>> > > devices
>> > > being reset during the various boot cycles?  If it is a hardware
>> > > reset
>> > > then some file systems are less resilient to it than others but I
>> > > would
>> > > expect in that case more fundamental boot issues.
>> >
>> > UBIFS, most device reset or boot cycles were calling halt or reboot,
>> > but it sometime it could just use power cycle.
>> >
>>
>> IIRC, UBIFS is safe from power cycles.
>>
>> Quentin
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>>
>> View/Reply Online (#48161):
>> https://lists.yoctoproject.org/g/yocto/message/48161
>> Mute This Topic: https://lists.yoctoproject.org/mt/70128245/3618354
>> Group Owner: yocto+owner@xxxxxxxxxxxxxxxxxxxxxx
>> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
>> [laurent.gauthier@xxxxxxxxxxxx]
>> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
> --
> Laurent Gauthier
> Phone: +33 630 483 429
> http://soccasys.com
>

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/