T4 - Dreaded bitflip strikes again

stevebow · Post by **stevebow** » Fri Sep 11, 2020 16:23

Just thought I'd mention I've had another bitflip in the NAND flashmem yet again. This is characterised by the T4 booting to the BW round blue logo, going no further, then rebooting. Reflashing the firmware has fixed this each time.

Below is the serial log from where things go astray, for anyone interested. Looking back in previous logs, the bitflip has been giving grief at the same address (0x2336fc00) each time, so I suspect the NAND device is a little flakey at that address, at least. Luckily I purchased a spare for the day it dies beyond resurrection.

I have logs of the last 3 times this has happened (01/18, 05/19, 09/20), and find it curious that this has happened 16 months apart each time. Well, at least I now have a good idea when to expect the next one.

Code: Select all

[...]
UBIFS: background thread "ubifs_bgt0_0" started, PID 70
UBIFS: recovery needed
brcmnand brcmnand.0: bitflips oob(0) data(1) at 0x2336fc00
brcmnand brcmnand.0: bitflips in apparent erased page at 0x2336f000
UBIFS error (pid 1): check_lpt_type: invalid type (15) in LPT node type 0
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.14.2 #1
Stack : 00000000 00000000 00000000 00000000 00000000 00000000 80986a52 00000033
  00000000 00000000 00000001 00010000 cfc22d08 808ecce7 8083139c 00000000
  00000001 809838d8 00000008 cf689b00 00023043 807681e0 10008703 8002c4b4
  219c4000 00000000 80837558 cfc39aac cfc39aac cfc22d08 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 cfc39a28
  ...
Call Trace:
[<8000c1f8>] show_stack+0x20/0x70
[<8076c794>] dump_stack+0xc0/0xf0
[<802c84a4>] check_lpt_type.constprop.9+0x4c/0x64
[<802c8500>] unpack_pnode+0x44/0x114
[<802c8818>] ubifs_get_pnode.part.8+0x248/0x318
[<802ca6b4>] ubifs_lpt_lookup_dirty+0x244/0x3b8
[<802bd958>] ubifs_replay_journal+0x1064/0x149c
[<802b1694>] ubifs_mount+0xffc/0x1778
[<800f9efc>] mount_fs+0x1c/0x100
[<80118398>] vfs_kern_mount.part.9+0x60/0x188
[<8011af74>] do_mount+0x1f0/0xb30
[<8011bd58>] SyS_mount+0xe8/0x18c
[<8093bda0>] do_mount_root+0x28/0xb4
[<8093c008>] mount_block_root+0x13c/0x26c
[<8093c338>] prepare_namespace+0xe0/0x1fc
[<8093bc28>] kernel_init_freeable+0x268/0x298
[<80763b7c>] kernel_init+0x10/0x100
[<80006b28>] ret_from_kernel_thread+0x14/0x1c

UBIFS error (pid 1): read_pnode: error -22 reading pnode at 8:143427
(pid 1) dumping pnode:
address cf689400 parent cf689500 cnext 0
flags 0 iip 1 level 0 num 0
0: free 0 dirty 0 flags 0 lnum 0
1: free 0 dirty 0 flags 0 lnum 0
2: free 0 dirty 0 flags 0 lnum 0
3: free 0 dirty 0 flags 0 lnum 0
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.14.2 #1
Stack : 00000000 00000000 00000000 00000000 00000000 00000000 80986a52 00000033
  00000000 00000000 00000001 00010000 cfc22d08 808ecce7 8083139c 00000000
  00000001 809838d8 00000008 cf689b00 00023043 807681e0 00000000 8002c4b4
  80986a52 00000000 80837558 cfc39afc cfc39afc cfc22d08 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 cfc39a78
  ...
Call Trace:
[<8000c1f8>] show_stack+0x20/0x70
[<8076c794>] dump_stack+0xc0/0xf0
[<802c8864>] ubifs_get_pnode.part.8+0x294/0x318
[<802ca6b4>] ubifs_lpt_lookup_dirty+0x244/0x3b8
[<802bd958>] ubifs_replay_journal+0x1064/0x149c
[<802b1694>] ubifs_mount+0xffc/0x1778
[<800f9efc>] mount_fs+0x1c/0x100
[<80118398>] vfs_kern_mount.part.9+0x60/0x188
[<8011af74>] do_mount+0x1f0/0xb30
[<8011bd58>] SyS_mount+0xe8/0x18c
[<8093bda0>] do_mount_root+0x28/0xb4
[<8093c008>] mount_block_root+0x13c/0x26c
[<8093c338>] prepare_namespace+0xe0/0x1fc
[<8093bc28>] kernel_init_freeable+0x268/0x298
[<80763b7c>] kernel_init+0x10/0x100
[<80006b28>] ret_from_kernel_thread+0x14/0x1c

UBIFS error (pid 1): read_pnode: calc num: 121
UBIFS: background thread "ubifs_bgt0_0" stops
List of all partitions:
1f00         2089984 mtdblock0  (driver?)
1f02            7168 mtdblock2  (driver?)
1f03             640 mtdblock3  (driver?)
1f04            1920 mtdblock4  (driver?)
1f05             512 mtdblock5  (driver?)
1f06             512 mtdblock6  (driver?)
1f07             256 mtdblock7  (driver?)
1f08             256 mtdblock8  (driver?)
1f09               8 mtdblock9  (driver?)
0800       976762584 sda  driver: sd
  0801       976760832 sda1 1b7805a8-7dbb-46c9-96e3-453b619bb54d
No filesystem could mount root, tried:  ubifs
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)
Rebooting in 180 seconds..

stevebow · Post by **stevebow** » Sun Dec 06, 2020 09:09

Just to mention, another flashmem bitflip occurred again, not 16 months this time but 3 months.

Same location as always: 0x2336fc00.

peteru · Post by **peteru** » Sun Dec 06, 2020 17:54

Sounds like faulty silicon.

stevebow · Post by **stevebow** » Mon Dec 07, 2020 12:43

Yep. I bought a spare a little while back, touch wood it won’t fail completely and I won't need it.

peteru · Post by **peteru** » Mon Dec 07, 2020 23:35

There may be a utility to map that bad block out. I've never had a reason to look into it in detail, but that's what UBI is for. I'm not sure if such a feature would be accessible from user space or if it may be a bootloader only thing.

stevebow · Post by **stevebow** » Thu Dec 10, 2020 15:46

That would be very handy!

Anything I could do to test if your get that far, let me know.