Please provide a short (approximately 100 word) summary of the following web Content, written in the voice of the original author. If there is anything controversial please highlight the controversy. If there is something surprising, unique, or clever, please highlight that as well. Content: Title: Avoid using Linux 6.3 if you have XFS filesystem Site: bugzilla.redhat.com (In reply to Rune Kleveland from comment #10 ) > Hi! > > All our "crawling servers" seems to crash consistently when using this > kernel, and it seems to always be the root partition crashing. The crawling > servers has high load and basically runs many instances of Google Chrome. > > Here is an attempt to provide information about a particular crash. This is > a server installed as Fedora 31 and upgraded to Fedora 38 step by step. It > has worked fine till now. > > After booting to kernel 6.3.3, the file system crashed almost immediately. > We did not have to start the processes. At other occasions it has taken > longer, up to a day, and on those the file system is created more recently. > It's just a hunch, we do not have that many observations. I believe we have > 8 crashes totally now of 12 servers running this kernel for some time. > > Here are links to what you hopefully requested for the particular crash: > > https://pascal.opoint.com/kernel6.3/fstab > https://pascal.opoint.com/kernel6.3/hardwareinfo > https://pascal.opoint.com/kernel6.3/kernellog > https://pascal.opoint.com/kernel6.3/metadump So this has failed at runtime because it tripped over an inode block full of garbage: May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): Metadata corruption detected at xfs_buf_ioend+0x154/0x5f0 [xfs], xfs_inode block 0x112ac740 xfs_inode_buf_verify May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): Unmount and run xfs_repair May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): First 128 bytes of corrupted metadata buffer: May 24 12:04:12 newton59.opoint.com kernel: 00000000: 31 45 20 24 41 41 46 46 32 44 39 46 38 44 43 43 1E $AAFF2D9F8DCC May 24 12:04:12 newton59.opoint.com kernel: 00000010: 46 41 30 32 33 31 33 33 36 36 41 38 36 41 46 46 FA02313366A86AFF May 24 12:04:12 newton59.opoint.com kernel: 00000020: 38 37 35 33 42 41 38 31 41 38 39 46 20 24 41 43 8753BA81A89F $AC May 24 12:04:12 newton59.opoint.com kernel: 00000030: 31 34 44 37 37 37 33 42 46 41 31 44 32 35 45 34 14D7773BFA1D25E4 May 24 12:04:12 newton59.opoint.com kernel: 00000040: 43 46 42 39 34 36 34 38 46 30 42 43 38 39 33 44 CFB94648F0BC893D May 24 12:04:12 newton59.opoint.com kernel: 00000050: 44 31 39 45 33 37 20 24 41 44 41 44 41 31 44 45 D19E37 $ADADA1DE May 24 12:04:12 newton59.opoint.com kernel: 00000060: 46 30 34 44 45 44 37 41 36 42 42 33 32 43 38 30 F04DED7A6BB32C80 May 24 12:04:12 newton59.opoint.com kernel: 00000070: 44 36 44 43 35 39 33 42 35 42 43 30 38 39 45 46 D6DC593B5BC089EF Note the address of this block: 0x112ac740 It is also tripping over a directory data block full of garbage: May 24 12:04:12 newton59.opoint.com kernel: XFS (sda3): metadata I/O error in "xfs_imap_to_bp+0x5c/0x80 [xfs]" at daddr 0x112ac740 len 32 error 117 May 24 12:06:11 newton59.opoint.com kernel: XFS (sda3): Metadata CRC error detected at xfs_dir3_block_read_verify+0xd4/0x110 [xfs], xfs_dir3_block block 0x112a9be0 May 24 12:06:11 newton59.opoint.com kernel: XFS (sda3): Unmount and run xfs_repair May 24 12:06:11 newton59.opoint.com kernel: XFS (sda3): First 128 bytes of corrupted metadata buffer: May 24 12:06:11 newton59.opoint.com kernel: 00000000: 35 44 33 43 31 39 30 35 42 43 20 24 46 45 41 46 5D3C1905BC $FEAF May 24 12:06:11 newton59.opoint.com kernel: 00000010: 30 41 42 30 39 30 38 35 33 31 35 43 44 33 39 35 0AB09085315CD395 May 24 12:06:12 newton59.opoint.com kernel: 00000020: 46 35 31 31 43 44 45 45 39 31 30 32 34 36 32 46 F511CDEE9102462F May 24 12:06:12 newton59.opoint.com kernel: 00000030: 35 38 39 35 0a 70 20 61 63 63 65 70 74 20 34 33 5895.p accept 43 May 24 12:06:12 newton59.opoint.com kernel: 00000040: 2c 35 33 2c 38 30 2c 31 31 30 2c 31 34 33 2c 31 ,53,80,110,143,1 May 24 12:06:12 newton59.opoint.com kernel: 00000050: 39 34 2c 32 32 30 2c 34 34 33 2c 38 37 33 2c 39 94,220,443,873,9 May 24 12:06:12 newton59.opoint.com kernel: 00000060: 39 31 2c 39 39 33 2d 39 39 35 2c 31 31 39 34 2c 91,993-995,1194, May 24 12:06:12 newton59.opoint.com kernel: 00000070: 31 32 39 33 2c 33 36 39 30 2c 34 33 32 31 2c 35 1293,3690,4321,5 Note the address of this block: 0x112a9be0 And when I try to replay the log from the metadata to mount the image, it's trips over a freespace btree block full of garbage at block 0x112a9b08 and goes no further. Lets put those three corruptions together: 0x112ac740, 0x112a9be0 and 0x112a9b08. They are all within a range of ~5.8MB of disk space. That's highly suspicious. So to do a quick scan of the state of all the metadata, I ran 'xfs_repair -n dump.img' to see what active metadata blocks it found that were corrupt: $ grep Metadata t.t Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9b08/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112ac150/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9bc0/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9b90/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_bnobt block 0x112a9b40/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112ac140/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112ab920/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112aa3c0/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112a9c80/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112ac158/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112a9828/0x1000 Metadata CRC error detected at 0x5625c07feecd, xfs_cntbt block 0x112aad30/0x1000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aa3a0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aa920/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aac00/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112aac20/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112abac0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ab980/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac1c0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac2c0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac2e0/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac740/0x4000 Metadata corruption detected at 0x5625c07eb846, xfs_inode block 0x112ac760/0x4000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9b20/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9d50/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9d58/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9d48/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112a9be0/0x1000 Metadata CRC error detected at 0x5625c08176c3, xfs_dir3_block block 0x112ab468/0x1000 Metadata CRC error detected at 0x5625c08177a3, xfs_dir3_data block 0x112a97d0/0x1000 Metadata CRC error detected at 0x5625c08177a3, xf