{"id":159,"date":"2006-08-02T13:21:21","date_gmt":"2006-08-02T13:21:21","guid":{"rendered":"http:\/\/www.starcoder.com\/wordpress\/2006\/08\/02\/another-one-bites-the-dust\/"},"modified":"2021-10-30T20:01:04","modified_gmt":"2021-10-30T20:01:04","slug":"another-one-bites-the-dust","status":"publish","type":"post","link":"https:\/\/www.starcoder.com\/wordpress\/2006\/08\/another-one-bites-the-dust\/","title":{"rendered":"Another one bites the dust&#8230;"},"content":{"rendered":"<p>I&#8217;m very thankful for RAID 5 at the moment&#8230;gotta love that parity thing.  I checked my server status emails this morning, only to find these lines in \/proc\/mdstat:<\/p>\n<pre>md1 : active raid5 sda4[0] sdc4[2] sdb4[3](F)\n      576283520 blocks level 5, 64k chunk, algorithm 2 [3\/2] [U_U]<\/pre>\n<p>Hmm, that little F doesn&#8217;t look too promising, and one of the U&#8217;s is missing.  So I look into this a bit further and find:<\/p>\n<pre>root@rio:\/rio# mdadm --detail \/dev\/md1\n\/dev\/md1:\n     Raid Level : raid5\n     Array Size : 576283520 (549.59 GiB 590.11 GB)\n    Device Size : 288141760 (274.79 GiB 295.06 GB)\n   Raid Devices : 3\n  Total Devices : 3\n Active Devices : 2\nWorking Devices : 2\n Failed Devices : 1<\/pre>\n<p>&#8230;<\/p>\n<pre>    Number   Major   Minor   RaidDevice State\n       0       8        4        0      active sync   \/dev\/sda4\n       1       0        0        -      removed\n       2       8       36        2      active sync   \/dev\/sdc4\n       3       8       20        -      faulty        \/dev\/sdb4<\/pre>\n<p>Sure enough, after checking \/var\/log\/messages, last night at around 8pm a disk failed&#8230;<\/p>\n<pre>kernel: ata2: status=0x25 { DeviceFault CorrectedError Error }\nkernel: SCSI error :  return code = 0x8000002\nkernel: sdb: Current: sense key: Hardware Error\nkernel:     Additional sense: No additional sense information\nkernel: end_request: I\/O error, dev sdb, sector 18912489\nkernel: RAID5 conf printout:\nkernel:  --- rd:3 wd:2 fd:1\nkernel:  disk 0, o:1, dev:sda4\nkernel:  disk 1, o:0, dev:sdb4\nkernel:  disk 2, o:1, dev:sdc4\nkernel: RAID5 conf printout:\nkernel:  --- rd:3 wd:2 fd:1\nkernel:  disk 0, o:1, dev:sda4\nkernel:  disk 2, o:1, dev:sdc4<\/pre>\n<p>I&#8217;m a bit surprised because the drives I used for this RAID are manufactured by Seagate, which I&#8217;ve had luck with in the past.  Fortunately, Seagate offers a 5 year warranty for all of it&#8217;s drives, so this one is going back to the manufacturer to be replaced.  In the mean time, I ordered another disk with overnight shipping&#8211;I need to take care of this before leaving for WWDC on Saturday. \ud83d\ude42<\/p>\n<p><i>Update (8\/4):<\/i> The replacement disk arrived yesterday afternoon and I was able to re-add partitions to the RAID volumes using <code>mdadm &lt;raid volume device&gt; --add &lt;disk device&gt;<\/code>. Rebuilding went pretty quick&#8211;\/usr finished rebuilding in less than a minute and the larger volume took just over an hour and a half:<\/p>\n<pre>Personalities : [raid5]\nmd1 : active raid5 sdb4[3] sda4[0] sdc4[2]\n      576283520 blocks level 5, 64k chunk, algorithm 2 [3\/2] [U_U]\n      [&gt;....................]  recovery =  2.7% (8006784\/288141760)\n      finish=98.3min speed=47479K\/sec\nmd0 : active raid5 sdb1[1] sda1[0] sdc1[2]\n      5863552 blocks level 5, 64k chunk, algorithm 2 [3\/3] [UUU]<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m very thankful for RAID 5 at the moment&#8230;gotta love that parity thing. I checked my server status emails this morning, only to find these lines in \/proc\/mdstat: md1 : active raid5 sda4[0] sdc4[2] sdb4[3](F) 576283520 blocks level 5, 64k chunk, algorithm 2 [3\/2] [U_U] Hmm, that little F doesn&#8217;t look too promising, and one [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-159","post","type-post","status-publish","format-standard","hentry","category-sys-admin","post-preview"],"_links":{"self":[{"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/posts\/159","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/comments?post=159"}],"version-history":[{"count":1,"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/posts\/159\/revisions"}],"predecessor-version":[{"id":633,"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/posts\/159\/revisions\/633"}],"wp:attachment":[{"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/media?parent=159"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/categories?post=159"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.starcoder.com\/wordpress\/wp-json\/wp\/v2\/tags?post=159"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}