Monday, February 9, 2015

smart - Cause of Current Pending Sector and Offline Uncorrectable increase and then decrease to zero




Over the course of a week, for a 3TB Seagate drive (ST3000DM001-1CH166), smartd reported a slowly increasing number of offline uncorrectable and currently unreadable (pending) sectors, then a decreasing number, until finally the count was 0 and the error condition reset. From the logs (showing only changes):



Jul  6 18:04:57 x smartd[462]: Device: /dev/sdb [SAT], 8 Currently unreadable (pending) sectors
Jul 6 18:04:58 x smartd[462]: Device: /dev/sdb [SAT], 8 Offline uncorrectable sectors
[...]
Jul 7 16:34:58 x smartd[462]: Device: /dev/sdb [SAT], 16 Currently unreadable (pending) sectors (changed +8)
Jul 7 16:34:58 x smartd[462]: Device: /dev/sdb [SAT], 16 Offline uncorrectable sectors (changed +8)
[...]
Jul 11 14:04:57 x smartd[462]: Device: /dev/sdb [SAT], 24 Currently unreadable (pending) sectors (changed +8)

Jul 11 14:04:57 x smartd[462]: Device: /dev/sdb [SAT], 24 Offline uncorrectable sectors (changed +8)
Jul 11 14:34:57 x smartd[462]: Device: /dev/sdb [SAT], 32 Currently unreadable (pending) sectors (changed +8)
Jul 11 14:34:58 x smartd[462]: Device: /dev/sdb [SAT], 32 Offline uncorrectable sectors (changed +8)
[...]
Jul 13 09:04:57 x smartd[462]: Device: /dev/sdb [SAT], 24 Currently unreadable (pending) sectors (changed -8)
Jul 13 09:04:57 x smartd[462]: Device: /dev/sdb [SAT], 24 Offline uncorrectable sectors (changed -8)
Jul 13 09:34:58 x smartd[462]: Device: /dev/sdb [SAT], 16 Currently unreadable (pending) sectors (changed -8)
Jul 13 09:34:58 x smartd[462]: Device: /dev/sdb [SAT], 16 Offline uncorrectable sectors (changed -8)
Jul 13 10:04:57 x smartd[462]: Device: /dev/sdb [SAT], 16 Currently unreadable (pending) sectors
Jul 13 10:04:57 x smartd[462]: Device: /dev/sdb [SAT], 16 Offline uncorrectable sectors

Jul 13 10:34:57 x smartd[462]: Device: /dev/sdb [SAT], No more Currently unreadable (pending) sectors, warning condition reset after 1 email
Jul 13 10:34:57 x smartd[462]: Device: /dev/sdb [SAT], No more Offline uncorrectable sectors, warning condition reset after 1 email


Furthermore, the reallocated sector count is also 0, so the sectors do not appear to have been re-mapped. Here is the complete (current) smartctl -a output for the drive:



smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.14.4-100.fc19.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1CH166
Serial Number: W1F30FK2
LU WWN Device Id: 5 000c50 06129a9a8
Firmware Version: CC27
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b

SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Jul 16 11:23:08 2014 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever

been run.
Total time to complete Offline
data collection: ( 584) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.

Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine

recommended polling time: ( 347) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 115 099 006 Pre-fail Always - 91131424
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0

4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 12
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 078 060 030 Pre-fail Always - 59138260
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5888
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 12
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0

189 High_Fly_Writes 0x003a 097 097 000 Old_age Always - 3
190 Airflow_Temperature_Cel 0x0022 053 049 045 Old_age Always - 47 (Min/Max 23/51)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 10
193 Load_Cycle_Count 0x0032 096 096 000 Old_age Always - 8086
194 Temperature_Celsius 0x0022 047 051 000 Old_age Always - 47 (0 22 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5703h+56m+25.808s

241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 11838196191
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 211237637103

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]



SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



I have downloaded but not yet run Seatools on the drive, but the current SMART status of the drive basically seems fine. What could cause such behavior?



UPDATE: For future readers, the drive was actually fine for a couple more months, at which point more sectors become uncorrectable/offline, and the number of reallocated sectors started increasing beyond 0, and the SMART long self-test started failing with a read error. So these messages appeared to be a useful early warning.


Answer



I've had this a few times recently with a similar Seagate ST3000DM001-1CH166 firmware CC24.
You don't need Seatools, just run a long smart test:



smartctl -t long /dev/sdc



Then if smarctl shows no error you're OK for now:



SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 9912 -


If it fails then send straight back to Seagate.

They replaced my last failed one in about a week.


No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...