Почему самопроизвольно зависает сервер на CentOS 6.5?

Question

[Z0nd0R]

Доброго времени суток!

Стоит сервер в Хетзнере. Конфигурация EX4.
Недавно начал самопроизвольно зависать раз в сутки с различным интервалом.
Найти закономерность в логах перед падением не получилось. (может навыков не хватает)

Единственное что было сделано, это примерно месяц назад обновлял ПО и с ним обновилось ядро.
Зависает насмерть, так что запросы на автоматическую перезагрузку не дают ни какого результата.

Самое интересное, что нагрузка на сервер практически нулевая, ничего сверхъестественного на нем не стоит.
PHP, Apache, Nginx, Clamav, ISPMgr, Ruby, MySQL.

Первый раз все началось 16 числа.

reboot   system boot  2.6.32-431.5.1.e Wed Feb 19 13:55 - 14:01  (00:06)
reboot   system boot  2.6.32-431.5.1.e Tue Feb 18 08:23 - 14:01 (1+05:38)
reboot   system boot  2.6.32-431.5.1.e Mon Feb 17 08:23 - 14:01 (2+05:38)
reboot   system boot  2.6.32-431.el6.x Sun Feb 16 01:15 - 14:01 (3+12:46)

Есть идеи в чем может проблема?

Comments

[Z0nd0R]

дамп dmesg

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-431.5.1.el6.x86_64 ([e-mail]) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Wed Feb 12 00:41:43 UTC 2014
Command line: ro root=/dev/md2 rd_NO_LUKS rd_NO_DM nomodeset crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009d800 (usable)
 BIOS-e820: 000000000009d800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 0000000020000000 (usable)
 BIOS-e820: 0000000020000000 - 0000000020200000 (reserved)
 BIOS-e820: 0000000020200000 - 0000000040004000 (usable)
 BIOS-e820: 0000000040004000 - 0000000040005000 (reserved)
 BIOS-e820: 0000000040005000 - 00000000cc608000 (usable)
 BIOS-e820: 00000000cc608000 - 00000000cca87000 (reserved)
 BIOS-e820: 00000000cca87000 - 00000000cca98000 (ACPI data)
 BIOS-e820: 00000000cca98000 - 00000000ccbc0000 (ACPI NVS)
 BIOS-e820: 00000000ccbc0000 - 00000000cd7f4000 (reserved)
 BIOS-e820: 00000000cd7f4000 - 00000000cd7f5000 (usable)
 BIOS-e820: 00000000cd7f5000 - 00000000cd838000 (ACPI NVS)
 BIOS-e820: 00000000cd838000 - 00000000cdc5e000 (usable)
 BIOS-e820: 00000000cdc5e000 - 00000000cdff4000 (reserved)
 BIOS-e820: 00000000cdff4000 - 00000000ce000000 (usable)
 BIOS-e820: 00000000cf000000 - 00000000df200000 (reserved)
 BIOS-e820: 00000000f8000000 - 00000000fc000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fed00000 - 00000000fed04000 (reserved)
 BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 000000041fe00000 (usable)
DMI 2.7 present.
SMBIOS version 2.7 @ 0xF04C0
DMI: System manufacturer System Product Name/P8H77-M PRO, BIOS 1008 01/24/2013
AMI BIOS detected: BIOS may corrupt low RAM, working around it.
e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x41fe00 max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR fixed ranges enabled:
  00000-9FFFF write-back
  A0000-BFFFF uncachable
  C0000-CFFFF write-protect
  D0000-E7FFF uncachable
  E8000-FFFFF write-protect
MTRR variable ranges enabled:
  0 base 000000000 mask C00000000 write-back
  1 base 400000000 mask FE0000000 write-back
  2 base 0E0000000 mask FE0000000 uncachable
  3 base 0D0000000 mask FF0000000 uncachable
  4 base 0CF000000 mask FFF000000 uncachable
  5 base 41FE00000 mask FFFE00000 uncachable
  6 disabled
  7 disabled
  8 disabled
  9 disabled
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
original variable MTRRs
reg 0, base: 0GB, range: 16GB, type WB
reg 1, base: 16GB, range: 512MB, type WB
reg 2, base: 3584MB, range: 512MB, type UC
reg 3, base: 3328MB, range: 256MB, type UC
reg 4, base: 3312MB, range: 16MB, type UC
reg 5, base: 16894MB, range: 2MB, type UC
total RAM covered: 16110M
Found optimal setting for mtrr clean up
 gran_size: 64K 	chunk_size: 32M 	num_reg: 8  	lose cover RAM: 0G
New variable MTRRs
reg 0, base: 0GB, range: 2GB, type WB
reg 1, base: 2GB, range: 1GB, type WB
reg 2, base: 3GB, range: 256MB, type WB
reg 3, base: 3312MB, range: 16MB, type UC
reg 4, base: 4GB, range: 4GB, type WB
reg 5, base: 8GB, range: 8GB, type WB
reg 6, base: 16GB, range: 512MB, type WB
reg 7, base: 16894MB, range: 2MB, type UC
e820 update range: 00000000cf000000 - 0000000100000000 (usable) ==> (reserved)
last_pfn = 0xce000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-00000000ce000000
 0000000000 - 00ce000000 page 2M
kernel direct mapping tables up to ce000000 @ 10000-15000
Use unified mapping for non-reserved e820 regions.
init_memory_mapping: 0000000100000000-000000041fe00000
 0100000000 - 041fe00000 page 2M
kernel direct mapping tables up to 41fe00000 @ 13000-21000
RAMDISK: 1f1ce000 - 1ffef11a
ACPI: RSDP 00000000000f0490 00024 (v02 ALASKA)
ACPI: XSDT 00000000cca8a078 0006C (v01 ALASKA    A M I 01072009 AMI  00010013)
ACPI: FACP 00000000cca95560 0010C (v05 ALASKA    A M I 01072009 AMI  00010013)
ACPI Warning: FADT (revision 5) is longer than ACPI 2.0 version, truncating length 0x10C to 0xF4 (20090903/tbfadt-288)
ACPI: DSDT 00000000cca8a180 0B3D9 (v02 ALASKA    A M I 00000022 INTL 20051117)
ACPI: FACS 00000000ccbbe080 00040
ACPI: APIC 00000000cca95670 00092 (v03 ALASKA    A M I 01072009 AMI  00010013)
ACPI: FPDT 00000000cca95708 00044 (v01 ALASKA    A M I 01072009 AMI  00010013)
ACPI: MCFG 00000000cca95750 0003C (v01 ALASKA    A M I 01072009 MSFT 00000097)
ACPI: HPET 00000000cca95790 00038 (v01 ALASKA    A M I 01072009 AMI. 00000005)
ACPI: SSDT 00000000cca957c8 0036D (v01 SataRe SataTabl 00001000 INTL 20091112)
ACPI: DMAR 00000000cca96fd8 000B0 (v01 INTEL      SNB  00000001 INTL 00000001)
ACPI: SSDT 00000000cca95b90 009AA (v01  PmRef  Cpu0Ist 00003000 INTL 20051117)
ACPI: SSDT 00000000cca96540 00A92 (v01  PmRef    CpuPm 00003000 INTL 20051117)
ACPI: Local APIC address 0xfee00000
Setting APIC routing to flat.
No NUMA configuration found
Faking a node at 0000000000000000-000000041fe00000
Bootmem setup node 0 0000000000000000-000000041fe00000
  NODE_DATA [0000000000020000 - 0000000000053fff]
  bootmap [0000000000100000 -  0000000000183fbf] pages 84
(8 early reservations) ==> bootmem [0000000000 - 041fe00000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0001000000 - 000201faa4]    TEXT DATA BSS ==> [0001000000 - 000201faa4]
  #3 [001f1ce000 - 001ffef11a]          RAMDISK ==> [001f1ce000 - 001ffef11a]
  #4 [000009d800 - 0000100000]    BIOS reserved ==> [000009d800 - 0000100000]
  #5 [0002020000 - 000202040e]              BRK ==> [0002020000 - 000202040e]
  #6 [0000010000 - 0000013000]          PGTABLE ==> [0000010000 - 0000013000]
  #7 [0000013000 - 0000020000]          PGTABLE ==> [0000013000 - 0000020000]
found SMP MP-table at [ffff8800000fd760] fd760
Reserving 130MB of memory at 48MB for crashkernel (System RAM: 16894MB)
 [ffffea0000000000-ffffea000e7fffff] PMD -> [ffff88002c600000-ffff88003a3fffff] on node 0
Zone PFN ranges:
  DMA      0x00000010 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x0041fe00
Movable zone start PFN for each node
early_node_map[8] active PFN ranges
    0: 0x00000010 -> 0x0000009d
    0: 0x00000100 -> 0x00020000
    0: 0x00020200 -> 0x00040004
    0: 0x00040005 -> 0x000cc608
    0: 0x000cd7f4 -> 0x000cd7f5
    0: 0x000cd838 -> 0x000cdc5e
    0: 0x000cdff4 -> 0x000ce000
    0: 0x00100000 -> 0x0041fe00
On node 0 totalpages: 4113863
  DMA zone: 56 pages used for memmap
  DMA zone: 117 pages reserved
  DMA zone: 3808 pages, LIFO batch:0
  DMA32 zone: 14280 pages used for memmap
  DMA32 zone: 819314 pages, LIFO batch:31
  Normal zone: 44793 pages used for memmap
  Normal zone: 3231495 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x408
ACPI: Local APIC address 0xfee00000
Setting APIC routing to flat.
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a701 base: 0xfed00000
SMP: Allowing 8 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 24
PM: Registered nosave memory: 000000000009d000 - 000000000009e000
PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
PM: Registered nosave memory: 0000000020000000 - 0000000020200000
PM: Registered nosave memory: 0000000040004000 - 0000000040005000
PM: Registered nosave memory: 00000000cc608000 - 00000000cca87000
PM: Registered nosave memory: 00000000cca87000 - 00000000cca98000
PM: Registered nosave memory: 00000000cca98000 - 00000000ccbc0000
PM: Registered nosave memory: 00000000ccbc0000 - 00000000cd7f4000
PM: Registered nosave memory: 00000000cd7f5000 - 00000000cd838000
PM: Registered nosave memory: 00000000cdc5e000 - 00000000cdff4000
PM: Registered nosave memory: 00000000ce000000 - 00000000cf000000
PM: Registered nosave memory: 00000000cf000000 - 00000000df200000
PM: Registered nosave memory: 00000000df200000 - 00000000f8000000
PM: Registered nosave memory: 00000000f8000000 - 00000000fc000000
PM: Registered nosave memory: 00000000fc000000 - 00000000fec00000
PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
PM: Registered nosave memory: 00000000fec01000 - 00000000fed00000
PM: Registered nosave memory: 00000000fed00000 - 00000000fed04000
PM: Registered nosave memory: 00000000fed04000 - 00000000fed1c000
PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000
PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000
PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
PM: Registered nosave memory: 00000000fee01000 - 00000000ff000000
PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
Allocating PCI resources starting at df200000 (gap: df200000:18e00000)
Booting paravirtualized kernel on bare hardware
NR_CPUS:4096 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff88002c200000 s94872 r8192 d23912 u262144
pcpu-alloc: s94872 r8192 d23912 u262144 alloc=1*2097152
pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 4054617
Policy zone: Normal
Kernel command line: ro root=/dev/md2 rd_NO_LUKS rd_NO_DM nomodeset crashkernel=130M@0M SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de
PID hash table entries: 4096 (order: 3, 32768 bytes)
xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Checking aperture...
No AGP bridge found
dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880024000000 - ffff880028000000
software IO TLB at phys 0x24000000 - 0x28000000
Memory: 15996412k/17299456k available (5326k kernel code, 844004k absent, 459040k reserved, 7013k data, 1276k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:472
Extended CMOS year: 2000
Console: colour VGA+ 80x25
console [tty0] enabled
allocated 66060288 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
hpet clockevent registered
Fast TSC calibration using PIT
Detected 3399.762 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 6799.52 BogoMIPS (lpj=3399762)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 9 MCE banks
CPU0: Thermal monitoring enabled (TM1)
using mwait in idle threads.
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 21773 entries in 86 pages
dmar: Host address width 36
dmar: DRHD base: 0x000000fed90000 flags: 0x0
dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
dmar: DRHD base: 0x000000fed91000 flags: 0x1
dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f010da
dmar: RMRR base: 0x000000cca09000 end: 0x000000cca1afff
dmar: RMRR base: 0x000000cf000000 end: 0x000000df1fffff
dmar: No ATSR found
IOAPIC id 2 under DRHD base 0xfed91000
HPET id 0 under DRHD base 0xfed91000
Enabled IRQ remapping in x2apic mode
Enabling x2apic
Enabled x2apic
APIC routing finalized to cluster x2apic.
  alloc irq_desc for 24 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 25 on node 0
  alloc kstat_irqs on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
alloc irq_2_iommu on node 0
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz stepping 09
Performance Events: PEBS fmt1+, 16-deep LBR, IvyBridge events, Intel PMU driver.
... version:                3
... bit width:              48
... generic registers:      4
... value mask:             0000ffffffffffff
... max period:             000000007fffffff
... fixed-purpose events:   3
... event mask:             000000070000000f
NMI watchdog enabled, takes one hw-pmu counter.
Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 Ok.
Brought up 8 CPUs
Total of 8 processors activated (54396.19 BogoMIPS).
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2648 bytes
devtmpfs: initialized
PM: Registering ACPI NVS region at cca98000 (1212416 bytes)
PM: Registering ACPI NVS region at cd7f5000 (274432 bytes)
regulator: core version 0.5
NET: Registered protocol family 16
ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
ACPI: bus type pci registered
PCI: MCFG configuration 0: base f8000000 segment 0 buses 0 - 63
PCI: MCFG area at f8000000 reserved in E820
PCI: Using MMCONFIG at f8000000 - fbffffff
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: EC: Look up EC in DSDT
ACPI: Executed 1 blocks of module-level executable AML code
ACPI Error (psargs-0359): [RAMB] Namespace lookup failure, AE_NOT_FOUND
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20090903/nsinit-347)
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: Power Resource [FN00] (off)
ACPI: Power Resource [FN01] (off)
ACPI: Power Resource [FN02] (off)
ACPI: Power Resource [FN03] (off)
ACPI: Power Resource [FN04] (off)
ACPI: No dock devices found.
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e])
pci_root PNP0A08:00: host bridge window [io  0x0000-0x0cf7]
pci_root PNP0A08:00: host bridge window [io  0x0d00-0xffff]
pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff]
pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff]
pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff]
pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff]
pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff]
pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff]
pci_root PNP0A08:00: host bridge window [mem 0xdf200000-0xfeafffff]
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff]
pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff]
pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff]
pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff]
pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff]
pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff]
pci_bus 0000:00: root bus resource [mem 0xdf200000-0xfeafffff]
pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
pci 0000:00:01.0: PME# disabled
pci 0000:00:02.0: reg 10: [mem 0xf7800000-0xf7bfffff 64bit]
pci 0000:00:02.0: reg 18: [mem 0xe0000000-0xefffffff 64bit pref]
pci 0000:00:02.0: reg 20: [io  0xf000-0xf03f]
pci 0000:00:16.0: reg 10: [mem 0xf7c06000-0xf7c0600f 64bit]
pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
pci 0000:00:16.0: PME# disabled
pci 0000:00:1a.0: reg 10: [mem 0xf7c04000-0xf7c043ff]
pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1a.0: PME# disabled
pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: PME# disabled
pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.4: PME# disabled
pci 0000:00:1d.0: reg 10: [mem 0xf7c03000-0xf7c033ff]
pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.0: PME# disabled
pci 0000:00:1f.2: reg 10: [io  0xf0b0-0xf0b7]
pci 0000:00:1f.2: reg 14: [io  0xf0a0-0xf0a3]
pci 0000:00:1f.2: reg 18: [io  0xf090-0xf097]
pci 0000:00:1f.2: reg 1c: [io  0xf080-0xf083]
pci 0000:00:1f.2: reg 20: [io  0xf060-0xf07f]
pci 0000:00:1f.2: reg 24: [mem 0xf7c02000-0xf7c027ff]
pci 0000:00:1f.2: PME# supported from D3hot
pci 0000:00:1f.2: PME# disabled
pci 0000:00:1f.3: reg 10: [mem 0xf7c01000-0xf7c010ff 64bit]
pci 0000:00:1f.3: reg 20: [io  0xf040-0xf05f]
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:00:01.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:00:01.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:00:1c.0: PCI bridge to [bus 02-02]
pci 0000:00:1c.0:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:00:1c.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:00:1c.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:03:00.0: reg 10: [io  0xe000-0xe0ff]
pci 0000:03:00.0: reg 18: [mem 0xf0004000-0xf0004fff 64bit pref]
pci 0000:03:00.0: reg 20: [mem 0xf0000000-0xf0003fff 64bit pref]
pci 0000:03:00.0: supports D1 D2
pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:03:00.0: PME# disabled
pci 0000:00:1c.4: PCI bridge to [bus 03-03]
pci 0000:00:1c.4:   bridge window [io  0xe000-0xefff]
pci 0000:00:1c.4:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:00:1c.4:   bridge window [mem 0xf0000000-0xf00fffff 64bit pref]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.RP01._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEG0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.RP05._PRT]
 pci0000:00: Requesting ACPI _OSC control (0x1d)
 pci0000:00: ACPI _OSC control (0x18) granted
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 *4 5 6 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs *3 4 5 6 10 11 12 14 15)
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:00:02.0
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
PCI: pci_cache_line_size set to 64 bytes
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
  alloc irq_desc for 26 on node 0
  alloc kstat_irqs on node 0
alloc irq_2_iommu on node 0
  alloc irq_desc for 27 on node 0
  alloc kstat_irqs on node 0
alloc irq_2_iommu on node 0
  alloc irq_desc for 28 on node 0
  alloc kstat_irqs on node 0
alloc irq_2_iommu on node 0
  alloc irq_desc for 29 on node 0
  alloc kstat_irqs on node 0
alloc irq_2_iommu on node 0
  alloc irq_desc for 30 on node 0
  alloc kstat_irqs on node 0
alloc irq_2_iommu on node 0
HPET: 8 timers in total, 5 timers will be used for per-cpu timer
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 26, 27, 28, 29, 30, 0
hpet0: 8 comparators, 64-bit 14.318180 MHz counter
hpet: hpet2 irq 26 for MSI
hpet: hpet3 irq 27 for MSI
hpet: hpet4 irq 28 for MSI
hpet: hpet5 irq 29 for MSI
hpet: hpet6 irq 30 for MSI
Switching to clocksource hpet
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:00: [io  0x0cf8-0x0cff]
pnp 00:00: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
pnp 00:01: [mem 0xfed40000-0xfed44fff]
pnp 00:01: Plug and Play ACPI device, IDs PNP0c01 (active)
pnp 00:02: [io  0x0000-0x001f]
pnp 00:02: [io  0x0081-0x0091]
pnp 00:02: [io  0x0093-0x009f]
pnp 00:02: [io  0x00c0-0x00df]
pnp 00:02: [dma 4]
pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
pnp 00:03: [mem 0xff000000-0xffffffff]
pnp 00:03: Plug and Play ACPI device, IDs INT0800 (active)
pnp 00:04: [mem 0xfed00000-0xfed003ff]
pnp 00:04: Plug and Play ACPI device, IDs PNP0103 (active)
pnp 00:05: [io  0x002e-0x002f]
pnp 00:05: [io  0x004e-0x004f]
pnp 00:05: [io  0x0061]
pnp 00:05: [io  0x0063]
pnp 00:05: [io  0x0065]
pnp 00:05: [io  0x0067]
pnp 00:05: [io  0x0070]
pnp 00:05: [io  0x0080]
pnp 00:05: [io  0x0092]
pnp 00:05: [io  0x00b2-0x00b3]
pnp 00:05: [io  0x0680-0x069f]
pnp 00:05: [io  0x1000-0x100f]
pnp 00:05: [io  0xffff]
pnp 00:05: [io  0xffff]
pnp 00:05: [io  0x0400-0x0453]
pnp 00:05: [io  0x0458-0x047f]
pnp 00:05: [io  0x0500-0x057f]
pnp 00:05: [io  0x164e-0x164f]
pnp 00:05: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:06: [io  0x0070-0x0077]
pnp 00:06: [irq 8]
pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active)
pnp 00:07: [io  0x0454-0x0457]
pnp 00:07: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
pnp 00:08: [io  0x0000-0xffffffffffffffff disabled]
pnp 00:08: [io  0x0000-0xffffffffffffffff disabled]
pnp 00:08: [io  0x0000-0xffffffffffffffff disabled]
pnp 00:08: [io  0x0290-0x029f]
pnp 00:08: [io  0x0000-0xffffffffffffffff disabled]
pnp 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:09: [io  0x0010-0x001f]
pnp 00:09: [io  0x0022-0x003f]
pnp 00:09: [io  0x0044-0x005f]
pnp 00:09: [io  0x0062-0x0063]
pnp 00:09: [io  0x0065-0x006f]
pnp 00:09: [io  0x0072-0x007f]
pnp 00:09: [io  0x0080]
pnp 00:09: [io  0x0084-0x0086]
pnp 00:09: [io  0x0088]
pnp 00:09: [io  0x008c-0x008e]
pnp 00:09: [io  0x0090-0x009f]
pnp 00:09: [io  0x00a2-0x00bf]
pnp 00:09: [io  0x00e0-0x00ef]
pnp 00:09: [io  0x04d0-0x04d1]
pnp 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:0a: [io  0x00f0-0x00ff]
pnp 00:0a: [irq 13]
pnp 00:0a: Plug and Play ACPI device, IDs PNP0c04 (active)
pnp 00:0b: [io  0x0060]
pnp 00:0b: [io  0x0064]
pnp 00:0b: [irq 1]
pnp 00:0b: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
pnp 00:0c: [mem 0xfed1c000-0xfed1ffff]
pnp 00:0c: [mem 0xfed10000-0xfed17fff]
pnp 00:0c: [mem 0xfed18000-0xfed18fff]
pnp 00:0c: [mem 0xfed19000-0xfed19fff]
pnp 00:0c: [mem 0xf8000000-0xfbffffff]
pnp 00:0c: [mem 0xfed20000-0xfed3ffff]
pnp 00:0c: [mem 0xfed90000-0xfed93fff]
pnp 00:0c: [mem 0xfed45000-0xfed8ffff]
pnp 00:0c: [mem 0xff000000-0xffffffff]
pnp 00:0c: [mem 0xfee00000-0xfeefffff]
pnp 00:0c: [mem 0xdf200000-0xdf200fff]
pnp 00:0c: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:0d: [mem 0x20000000-0x201fffff]
pnp 00:0d: [mem 0x40004000-0x40004fff]
pnp 00:0d: Plug and Play ACPI device, IDs PNP0c01 (active)
pnp: PnP ACPI: found 14 devices
ACPI: ACPI bus type pnp unregistered
system 00:01: [mem 0xfed40000-0xfed44fff] has been reserved
system 00:05: [io  0x0680-0x069f] has been reserved
system 00:05: [io  0x1000-0x100f] has been reserved
system 00:05: [io  0xffff] has been reserved
system 00:05: [io  0xffff] has been reserved
system 00:05: [io  0x0400-0x0453] has been reserved
system 00:05: [io  0x0458-0x047f] has been reserved
system 00:05: [io  0x0500-0x057f] has been reserved
system 00:05: [io  0x164e-0x164f] has been reserved
system 00:07: [io  0x0454-0x0457] has been reserved
system 00:08: [io  0x0290-0x029f] has been reserved
system 00:09: [io  0x04d0-0x04d1] has been reserved
system 00:0c: [mem 0xfed1c000-0xfed1ffff] has been reserved
system 00:0c: [mem 0xfed10000-0xfed17fff] has been reserved
system 00:0c: [mem 0xfed18000-0xfed18fff] has been reserved
system 00:0c: [mem 0xfed19000-0xfed19fff] has been reserved
system 00:0c: [mem 0xf8000000-0xfbffffff] has been reserved
system 00:0c: [mem 0xfed20000-0xfed3ffff] has been reserved
system 00:0c: [mem 0xfed90000-0xfed93fff] could not be reserved
system 00:0c: [mem 0xfed45000-0xfed8ffff] has been reserved
system 00:0c: [mem 0xff000000-0xffffffff] has been reserved
system 00:0c: [mem 0xfee00000-0xfeefffff] could not be reserved
system 00:0c: [mem 0xdf200000-0xdf200fff] has been reserved
system 00:0d: [mem 0x20000000-0x201fffff] has been reserved
system 00:0d: [mem 0x40004000-0x40004fff] has been reserved
PCI: max bus depth: 1 pci_try_num: 2
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0:   bridge window [io  disabled]
pci 0000:00:01.0:   bridge window [mem disabled]
pci 0000:00:01.0:   bridge window [mem pref disabled]
pci 0000:00:1c.0: PCI bridge to [bus 02-02]
pci 0000:00:1c.0: PCI bridge to [bus 02-02]
pci 0000:00:1c.0:   bridge window [io  disabled]
pci 0000:00:1c.0:   bridge window [mem disabled]
pci 0000:00:1c.0:   bridge window [mem pref disabled]
pci 0000:00:1c.4: PCI bridge to [bus 03-03]
pci 0000:00:1c.4: PCI bridge to [bus 03-03]
pci 0000:00:1c.4:   bridge window [io  0xe000-0xefff]
pci 0000:00:1c.4:   bridge window [mem disabled]
pci 0000:00:1c.4:   bridge window [mem 0xf0000000-0xf00fffff 64bit pref]
  alloc irq_desc for 16 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
pci 0000:00:01.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:01.0: setting latency timer to 64
pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:1c.0: setting latency timer to 64
pci 0000:00:1c.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:1c.4: setting latency timer to 64
pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000d3fff]
pci_bus 0000:00: resource 8 [mem 0x000d4000-0x000d7fff]
pci_bus 0000:00: resource 9 [mem 0x000d8000-0x000dbfff]
pci_bus 0000:00: resource 10 [mem 0x000dc000-0x000dffff]
pci_bus 0000:00: resource 11 [mem 0x000e0000-0x000e3fff]
pci_bus 0000:00: resource 12 [mem 0x000e4000-0x000e7fff]
pci_bus 0000:00: resource 13 [mem 0xdf200000-0xfeafffff]
pci_bus 0000:03: resource 0 [io  0xe000-0xefff]
pci_bus 0000:03: resource 2 [mem 0xf0000000-0xf00fffff 64bit pref]
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:02.0: Boot video device
  alloc irq_desc for 23 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
pci 0000:00:1a.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
pci 0000:00:1a.0: PCI INT A disabled
pci 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
pci 0000:00:1d.0: PCI INT A disabled
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 14468k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1392814497.320:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 31271
SELinux:  Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 46712A08B7477BD
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pcieport 0000:00:01.0: setting latency timer to 64
  alloc irq_desc for 31 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
pcieport 0000:00:01.0: irq 31 for MSI/MSI-X
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
intel_idle: MWAIT substates: 0x1120
intel_idle: v0.4 model 0x3A
intel_idle: lapic_timer_reliable_states 0xffffffff
ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding default-specified kcs state machine
ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
ipmi_si: Interface detection failed
ipmi_si: Adding default-specified smic state machine
ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0
ipmi_si: Interface detection failed
ipmi_si: Adding default-specified bt state machine
ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
ipmi_si: Interface detection failed
ipmi_si: Unable to find any System Interface(s)
input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
ACPI: Power Button [PWRB]
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
ACPI: Power Button [PWRF]
ACPI: Fan [FAN0] (off)
ACPI: Fan [FAN1] (off)
ACPI: Fan [FAN2] (off)
ACPI: Fan [FAN3] (off)
ACPI: Fan [FAN4] (off)
ACPI: acpi_idle yielding to intel_idleACPI: SSDT 00000000cca34018 0083B (v01  PmRef  Cpu0Cst 00003001 INTL 20051117)
ACPI: SSDT 00000000cca35a98 00303 (v01  PmRef    ApIst 00003000 INTL 20051117)
ACPI: SSDT 00000000cca36c18 00119 (v01  PmRef    ApCst 00003000 INTL 20051117)
ACPI Exception: AE_NOT_FOUND, No or invalid critical threshold (20090903/thermal-386)
thermal LNXTHERM:01: registered as thermal_zone0
ACPI: Thermal Zone [TZ00] (28 C)
thermal LNXTHERM:02: registered as thermal_zone1
ACPI: Thermal Zone [TZ01] (30 C)
GHES: HEST is not enabled!
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input2
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1a.0: setting latency timer to 64
ehci_hcd 0000:00:1a.0: EHCI Host Controller
ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.0: debug port 2
ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1a.0: irq 23, io mem 0xf7c04000
ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-431.5.1.el6.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.0: setting latency timer to 64
ehci_hcd 0000:00:1d.0: EHCI Host Controller
ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.0: debug port 2
ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1d.0: irq 23, io mem 0xf7c03000
ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-431.5.1.el6.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
rtc_cmos 00:06: RTC can wake from S4
rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
GRE over IPv4 demultiplexor driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:06: setting system clock to 2014-02-19 12:54:58 UTC (1392814498)
Initalizing network drop monitor service
Freeing unused kernel memory: 1276k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 800k freed
Freeing unused kernel memory: 1584k freed
dracut: dracut-004-336.el6_5.2
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.24.6-ioctl (2013-01-15) initialised: [e-mail]
udev: starting version 147
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
[drm] Initialized drm 1.1.0 20060810
pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[drm:drm_pci_agp_init] *ERROR* Cannot initialize the agpgart module.
DRM: Fill_in_dev failed.
pci 0000:00:02.0: PCI INT A disabled
dracut: Starting plymouth daemon
ahci 0000:00:1f.2: version 3.0
  alloc irq_desc for 19 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ahci 0000:00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
  alloc irq_desc for 32 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ahci 0000:00:1f.2: irq 32 for MSI/MSI-X
ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3 impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq led clo pio slum part ems apst 
ahci 0000:00:1f.2: setting latency timer to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m2048@0xf7c02000 port 0xf7c02100 irq 32
ata2: SATA max UDMA/133 irq_stat 0x00400040, connection status changed irq 32
ata3: DUMMY
ata4: DUMMY
ata5: DUMMY
ata6: DUMMY
usb 1-1: new high speed USB device number 2 using ehci_hcd
usb 1-1: New USB device found, idVendor=8087, idProduct=0024
usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-1: configuration #1 chosen from 1 choice
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 6 ports detected
usb 2-1: new high speed USB device number 2 using ehci_hcd
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ACPI _SDD failed (AE 0x5)
usb 2-1: New USB device found, idVendor=8087, idProduct=0024
usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 2-1: configuration #1 chosen from 1 choice
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 8 ports detected
Refined TSC clocksource calibration: 3399.999 MHz.
Switching to clocksource tsc
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata2.00: ACPI _SDD failed (AE 0x5)
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ACPI _SDD failed (AE 0x5)
ata1.00: ACPI: failed the second time, disabled
ata1.00: ATA-8: TOSHIBA DT01ACA300, MX6OABB0, max UDMA/133
ata1.00: 5860533168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata2.00: ACPI _SDD failed (AE 0x5)
ata2.00: ACPI: failed the second time, disabled
ata1.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access     ATA      TOSHIBA DT01ACA3 MX6O PQ: 0 ANSI: 5
ata2.00: ATA-8: TOSHIBA DT01ACA300, MX6OABB0, max UDMA/133
ata2.00: 5860533168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
ata2.00: configured for UDMA/133
scsi 1:0:0:0: Direct-Access     ATA      TOSHIBA DT01ACA3 MX6O PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
sd 1:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
sd 1:0:0:0: [sdb] 4096-byte physical blocks
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb:
sd 0:0:0:0: [sda] 4096-byte physical blocks
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 sda4 < sdb1 sdb2 sdb3 sdb4 < sda5 >
sd 0:0:0:0: [sda] Attached SCSI disk
 sdb5 >
sd 1:0:0:0: [sdb] Attached SCSI disk
md: bind<sdb5>
md: bind<sdb2>
md: bind<sdb1>
md: bind<sdb3>
md: bind<sda1>
md: raid1 personality registered for level 1
bio: create slab <bio-1> at 1
md/raid1:md0: active with 2 out of 2 mirrors
md0: detected capacity change from 0 to 8589922304
 md0: unknown partition table
md: bind<sda5>
md/raid1:md3: active with 2 out of 2 mirrors
md3: detected capacity change from 0 to 1891949166592
 md3: unknown partition table
md: bind<sda3>
md/raid1:md2: active with 2 out of 2 mirrors
md2: detected capacity change from 0 to 1099511488512
 md2: unknown partition table
md: bind<sda2>
md/raid1:md1: active with 2 out of 2 mirrors
md1: detected capacity change from 0 to 536858624
 md1: unknown partition table
EXT4-fs (md2): INFO: recovery required on readonly filesystem
EXT4-fs (md2): write access will be enabled during recovery
EXT4-fs (md2): orphan cleanup on readonly fs
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 15470064
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 15469793
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 57934996
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 57934495
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 30408714
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 21103413
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 21103410
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 21103409
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 30408711
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 30408710
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 30408709
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 30408708
EXT4-fs (md2): ext4_orphan_cleanup: deleting unreferenced inode 30408707
EXT4-fs (md2): 13 orphan inodes deleted
EXT4-fs (md2): recovery complete
EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: 
dracut: Remounting /dev/md2 with -o grpquota,usrquota,ro
EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: 
dracut: Mounted root filesystem /dev/md2
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1392814512.200:2): selinux=0 auid=4294967295 ses=4294967295
dracut: 
dracut: Switching root
udev: starting version 147
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ACPI: resource (null) [io  0x0428-0x042f] conflicts with ACPI region PMIO [io 0x400-0x47f]
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
ACPI: resource (null) [io  0x0500-0x053f] conflicts with ACPI region GPIO [io 0x500-0x563]
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
lpc_ich: Resource conflict(s) found affecting gpio_ich
sd 0:0:0:0: Attached scsi generic sg0 type 0
sd 1:0:0:0: Attached scsi generic sg1 type 0
  alloc irq_desc for 18 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
i801_smbus 0000:00:1f.3: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ACPI: resource 0000:00:1f.3 [io  0xf040-0xf05f] conflicts with ACPI region SMBI [io 0xf040-0xf04f]
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
r8169 0000:03:00.0: setting latency timer to 64
  alloc irq_desc for 33 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
r8169 0000:03:00.0: irq 33 for MSI/MSI-X
r8169 0000:03:00.0: eth0: RTL8168f/8111f at 0xffffc900066b0000, 08:60:6e:68:0f:8c, XID 08000880 IRQ 33
r8169 0000:03:00.0: eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: Found a Panther Point TCO device (Version=2, TCOBASE=0x0460)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
EXT4-fs (md2): warning: checktime reached, running e2fsck is recommended
kjournald starting.  Commit interval 5 seconds
EXT3-fs (md1): warning: checktime reached, running e2fsck is recommended
EXT3-fs (md1): using internal journal
EXT3-fs (md1): recovery complete
EXT3-fs (md1): mounted filesystem with ordered data mode
EXT4-fs (md3): warning: checktime reached, running e2fsck is recommended
EXT4-fs (md3): recovery complete
EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: 
Adding 8388588k swap on /dev/md0.  Priority:-1 extents:1 across:8388588k

[Z0nd0R]

дамп messages

...
Feb 19 08:01:01 onss rotated[26254]: Checking links to log files
Feb 19 08:01:01 onss rotated[26254]: Checking links finished
Feb 19 08:01:01 onss rotated[26254]: Rotation finished. 0 log files was processed. 0 seconds left
...
Feb 19 08:01:01 - Feb 19 09:01:01 FTP session opened. - FTP session closed. (много переподключений к ftp)
...
Feb 19 09:01:01 onss rotated[1462]: Checking links to log files
Feb 19 09:01:01 onss rotated[1462]: Checking links finished
Feb 19 09:01:01 onss rotated[1462]: Rotation finished. 0 log files was processed. 0 seconds left
...
Feb 19 09:01:01 - Feb 19 10:01:01 FTP session opened. - FTP session closed. (много переподключений к ftp)
...
Feb 19 10:01:01 onss rotated[11052]: Checking links to log files
Feb 19 10:01:01 onss rotated[11052]: Checking links finished
Feb 19 10:01:01 onss rotated[11052]: Rotation finished. 0 log files was processed. 0 seconds left
Feb 19 10:23:11 onss named[1351]: client 122.136.196.116#65449: query (cache) 'www.jrdga.info/A/IN' denied
Feb 19 10:58:46 onss named[1351]: client 93.174.93.93#7678: query (cache) 'gerdar3.ru/ANY/IN' denied
Feb 19 11:01:01 onss rotated[14670]: Checking links to log files
Feb 19 11:01:01 onss rotated[14670]: Checking links finished
Feb 19 11:01:01 onss rotated[14670]: Rotation finished. 0 log files was processed. 0 seconds left
Feb 19 12:01:01 onss rotated[18742]: Checking links to log files
Feb 19 12:01:01 onss rotated[18742]: Checking links finished
Feb 19 12:01:01 onss rotated[18742]: Rotation finished. 0 log files was processed. 0 seconds left
Feb 19 13:01:01 onss rotated[22922]: Checking links to log files
Feb 19 13:01:01 onss rotated[22922]: Checking links finished
Feb 19 13:01:01 onss rotated[22922]: Rotation finished. 0 log files was processed. 0 seconds left
--- Завис, перезагрузка ---
Feb 19 13:55:18 onss kernel: imklog 5.8.10, log source = /proc/kmsg started.
Feb 19 13:55:18 onss rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1339" x-info="http://www.rsyslog.com"] start
Feb 19 13:55:18 onss kernel: Initializing cgroup subsys cpuset
Feb 19 13:55:18 onss kernel: Initializing cgroup subsys cpu
Feb 19 13:55:18 onss kernel: Linux version 2.6.32-431.5.1.el6.x86_64 ([e-mail]) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Wed Feb 12 00:41:43 UTC 2014
Feb 19 13:55:18 onss kernel: Command line: ro root=/dev/md2 rd_NO_LUKS rd_NO_DM nomodeset crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de
Feb 19 13:55:18 onss kernel: KERNEL supported cpus:
Feb 19 13:55:18 onss kernel:  Intel GenuineIntel
Feb 19 13:55:18 onss kernel:  AMD AuthenticAMD
Feb 19 13:55:18 onss kernel:  Centaur CentaurHauls
...

[Z0nd0R]

В общем обновление.

Разобраться в чем причина не удалось. В процессе проверки раздела /boot он рассыпался из-за ошибок.
В итоге все данные были перенесены на бекап сервер. Заного переустановлена операционная система, обновлено ПО и т.д. Система проработала меньше суток и опять повисла.

Не понятно с чем проблема, с винтами или с железом :(

[grossws]

Система повисла на новом сервере?

[Z0nd0R]

Сервер тот же. Начисто переустановил CentOS 6.5. Проблема сохранилась. Логи чистые. Т.е. перед падением никаких подозрительных записей нет. Машина просто перестает реагировать на внешние воздействия.

[grossws]

Меняйте диски, в первую очередь. Если у вас зеркало, то можно выдернуть один диск (через mdadm /dev/mdX -f /dev/sdaXY) и посмотреть на работу на втором. Крайне желательно при этом иметь подключенный KVM и какой-нибудь rescue-образ под рукой.

Пробовали ли вы воспользоваться рекомендациями от @brutal_lobster?

[Z0nd0R]

Закончили делать бекапы. Сейчас запустим диагностику винтов "smartctl --test=long".
И пройдемся по рекомендациям @brutal_lobster.

P.S. fsck ошибок не выдал при проверке с rescue.
P.S.S. Есть одна строчка которая насторожила, так как на ней на секунду ось при загрузке задумывается. Через LARA(KVM) было видно:

ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ACPI _SDD failed (AE 0x5)

[Z0nd0R]

Обнаружил интересную статистику.
ISPManager ведет статистику по трафику. Есть один сайт(соц.сеть)(В основном сайт посещают поисковые боты)
Входящий HTTP трафик: 77 мегабайт
Исходящий: 8.9 Гигабайт.
Это за примерно за два дня(после переустановки уже). Это вообще реально?
У остальных пользователей трафик в пределах нормы.

[grossws]

Зависит от посещаемости. Стоит учитывать, что сайт, в основном, отдает информацию. Поэтому такой перекос вполне реален.

Answer 1

[Z0nd0R]

В общем прошло 8 месяцев с момента постановки вопроса. Где-то месяц назад кончилось терпение и написал в техподдержку Хетзнера по поводу проблем.

Попросил провести диагностику винтов и посмотреть почему происходили падения сервера.
Через час после начала диагностики тех.поддержка разродилась.

Написали что винты в порядке. И они обновили биос на материнке. И как по волшебству все прекратилось. Прошел уже месяц и ни одного падения.

Answer 2

[brutal_lobster]

Включите создание аварийных дампов ядра.
https://access.redhat.com/site/documentation/en-US...

И пройдитесь по возможным ошибкам.
www.dedoimedo.com/computers/crash-analyze.html

В дампе скорее всего найдете причину зависаний. Если проблема в железке - покажете трейсы и логи местным инженерам - что-нибудь заменят.

Answer 3

[grossws]

Выхлоп dmesg в студию, для начала.

Comments

[grossws]

Загоните выхлоп dmesg'а под спойлер. И выложите /var/log/messages.

Orphaned inode у вас скорее всего после ребута. Проблемные записи по acpi были и раньше или добавлялось какое-нибудь оборудование? Запущен ли acpid?

Я бы также предложил заглянуть в /proc/mdstat, smart дисков, прогнать smart-тесты (smartctl --test=long /dev/sdX): у хетзнера бывают проблемы с дисками. Не было ли последнее время просадки производительности на дисковых операциях?

[Z0nd0R]

Глупый вопрос: А как убрать под спойлер?

1. На записи ACPI не обращал внимания
2. Оборудование не добавлялось.
3. ACPID вроде работал. Сейчас перезагрузил на всякий пожарный.
4. Запустил тест smart
5. Изменения в производительности сложно заметить было, так как средняя нагрузка 0,09 - 0,90%
6. Файлик messages достаточно большой, около полумегабайта.

[grossws]

<spoiler title="заголовок">...</spoiler>

При проблемах с acpi запущенный acpid может создавать проблемы, но это диагностировать сложно.

Попробуйте погонять какие-нибудь дисковые операции (dd if=/dev/null of=/some-tmp-file oflag=direct bs=1M), посмотрите на производительность.

Можно глянуть tail -n 1000 /var/log/messages. Если у вас логгирование не очень плотное, то хватит. Если плотное -- то выдернуть кусок около момента перезагрузки.

Ещё может быть полезно посмотреть на показания сенсоров. Учитывая, что эти товарищи любят десктопное железо -- хз что там с температурными режимами.

[Z0nd0R]

Добавил дамп messages. Убрал повторяющиеся операции подключения к FTP
Данные сенсоров показывают температуру 27-29 градусов.

У меня в грабе есть 4 конфигурации.
Если попытаться запуститься с предыдущим ядром, может помочь?
Или какие-то дополнительные операции надо делать?
Есть подозрение что проблемы начались после обновления ядра до последней версии.

[Z0nd0R]

dd if=/dev/zero of=/test oflag=direct bs=1M
15209+0 records in
15209+0 records out
15947792384 bytes (16 GB) copied, 98.7405 s, 162 MB/s

вроде нормально.

[grossws]

Попробуйте. Достаточно выбрать опцию в grub.

Answer 4

[m1rl0b]

а попробовать откатить и загрузиться на старом ядре?

Comments

[Z0nd0R]

После несольких перезагрузок операционка вообще перестала грузиться с ошибкой граба: Error 2.

После загрузки из rescue и проверки раздела на нем ничего не осталось. Было много ошибок при проверке. Самое интересное что раздел с данными оказался без ошибок.

[grossws]

Переносите данные на новый диск. Скорее всего, проблема с дисками.
У меня недавно попался один seagate, который вёл себя, как живой, но при наличии его в рейде -- не давал запустить систему: то udev упадёт при загрузке, то sssd+slapd не отзываются.

Answer 5

[Дмитрий Алексеев]

Смотрите вывод sar на предмет высоких скачков нагрузки или iowait перед падением

Так же хорошо бы глянуть на загрузку памяти непосредственно перед падением, может быть снаружи по SNMP.

Посмотрите внимательно логу messages, была ли активность системы в моменты недоступности? Может сетевой контроллер отвалился?

Answer 6

[1serfer]

Сталкивался с подобной проблемой.
Пройдитесь по всем логам указанного Вами ПО:
PHP, Apache, Nginx, Ruby, MySQL