accessing the Core dump File

This section forms part of my diagnosis on a failed subscriber and the identification onto why it failed, The CUCM creates a core dump file when it experiences a fault.

This can be found on the RTMT Alert central and the Application Event log:-

 Nov 30 00:19:35 BMML-SUB1 local7 2 : 0: Nov 30 00:19:35.810 UTC :  %CCM_LPM-LPMTCT-2-CoreDumpFileFound: The new core dump file(s) have been found in the system. TotalCoresFound:1 CoreDetails:The following lists up to 6 cores dumped by corresponding applications. Core1:Cisco CallManager (core.2951.6.ccm.1322612299) App ID:Cisco Log Partition Monitoring Tool Cluster ID: Node ID:BMML-SUB1
/CiscoSyslog/

2011/11/30 00:19:35.598 | Recovering corefile /var/log/active/core/core.2951.6.ccm.1322612299
2011/11/30 00:19:35.650 |   total opened: 65538, total closed/freed 65535, current opened: 3
2011/11/30 00:19:35.651 |     logfile /var/log/active/cm/trace/dbl/sdi/ccm/dbl_ccm00000198.log: recovered 5839 bytes
2011/11/30 00:19:35.651 |     logfile /var/log/active/cm/trace/dbl/sdi/ccm/dbnotify_ccm00000212.log: recovered 15519 bytes
2011/11/30 00:19:35.652 |     logfile /var/log/active/cm/trace/ccm/sdi/ccm00000353.txt: recovered 20686 bytes
2011/11/30 00:19:35.652 |   Corefile /var/log/active/core/core.2951.6.ccm.1322612299 now recovered.
/cfrt/

Listing the Core dump file

admin:utils core active list

Size         Date            Core File Name
=================================================================
176792 KB   2011-11-30 00:18:20   core.2951.6.ccm.1322612299
admin:

Analysing the core dump

admin:utils core active analyze core.2951.6.ccm.1322612299

This command may use a considerable amount of I/O
and running it may impact system performance.

It is highly recommended that this command be
run off-hours.

Continue (y/n)?y
Using host libthread_db library “/lib/tls/libthread_db.so.1”.
Core was generated by `/usr/local/cm/bin/ccm’.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/csa/libcsa.so.6…done.
Loaded symbols for /lib/csa/libcsa.so.6
Reading symbols from /lib/csa/sse2/sse2_boost.so.1…done

!
!
====================================
backtrace
===================================
#0  0x00cc27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x05337825 in raise () from /lib/tls/libc.so.6
#2  0x05339289 in abort () from /lib/tls/libc.so.6
#3  0x084c9c4e in preabort () at ProcessCMProcMon.cpp:101
#4  0x084c9c66 in IntentionalAbort (reason=0xaa57ad8 “CallManager unable to process signals. This may be due to CPU or blocked function. Attempting to restart CallManager.”) at ProcessCMProcMon.cpp:106
#5  0x084ca2c7 in CMProcMon::monitorThread () at ProcessCMProcMon.cpp:597
#6  0x00feebd7 in ACE_OS_Thread_Adapter::invoke (this=0xb448f6b0) at OS_Thread_Adapter.cpp:94
#7  0x00faf087 in ace_thread_adapter (args=0x0) at Base_Thread_Adapter.cpp:137
#8  0x002b33cc in start_thread () from /lib/tls/libpthread.so.0
#9  0x053db96e in clone () from /lib/tls/libc.so.6
!
!

in this case the issue was with a blocked function that caused the CUCM service to restart. I will be looking into this further in the next section where i will diagnose the issue

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s