This section forms part of my diagnosis on a failed subscriber and the identification onto why it failed, The CUCM creates a core dump file when it experiences a fault.
This can be found on the RTMT Alert central and the Application Event log:-
Nov 30 00:19:35 BMML-SUB1 local7 2 : 0: Nov 30 00:19:35.810 UTC : %CCM_LPM-LPMTCT-2-CoreDumpFileFound: The new core dump file(s) have been found in the system. TotalCoresFound:1 CoreDetails:The following lists up to 6 cores dumped by corresponding applications. Core1:Cisco CallManager (core.2951.6.ccm.1322612299) App ID:Cisco Log Partition Monitoring Tool Cluster ID: Node ID:BMML-SUB1
2011/11/30 00:19:35.598 | Recovering corefile /var/log/active/core/core.2951.6.ccm.1322612299
2011/11/30 00:19:35.650 | total opened: 65538, total closed/freed 65535, current opened: 3
2011/11/30 00:19:35.651 | logfile /var/log/active/cm/trace/dbl/sdi/ccm/dbl_ccm00000198.log: recovered 5839 bytes
2011/11/30 00:19:35.651 | logfile /var/log/active/cm/trace/dbl/sdi/ccm/dbnotify_ccm00000212.log: recovered 15519 bytes
2011/11/30 00:19:35.652 | logfile /var/log/active/cm/trace/ccm/sdi/ccm00000353.txt: recovered 20686 bytes
2011/11/30 00:19:35.652 | Corefile /var/log/active/core/core.2951.6.ccm.1322612299 now recovered.
Listing the Core dump file
admin:utils core active list
Size Date Core File Name
176792 KB 2011-11-30 00:18:20 core.2951.6.ccm.1322612299
Analysing the core dump
admin:utils core active analyze core.2951.6.ccm.1322612299
This command may use a considerable amount of I/O
and running it may impact system performance.
It is highly recommended that this command be
Using host libthread_db library “/lib/tls/libthread_db.so.1”.
Core was generated by `/usr/local/cm/bin/ccm’.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/csa/libcsa.so.6…done.
Loaded symbols for /lib/csa/libcsa.so.6
Reading symbols from /lib/csa/sse2/sse2_boost.so.1…done
#0 0x00cc27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x05337825 in raise () from /lib/tls/libc.so.6
#2 0x05339289 in abort () from /lib/tls/libc.so.6
#3 0x084c9c4e in preabort () at ProcessCMProcMon.cpp:101
#4 0x084c9c66 in IntentionalAbort (reason=0xaa57ad8 “CallManager unable to process signals. This may be due to CPU or blocked function. Attempting to restart CallManager.”) at ProcessCMProcMon.cpp:106
#5 0x084ca2c7 in CMProcMon::monitorThread () at ProcessCMProcMon.cpp:597
#6 0x00feebd7 in ACE_OS_Thread_Adapter::invoke (this=0xb448f6b0) at OS_Thread_Adapter.cpp:94
#7 0x00faf087 in ace_thread_adapter (args=0x0) at Base_Thread_Adapter.cpp:137
#8 0x002b33cc in start_thread () from /lib/tls/libpthread.so.0
#9 0x053db96e in clone () from /lib/tls/libc.so.6
in this case the issue was with a blocked function that caused the CUCM service to restart. I will be looking into this further in the next section where i will diagnose the issue