Revision 3 posted to Systems Management - Wiki by Peter Tsai on 10/20/2010 2:13:01 PM
What are all These Baseboard Management Controller (BMC) System Event Log(SEL) Records Logged by Windows?
IntroductionDid you ever notice a bunch of entries in the Baseboard Management Controller’s (BMC) System Event Log (SEL) that seem to be coming from Windows? Not the Windows system event log, but rather the SEL that is accessible from the Baseboard Management Controller (BMC).It may seem that this log is only intended for hardware events and errors, but there is an OS driver that also logs events to it. Starting with Windows Server 2003 R2, Microsoft began shipping the Intelligent Platform Management Interface (IPMI) driver. One of its functions is to log certain OS events to the SEL. The IPMI driver writes multiple records to the SEL for these three types of events: boot up, shutdown and bugcheck events.The BMC SEL log is accessible via the sideband BMC interface or the Dell Remote Access Controller (DRAC) console and these log records can be useful in troubleshooting a failed system. Each SEL record packs a lot of information in only 16 bytes, and given their limited size, the abbreviated identifiers can be hard to decode. Decoding the SEL Record
Each SEL record is 16 bytes in length and the format is defined in the IPMI spec. The table below summarizes the format of records in the SEL log. Each record contains nformation relevant to the OS events. For example, the Record Type field indicates whether a record is a System Event Record or an OEM defined event record. System Event records are defined in the IPMI Specification, and OEM records are defined by the OEM. Each event logged by the IPMI driver (whether boot up, shutdown, or bugcheck) will cause one system event record and one or more OEM record to be logged to the SEL. The system event record format is defined in the table below.| Offset | Length(bytes) | Field | Description |
| 0x0 | 2 | SEL Record ID | 16 bit identifier 0000h: Reserved 0001h to FFFEh: SEL Record ID FFFFh: Reserved |
| 0x2 | 1 | Record Type | 02h: System Event Record C0h to DFh: OEM Time stamped Record (bytes 8-16 are OEM Defined) E0h to FFh: OEM Non-Time stamped Record (bytes 4-16 are OEM Defined) |
| 0x3 | 4 | SEL Timestamp | Time when event was logged. Least Significant byte first |
| 0x7 | 2 | Generator ID | Byte 0 (LSB): System Software ID or IPMB Slave Address=============[7:1]: System Software ID, or7-bit I2C slave address[0]: 0b means bits 7-1 are the IPMB Slave Address1b means bits 7-1 are the System Software IDByte 1 (MSB): Channel/LUN==============[7:4]: Channel Number. Must be 0000b if the event was received via the system interface, primary IPMB, or internally generated by the BMC[3:2]: Reserved. Must be 00b[1:0]: IPMB device LUN if Byte 0 holds a slave address00b if Byte 0 holds a System Software ID |
| 0x9 | 1 | EvMRev | 03h: IPMI 1.0 04h: IPMI 2.0 |
| 0xA | 1 | Sensor Type | Sensor type code as specified in the IPMI spec |
| 0xB | 1 | Sensor Number | Number of the sensor that generated the event |
| 0xC | 1 | Event Type | [7]: Event Direction0b: Assertion Event1b: De-assertion Event[6:0]: 7 bit Event/Reading Type Code(See Table 42-1 in IPMI spec)00h: Event/Reading Type unspecified01h: Threshold(See Table 42-2 in IPMI spec)02h-0Ch: Generic Discrete(See Table 42-2 in IPMI spec)6Fh: Sensor-Specific Discrete(See table 42-3 in IPMI spec)70h-7Fh: OEM Discrete. Indicates the discrete state info is specific to the OEM identified by the Manufacturer ID for the IPMI device that is providing access to the sensor |
| 0xD | 1 | Event Data 1: | Event Field Contents. See Table 29-6 in IPMI spec |
| 0xE |
| Event Data 2 | Event Field Contents. See Table 29-6 in IPMI spec |
| 0xF |
| Event Data 3 | Event Field Contents. See Table 29-6 in IPMI spec
|
Boot Up EventsDuring an OS system boot, two records are written to the SEL. The first is a type 2 System Event, and the second is an OEM event. Type 0x02 Boot Up Record
The first 10 bytes of the type 2 System boot up record are general header information, and the boot-relevant information starts at offset 0xA.| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0x02 | Record Type | 2 indicates a system event record |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 2 | 0x0041 | Generator ID | 0x0041 indicates System Software id 0x20 |
| 0x9 | 1 | 0x04 | EvMRev | IPMI 2.0 Event Message Revision |
| 0xA | 1 | 0x1F | Sensor Type:OS boot | Sensor type 0x1f is OS Boot as specified in the IPMI spec |
| 0xB | 1 | 0x00 | Sensor Number |
|
| 0xC | 1 | 0x6F | Event Type | [7]: 0b: indicates an assertion event [6:0]: 6Fh: indicates a Sensor-specific Discreet Type code. This means use the sensor specific offsets to interpret the event data bytes |
| 0xD | 1 | 0x01 | Event Data 1: C: Boot Completed0000 0001b | [7:6]: 00b = unspecified byte 2 event data [5:4]: 00b = unspecified byte 3 event data [3:0]: Offset from Event/Reading Code 0h: A: boot completed 1h: C: boot completed 2h: PXE boot completed 3h: Diagnostic boot completed 4h: CD-ROM boot completd 5h: ROM boot completed 6h: boot completed-boot device not specified |
| 0xE | 1 | 0xFF | Event Data 2 | Not specified per bits 7-6 in event data 1 |
| 0xF | 1 | 0xFF | Event Data 3 | Not specified per bits 5-4 in event data 1 |
Type 0xDC Boot Up OEM Event Record
The OEM Event record for bootup does not provide much more information. Nevertheless, the type 0xDC OEM Boot up event record looks similar to this:| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0xDC | Record Type | OEM Time stamped Record (bytes 0x7-0xF are OEM Defined)0xDC: OS Boot up |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 3 | 0x000137 | IPMI Manufacturer ID | 0x137 (311d) is the IANA enterprise number for Microsoft |
| 0xA | 1 |
| Sequence Number | Sequence number used to concatenate the OEM Data bytes from multiple SEL entries |
| 0xB | 4 |
| Boot Time | The OS Boot time |
| 0xF | 1 | 0x00 | Reserved | Reserved |
Shutdown EventsSystem shutdown events can cause many records to be logged to the SEL. In fact, shutdown events are usually the reason that SEL logs become unexpectedly full. Again, there is one system event record type 2, one OEM record type 0xDD for the shutdown reason code, and zero or more OEM type 0xDD records for the shutdown comment.Type 0xDC System Shutdown Event Record
Again, the first 10 bytes of the type 0x02 System Event Record is all header information. The information relevant to the system shutdown starts at offset 0xA.| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0x02 | Record Type | 2 indicates a system event record |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 2 | 0x0041 | Generator ID | 0x0041 indicates System Software id 0x20 |
| 0x9 | 1 | 0x04 | EvMRev | IPMI 2.0 Event Message Revision |
| 0xA | 1 | 0x20 | Sensor Type:OS Stop/Shutdown | Sensor type 0x20 is OS Stop/Shutdown as specified in the IPMI spec |
| 0xB | 1 | 0x00 | Sensor Number |
|
| 0xC | 1 | 0x6F0101 1111b | Event Type | [7]: 0b: indicates an assertion event [6:0]: 6Fh: indicates a Sensor-specific Discreet Type code. This means use the sensor specific offsets to interpret the event data bytes |
| 0xD | 1 | 0x030000 0011b | Event Data 1: OS Graceful Shutdown | [7:6]: 00b = unspecified byte 2 event data [5:4]: 00b = unspecified byte 3 event data [3:0]: Offset from Event/Reading Code 0h: Critical stop during OS load/initialization. Unexpected error during system startup. Stopped waiting for input or power cycle/reset 1h: Run-time critical stop (aka ‘core dump’ or ‘blue screen’ 2h: OS Graceful stop (system powered up, but normal OS operation has shut down and system is awaiting reset pushbutton, power-cycle or other external input) 3h: OS Graceful Shutdown (system graceful power down by OS) 4h: Soft Shutdown initiated by PEF 5h: Agent Not Responding. Graceful shutdown request to agent via BMC did not occur due to missing or malfunctioning local agent |
| 0xE | 1 | 0xFF | Event Data 2 | Not specified per bits 7-6 in event data 1 |
| 0xF | 1 | 0xFF | Event Data 3 | Not specified per bits 5-4 in event data 1 |
Type 0xDD Shutdown Reason OEM Event Record
The IPMI driver reads the shutdown reason code is from the DWORD registry value at HKLM/Software/Microsoft/Windows/CurrentVersion/Reliability/shutdown/ReasonCode (DWORD)The reason code is then logged to the SEL with the following OEM Type 0xDD record:| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0xDD | Record Type | OEM Time stamped Record (bytes 0x7-0xF are OEM Defined)0xDD: OS Shutdown |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 3 | 0x000137 | IPMI Manufacturer ID | 0x137 (311d) is the IANA enterprise number for Microsoft |
| 0xA | 1 |
| Sequence Number | Sequence number used to concatenate the OEM Data bytes from multiple SEL entries |
| 0xB | 4 |
| Shutdown Reason | Shutdown Reason which is read from the registry:HKLM/Software/Microsoft/Windows/CurrentVersion/Reliability/shutdown/ReasonCode (DWORD) |
| 0xF | 1 | 0x00 | Reserved |
|
Type 0xDD Shutdown Comment OEM Event Record
Similarly, the shutdown comment is a REG_SZ value read from the same registry key. HKLM/Software/Microsoft/Windows/CurrentVersion/Reliability/shutdown/ReasonCode (DWORD)Whenever a user shuts down a system interactively (Start | Shutdown), the reason and comment entered are saved in the registry, and the IPMI driver logs them to the SEL. This can become problematic because as you can see below, there are only 4 bytes available for the comment per SEL entry, and the comment is a Unicode string, so it can take a lot of SEL records to save the whole comment. The sequence number is used to concatenate all the SEL records used to save the comment.| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0xDD | Record Type | OEM Time stamped Record (bytes 0x7-0xF are OEM Defined)0xDD: OS Shutdown |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 3 | 0x000137 | IPMI Manufacturer ID | 0x137 (311d) is the IANA enterprise number for Microsoft |
| 0xA | 1 |
| Sequence Number | Sequence number used to concatenate the OEM Data bytes from multiple SEL entries |
| 0xB | 4 |
| Shutdown Reason | Shutdown Comment which is read from the registry:HKLM/Software/Microsoft/Windows/CurrentVersion/Reliability/shutdown/Comment (REG_SZ) |
| 0xF | 1 | 0x00 | Reserved |
|
Disabling the Type 0xDD Shutdown Comment records
Because the shutdown comment can take so many records to log, it can cause the SEL to become full very quickly. Microsoft released a hotfix to allow for disabling the shutdown log comments for Windows Server 2008 SP1. This hotfix can be found at http://support.microsoft.com/kb/962920Starting with Windows Server 2008 R2, the functionality is present in the OS without applying the hotfix. So for windows Server 2008 SP1, you must apply the hotfix before modifying the registry, but for Windows Server 2008 SP2 and later you can modify the registry without applying the hotfix.1.Open regedit and navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\IPMI2.Create a new DWORD Value named DisableSELShutdownComment3.Right-click DisableSELShutdownComment, and then click Modify. 4.In the Value data box, type 1, and then click OK. 5.Close regedit and rebootBugcheck EventsA system bugcheck (bluescreen) will cause six records to be written to the event log. A type 0x2 System Event records is written, followed by five type 0xDE OEM event records: One for the bugcheck code, and one for each of the four parameters to the bugcheck.Type 0x02 Bugcheck System Event Record
Like the other Type 0x02 System event records logged by the IPMI driver, the relevant information starts after the header at offset 0xA. In the case of the bugcheck, the sensor type is 0x20 which is the same as the Shutdown system event record, but the event data byte at offset 0xD indicates runtime critical stop.| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0x02 | Record Type | 2 indicates a system event record |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 2 | 0x0041 | Generator ID | 0x0041 indicates System Software id 0x20 |
| 0x9 | 1 | 0x04 | EvMRev | IPMI 2.0 Event Message Revision |
| 0xA | 1 | 0x20 | Sensor Type: OS Stop/Shutdown | Sensor type 0x20 is OS Stop/Shutdown as specified in the IPMI spec |
| 0xB | 1 | 0x00 | Sensor Number |
|
| 0xC | 1 | 0x6F0101 1111b | Event Type | [7] 0b: indicates an assertion event [6:0] 6Fh: indicates a Sensor-specific Discreet Type code. This means use the sensor specific offsets to interpret the event data bytes |
| 0xD | 1 | 0x010000 0001b | Event Data 1: Run-time Critical Stop | [7:6]: 00b = unspecified byte 2 event data [5:4]: 00b = unspecified byte 3 event data [3:0]: Offset from Event/Reading Code 0h: Critical stop during OS load/initialization. Unexpected error during system startup. Stopped waiting for input or power cycle/reset 1h: Run-time critical stop (aka ‘core dump’ or ‘blue screen’ 2h: OS Graceful stop (system powered up, but normal OS operation has shut down and system is awaiting reset pushbutton, power-cycle or other external input) 3h: OS Graceful Shutdown (system graceful power down by OS) 4h: Soft Shutdown initiated by PEF 5h: Agent Not Responding. Graceful shutdown request to agent via BMC did not occur due to missing or malfunctioning local agent |
| 0xE | 1 | 0xFF | Event Data 2 | Not specified per bits 7-6 in event data 1 |
| 0xF | 1 | 0xFF | Event Data 3 | Not specified per bits 5-4 in event data 1 |
Type 0xDE Bugcheck Code OEM Event Record
The bugcheck code is the stop code displayed on the bluescreen. It is used to indicate the type of failure that caused the bugcheck. The OEM Type 0xDE event record saves the bugcheck code to the SEL.| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0xDE | Record Type | OEM Time stamped Record (bytes 0x7-0xF are OEM Defined)0xDE: OS Bugcheck |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 3 | 0x000137 | IPMI Manufacturer ID | 0x137 (311d) is the IANA enterprise number for Microsoft |
| 0xA | 1 |
| Sequence Number | Sequence number used to concatenate the OEM Data bytes from multiple SEL entries |
| 0xB | 4 |
| Bugcheck Stop Code | Stop code listed on the BSOD |
| 0xF | 1 |
| System Architecture | 0x00: 32bit OS0x01: 64bit OS |
Type 0xDE Bugcheck Parameter OEM Event Record
When the system encounters a condition or error it can’t handle, a system call is made to bugcheck the system. In addition to the bugcheck code described above, four parameters are passed to further describe the failure. This information can be used to debug the system and figure out why the crash occurred. The OEM Type 0xDE Bugcheck Parameter Event record is described below:| Offset | Length (bytes) | Value | Field Name | Description |
| 0x0 | 2 |
| SEL Record ID |
|
| 0x2 | 1 | 0xDE | Record Type | OEM Time stamped Record (bytes 0x7-0xF are OEM Defined)0xDE: OS Bugcheck |
| 0x3 | 4 |
| SEL Timestamp |
|
| 0x7 | 3 | 0x000137 | IPMI Manufacturer ID | 0x137 (311d) is the IANA enterprise number for Microsoft |
| 0xA | 1 |
| Sequence Number | Sequence number used to concatenate the OEM Data bytes from multiple SEL entries |
| 0xB | 4 |
| Bugcheck Parameter | Argument to Bugcheck. These are the same arguments listed on the BSOD |
| 0xF | 1 |
| System Architecture | 0x00: 32bit OS0x01: 64bit OS |