84 lines
8.5 KiB
Markdown
84 lines
8.5 KiB
Markdown
**Condition:**
|
||
When these SCOM alerts in the hundreds role in,
|
||
|
||
- LSASS Process Monitor. The core process (lsass.exe) for Active Directory Domain Services is consuming a lot of CPU resources.
|
||
|
||
If the overall CPU utilization on the server is too high, users and services that rely on Active Directory Domain Services may experience delays.
|
||
|
||
- AD Replication Partner Count monitor. This monitor verifies that the domain controller has a suitable number of replication partners.
|
||
|
||
- This monitor verifies the availability of Active Directory using an LDAP (Lightweight Directory Access Protocol) request. It periodically binds to the Domain Controller to verify the availability using an LDAP query.
|
||
|
||
This monitor allows for monitoring of non-SSL, SSL, and both types of bindings.
|
||
|
||
- AD Replication Partner Count monitor. This monitor verifies that the domain controller has a suitable number of replication partners.
|
||
|
||
If the number of replication partners is too low or high, the overall replication health may be affected.
|
||
|
||
|
||
*****A bad Splunk configuration issue has caused this issue two times.*****
|
||
|
||
**Detection:**
|
||
|
||
**Review Wireshark capture on a affected domain controller.**
|
||
|
||
**Identify a workstation that you can RDP into**
|
||
|
||
- **On affected computer rereview last modified date on** "C:\Program Files\SplunkUniversalForwarder\etc\apps\cbp_all_deploymentclient\local\deploymentclient.conf" **A current date would place Splunk at the top of the probable cause list. (Servers and workstation have different policies, it's possible to see different dates on these two platforms.)**
|
||
- **Install Wireshark Set filter "**DRSUAPI" **you should see events, now stop** SplunkForwarder **service you should see the events stop**
|
||
|
||
|
||
**Previous Incidents:**
|
||
|
||
|
||
1. Please see attached and below is the attached but in the email message instead:
|
||
|
||
Problem start date/time; problem end date/time:
|
||
11/16/2024 12:50 ET - 11/16/2024 16:58 ET 2. What system/application(s) were impacted? (Summarize functions / importance)
|
||
EDCOG DCO Windows Services Degradation 3. State end user/business impacts in plain English (if known), specifically who couldn't do what and how CBP operations were affected (view of end user).
|
||
Replication Partner Count health monitor failed. There was no production impact. 4. Incident problem description (detailed paragraph):
|
||
Due to a change in Ansible cofig. file pointing to wrong set of servers, replication Partner Count health monitor failed. 5. List team(s) involved in incident management and track their status on bridges.
|
||
EIOD, DMG, CSD, WOs, TOC, WSG
|
||
|
||
6. How was the incident resolved and by which team(s)? (detailed paragraph)?
|
||
At 1658 ET SMEs run the correct Ansible configuration file to right servers resolving issue. 7. What was the cause? (provide if identified - Human error, hardware failure, etc.)
|
||
Splunk CR CHG00152872 | Change Request | CBP ServiceNow Production Instance caused this issue to occur. CBP Splunk engineering will research further to better understand why this occurred when the CR was executed. Restarting of servers may have caused a push of incorrect or older configurations. Summary provided by Splunk SME: Within the Technical Addon for Splunk, the default setting for inputs.conf is:
|
||
evt_resolve_ad_obj = 1
|
||
This default setting continuously checks AD for GUID and SID via lookups until it is resolved, hence spamming the Active Directory server. This caused the resource utilization to max out when UFs were unable to resolve AD objects. In order to temporarily remediate this issue, we changed this setting to:
|
||
evt_resolve_ad_obj = 0 Then this setting change caused all UFs to restart gracefully and resource utilization dropped to normal levels. 8. If the incident was caused by a change, indicate source (CR, ECR, etc.) and provide corresponding number(s).
|
||
CHG00152872 | Change Request | CBP ServiceNow Production Instance Splunk CR 9. What follow on actions are team(s) taking and when are they planned? (if any)
|
||
No further action is required, all settings are now uniform and this setting should be properly set for servers and workstations. 10. Provide other ticket information as available (ex: vendor incident ticket #s)
|
||
INC2932437 11. How were teams notified? Provide date/time teams were aware of the incident, source of the information, and method. (Ex: TSD, 17:00 on 6/3, User, SNOW Ticket)
|
||
* SMEs – 11/16/24 1625 ET Email
|
||
TOC – 11/16/24 1641 ET Chat/Email
|
||
TSD – 11/16/24 1641 ET Chat/Email
|
||
WO – 11/16/24 1625 ET Email * Subject Matter Experts such as Application and Infrastructure teams, or other functional teams like NOC and SOC) 12. What was the time of the first alert and what monitoring system(s) alerted?
|
||
11/16/24 12:50 ET NOI 13. What time did NOI alert (if at all)?
|
||
11/16/24 12:50 ET
|
||
14. Provide detailed timeline of key actions taken which were not captured in the EOC Significant Incident Channel thread.
|
||
Significant Channel thread
|
||
|
||
********************************************************************************************************************************************
|
||
|
||
1. Problem start date/time; problem end date/time:
|
||
10/05/2024 1230ET
|
||
10/06/2024 1012ET 2. What system/application(s) were impacted? (Summarize functions / importance)
|
||
Active Directory Domain Services: This service serves as CBP’s user and computer management platform, primarily providing authentication services for desktop logon, asset management, Office 365 authentication and single-sign on. 3. State end user/business impacts in plain English (if known), specifically who couldn't do what and how CBP operations were affected (view of end user).
|
||
==No direct user impact was recorded because the events presented outside of peak demand. Active Directory health degradation was reported by the “SCOM” monitoring platform.== 4. Incident problem description (detailed paragraph):
|
||
Active Directory domain controllers located in the field were saturated with requests from desktop and server installed Splunk forwarders, causing CPU starvation. This degraded critical domain services because the domain controllers were overwhelmed. 5. List team(s) involved in incident management and track their status on bridges.
|
||
EIOD WSG, DMG, CSD, WOs, TOC, Microsoft support 6. How was the incident resolved and by which team(s)? (detailed paragraph)?
|
||
The Slunk SME modified a configuration that instructs all CBP Windows computers to skip Active Directory lookups. This is expected to be a temporary workaround and the setting will be reverted once all computers have cycled the Splunk forwarder service. 7. What was the cause? (provide if identified - Human error, hardware failure, etc.)
|
||
Splunk CR CHG00152202 caused this issue to occur. CBP Splunk engineering will research further to better understand why this occurred when the CR was executed. Summary provided by Splunk SME: Within the Technical Addon for Splunk, the default setting for inputs.conf is:
|
||
evt_resolve_ad_obj = 1
|
||
This default setting continuously checks AD for GUID and SID via lookups until it is resolved, hence spamming the Active Directory server. This caused the resource utilization to max out when UFs were unable to resolve AD objects. In order to temporarily remediate this issue, we changed this setting to:
|
||
evt_resolve_ad_obj = 0 Then this setting change caused all UFs to restart gracefully and resource utilization dropped to normal levels.
|
||
8. If the incident was caused by a change, indicate source (CR, ECR, etc.) and provide corresponding number(s).
|
||
CHG00152202 Splunk CR 9. What follow on actions are team(s) taking and when are they planned? (if any)
|
||
The SMEs will continue to research to determine when/if the setting can be returned to default, and also research to determine how this could occur given the scope of the CR. 10. Provide other ticket information as available (ex: vendor incident ticket #s)
|
||
INC2864726 11. How were teams notified? Provide date/time teams were aware of the incident, source of the information, and method. (Ex: TSD, 17:00 on 6/3, User, SNOW Ticket)
|
||
* SMEs – 10/05/24 1308ET Email
|
||
TOC – 010/05/24 1304ET Chat/Email
|
||
TSD –
|
||
WO – 10/05/24 1308ET Chat/Email * Subject Matter Experts such as Application and Infrastructure teams, or other functional teams like NOC and SOC) 12. What was the time of the first alert and what monitoring system(s) alerted?
|
||
10/5/24 12:48 ET SCOM 13. What time did NOI alert (if at all)?
|
||
1. 10/5/24 12:48 ET SCOM 15. Provide detailed timeline of key actions taken which were not captured in the EOC Significant Incident Channel thread. |