Potential issues loading external services

Incident Report for HBK Where

Postmortem

I ran WinDirStat to determine where the large files were coming from. I came across a Windows folder, C:/Windows/Logs/CBS, that was taking up over 100GB. I first thought I would need to just rotate the logs, but after fiddling with that for a while, I Googled the folder, where I came up with this: https://www.computerworld.com/article/3112358/windows-7-log-file-compression-bug-can-fill-up-your-hard-drive.html

I followed the steps – with the addition of needing to manually force stop the TrustedInstaller service from the Command Prompt, as it wasn’t responding to the services.msc UI. This cleared up over 100GB.

I will also install alerting in DataDog to ensure we can catch problems like this before they become operational issues.

Posted May 11, 2021 - 21:07 CDT

Resolved

We have identified and fixed the source of the issue, a Windows bug.

Posted May 11, 2021 - 21:03 CDT

Investigating

There are still some issues with disk space usage on the ArcGIS server that we have yet to resolve. There is a potential for this issue to cause outages with external services. We are investigating the problem and working on resolving it, and we appreciate your patience should the ArcGIS server become unavailable before we are able to fix it.

Posted May 11, 2021 - 16:50 CDT

This incident affected: External Services (ArcGIS Server).