We are marking this incident as resolved and will continue updates on a new incident.
Posted Aug 04, 2021 - 06:17 CDT
Update
We are continuing to investigate this issue.
Posted Aug 01, 2021 - 13:58 CDT
Update
Thank you for your continued patience as we address the performance issues.
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. We have deployed the release candidate that includes NHibernate fixes to a targeted stack and we are monitoring performance.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement. This tool has been fully implemented as of the latest update cycle.
In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization. The Cloud Operations team began deploying additional auto scaling rules to targeted server stacks this week. We are currently monitoring performance and plan to deploy the new auto scaling rules to all server stacks in the coming weeks.
Posted Jul 30, 2021 - 09:18 CDT
Update
Thank you for your continued patience as we address the performance issues.
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. We are in the process of finalizing a release candidate that includes the NHibernate fixes to improve latency issues.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement. A portion of the tool was deployed to all server stacks this week and the remaining configurations will be deployed in the coming weeks.
In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization. The Cloud Operations team began deploying additional auto scaling rules to targeted server stacks this week. We are currently monitoring performance and plan to deploy the new auto scaling rules to all server stacks in the coming weeks.
Posted Jul 16, 2021 - 16:38 CDT
Update
Thank you for your continued patience as we address the performance issues.
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. The hot fix deployed this week focuses on NHibernate processes.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.
In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization.
Posted Jul 09, 2021 - 15:29 CDT
Update
Thank you for your continued patience as we address the performance issues
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.
In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization.
Posted Jul 02, 2021 - 07:40 CDT
Update
Thank you for your continued patience as we address the performance issues
We are investigating several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
Targeted hot fixes are continuing to be deployed to isolated groups of servers and the performance is being monitored.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.
Posted Jun 24, 2021 - 06:39 CDT
Update
Thank you for your continued patience as we address the performance issues. These types of issues can be very complex and solutions elusive. We understand that performance issues like this make daily operations more difficult. Our highest priority across the organization is addressing performance, and we are continuing to make incremental progress on this issue. Here are the steps we are taking to improve performance across our fleet.
1. We are adding additional resources to our infrastructure to help alleviate the performance degradation as a short-term solution. 2. Three incremental hot fixes have been canary released to date based on weekly sprints, and we are continuing to deploy incremental fixes every week. 3. Additional development resources have been diverted from our product roadmap to focus on this issue.
Our communication will become more proactive and frequent. Status Page updates will be provided on a weekly basis going forward until this issue is resolved.
Posted Jun 09, 2021 - 11:35 CDT
Update
We are continuing to investigate this issue.
Posted Apr 22, 2021 - 15:25 CDT
Update
We are continuing to investigate this issue.
Posted Apr 12, 2021 - 08:29 CDT
Update
We are continuing to investigate this issue.
Posted Apr 09, 2021 - 14:21 CDT
Update
We are continuing to investigate this issue.
Posted Apr 08, 2021 - 13:06 CDT
Investigating
We are currently investigating an issue with memory utilization across our application servers.