Update - Thank you for your continued patience as we address the performance issues.

We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. We are in the process of finalizing a release candidate that includes the NHibernate fixes to improve latency issues.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement. A portion of the tool was deployed to all server stacks this week and the remaining configurations will be deployed in the coming weeks.

In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization. The Cloud Operations team began deploying additional auto scaling rules to targeted server stacks this week. We are currently monitoring performance and plan to deploy the new auto scaling rules to all server stacks in the coming weeks.
Jul 16, 16:38 CDT

Update - Thank you for your continued patience as we address the performance issues.

We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. The hot fix deployed this week focuses on NHibernate processes.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.

In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization.
Jul 9, 15:29 CDT

Update - Thank you for your continued patience as we address the performance issues

We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.

In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization.
Jul 2, 07:40 CDT

Update - Thank you for your continued patience as we address the performance issues

We are investigating several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes are continuing to be deployed to isolated groups of servers and the performance is being monitored.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.
Jun 24, 06:39 CDT

Update - Thank you for your continued patience as we address the performance issues. These types of issues can be very complex and solutions elusive. We understand that performance issues like this make daily operations more difficult. Our highest priority across the organization is addressing performance, and we are continuing to make incremental progress on this issue. Here are the steps we are taking to improve performance across our fleet.

1. We are adding additional resources to our infrastructure to help alleviate the performance degradation as a short-term solution.
2. Three incremental hot fixes have been canary released to date based on weekly sprints, and we are continuing to deploy incremental fixes every week.
3. Additional development resources have been diverted from our product roadmap to focus on this issue.

Our communication will become more proactive and frequent. Status Page updates will be provided on a weekly basis going forward until this issue is resolved.
Jun 9, 11:35 CDT

Update - We are continuing to investigate this issue.
Apr 22, 15:25 CDT

Update - We are continuing to investigate this issue.
Apr 12, 08:29 CDT

Update - We are continuing to investigate this issue.
Apr 9, 14:21 CDT

Update - We are continuing to investigate this issue.
Apr 8, 13:06 CDT

Investigating - We are currently investigating an issue with memory utilization across our application servers.
Apr 8, 11:20 CDT