The release candidate that includes the NHibernate fixes has been deployed across the TCP server fleet. We have seen a dramatic improvement in server performance over the last several weeks and will be marking this incident as resolved.
TCP truly appreciates the patience and support we received from our customers as we worked through the performance issues over the last several months.
Posted Sep 22, 2021 - 08:43 CDT
Update
Thank you for your continued patience as we address the performance issues.
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
We have deployed the release candidate that includes NHibernate fixes to the following server stacks: • Group2.tcplusondemand.com • Group5.tcplusondemand.com • Prod01.tcplusondemand.com • Prod05.tcplusondemand.com • Prod02.tcplusondemand.com • Prod03.tcplusondemand.com • Prod10.tcplusondemand.com • Prod11.tcplusondemand.com • Prod12.tcplusondemand.com • Prod14.tcplusondemand.com • Prod15.tcplusondemand.com • Prod18.tcplusondemand.com • Prod23.tcplusondemand.com • Prod25.tcplusondemand.com • Prod31.tcplusondemand.com • Prod32.tcplusondemand.com • Prod34.tcplusondemand.com • Prod35.tcplusondemand.com
Based on monitoring and performance of the targeted stacks, we are confident that this release candidate will significantly improve performance and efficiency of the TimeClock Plus application.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. The tool has been deployed across the entire server fleet, and we are analyzing results and metrics to further improve the performance of TCP software.
We will continue scaling back additional resources allocated to servers as this release candidate is deployed. We are still on track to have it deployed across the entire server fleet by 9/2.
Posted Aug 30, 2021 - 07:11 CDT
Update
Thank you for your continued patience as we address the performance issues.
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
We have deployed the release candidate that includes NHibernate fixes to two additional targeted stacks.
Based on monitoring and performance of the targeted stacks, we are confident that this release candidate will significantly improve performance and efficiency of the TimeClock Plus application.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. As of today, this tool has been deployed across the entire server fleet.
We will begin scaling back additional resources allocated to servers as this release candidate is deployed. We expect to have it deployed across the entire server fleet by 9/2.
Posted Aug 20, 2021 - 12:59 CDT
Investigating
Thank you for your continued patience as we address the performance issues.
We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.
Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. We have deployed the release candidate that includes NHibernate fixes to a targeted stack and we are monitoring performance.
We anticipate deploying the release candidate that includes NHibernate fixes to two additional targeted stacks on 8/12/2021 to continue testing and monitoring performance.
The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement. This tool will be fully implemented once 7.1.13.3 has been deployed to all server stacks.
In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization. The Cloud Operations team began deploying additional auto scaling rules to targeted server stacks this week. We are currently monitoring performance and plan to deploy the new auto scaling rules to all server stacks in the coming weeks.