Time and Attendance On Demand.
Update - v7 OnDemand updates to 7.1.13.3 for the CloudOps Small Server Stacks are scheduled for Thursday, July 29th between 9:00 PM CST - 3:00 AM CST.

While the updates are taking place, you may experience intermittent access to your v7 OnDemand website, which includes the following applications: Manager, WebClock, TelClock and TCP Smartphone apps. If you have purchased 200 or 400 Series Remote Data Terminals or Hand scanners, they will continue to operate seamlessly in fallback mode.

Although we don’t expect the software updates to take the entire time allotted and only some customers will experience a disruption in service, we apologize in advance for any inconvenience this may cause. Normal operations will resume upon completion of the updates.

If you have any questions or concerns, please feel free to contact TCP technical support at 325-223-9300, or email us at SaaS.Tickets@TCPSoftware.com and make sure to stay informed about future maintenance and update events with our Status Page: https://status.tcplusondemand.com/
Jul 28, 16:24 CDT
Monitoring - A fix has been implemented and we are monitoring the results.
Jul 27, 05:55 CDT
Update - We are continuing to work on a fix for this issue.
Jul 26, 11:58 CDT
Update - We are continuing to work on a fix for this issue.
Jul 23, 17:11 CDT
Identified - The issue has been identified and a fix is being implemented.
Jul 23, 12:03 CDT
Investigating - TCP made the decision to cancel last night's v7 OnDemand updates as a precaution due to an issue identified during pre-deployment quality checks.

We tentatively expect to deploy 7.1.13.2 updates to the small server stack groups next Thursday, July 29th. A follow up email will be sent early next week to confirm the date and time of the updates.

We will be working diligently to identify and troubleshoot the issue as quickly as possible.

We appreciate your patience and support and look forward to delivering the next revision of TCP soon.
Jul 23, 07:12 CDT
Update - Thank you for your continued patience as we address the performance issues.

We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. We are in the process of finalizing a release candidate that includes the NHibernate fixes to improve latency issues.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement. A portion of the tool was deployed to all server stacks this week and the remaining configurations will be deployed in the coming weeks.

In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization. The Cloud Operations team began deploying additional auto scaling rules to targeted server stacks this week. We are currently monitoring performance and plan to deploy the new auto scaling rules to all server stacks in the coming weeks.
Jul 16, 16:38 CDT
Update - Thank you for your continued patience as we address the performance issues.

We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements. The hot fix deployed this week focuses on NHibernate processes.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.

In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization.
Jul 9, 15:29 CDT
Update - Thank you for your continued patience as we address the performance issues

We are continuing to investigate several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes continue to be deployed weekly to isolated groups of servers and the performance is being monitored for improvements.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.

In addition to implementing new instrumentation for higher quality monitoring, the Cloud Operations team is testing and reviewing strategies to increase available resources for servers experiencing increased RAM utilization.
Jul 2, 07:40 CDT
Update - Thank you for your continued patience as we address the performance issues

We are investigating several environmental settings to the .NET Core Runtime to increase the activity of the garbage collection behavior and release a significant memory footprint.

Targeted hot fixes are continuing to be deployed to isolated groups of servers and the performance is being monitored.

The Cloud Operations team is implementing new instrumentation to allow for much higher quality monitoring and visibility into the environment health. This tool is going to provide insights to help identify additional areas of improvement.
Jun 24, 06:39 CDT
Update - Thank you for your continued patience as we address the performance issues. These types of issues can be very complex and solutions elusive. We understand that performance issues like this make daily operations more difficult. Our highest priority across the organization is addressing performance, and we are continuing to make incremental progress on this issue. Here are the steps we are taking to improve performance across our fleet.

1. We are adding additional resources to our infrastructure to help alleviate the performance degradation as a short-term solution.
2. Three incremental hot fixes have been canary released to date based on weekly sprints, and we are continuing to deploy incremental fixes every week.
3. Additional development resources have been diverted from our product roadmap to focus on this issue.

Our communication will become more proactive and frequent. Status Page updates will be provided on a weekly basis going forward until this issue is resolved.
Jun 9, 11:35 CDT
Update - We are continuing to investigate this issue.
Apr 22, 15:25 CDT
Update - We are continuing to investigate this issue.
Apr 12, 08:29 CDT
Update - We are continuing to investigate this issue.
Apr 9, 14:21 CDT
Update - We are continuing to investigate this issue.
Apr 8, 13:06 CDT
Investigating - We are currently investigating an issue with memory utilization across our application servers.
Apr 8, 11:20 CDT

About This Site

For more information regarding which components effect the availability of your TimeClock Plus v7 On Demand website, please contact our support department at support@timeclockplus.com.

Application Health Degraded Performance
Group1.tcplusondemand.com Operational
Group2.tcplusondemand.com Operational
Group3.tcplusondemand.com Operational
Group4.tcplusondemand.com Operational
Group5.tcplusondemand.com Operational
Group10.tcplusondemand.com Degraded Performance
Group10-2.tcplusondemand.com Operational
Group12.tcplusondemand.com Operational
Prod01.tcplusondemand.com Operational
Prod02.tcplusondemand.com Operational
Prod03.tcplusondemand.com Operational
Prod04.tcplusondemand.com Operational
Prod05.tcplusondemand.com Operational
Prod06.tcplusondemand.com Operational
Prod07.tcplusondemand.com Operational
Prod08.tcplusondemand.com Operational
Prod09.tcplusondemand.com Operational
Prod10.tcplusondemand.com Degraded Performance
Prod11.tcplusondemand.com Operational
Prod12.tcplusondemand.com Operational
Prod13.tcplusondemand.com Degraded Performance
Prod14.tcplusondemand.com Degraded Performance
Prod15.tcplusondemand.com Degraded Performance
Prod16.tcplusondemand.com Operational
Prod17.tcplusondemand.com Operational
Prod18.tcplusondemand.com Operational
Prod19.tcplusondemand.com Degraded Performance
Prod20.tcplusondemand.com Degraded Performance
Prod21.tcplusondemand.com Degraded Performance
Prod22.tcplusondemand.com Operational
Prod23.tcplusondemand.com Operational
Prod24.tcplusondemand.com Operational
Prod25.tcplusondemand.com Operational
Prod26.tcplusondemand.com Operational
Prod27.tcplusondemand.com Operational
Prod28.tcplusondemand.com Operational
Prod29.tcplusondemand.com Operational
Prod30.tcplusondemand.com Operational
Prod31.tcplusondemand.com Operational
Prod32.tcplusondemand.com Operational
Prod33.tcplusondemand.com Operational
Prod34.tcplusondemand.com Operational
phr1.tcplusondemand.com Operational
phr2.tcplusondemand.com Degraded Performance
phr3.tcplusondemand.com Operational
phr4.tcplusondemand.com Operational
phr5.tcplusondemand.com Operational
phr6.tcplusondemand.com Operational
phr7.tcplusondemand.com Operational
Prod36.tcplusondemand.com Operational
Prod35.tcplusondemand.com Operational
Prod37.tcplusondemand.com Operational
AWS Infrastructure ? Operational
AWS ec2-us-east-1 Operational
AWS ec2-us-east-2 Operational
AWS rds-us-east-1 Operational
AWS rds-us-east-2 Operational
AWS s3-sa-east-1 Operational
AWS route53-us-east-1 Operational
AWS elb-us-east-1 Operational
AWS elb-us-east-2 Operational
AWS s3-us-east-2 Operational
Web Application Firewall ? Operational
Incapsula Ashburn, VA Operational
Incapsula Dallas, TX Operational
Incapsula Los Angeles, CA Operational
Incapsula Newark, NJ Operational
Incapsula Miami, FL Operational
Incapsula Atlanta, GA Operational
Incapsula Chicago, IL Operational
Incapsula Seattle, WA Operational
Incapsula San Jose, CA Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
CloudOps Updates - 7.1.13.3 | Small Server Stacks Jul 29, 21:00 - Jul 30, 03:00 CDT
Update - We will be undergoing scheduled maintenance during this time.
Jul 28, 16:22 CDT
Scheduled - Software updates for v7 OnDemand are scheduled to occur Thursday, July 29th from 9:00 PM - 3:00 AM (CST) for the following groups. Some downtime is expected to occur while the updates are being applied.

prod02.tcplusondemand.com
prod03.tcplusondemand.com
prod10.tcplusondemand.com
prod11.tcplusondemand.com
prod12.tcplusondemand.com
prod14.tcplusondemand.com
prod15.tcplusondemand.com
prod18.tcplusondemand.com
prod23.tcplusondemand.com
prod25.tcplusondemand.com
prod26.tcplusondemand.com
prod31.tcplusondemand.com
prod32.tcplusondemand.com
​​​​​​​prod34.tcplusondemand.com
​​​​​​​prod35.tcplusondemand.com
Jul 27, 17:52 CDT
Software updates for v7 OnDemand are scheduled to occur Thursday, August 5th from 9:00 PM - 3:00 AM (CST) for the following groups. Some downtime is expected to occur while the updates are being applied.

group2.tcplusondemand.com
group5.tcplusondemand.com
prod01.tcplusondemand.com
prod04.tcplusondemand.com
prod05.tcplusondemand.com
prod06.tcplusondemand.com
prod07.tcplusondemand.com
prod08.tcplusondemand.com
prod09.tcplusondemand.com
prod13.tcplusondemand.com
prod16.tcplusondemand.com
prod17.tcplusondemand.com
prod19.tcplusondemand.com
prod20.tcplusondemand.com
prod21.tcplusondemand.com
prod22.tcplusondemand.com
prod24.tcplusondemand.com
prod27.tcplusondemand.com
prod28.tcplusondemand.com
prod29.tcplusondemand.com
Posted on Jul 27, 06:24 CDT
CloudOps Updates - 7.1.13.3 | Prod30 Aug 5, 23:00 - Aug 6, 02:00 CDT
Software updates for v7 OnDemand are scheduled to occur Thursday, August 5th from 11:00 PM - 3:00 AM (CST) for the following groups. Some downtime is expected to occur while the updates are being applied.

prod30.tcplusondemand.com
Posted on Jul 27, 06:27 CDT
Past Incidents
Jul 28, 2021

Unresolved incident: CloudOps 7.1.13.2 OnDemand Updates Cancelation.

Jul 27, 2021
Jul 26, 2021
Jul 25, 2021

No incidents reported.

Jul 24, 2021

No incidents reported.

Jul 23, 2021
Jul 22, 2021

No incidents reported.

Jul 21, 2021

No incidents reported.

Jul 20, 2021

No incidents reported.

Jul 19, 2021

No incidents reported.

Jul 18, 2021

No incidents reported.

Jul 17, 2021

No incidents reported.

Jul 16, 2021
Resolved - This incident has been resolved.
Jul 16, 10:16 CDT
Monitoring - A fix has been implemented and we are monitoring the results.
Jul 16, 10:06 CDT
Update - Two application servers have been placed back in rotation. Performance has stabilized and all companies should be accessible.
Jul 16, 09:57 CDT
Update - One application server has been placed back in rotation. Partial accessibility has been restored.

We are working to place the remaining two application servers in rotation.
Jul 16, 09:49 CDT
Identified - High memory utilization has caused application servers to become unresponsive. Application servers are being rotated and restarted.
Jul 16, 09:25 CDT
Jul 15, 2021
Completed - The scheduled maintenance has been completed.
Jul 15, 23:04 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 15, 21:01 CDT
Scheduled - Software updates for v7 OnDemand are scheduled to occur Thursday, July 15th from 9:00 PM - 3:00 AM (CST) on the following groups. Some downtime is expected to occur while the updates are being applied.

group12.tcplusondemand.com
group03.tcplusondemand.com
Jul 13, 07:30 CDT
Jul 14, 2021

No incidents reported.