We are now closing this incident. Thank you for your patience.
Posted Mar 15, 2024 - 14:53 PDT
Update
We have completed the hardware and network operations for this incident. We are keeping this in monitoring for 24 more hours to allow all changes over the last week to settle across the platform. At the end of this monitoring period we will be closing this incident as resolved. Thank you for your patience.
Posted Mar 14, 2024 - 14:35 PDT
Update
We made several changes last night and this morning to move traffic to new endpoints. We will have more maintenance windows in the coming days to move more traffic and increase throughput. Until then we will continue to monitor these changes. Most devices should now be up to date with minor delays to portal reporting.
Posted Mar 12, 2024 - 14:31 PDT
Monitoring
We have made the switchover with our software and hardware updates and are monitoring the issue closely. We will update this message 3/12/24 9:00 AM PST.
Posted Mar 11, 2024 - 23:03 PDT
Update
We have completed the upgrade and cutover of our network devices. We see an improved infrastructure performance. The system is still processing the data backlog from the original issue, and we expect this to continue through the weekend. We will continue to evaluate the system through and decide on any other further action once the backlog clears.
Posted Mar 08, 2024 - 13:35 PST
Update
We are continuing to make progress after doing additional testing and are actively upgrading specific infrastructure with our provider. That action requires swapping of equipment and other configuration information which has been going forward with an urgent level of support over the weekend between Stem and our provider. We will provide daily updates as we go forward.
Posted Mar 05, 2024 - 16:53 PST
Update
STEM / Also Energy support and engineering teams have discovered network congestion in one of our data pipelines. This is causing delayed data ingestion for a subset of our fleet. We are working on steps to reduce congestion and increase bandwidth in the coming days. These steps should reduce network latency and will help with timely ingestion of data across our fleet.
Posted Mar 04, 2024 - 10:42 PST
Identified
STEM / AlsoEnergy support and engineering is diligently working on troubleshooting a delay in data ingestion affecting a subset of sites with Acquisuite loggers uploading data to our monitoring platform. We have identified the issue and are working closely with our 3rd party vendors to resolve it as soon as possible.
Posted Feb 29, 2024 - 11:50 PST
Investigating
We are currently investigating this issue further.
Posted Feb 27, 2024 - 12:27 PST
Update
We are continuing to monitor for any further issues after the scale-out undertaken late Friday evening
Posted Feb 19, 2024 - 17:06 PST
Update
The underlying service contributing to latency is scheduled for scale out today, and we are continuing to monitor.
Posted Feb 16, 2024 - 13:02 PST
Monitoring
Backlogged devices are checking in while in night quiescence for generation. We are continuing to monitor latency reduction.
Posted Feb 12, 2024 - 19:59 PST
Identified
The issue has been identifed, a workaround implemented, with good initial results for device upload cadence.
Posted Feb 12, 2024 - 18:13 PST
Investigating
We are seeing increase delay in some site data acquisition and ingest into the system . We are currently investigating the affected lagging sites.
Posted Feb 12, 2024 - 08:48 PST
This incident affected: LocusNOC (Data Acquisition, Data Ingests) and PowerTrack US (Obvius Ingest).