Here you can find out what’s up with GuidedTrack. We aim to post an update within fifteen minutes after the start of an incident. Please forgive us when the wording is rough. We try not to panic, but when the machines go wild, we don’t have time polish our prose. If you have questions or comments, we invite you to join the GuidedTrack community Slack or email us at hello@guidedtrack.com.
GuidedTrack is behaving well. We’ve installed a new SSL certificate.
Our SSL certificate has expired. You won’t be able to access GuidedTrack until it’s renewed. We’re working on it.
GuidedTrack is behaving well. The infrastructure change appears to have gone almost without an issue.
GuidedTrack is behaving well so far.
We’re about to make an infrastructure change that will disrupt things if it goes wrong. It could have effects for a few days. If you find anything misbehaving, please email hello@guidedtrack.com.
It looks like things are back to normal.
We’ve migrated everything to our new setup. Emails have been going out successfully for about a day and we haven’t seen any further suspicious email activity.
We’re working on improving our detection tools to make this sort of incident less likely in the future. We’ll publish a postmortem once we’ve completed the most pressing tasks and had time to reflect. If you want to get our analysis, subscribe to our mailing list by sending an email to ~guidedtrack/releases+subscribe@lists.sr.ht.
We’ve set up a different IP address to send our email from and are now routing emails through it. It looks like email is being delivered successfully. We’ll monitor things and start migrating settings.
Emails from GuidedTrack programs are being held up by our email sending provider. So if you’re testing a program and not getting email from it, that’s why. We’ll update this page once the problem resolves.
Everything else should be behaving normally.
Everything is behaving normally again.
We are still looking for a way to bypass the index rebuilding operation. We are considering setting up a new server, so new runs can be started, but old ones will keep failing until the reindexing completes.
What happened is that we caused an index rebuilding operation to start, which is blocking everything. The operation is going to take up to nine more hours to run. We’re looking for a way to bypass it. If we find one, GuidedTrack is going to be available again sooner.
Programs are not loading. We’re looking into it.
GuidedTrack is behaving well. The Q&A site, https://answers.guidedtrack.com/, is back, too.
GuidedTrack is behaving well. The Q&A site, https://answers.guidedtrack.com/, is down. We’ll look into it soon, but we don’t consider it on fire since the site is rarely used, as far as we know.
The site is behaving well. The CSV export should work for data of all sizes again.
The site is mostly behaving well, but the CSV export is broken for large files (ie. anything that takes longer than 30 s to download). We’re looking into it.
The site is behaving well. http://guidedtrack.com is working again.
The site is behaving well if you access it at https://www.guidedtrack.com. http://guidedtrack.com, which used to redirect to https://www.guidedtrack.com, has stopped working. We’re looking into it.
The site has been behaving well for a while. The repair is finished and probably the performance is the same as before.
The site is mostly behaving well. The repair procedure mentioned in the previous entries (automatic indexing operation) is still going on. All estimates were wrong.
Amendment to the previous entry: It looks like it’s going to take up to two hours.
The site is mostly behaving well. Only the pages that list answers to questions are slow or timing out. That’s because we made a change to the database that had unforeseen side-effects. We’ve started the procedure that should repair the problem, but that’s going to take up to an hour to complete.
The site is behaving well again.
We thought we had gone through the steps to renew the SSL certificate, but for some reason, our provider was still configured with the old certificate, which expired. We’ll figure out what went wrong and make sure it doesn’t happen next time.
The GuidedTrack website (including program pages) is not available due to some issues with our SSL certificate. We’re working on it!
The server is stable again.
The server is having some trouble, which results in some programs data not being able to load or to sync their data, and possibly some timeouts. We suspect the connection with the database is poor at the moment but we’re still investigating.
The server is stable again.
The server is having some trouble, which results in some programs data not being able to sync their data and possibly some timeouts. We suspect the connection with the database is poor at the moment but we’re still investigating.
The problem has resolved itself. I’ll find out how to switch to the backup database in case this kind of problem escalates.
The server is having some trouble, which is causing programs not to load and ‘no internet connection’ warnings. We’re working on it.
What is happening is that a backend database has many cleanup operations running at once and those are slowing down everything else. Unfortunately, we don’t have the level of access to the database (it’s a SaaS) that would allow us to easily abort those cleanup operations.
The site is behaving well again.
The server is having some trouble, which is causing some programs to be unavailable. We’re working on it.
The site is behaving well again.
The server is having some trouble, which probably affects a few users. We’re working on it.
The DNS problem appears to be fixed. Everything else continues to work normally.
The server is up, but there seems to be a problem with DNS. When you type ‘guidedtrack.com’ in your browser, you might end up on a page that says: ‘guidedtrack.com is a totally awesome idea still being worked on. Check back later.’ Or it might seem to load forever. In either case, go to https://www.guidedtrack.com and it should work as usual. We’re looking into restoring the previous behaviour.
The site is behaving well. The problem from the previous entry still exists, but all the embedding pages it affects have been fixed. If you run into it, let us know and we’ll help you fix your embedding pages.
The site is behaving well, but there is a defect, which affects a few embedded programs (probably not yours).
If a program is embedded the old-fashioned way with a numerical ID –
<div class="guidedtrack program_container" id="1234">
– *service fails. We’re thinking about how to address this. It’s unlikely we’ll fix it before Monday (28 Nov).
If your program is affected, you can work around the problem by replacing the numerical ID with the program key. How to find the program key? Go to
https://www.guidedtrack.com/programs/<numerical ID>/
and click on Preview. Now you can copy the program ID from the URL your browser has navigated to, which is in this format:
https://www.guidedtrack.com/programs/<program key>/preview
The site is behaving well. Traffic from the study appears to have subsided.
We’ve unpaused the study. We’re monitoring our systems, ready to react if the additional traffic starts to affect performance.
The site is experiencing some performance issues due to a large study, which has been paused.
The site is behaving well. The defect described in the entry from 2022-10-08 is fixed.
The site is behaving well. We have fix for the defect described in the last entry, but not released it yet.
This status site is now available at https://status.guidedtrack.com.
The site is behaving well, but there is a defect, which program authors are noticing.
Currently the first Preview, Split or run after changing a program fails. It says: ‘Sorry, something went wrong.’ We’ve found the reason and are going to deploy a fix by Monday for the latest.
You can work around the problem by clicking Preview/Split a second time or refreshing the run page. From then on it works until the program is changed again.
We don’t log all defects on this site. But this one is affecting all active program authors.
New Relic now reports their service has come back to normal. This means GuidedTrack systems are working as expected.
GuidedTrack is operating well, but our monitoring infrastructure, provided by New Relic, is experiencing service interruptions. This shouldn’t directly affect GuidedTrack, but it prevents our team from investigating performance issues, should they arise.
As long as we don’t experience a sudden spike in traffic, the site should work fine.
More insight into the incident on New Relic’s side here: status.newrelic.com
The site is behaving well.
The status site (= what you’re looking at) should also look as before and show https://guidedtrack.github.io/gtstatus in the address bar. Richard is done trying DNS settings.
The site is behaving well.
If you’re wondering why you ended up on status.littlepluses.com when opening this page: Richard is trying out some DNS settings so that he can give the person who takes care of the GuidedTrack DNS precise instructions on how to set up status.guidedtrack.com.
The site was unstable for five minutes after I changed the configuration and restarted the server. Now it is behaving well again. We’re working on something to make server restarts less disruptive.
We’ve reduced the number of resources back to the usual amount. The site is behaving well.
The servers are running fine.
We’re planning to change how much resources are allocated to the site between 06:00 and 07:00 UTC. If you notice anything more than a few glitches, please let us know. Of course, we’ll be watching the site’s performance, too.