i work tech at a university that's impacted by this. while it doesn't impact me directly, many many other staff and instructors i know are heavily affected by this outage. the students are absolutely outraged, mostly because the university hasn't been providing updates as quickly as they'd like, but since the staff/admin are waiting on word from instructure -- and there hasn't been a lot from them, it just generally sucks for all of us.
this is really, really, REALLY bad. it's not great that names/emails/etc will potentially be leaked, but also private messages between students and instructors. and since many of the campus systems rely on canvas integration, things have pretty much ground to a halt a week before finals.
after they were breached on the 1st of this month, instructure had an announcement yesterday that "everything is great! we're good! hackers are gone! we've rotated our keys!".
Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.
Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.
It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)
---
Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...
---
(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
It's possible that Instructure's servers got compromised:
dig canvas.ucdavis.edu
[...]
;; ANSWER SECTION:
canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com.
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18
dig canvas.duke.edu
;; ANSWER SECTION:
canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com.
duke-vanity.instructure.com. 60 IN A 18.173.121.125
duke-vanity.instructure.com. 60 IN A 18.173.121.18
duke-vanity.instructure.com. 60 IN A 18.173.121.103
duke-vanity.instructure.com. 60 IN A 18.173.121.15
i did the circleci --> github actions migration for my job 1.5 years ago, and things seemed great... at first. at the time, we'd been dealing w/circleci's semi-regular (but thankfully short) outages for over two years, and we were excited to move to a more stable system.
From CircleCI here. A big effort and investment went into resolving those outage issues you're referring to. Results have been stellar for a while now. Here's the latest: https://status.circleci.com
Apologies for shameless plug... have you considered Buildkite? Our statuspage is a sea of green even as we hit 1.3b minutes/wk (GHA sits at 2.1/wk now). Much less maintenance overhead than Jenkins, more dynamic featureset. The trial is all-access, unlocks the full product and can be extended past 30 days. Real human eng on standby throughout.
funny, but if you read the mecha-hitler tech debrief, mecha hitler was a 'sycophancy' bug, a-la gpt4o, if you gave gpt4o all your edge-lord tweets, and told it to be funny back to you and connect with you. Probably not grok's default posture, just sayin
> Am I reading this right, possibly a $100k fine and up to 20 years in prison for a biological man posting a picture of themself dressing in clothing considered "female"?
yep, i believe you are. i wonder how they'll deal with men wearing scottish kilts? or women wearing pants?
this is insane and i can't see how this would even come close to passing in to law...
this is really, really, REALLY bad. it's not great that names/emails/etc will potentially be leaked, but also private messages between students and instructors. and since many of the campus systems rely on canvas integration, things have pretty much ground to a halt a week before finals.
after they were breached on the 1st of this month, instructure had an announcement yesterday that "everything is great! we're good! hackers are gone! we've rotated our keys!".
no. nothing is great. we are not good.
reply