They're just more efficient for certain kinds of workflows - both cheaper and faster.
If you're building for arm64 targets or android, this is a no brainer because it sidesteps the emulation requirements.
We've been offering arm64 runners for a few months now (up to 32 vcpu) with WarpBuild and our users love it.
In general, there are lots of inefficiencies in CI starting from the lack of visibility, debugging ease, performance, runner config customization, persistence, etc. We're out to solve it and make the world's fastest CI cloud ecosystem on top of existing CI providers with WarpBuild.
We at FlyCI offer macOS runners of GitHub Actions and have free 500 minutes per month for public repositories. Also, we are about to release FlyCI Wingman - an AI agent that analyzes and fixes failing builds. Feel free to contact us at contact@flyci.net and tell us more about your needs. We'll be happy to help you with a solution.
Second, I feel runs-on's analysis is a bit unfair to us. Ubicloud's x64 / arm64 performance and queue times above are as good as any (we deprecated the AMD EPYC 7502 line). For example, RunsOn arm64 queue times are 31|42 seconds and Ubicloud queue times are 17|24 seconds.
But the analysis says the following, "Be aware that Ubicloud has low CPU speeds and high queue times, which means they won’t necessarily end up being cheaper than competitors." I fail to see this conclusion from the numbers above. What am I missing?
The queue time for x64 appears to vary quite a bit still. But you are correct that it is much better than what it was a few weeks back, when the analysis was written (it was not unusual to see 60s+ spikes).
Also note that I specifically mentioned that "all third-parties are good on that front, expected [sic] Ubicloud (but that may change)."
Also agree that now that you removed the outdated CPUs (was still active end of May), the analysis should mention that you have OK CPUs by now. I will fix it.
For providers that are on this page, do not hesitate to write to me if you find the analysis outdated for your specific service.
Hey there, thank you for this update. It's your analysis, so you're free to mention Ubicloud as you'd like.
Still, you updated the above statement to, "Be aware that Ubicloud has a slightly lower CPU speed and somewhat variable queue times for x64 (but improving)."
Could you clarify what you meant by variable queue times?
According to the benchmarks, Ubicloud queue times for x64 were 18|44 secs at p50|p95 over the past month. I agree that our p95 number is higher. At the same time, RunsOn's AWS numbers are 31|36 secs. So I feel that driving a conclusion based on our p95 variance number for x64 is a bit unfair to Ubicloud.
Thank you for compiling this benchmark btw. Now, we'll follow it closely.
RunsOn is the only on-premise solution of the bunch, so it is single tenant, and as such it cannot easily have pools of 10s of machines on standby to start within 10s. So on queue time I’m comparing Ubicloud against the other SaaS providers, and anything higher than 20s is not especially great in that case. Plus you have high variance, which I personally consider an issue.
Doesn’t GitHub use Azure, being owned by Microsoft. So perf difference could be Microsoft azure vs AWS graviton offering differences. Or Grubhub using an older machine type that provides greater capacity at lower costs.
Also a bit weird that they now seem to delegate the image building to third-party vendors (first GPU images, now ARM). Would be nice to know how they validate base images before distributing them to everyone.
We're a small team (four engineers) and we purchased four Dell mini PCs to toss into our small office space. The break even compared to our AWS hosted runners (which were much faster and cheaper than the GH ones) was like four or five months, plus they cut the time again by over half. We have to deal with them if something weird happens, but it's been such a low time investment that it has been such a great choice.
Plus, who doesn't love having some new hardware to play with?
Not OP, but I went with MOREFINE AMD Ryzen mini PCs. They can be configured with a ton of CPU/RAM (including ECC). They're really compact, support 2.5 Gb ethernet, and are great to work with. A comparable Dell workstation with ECC RAM would've been significantly more expensive, and you have to deal with their business folks to purchase them (or at least I couldn't figure out how to buy them direct).
Wasn't my choice, but one wasn't enough so we threw a few more at it. So mostly for performance, but the redundancy is an added benefit. They were a pretty decent space, power/heat, performance, and noise ratio for us.
Our office is a single room with six desks, so space, noise, and heat are a larger consideration for us. I use a mini PC at home for a personal server as well, but that's just because used enterprise equipment is often a great deal.
Three years sounds like a reasonably good return on investment. At that point the teams either big enough to justify paying $$ for a hosted solution, or still cheaper to buy dedicated hardware.
You can also just rent dedicated servers from hetzner or ovh for far cheaper than AWS instances with more horsepower. More expensive than a dell mini in a closet. But you get datacenter Internet and power backup.
We're about eight months in at the moment, so here are some current stats:
- No OS upgrades (running Ubuntu 22 LTS), but regular update
- We've had a power outage, but not during the day, so when they rebooted, they went right back to action. Same issue with a network outage in the office.
- We have hit disk space issues due to not clearing all the data we needed to. A quick SSH and an rm or two did the job, plus another 30 minutes to an hour to update a cron to remove that portion of the cache
I see what you mean for sure, and there will undoubtedly be more things that come up at some point, but with the savings, it would take quite a few engineering hours per month to lose the savings at this point.
And I won't lie, when something goes wrong, it's a nice change of pace to deal with a server instead of code once and a while. Can't actually use that for calculations, but I don't hate when it happens so far.
if the break even was 4-5 months, it means compared to going to cloud vendors, in 36 months, the self hosted solution would have saved 7.2 times the cost!!
Even if within 3 years there needs to be OS upgrades, you can just e-cycle the old machines and buy brand new minipc's (with added benefit of increased performance) every time you need to upgrade, and you'd still be ahead!!
I considered this for my personal business but the minutes are stupid cheap and to pay off a Mac mini it would take me many, many years to break even. For a company with 10's-100's-1000's+ employees writing code and needing CI/CD the math will be different. I came close to buying a Mac Mini anyways, costs be damned, to get away from GH's slow runners.
At smaller scales, as long as your builds aren't stupud-slow, cloud providers should be cheaper just because they can utilize their hardware more (10x?) efficiently.
Totally agree, and I called out that my scale is small so cloud makes a lot more financial sense for me (especially as a solo founder, I don't want to manage more hardware than I have to already).
I get that, but you can run GH runners on your own hardware or someone else's. Switching to WarpBuild was literally as easy as linking my GH account and changing the running image from `macOS-latest` to `warp-macos-14-arm64-6x` and everything worked. Secrets and everything.
Ditto. I came for the cheaper Apple Silicon runners (before GitHub Actions had them available for free) and stayed for the faster runs across the board. I cut a 24 minute build down to 10 minutes just by switching over to WarpBuild runners. Not a shill, just a happy customer.
Can't agree more with this, I recently migrated my company to use WarpBuild and our CI got 50% faster, not even kidding. We were using self-hosted runners which were not as efficient as we thought so we spawned EC2 instances and connected them to the github network to be runners, but we had such an amount of issues that we wanted to go back to non-self hosted. At the end a tech lead of the company suggested using Warp and we could not be happier with the results.
Also the CEO Surya is such a great guy, we had a small networking issue that lasted for about two hours and they gave us credits for all the runners affected with that, and their customer support is blazingly fast.
> We were using self-hosted runners which were not as efficient as we thought so we spawned EC2 instances and connected them to the github network to be runners, but we had such an amount of issues that we wanted to go back to non-self hosted
For those hitting that same kind of issue (and you are many), you should check out my product https://runs-on.com. 1-click install, 1-click upgrades, and from 7x to 16x cheaper runners on AWS.
Brilliant idea!! I just dug through your docs. You significantly undercut GitHub on runner pricing, using your customers’ infrastructure for both runners and their orchestration, which is managed by your software, some cloudformation, and amis that you maintain and license in return for a small slice of that savings. I love it, and if our cloudops org ever threatens to charge back CI to our budget I’ll be a customer.
GitHub Actions is a joke. It baffles me that they've been breaking file permissions on upload for years and they haven't fixed it yet. God forbid you want to ship an executable in a tarball or save it for a later stage of your build.
Different needs, different tools!
I work on a open-source project, and not having to pay for CI/CD with a "simple" yml file configuration is simpler and easy to manage.
You can try FlyCI Wingman - an AI agent that analyzes and fixes your failing builds in GitHub actions. We're about to release it tomorrow, so feel free to subscribe and be among the first to try: https://www.flyci.net/ .
Assuming the question was reg. WarpBuild - we provide blazing fast caches with unlimited sizes. These still need to be downloaded to machines and setup on each workflow job.
We don't (yet) have support for pre-pulled repos on runners per customer. Are the large repo sizes despite fetching only the refs required for the specific context [1][2] which can help speed things up significantly.
(WarpBuild founder here)
We have had a lot of users evaluate multiple tools including ubicloud but chose us. Typically, they've mentioned our runners being faster, our caching is very useful, and queueing delays being lower.
However, my last anecdata was from a few weeks ago. You can try us out and take a call for yourself.
Our focus is on CI runners and efficiency and I think that gives us an edge on the devex. Ubicloud is definitely cheaper though.
I think I skipped over evaluating ubicloud because they don't have macOS runners. There was another place I liked but I needed macOS runners they didn't have them either.
> Larger runners are only available for organizations and enterprises using the GitHub Team [$4 per user/month] or GitHub Enterprise Cloud [$21 per user/month] plans.
At Zig Software Foundation we rent an aarch64 Hetzner server for about 100 euros a month and it has 64 processors. It can handle a lot more CI work than the GitHub provided runners for a fraction of the cost.
Hmm... this is neat. I develop on MacBook so ARM64, and I do use GitHub Actions, which can not be ARM64, but unfortunately Google Cloud Run doesn't support ARM64 images yet:
It's nice to know that we have an ARM64 fallback if our perfectly robust and cheaper self-hosting of GHA (gha-runner-scale-set* components) were ever to go down, I suppose, but I can't help but feel like I am part of a huge market segment that GitHub lost during the ARM64 runner absence. Seems a little too late given they have natively supported action-runners images in ARM64 for some time.
I've been using github-act-runner on top of a local Nomad cluster to run some of my builds on risc-v, arm64 & x86-64, it's fine, but a bit of a hassle to set up
For some reason I expected a "not-first-class" experience while using my own hosted runner but that was not the case. It works and behaves exactly as their own runners do. Maybe s/surprisingly/extremely would be more what I meant.
Just wanted to add, I loved your post on logging Nomad allocs using Loki. Was a huge help while I was setting up our clusters at $CURRENT_JOB
> For some reason I expected a "not-first-class" experience while using my own hosted runner but that was not the case. It works and behaves exactly as their own runners do.
Interesting, I've used GitLab extensively and hosted/self-hosted runners were trivial to use/run, so I would have expected nothing less from GitHub. They probably make a decent margin on the hosted runners at the cost of significant complexity, so if they can give that away to the orgs that will have their own (virtual) hardware anyway, with their own contracts and networking and etc., why not?
> Just wanted to add, I loved your post on logging Nomad allocs using Loki. Was a huge help while I was setting up our clusters at $CURRENT_JOB
Glad you found it helpful! If you ever want to chat about it, feel free to hit me up (contacts are on my website available in my profile).
I had an issue where a Rust application, which I had compiled on a Radxa ROCK 5B, would nor run on a Raspberry Pi 4; I had to compile it again for it. Both were running Ubuntu 22.04, 64 bit.
Why would that be and how would this affect these Arm64 compiles on GitHub?
If you compile a normal application for amd64, it will work on any 64 bit Intel and AMD machine, unless you start using specialized features, which wasn't the case in the above mentioned case.
This reminds me of one issue that I've encountered at Rustup. Early versions of ARM64 Pi OS actually use a 32-bit userland. As a result, this has caused user complaints since our installer sees a 64-bit kernel and will happily download a 64-bit binary (which will then refuse to run): https://github.com/rust-lang/rustup/issues/3307#issuecomment...
Maybe but I’d recommend checking out Depot.Dev for that, they expose Arm and x86-64 runners to buildkit/buildx. Not associated with them, just a happy customer.
it's a pain, you'll need to build the images on separate jobs then merge them into a multi-arch manifest. I moved to namespace.so for remote BuildKit builds a while ago and haven't looked back. depot.dev is also good, but pretty expensive
One thing that consistently bugs me is how upgrading from the lowest type of runners to _literally_ anything else renders your paid for included minutes useless. I do not understand who those wouldn't just be multiples of the base runner, much like how Mac and Windows runners are. Seems like the crediting logic is there, but knowing software, there is probably some accidental complexity causing this part not to leverage it. That said, it's very frustrating from a paying customer perspective.
The same goes for minute crediting, where splitting things up to run concurrently is actively discouraged because they individually all round up to the next minute. For example; have 3 concurrent tasks that run 1m10s? You're billed 6 minutes. I get that a run would be rounded up, but come on.
Sorry, my post was rather lacking context wasn't it. What I meant was that the vast majority of my work is now done end to end on ARM CPUs and even the bits that haven't been, like GitHub runners, are starting to make ARM an option. Customers have been asking Digital Ocean for ARM servers for years [0], and every time they do, DO just tell you to upvote the suggestion on their feedback site, but you can't, because they closed it [1].
edit for additional context: An ARM server on Hetzner costs roughly 1/10th that of an x86 one on DO with the same core count and memory, and in my tests out-performs it too.
We waited on this issue for 2 years before migrating off EKS Fargate to Managed Node Groups due to this very reason. In the end, it turned out to be much better of an environment anyway, because you didn't have to deal with Kubernetes oddities (like lack of Daemonsets) requiring anti-patterns and ultimately resulted in cheaper pricing due to better bin-packing. If you're doing K8s orchestration with Fargate, I highly recommend the switch.
(founder of WarpBuild - we offer hosted GHA runners)
This is a common issue.
A common pain point we see with users approaching some scale with self-hosting on k8s is that the k8s node autoscaling can become inefficient because of spiky loads.
We have a lot of users migrating off self-hosted setups using `actions-runner-controller` to ours because of this. Essentially, not having to deal with bin-packing is more efficient and concurrency, uptime guarantees are nice.
Sidecars are definitely a workaround but are hard to manage lifecycle for in conjunction with the primary container. This is now easier to do in 1.29+ with Sidecars officially supported via restartPolicy, but it was a colossal hassle prior to the advent of that. Also, we had noticed (maybe this has changed) that often logging frameworks were distro'd as DaemonSets (Helm, etc) and you'd have to shoe-horn the sidecar approach a bit.
They're just more efficient for certain kinds of workflows - both cheaper and faster.
If you're building for arm64 targets or android, this is a no brainer because it sidesteps the emulation requirements.
We've been offering arm64 runners for a few months now (up to 32 vcpu) with WarpBuild and our users love it.
In general, there are lots of inefficiencies in CI starting from the lack of visibility, debugging ease, performance, runner config customization, persistence, etc. We're out to solve it and make the world's fastest CI cloud ecosystem on top of existing CI providers with WarpBuild.