I hope this will allow us to contribute toward fixing some of the longstanding issues. One that I'm still surprised hasn't been addressed is the way that it grants irrevocable admin permissions for the IAM security principal that created the cluster. And it does it invisibily–there's no way to see what IAM role or user has system:masters since the permission doesn't appear in the configmap or any other user-visible configuration. Last I checked you can't even see it in the dashboard and have to make note of it yourself for future reference.
Its even worse than you describe, its by name. It doesn't have to be the same IAM user, that user just have to have the same username as the original creator. Support told me this when we got locked out of our cluster and the creator had moved on to other things.
Yeah, unfortunately for security minded folks the unique ID of the IAM security principal isn't used instead of (or in conjunction with) the ARN. Or perhaps fortunately if you are blindsided by a security setting that makes you wonder how thorny it is on the backend if this shipped to production in the first place. I experienced something similar. We tried recreating it on a whim thinking it wouldn't work but needless to say everyone on that call was surprised.
I know they did (do?) some weird stuff exposing the control plane with VPC Endoints that seem like some kind of weird hybrid of AWS-managed endpoints and privatelink. I always wondered how scrappy the team had to be at the beginning when they were still pushing ECS hard.
Agreed. This is extra bizarre considering their own documentation on the purpose of IAM unique IDS states:
>"However, every IAM user has a unique ID, even if you create a new IAM user that reuses a friendly name you deleted before. In the example, the old IAM user David and the new IAM user David have different unique IDs. You can create resource-based policies that grant access by unique ID and not just by user name. Doing so reduces the chance that you could inadvertently grant access to information that an employee should not have"[1]
> When you create an Amazon EKS cluster, the IAM entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster's RBAC configuration. This access cannot be removed and is not managed through the aws-auth ConfigMap. Therefore it is a good idea to create the cluster with a dedicated IAM role and regularly audit who can assume this role. This role should not be used to perform routine actions on the cluster, and instead additional users should be granted access to the cluster through the aws-auth ConfigMap for this purpose. After the aws-auth ConfigMap is configured, the role can be deleted and only recreated in an emergency / break glass scenario where the aws-auth ConfigMap is corrupted and the cluster is otherwise inaccessible. This can be particularly useful in production clusters which do not usually have direct user access configured.
> Yes, you can install EKS Connector to connect your EKS Anywhere cluster to AWS EKS. EKS Connector is a software agent that you can install on the EKS Anywhere cluster that enables the cluster to communicate back to AWS. Once connected, you can immediately see the EKS Anywhere cluster with workload and cluster configuration information on the EKS console, alongside your EKS clusters.
My guess is that this is similar to Google's Anthos Connect agent https://cloud.google.com/anthos/multicluster-management/conn... that initiates the bi-directional connection from the on-prem to cloud (as opposed to opening a firewall rule for the cloud to access on-prem). That way, the cluster shows up and is manageable through a UI that is hosted on AWS Console.
Another interesting point is that I don't see a fee or price for installing this to your local VMware set up. I am not sure if it has to dial to AWS APIs to work (my guess: not really). So it can be treated as just another Kubernetes packaging that runs the AWS' distro by default.
Props to AWS for putting this out there and doing it in the open. I look forward to reading on how it compares with other Kubernetes installation methods given this doesn't manage the on-prem Kubernetes for you.
It is by far the easiest installation method I’ve encountered for vmware/vcenter. I plugged in a few variables (Datacenter, VSAN storage pool name, Network name, and resource pool) and in 30 minutes I had a fully production ready cluster installed with all the vsphere cloud provider / csi plugins / etc. installed also.
AWS is lagging behind on managed K8s on-prem for air-gapped environments, consider Google had GKE on-prem (now a part of Anthos) long ago, and Azure offered AKS on Azure Stack HCI. Finally AWS caught up, feature parity between the big 3.
Adoption will be interesting to see.
I've worked on gravitational (gravity / telekube) flavoured K8s distribution designed for on-prem data centre deployment models for the past few years. Those customers will eventually move to 1. managed K8s (good on them) or 2. continue to self-host K8s to satisfy their building one's own PaaS goal lol (in that case, it's better be managed K8s' on-prem variant from GKE/AKS/EKS). For the same K8s solution's other model - BYO K8s, I see rapid adoption of AKS (vs GKE, EKS is the dominate from beginning as it was the first supported).
On gravitational: Recently I heard gravitational has shifted focus (and rebranded) to teleport (secure cloud access gateway - used to be a part of gravity), around their Series B funding round, after all, investors are looking for a business model that can bring them profit (and exit). On the other hand, `gravity` - `pull the stack from cloud` - where the name came from - which addresses specify niche market does not seem to have generated much revenue over the last couple of years.
>"I've worked on gravitational (gravity / telekube) flavoured K8s distribution designed for on-prem data centre deployment models for the past few years."
This sounds interesting. Can you say what K8S distro this is?
The existing appliance model will continue to be maintained (until upstream EOL it - or when the support agreement ends).
In the meantime, to address the same requirements from customers who wants some control (K8s stack, VMs, networking and storage are theirs), managed K8s' on-prem distribution may be the answer, I'll be exploring EKS Anywhere see how it works and see if it's a viable option to replace the appliance model moving forward.
The big selling point for k8s vs ECS is that it (theoretically) allows you to lift and shift your cluster to another cloud provider without too much pain because load balancers/storage/etc sit behind an abstraction layer. When you are AWS that's not really something you're even going to consider, so you may as well build with the underlying primitives because its absolutely guaranteed no one is ever going to ask you to migrate to Google Cloud.
ECS is 3 years older than EKS (2015 vs 2018) and a lot of AWS services launched (or planned to launch) during that time which is a very valid reason lots of customers picked ECS.
We've been running on ECS for 6 years now (basically within a few months of GA). When EKS launched we talked about maybe migrating over but ECS just feels more first class wrt cloudformation and stuff. It makes sense that internal teams rely on it.
Literally everything is a Lambda nowadays it seems, so it seems like it's just Lambda -> ECS -> -> -> raw EC2 unless you're building directly on bare metal.
to me the 4 main benefits of EKS is
1. managed control plain.
2. ebs storage integration.
3. amazon vpc network integration.
4. IAM integration.
EKS anywehere provides none of these, at best maybe it is providing similar networking behaviour, but that is actually happening with a 3rd part CNI, so even if it is consistent which i am no convinced it is, i don't need eks anywhere for it at all.
I would have expected it to be some sort of system that would let me use my EKS master to run nodes anywhere
For what it's worth, the analogue to Anthos is AWS Outpost.
But you could argue Outpost still has a more clear name for what it is, although I wonder if they actually thought through the definition of the word: a small military camp or position at some distance from the main force, used especially as a guard against surprise attack.
In 8-12 months EKSA will either be very successful or will die, because it’s up against incumbent on-prem k8s so teams will either quickly see the benefits or it’ll quickly be dismissed as useless.
The target market seems to be infrastructure admins who have on-prem VMs and an AWS account and don’t like running their own K8s clusters. No idea what that market looks like in terms of size or interest in something like this, but I’m sure AWS did their homework.
A cloud-hosted AWS console changing configs on an on-prem vSphere cluster doesn't sound like something that's going to appeal to a broad audience to me. There's also the option of having your on-prem EKSA cluster have a hard dependency on the cloud AWS IAM. Youch.
I'm not convinced the target market is people who run their own data centres, those people are highly unlikely to migrate the control plane to AWS. To me the big target markets here are:
* People running compute on AWS, but content distribution on their own hardware co-located at ISPs, those CDN nodes need to have the software running on them managed, and I can see the appeal of doing so with the same control plane everything else uses.
* Organisations running heavily distributed compute, either for the purposes of latency or resilience. Many large retail chains have small remote nodes running the PoS systems at each site so that they can continue making sales and tracking inventory in the case of a central outage. Again, seems like it would be nice to just treat those nodes as an extension of the central servers on AWS.
* (Somewhat more dubiously) IoT providers. AWS have a couple of services which are much better suited to managing IoT devices, but I could see some arguments for treating an IoT hub as a k8s node and just deploying new containers to it when updating.
My company committed to AWS-native EKS, didn't want to deal with the management headache of on-prem or self-deployed
Kubernetes in AWS/GCP, presumably AWS support is pretty good, DevOps is happy so far. One anecdotal win for Amazon.
I'm not understanding the question. EKS Anywhere installs EKS-D on-prem and Anthos installs whatever the GKE distro that Google maintains onto your on-prem hardware no? I didn't think you could bring you own distro with either provider. I thought the whole point was to have somewhat unified environment between both sides of a hybrid cloud.
a slightly different way to view it is that there is a class of customers that needs to do something re: kubernetes on-prem and in-cloud. in an ideal world they would like to have someone solve all of those use cases for them, preferably in a way that is consistent and has all of the icky operations delegated away.
also, I didn't downvote you but your comment was uncharitable to the people building eks anywhere and was basically conspiracy theory. iirc, it was something to the effect of this product making people hate on-prem and ultimately moving to the cloud.
thanks for the insight; I didn't think my comment would be interpreted that way but I can see how it might've been so I appreciate you shedding some light on that.
My intent was to call into question how Amazon would benefit from building an on-prem EKS solution. I had questioned whether or not they hoped it would lead to more cloud hardware sales, or whether or not selling support for this (which they clearly call out as an offering) would be the primary benefit for them.
I didn't think it was unfair or conspiratorial to question this, but I can see now maybe people interpreted that as "zomg amazon is trying to kill on-prem!!" FUD when, really, I was just genuinely asking because it seemed out of character for them to me.
so caveat this with the fact that i used to work at aws, but one thing aws is really good at is listening to customers and reacting. a few years back aws messaging was very much so single-cloud and no on-prem. but after listening to customers, there’s been quite a shift to prioritizing them through things like outposts and eks/ecs anywhere. even for something that’s effectively free like eks-a, it can drive revenue by keeping customers all-in with aws and helps with brand loyalty and trust.
