It's a really cool demo and I'm impressed by the dexterity of the robot, however I'm a bit underwhelm by what's shown here in the sense that the speech and reasoning capabilities is just obvious to anyone who's been paying attention and has experience with GPT4. The function calling was great, but it had a very simple "world" to interact with.
It's really interesting to see it integrated with a robot that can interact with the world though. I think that what's really holding back the current crop of Gen AI is inference cost and speed. When we figure out to get thousands of token per seconds for cheap, I think we will be able to bruteforce many hard problems and actually start seeing amazing applications like this one (but in production rather than a cool demo).
I take your point about familiarity with GPT4 making this less immediately impressive - but just as an end to end demo it's absolutely mind-blowing how far we've come.
Can you imagine seeing this ten years ago? Moving so far in such a short time frame would have been unbelievable.
This gave me the "I'm actually living in the future" vibes that I always imagined I'd get from flying cars.
Yea of course this is just an LLM interacting through a very crude interface with some control algorithms, but I think it is amazing that we sort of have an approach for both ends of the complexity spectrum now: LLMs for high level, vague, common sense reasoning, and traditional robot control, planning, machine learning methods for the physicak execution of simple movements. We just have to gradually connect these two systems.
I think it's always going to be the case that we'll get to AGI through a collection of neural networks and symbolic logic systems and not through just a single neural network. For some reason a lot of people think that's "cheating", though.
The way I see it, a symbolic logic system is not an integral part to an AGI, but rather a tool that it can use the same way humans use tools.
Tools can be useful for many things, but tend to be inflexible compared to neural nets.
The way I see it, the main benefit of such tools come when the part covered by the tool is very repetitive and well defined. In such cases, algorithmic solutions may offer better precision and lower power consumption than neural nets.
Anyway, once most of these algorithms are actually WRITTEN by AGI's, the difference may cease to matter.
Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Using GPT-4V to describe a scene is also pretty simple. The most impressive things here to me are the speed of picking up the trash and the fluid passing of objects between hands.
It's unclear how general these movement policies are though. The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation.
I don't want to be too negative here though. I think it's a great demo and I can't wait to see more.
The Ok-robot demo shows that the technology for it to be fairly general is there, though no idea if figure one is using their technology or not. Simply being able to command a robot instead of moving a turtle with gcode is nothing short of astounding to those who aren’t deeply involved and tracking the sota progress in this area.
> Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking.
Yes. Compare "Put That There" (1979).[1]
> The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation.
Yes. Unstructured manipulation is hard. Structured robotic manipulation is pretty standard.
Picking isolated objects is a solved problem.
Here's a robot recycling sorting system, "Max-AI".[2] That's been in use for years.
San Francisco recycling uses those robots. So do many other cities.
(That's from "Bulk Handling Systems", a company which does exactly what their name says. Recycling and trash come in bulk, and their machines handle it. Shakers, magnets, screens, and vision-based air sorters do 95% of the sorting. The robots only handle the hard cases. This is the no-bullshit end of AI.)
I suspect it's less pre-trained than you think, the tweet is prefaced with a message that all actions are driven by neural networks indicating that it's probably adjusting for the objects, the environment etc.
I think it's true that it can adjust, somewhat. I expect that it could handle slight variations such as moving any of the items on the table around by a few inches, or adding or removing a couple of plates from the rack. However, I do not expect that it could handle larger variations like replacing the apple with a pineapple, picking trash out of the dish rack or the cup instead of the plate, replacing the cup with a coffee mug, or sliding the dish rack over to the other side of the robot.
I'd love to be wrong but I expect that if they had that much flexibility in their controllers they would have demonstrated it.
I'm sure there is definitely some limitation in this demo, but who knows, Google RT-2 exists after all. Enough shapes and object training will save the day, there's still objects me as a human would be new to handling.
This is cool. But why would the plate go in the drying rack? It is obviously dirty since there was trash and an apple on it. It should have been washed first.
This shows the real utility of the Groq low latency inference. That delay in responding makes it much less impressive (it’s still really impressive obviously!)
That delay will be eliminated very soon. IMO low latency natural voice conversations are going to be bigger than ChatGPT. It's going to blow people's minds when they can converse with these AIs casually just like with their real life friends. It won't be anything like Siri or Alexa anymore.
I’ll be thrilled when Siri can spell my wife’s name correctly after 13 years of continuous usage and explicitly training it in her name. Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada
Also my radio-trained voice is so generic a caller every week-ish assumes I am a bot, so I’m pretty sure the problem isn’t me enunciation or accent.
> Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada
Aye.
I was surprised this morning when it decided I had was talking about a "Mark of chain". 1/3rd of the time it hears "bedroom 100%" as "bedroom off".
When cooking dinner today, I asked for a "ten minute timer", it responded "for how long?" then confirmed my "ten minute minute timer".
Still better than Alexa, which kept telling us it couldn't find «kitchen» on Spotify even though we didn't even have Spotify.
And way better than the voice control on Mac OS Classic; back in the late 90s/early 00s, it interpreted 75% of my attempts to use it as "tell me a joke" (it wasn't even a good joke), and ignored 20%.
Can we stop this kind of misinformation? Training a model to map a token to individual letters are no harder than training a model to be fluent at English. Arithmetics with small number of digits are achievable as well. You can just try a small 7B model yourself. If you don't know where to start, try the mistral instruct v0.2, and this is how it goes,
> [INST] Spell out the following word letter by letter: margorczynski [/INST] m - a - r - g - o - r - c - z - y - n - s - k - i
> So, the word "margorczynski" spelled out letter by letter is: m-a-r-g-o-r-c-z-y-n-s-k-i.
The text between `[INST]` and `[/INST]` is the input. The text after `[/INST]` is the output.
The easy solution is to create an additional dataset that is token aware. I.e. you take 1% of the dataset and take random tokens and split them into smaller tokens while expecting the same answer on the character level. This should force the model to learn multiple token representations of the same character strings.
demod this. The ability to interrupt the language model is very cool, However, i notice, it failed to move onto the next slide often. It could never get to the final slide without explicit mention to go there, and when i got to the last slide, i asked to go back to the first slide, it would say "ok lets go to the last slide" everytime, these are probably more control issues than language model issues but i thought id point them out, just in case.
Absolutely. People on X keep making the mistake of assuming cloud / network latency is the problem here.
The vast majority of America is within 10ms of a data center. That's nothing.
The current challenge for most interaction is ASR -> prompt processing latency. This will be improved with multimodal models on specialized hardware like Groq.
The bottleneck here is the multimodal vision processing, at least if my experience building this kind of thing is any indication. Afaik Groq has not demonstrated the speeds they have for this. (Obviously they'll be better than OpenAI, but it still may be slow enough to leave people disappointed.)
IME I can get about 0.2s to get the first chunk from Mistral (i.e. Mistral API, using Mixtral model (`mistral-small`), not Mixtral on Groq) (and note the Mistral sends larger chunks, unlike ChatGPT which sends individual tokens)
and another 0.6s or so to get first voice chunks from PlayHT
measuring STT latency is harder, I need to implement a local VAD model first to properly measure it, but I think it's on the order of 0.5s
So this has nothing to do with Groq, really. ChatGPT is just slow (too slow for realtime voice communication).
Unless the only thing you want to do with the robot is talk, you need to do a lot more reasoning and execution planning first (= multiple LLM round trips; tool calling) before you even know whether talking is the correct action to take. So the naive time-to-first-chunk estimate will be way off.
Which is so cool because it's an evolutionary/language thing.
Why do we add junk words while we think? I think it's probably because we're social animals, we want to hold that person's attention as we think as periods of silence are likely to make them become disengaged. But who knows really.
The ability to translate between text to servo movement is unreal, and it looks like gpt4 vision + whisper are heavily used. They're also using the term "reasoning" which is... new.
Can you call this an AI wrapper company? Kinda! The medium is a little different than an app, of course.
Lots of amazing applications of AI even if frontier AI development froze today.
yeah this was super impressive. If this is at the point where you can put an arbitrary object in front of it and ask it to move it somewhere, that's going to be huge for industrial automation type stuff I'd imagine.
I do wonder how much of that demo was pre-baked/trained though. Could they repeat the same thing with a banana? What if the table was more cluttered? What if there were two people in the frame?
Knowing as many people in the robotics space as I do, I suspect the demo may not be completely "pre-baked" but it is almost certaintly highly selected. Often they'll try the demo many many times until they get a clean run-through without mistakes. The circumstances are also likely pretty idealized, like they pick objects and settings that they know it performs well in.
Not true. Boston Dynamics has been demoing long and complicated single take videos involving walking, running, jumping and manipulation, in less than ideal conditions.
This humanoid form plus the voice really gives of a different feeling than the pure chat version. I think it will feel even more profound if they can add eyes and eye contact. Imagining demoing this to a random person.
The speech to servo movement is impressive as others pointed out. What strikes me as amazing is the speed with which it is performing tasks that require dexterity. This is the first object manipulation robot demo I have seen that didn't require speeding up the video for it to look "natural".
I am 100% convinced that the demo is partly fabricated, or at least very far from the robot’s capabilities. The inference part was probably true, but the dexterity would put them waaaaay ahead of what robots are capable of, if not programmed finely for a specific task. Industrial robots obviously have that kind of dexterity and precision because they’re specifically programmed for a task. General purpose robots however are nowhere as close to this level of accuracy or fluidity in their movements.
OK-Robot is super impressive, but there is a huge difference in the manipulation ability OK-Robot shows with a simple 1-DOF gripper and parallel-axis arm to pick up objects and place them largely on flat surfaces, and the human-analog arms and fingers of Figure 01 picking up a plate in both hands and placing it in the right slot in a drying rack, or dropping the apple into a person's moving hand.
It would be absolutely amazing if they really are at that level of manipulation in general, and it would put them vastly beyond what anyone has been able to do date. However robotics demos have a great history of being a mix of slight of hand (partial/full tele-op), heavily cherry picked, or tuned to a extremely specific example.
Because it's such a leap being implied by this video, it's reasonable to want significantly more evidence before believing they can do this type of interaction and manipulation in a general way. But even if it is heavily leaning say on imitation learning for this exact scenario, there are tons of potential applications for this level of capability.
Yes, there are two half’s to this. This is being able to understand the command which OK robot shows is possible, and then there’s the control of motors and actuators as you pointed out. Not needing a team of engineers to laboriously write gcode to manipulate motors is part two and is the other astounding thing here. I haven’t been following this sort of thing, so I have no idea what previous state of the art was, so I can’t judge how much of an advancement it really is.
How though. It’s probably just predefined actions that are triggered by the LLM output. At the same time it would be impressive if the LLM determined the right function to call in real time, was able to deal with the ambiguity in placement of the garbage and bonus points if it could do that in a scenario that wasn’t hardcoded to exactly standing behind that table in that spot.
Why does it then after performing an action or whole plan return to the "default" position with the hands in that strange stance? Looks kinda like there is some "hard coded" flow that simply uses the LLM(s) to perform actions.
Is it just me or are people easily impressed by these robot demo that moves nothing like humans and doing the simplistic tasks like passing a large object. The freaking Apple is within his arms reach and he asked to pass the Apple. I almost laughed out loud.
I wish I was a 5 year old that didn't actually know how infeasible and useless this all is, and could just be positive about the future for once.
But humans cant even figure out you can't operate an "imaginary number go up" underneath the basic human requirement of rent for shelter, there is no way they can make this technology useful or affordable or reliable or good
I don't understand this negative sentiment. We are finally making progress on real physical robot labor. This has nothing to do with "imaginary number go up", this is about more labor being available to assemble, clean, transport, cook, cut, harvest, build, etc. i.e. things that really matter and are the bottlenecks to our livves being more comfortable.
This stuff is not infeasible, these robots can be built for the price of a car. There is still a ways to go from this to androids, but it is legitimately technologically feasible now.
Fully agreed. This is going to turn into novelty nonsense, like some fashion designer will have a robot runway event and it'll drum up a bunch of stupid press. Then the military will take the rest over into advancing drone technology.
The most impressive part of this demo, to me, is the robot "seeing" and picking up objects with human-like appendages. I must have missed something, but I was under the impression that this was very hard. As I understand it inverse kinematics is pretty hard - did they solve it with NNs?
OpenAI's TTS does this. You can hear it in regular ChatGPT's voice mode (which this demo is based on, it uses the same animations on the robot's face). It will also sometimes randomly hallucinate syllables or whole nonsense words, although that is rarer.
Is there a setting for this in the ChatGPT app? I have never once noticed it produce an "uh" or repeated syllable like "I... I think I did pretty well."
Really? Have you used it much? I haven't used it a ton but it definitely says "uh" and has various other artifacts. Maybe they have improved it recently but it was quite obvious when I first got access. Or maybe some of the voices are more prone to it than others.
The naturalness of the speech is extremely good, though.
It sounds so human, a person would also stutter at an introspective question like this. I wonder if their text to speech was trained on human data and produces these artifacts of human speech, or if it is intentional.
I believe it's Eleven Labs API with the Stability setting turned down a little bit. It is definitely trained on human speech and when you use a somewhat lower setting than default, it will insert those types of natural imperfections or pauses and is very realistic.
I'm not sure when OpenAI added them, but you can hear similar things when using the ChatGPT voice mode on iOS. Sometimes it feels almost like a latency stutter and other times it feels intentionally human.
I use ChatGPT voice a lot, and it is prone to this exact type of stutter. I don’t think it’s intentional. I think there are certain phonetic/tonal linkages that are naturally “weird” (uncommon in the training corpus) and that AI struggle with them. Why this struggle manifests as a very human-like stutter is a fascinating question.
Yes, perhaps the demonstration can be "demystified," but I can't resist but be astounded by the robot. A few years ago this was unimaginable and only seen in science fiction movies.
Someone, please ask OpenAI to stop artificially dumbing down ChatGPT by adding "um" to the audio output. I get that it is supposed to make it more human-like or something, but every time I hear it do that, I cringe and feel sad for humanity.
What if it just inserts filler words when the text generation is too slow, to make it sound more natural. It's exactly what people do when they're thinking about what to say next.
If they are using Eleven Labs, this is just the Stability setting. Turning it down will make it more realistic and closer to the training data. That is what causes the pauses and imperfections.
You can sign up and use their Voice Lab for free or maybe a few bucks and experiment with the slider for Stability and the other setting.
In my opinion, turning Stability down just a little bit to demo extremely realistic speech is a no-brainer. They could have turned it up and made it ultra-smooth, but that makes no sense. Why make your robot demo less realistic deliberately?
Question is how much this is cherry-picked as we all remember the "demo" of Bard. It would be nice to see it thrown into some random environment and then asked to do stuff, otherwise this has little value aside from marketing.
One of my colleagues predicted, when ChatGPT was first released, that AI would reduce the value of knowledge work relative to manual labour, but I disagreed as I think the main thing holding back robots from replacing many manual labour jobs at the moment is the difficulty communicating with them. I argued that ChatGPT indicated that we were not too far away from being able to tell a robot to pick the apples from a particular row and put them in the green barn as the usual red barn was being painted today. This video suggests that I was right, and I in fact suspect that such manual labour is a more realistic type of work for AI (with robots) to substantially replace first than most knowledge work, where I think it will remain as an assistant for some time.
Assuming you already have a robot, it is going to be easier to teach it to clean your toilet than to make it write a scientific paper that is worth publishing.
The singularity is nigh! We must work towards it so humans can be free from the drudgery of work and can work towards whatever their heart desires! Join us on our quest for Digital Salvation.
after that faked google gemini AI video from awhile back, I've got a healthy dose of skepticism about these next-gen demos. obviously they've done a lot though, kudos.
One of the remarkable things about LLM IMO is its ability to perform something akin to abductive semantic reasoning, which this is an example of. It inferred probabilistically from the semantic context that’s the appropriate action. This is an absolutely remarkable ability of LLMs that IMO is the most important capability they bring to the table over all prior AI techniques.
(All usual caveats of stochastic parrots and non reasoning apply)
It's really interesting to see it integrated with a robot that can interact with the world though. I think that what's really holding back the current crop of Gen AI is inference cost and speed. When we figure out to get thousands of token per seconds for cheap, I think we will be able to bruteforce many hard problems and actually start seeing amazing applications like this one (but in production rather than a cool demo).