I also don't think you have to disprove Godel's theorem to create an AI that thinks outside the box, however, I believe you would have to for it to be a rational agent. A rational agent by definition operates strictly within the domain of a formal system. Thus by definition, a rational agent is in direct conflict with the notion of "thinking outside the box."
Of course any simulation is implemented in some formal system, but that's not what I'm saying. I'm saying the inherent problem with a rational agent is that they maximize utility within a formal system -- that is what they do, that is their entire purpose. Once we start talking about rational agents that can "think outside the box", or think beyond the formal system in which they're defined, we're no longer talking about rational agents.
I agree with your belief that we may some day build AIs that think outside the box, I just don't see how it can be a rational agent. I think it will be much closer to a neural network, or probabilistic reasoning model, or some extremely "organic" or "fluid" device from which intelligent behavior organically emerges. Because if intelligence emerges from a strictly formalized framework, then Godel's theorems prove some very crippling limitations on what's possible within that fixed framework.
FYI there is definitely a connection between Godel's theorem and intelligence, though it may not be immediately obvious. I would highly recommend this as fun learning material: http://ocw.mit.edu/high-school/courses/godel-escher-bach/
Wait, wasn't a rational agent simply an agent that optimized a certain result given a set of knowledge about it and it's environment? I mean in engineering and physics, you do try to build the perfect something, but you also realize you will never ever build the perfect something. You'll never build the perfect computer, the perfect op amp, nor the perfect rational agent, just as you'll never find the biggest number, because there's no upper bounds on these kind of things.
Try to implement a rational agent (the AI or mathematically formalized variety) in software, and you'll see what I mean. You have to define some utility function precisely, and some algorithm to maximize it.
The problem with this is it won't really produce a "general intelligence" that can think outside the box, because it will always be maximizing some utility function defined in some rigid formal system. In other words, it will be completely unable to "understand" things outside of the formal system in which you define it.
Of course any simulation is implemented in some formal system, but that's not what I'm saying. I'm saying the inherent problem with a rational agent is that they maximize utility within a formal system -- that is what they do, that is their entire purpose. Once we start talking about rational agents that can "think outside the box", or think beyond the formal system in which they're defined, we're no longer talking about rational agents.
I agree with your belief that we may some day build AIs that think outside the box, I just don't see how it can be a rational agent. I think it will be much closer to a neural network, or probabilistic reasoning model, or some extremely "organic" or "fluid" device from which intelligent behavior organically emerges. Because if intelligence emerges from a strictly formalized framework, then Godel's theorems prove some very crippling limitations on what's possible within that fixed framework.
FYI there is definitely a connection between Godel's theorem and intelligence, though it may not be immediately obvious. I would highly recommend this as fun learning material: http://ocw.mit.edu/high-school/courses/godel-escher-bach/