This seems to be an excuse to start looking for a new job, depending on how your company rewards high performers. Since everyone around you is unproductive, this suggests that there is not really a competitive environment for performance bonuses. No one wants to be in that 20% that does 80% of the work without some payoff at the end.
That's the thing, I'm self-employed and I'm the only one working on (highly experimental) open-source software development tools. It's not so much that there are unproductive people around me, it's that there are no people. My real-life friends are mostly doing their PhDs and none of them are coders/developers.
I quit a fairly toxic job last summer. For about six months, I worked by myself on my own projects. However, I felt my productivity begin to slip, and frankly, not being around people began to weird me out. I'm now in the current batch of hackerschool.com, and it's great. I'm still pretty much working solo on my own project, but it really helps to be in a space surrounded by fellow, productive nerds.