There's a big difference between "would you like to go out for a drink?' and "I want to have sex with you". One poisons a working relationship, the other doesn't (if done with tact).

Basically, if you wouldn't do it in front of your mom, don't do it to a colleague.