>But a US border guard looking at your passport and rejecting your entry based on that alone feels like overreach.

This is at least such a common practice, that certain countries issue their entry visas in such a way, that they can be removed from the passport. I'd expect issues entering the US, if I had an Iranian stamp in my passport.