A ZIP code is an identifier made of numerals, it is not a number in the mathematical sense. It does not make sense to multiply, add, subtract, or divide them - even though the numerals are assigned in apparently ascending order. There's nothing preventing a future decree from making one that starts with a Z, or a 0, or putting a dash in the middle of it. It is foolish to store it as an integer type.

You can cluster them for distance without storing them as an integer. Just throw them in a graph data type and compute edges having actual integral weights. The ZIPs themselves are not what you want to be doing your math on.