There might be an accuracy/speed tradeoff. mpv encourages skipping with keyboard shortcuts rather than clicking in a timeline, but I don't think it actually guarantees those shortcuts will skip the same time ahead.

It also might be pre-decoding to enable trick play (rewinding etc).

No tricks, mpv just jumps to the nearest keyframe, which allows near instant skips. If you want to do time-exact skips, you need to skip to the keyframe previous to your target time and then decode normally from there, causing a delay. I presume VLC does the latter by default and can be configured to do the former.

If you only want to go forward it's not quite as bad - in a lot of files most frames are skippable (B frames) and you don't have to decode those unless the target frame is in that group (miniGOP).