Any point on the shell, whether the shell has zero thickness or not, has zero mass, but a finite mass density is assigned to it. Any finite mass is spread out, so you cannot have zero distance to enough of its points to feel infinite force.