Suppose you can distinguish 16 colors. Somebody else can distinguish only 4. To accommodate the sees-only-4-colors person, you need to make sure that game elements are not differentiated by colors that look the same to the sees-only-4-colors person. One way to do this is to choose a color palette having only 4 colors and designing the game to still make sense that way. Also make sure that sees-only-4-colors can distinguish the 4 colors you chose.
I believe the red-green blind means you can’t distinguish between different shades of red and green ., not that you can’t see red and green.
So you’re safe as long as you’re not using multiple red and green shades