Since you definitely need to let some IP addresses in, you can add a single or range of addresses specifically.
For instance, if I wanted to let only my development PC into the server and its IP address is 10.160.30.90 then I would add an Allow Restriction Rule for my specific IP address: