Order deny,allow Deny from All Allow from 184.108.40.206 Allow from 220.127.116.11
There is a way of checking the real sender address, though, because Varnish puts it in the HTTP header X-Forwarded-For.
A few additional lines make Apache check this HTTP header, too:
Order deny,allow Deny from All SetEnvIF X-Forwarded-For "18.104.22.168" AllowIP SetEnvIF X-Forwarded-For "22.214.171.124" AllowIP Allow from env=AllowIP Allow from 126.96.36.199 Allow from 188.8.131.52
This ruleset will work both for requests handled by Varnish (Port 80) and directly handled by Apache (Port 81).