This is a roadmap item for Key Vault.  The solution will provide a private IP address within your VNET that maps to your keyvault instance. The Private IP will be accessible over ER, S2S VPN, P2S VPN.   In the short-term a potential workaround could be using AzFW as a TCP Broker.  AzFW provides a private IP facing on-premises (S2S VPN) and you enable service endpoints on the AzFW subnet and you white-list the vnet/subnet/azfw to have access to keyvault.   You can further whitelist the FQDN of KeyVault on AzFW as well.