This content originally appeared on DEV Community and was authored by DEV Community
TL;DR
When operating Azure API Management in an internal virtual network, already integrated with Azure Application Gateway, an upcoming feature AllowApplicationGatewayPrivateLink allows you to connect this configuration to another virtual network using Private Link and Private Endpoint.
Motivation
In a post I made beginning of February 2022 I linked a virtual network with limited address space - like in a corporate / ExpressRoute / SD-WAN connected scenario - to a Container Apps environment:
That setup works as long as ingress only goes from resources in Hub network to Spoke network. But if Container Apps needs to call (as in my target scenario) Azure API Management - calling from Spoke network into Hub network - that solution is not sufficient.
Looking for options I had this post which Marcel.L pointed me to back then, where he forwards calls to API Management in another virtual network using a Private Endpoint/Link and Virtual Machine Scale Set combination.
However, as pointed out in my earlier post, my endgame should be to replace Azure Service Fabric IaaS container hosting with a higher level abstraction, PaaS like Container Apps, hence I did not necessarily want to add just another IaaS in the process - even one with a lower complexity.
Searching for alternatives I checked on private linking capabilities of Azure API Management itself. However this cannot be used and mixed when it is already operated in external or internal virtual network mode. Hence no option for me.
"As with container compute I need my API Management with 100% ingress and egress within a virtual network"
Researching on private IP address options on Azure Application Gateway I stumbled over a private link feature in Azure CLI which - lacking tangible documentation - I exploited here and converted to Bicep
Solution Elements
To achieve this configuration
following solution elements additionally to my earlier post are required:
- a Private Link Configuration on the Application Gateway; still to subnet within Hub virtual network
- a Private Endpoint in Spoke virtual network
- a Private DNS Zone linked to Spoke virtual network so that Container Apps resolve to Private Endpoint IP address
complete configuration for all snippets shown in this post can be found in this repo / tag; I'll state the
filenamesso that snippets can be found more easily
Prerequisites
- Azure CLI
- Bicep
- Linux shell / bash / ...
Stage 1 - get preview feature working (as of Feb'2022)
When first trying to deploy Application Gateway with specifying privateLinkConfiguration, I was rewarded with a nice error code SubscriptionNotRegisteredForFeature and a message like Subscription /subscriptions/... is not registered for feature PrivateLinkConfigurations required to carry out the requested operation. Finding no suitable documentation I tried to find the feature switch myself with
az feature list --namespace Microsoft.Network -o table | grep Priv
I then activated what seemed to make sense for my case:
az feature register --name AllowApplicationGatewayPrivateLink --namespace Microsoft.Network
az feature register --name AllowAppGwPublicAndPrivateIpOnSamePort --namespace Microsoft.Network
az provider register -n Microsoft.Network
Stage 2 - configuration Application Gateway
To add private link, an entry in privateLinkConfigurations section is required. It needs to be put in the same virtual network but a different subnet as the gateway itself (once Application Gateway is deployed to a subnet, it only allows for other Application Gateways to share this subnet, but no other resources).
That privateLinkConfiguration is then to be referenced on the frontendIPConfiguration. In my sample I share it with the public IP, but I guess, this also can be separated to allow various forwarding rules depending from where ingress is coming from.
appgw-priv.bicep
...
frontendIPConfigurations: [
{
name: 'default'
properties: {
publicIPAddress: {
id: pip.id
}
privateIPAllocationMethod:'Dynamic'
privateLinkConfiguration:{
id: resourceId('Microsoft.Network/applicationGateways/privateLinkConfigurations', appGwName, 'private')
}
}
}
]
privateLinkConfigurations: [
{
name: 'private'
properties: {
ipConfigurations: [
{
name: 'private-ip'
properties: {
privateIPAllocationMethod:'Dynamic'
subnet: {
id: subnetJumpHubId
}
}
}
]
}
}
]
...
for simplification in my sample I am using HTTP on port 8080; in production this will be replaced by a proper HTTPS, FQDNs and certificates
Based on private link configuration in Hub network now a private endpoint can be added in Spoke network:
appgw-priv.bicep
resource pep 'Microsoft.Network/privateEndpoints@2021-05-01' = {
name: 'pep-priv-gateway'
location: location
properties: {
subnet: {
id: subnetJumpSpokeId
}
privateLinkServiceConnections: [
{
properties: {
privateLinkServiceId: appgw.id
groupIds: [
'default'
]
}
name: 'pep-priv-gateway'
}
]
}
}
Stage 3 - private DNS zone
Spoke network needs a private DNS zone so that traffic towards the Private Endpoint is resolved correctly:
appgw-priv-dns.bicep
param vnetSpokeId string
param apiName string
param pepIp string
var privateDNSZoneName = 'internal-api.net'
resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {
name: privateDNSZoneName
location: 'global'
}
resource privateDnsZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
parent: privateDnsZone
name: '${privateDnsZone.name}-link'
location: 'global'
properties: {
registrationEnabled: false
virtualNetwork: {
id: vnetSpokeId
}
}
}
resource privateDnsZoneEntry 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
name: apiName
parent: privateDnsZone
properties: {
aRecords: [
{
ipv4Address: pepIp
}
]
ttl: 3600
}
}
again for simplification I use DNS zone
internal-api.netbecause I am not yet sure whether it is suitable here to go withprivatelink.azure-api.netas the configuration I use is not purely an API Management private link configuration
Stage 4 - deploy API Management, Application Gateway and DNS zone
I had to split deployment of API Management & Application Gateway, return and extract private IP address information from private endpoint and then continue with private DNS deployment. I was not able (or just too lazy) to wire this up within one Bicep file.
deploy-stage-2.sh
...
az deployment group create --resource-group $RESOURCE_GROUP \
--template-file apim.bicep \
--parameters apimName=$APIMNAME \
appInsightsName=$APPINSIGHTNAME \
logAnalyticsWorkspaceName=$LOGANALYTICSNAME \
fapp1Fqdn=$fapp1Fqdn \
fapp2Fqdn=$fapp2Fqdn
VNET_SPOKE_ID=`az network vnet list --resource-group ${RESOURCE_GROUP} --query "[?contains(name,'spoke')].id" -o tsv`
PEP_NIC_ID=`az network private-endpoint list -g $RESOURCE_GROUP --query "[?name=='pep-priv-gateway'].networkInterfaces[0].id" -o tsv`
PEP_IP=`az network nic show --ids $PEP_NIC_ID --query ipConfigurations[0].privateIpAddress -o tsv`
az deployment group create --resource-group $RESOURCE_GROUP \
--template-file appgw-priv-dns.bicep \
--parameters "{\"pepIp\": {\"value\": \"$PEP_IP\"},\"vnetSpokeId\": {\"value\": \"$VNET_SPOKE_ID\"},\"apiName\": {\"value\": \"$APIMNAME\"}}"
along with API Management instance I deploy an API to test calls to Function Apps hosted in Container Apps environment;
appgw-priv.bicepshown above is a module ofapim.bicep, hence deployed in the first block
Stage 5 - testing
To check, that I have not introduced regressions I first test access to Function Apps over public IP of Application Gateway:
test-api.sh
...
APIMID=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query id -o tsv`
APIMURL=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query gatewayUrl -o tsv`
GWURL=`az network public-ip show -n ${APIMNAME}-priv-gateway-pip -g $RESOURCE_GROUP --query dnsSettings.fqdn -o tsv`
GWPORT=`az network application-gateway show -n ${APIMNAME}-gateway -g $RESOURCE_GROUP --query frontendPorts[0].port -o tsv`
SUBKEY=`az rest --method post --uri ${APIMID}/subscriptions/test-subscription/listSecrets?api-version=2021-08-01 --query primaryKey -o tsv`
curl -s ${GWURL}:${GWPORT}/test/fapp1?subscription-key=$SUBKEY
printf '\n'
curl -s ${GWURL}:${GWPORT}/test/fapp2?subscription-key=$SUBKEY
printf '\n'
To make more thorough tests of the Functions Apps in combination with calls to API Management I have to hop on a jump VM in Spoke network:
test-fapps.sh
...
IP=$(az vm list-ip-addresses -g $RESOURCE_GROUP --query "[?contains(virtualMachine.name, 'hub')].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv)
declare -a apps=("fapp1" "fapp2")
for app in "${apps[@]}"
do
echo "$app"
fqdn=$(az containerapp show -n $app -g $RESOURCE_GROUP --query configuration.ingress.fqdn -o tsv --only-show-errors)
ssh ca@$IP curl -s https://$fqdn/api/health
echo " <<-- check APIM internal status"
ssh ca@$IP curl -s https://$fqdn/api/apim-status
echo " <<-- check $app APIM health"
ssh ca@$IP curl -s https://$fqdn/api/apim-internal-status
echo " <<-- check $app APIM internal status"
done
where
-
curl -s https://$fqdn/api/healthchecks Function App only with a call to a HTTP trigger -
curl -s https://$fqdn/api/apim-statusandcurl -s https://$fqdn/api/apim-internal-statuscall HTTP triggers which themselves forward calls to API Management instance (using theinternal-api.netdomain)
[FunctionName("Apim-Status")]
public static async Task<IActionResult> ApimStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-status")] HttpRequest req)
=> new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/status-0123456789abcdef"));
[FunctionName("Apim-Internal-Status")]
public static async Task<IActionResult> ApimInternalStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-internal-status")] HttpRequest req)
=> new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/internal-status-0123456789abcdef"));
Conclusion
With this solution a major roadblock for me is out of the way and I do not need to operate my own compute resources for network traffic forwarding. It still requires a some polishing and hardening before I can use it even close to our production resources.
A positive effect for me is that with reconfiguration of the existing Application Gateway and a very low footprint of IP addresses consumed in my corporate virtual network, I can even operate the existing environment with Service Fabric in coexistence with the new Container Apps environment, which reduces migration risks tremendously.
This content originally appeared on DEV Community and was authored by DEV Community
DEV Community | Sciencx (2022-02-26T16:48:16+00:00) Use Azure Application Gateway private link configuration for an internal API Management. Retrieved from https://www.scien.cx/2022/02/26/use-azure-application-gateway-private-link-configuration-for-an-internal-api-management/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.