cloudshell and vnet for nodes and not using public ip

[dashap]

At https://github.com/actions/virtual-environments/blob/main/docs/create-image-and-azure-resources.md, it has a section called:

Azure DevOps self-hosted pool requirements

It seems to allude to the issue with public ip I see due to compmany security policy and some way to address this with options. Can I get more details on what steps are needed to have it not use public ip and instead of a private ip? One additional thing is I was using Azure and shell.azure.com method to get a cloudshell, which has a subnet that is unfortunately not in the vnet where the virtual environments will get made.

I was looking for more details on how you set this up. Did you end up making your own vm in the vnet for the environment with all the tools? I was hoping to avoid that since the cloudshell seems to have all the tools.

[al-cheb]

Hello, @dashap

I don't think so it's possible to generate an image using cloud shell that's why we use a dedicated virtual machines.

Preq:

1. Create Azure VirtualNetworkName, VirtualNetworkRG, VirtualNetworkSubnet resources to your own environment
2. Create windows agent vm with VirtualNetworkName/VirtualNetworkSubnet settings and
3. Pre-install software - https://github.com/actions/virtual-environments/blob/main/docs/create-image-and-azure-resources.md#how-to-prepare-windows-build-agent

Generating image process:
All our pipelines and scripts are stored at the https://github.com/actions/virtual-environments/tree/main/images.CI/linux-and-win location for image generation or use https://github.com/actions/virtual-environments/blob/main/helpers/GenerateResourcesAndImage.ps1 script instead.

To start image generation process without public ip address run the build-image.ps1 script and pass necessary parameters - https://github.com/actions/virtual-environments/blob/main/images.CI/linux-and-win/build-image.ps1

E.g.:

. '\build-image.ps1' -ResourcesNamePrefix 115442 -ClientId 12345 -ClientSecret 12345 -TemplatePath C:\images\win\windows2019.json -ResourceGroup my_rg_name -StorageAccount images -SubscriptionId 12345 -TenantId 12345 -Location EUS2 -VirtualNetworkName my_vnet -VirtualNetworkRG my_net_rg -VirtualNetworkSubnet default

Create a template from packer image:
You shoud slightly change the script to remove public ip address assignment and pass the same VirtualNetworkName, VirtualNetworkSubnet -https://github.com/actions/virtual-environments/blob/main/helpers/CreateAzureVMFromPackerTemplate.ps1

$nicName = $VirtualMachineName + [System.GUID]::NewGuid().ToString().ToUpper()

$subnetId = az network vnet subnet show --vnet-name $VnetName `
                                        --name $VnetSubnetName `
                                        --resource-group $VnetResourceGroup `
                                        --query "id"

$networkId = az network nic create -g $ResourceGroup `
                                    --subnet $subnetId `
                                    -n $nicName `
                                    --subscription $SubscriptionId `
                                    --location $Location `
                                    --query "NewNIC.id"

Write-Host "`nCreating the VM"
az group deployment create -g $ResourceGroupName -n $VirtualMachineName --subscription $subscriptionId --template-file $templateFilePath --parameters vmSize=$vmSize vmName=$VirtualMachineName adminUserName=$AdminUsername adminPassword=$AdminPassword networkInterfaceId=$networkId

[dashap]

Thank you for the response. I will try a windows server as suggested. I think we just need the image that we pass to terraform, so I am not sure we need that last step to remove public ip or not.

Note that the options we used for the linux vm we tried to use to set it to use private ip:

$Env:VNET_RESOURCE_GROUP = "azuse-GithubRunners-vpn-rg"
$Env:VNET_NAME = "azuse-GithubRunners-vpn-vnet"
$Env:VNET_SUBNET = "Azuse-GithubRunners-subnet"
$Env:PACKER_LOG=1
$Env:PACKER_LOG_PATH="/home/adminuser/build/virtual-environments/packer.log"

In addition, I pain-stackingly downloaded manually all the powershell modules it needed on the server running GenerateResourcesAndImage.ps1 so that it could run.

This allowed us to start the GenerateResourcesAndImage script with private ip. The GenerateResourceAndImage script was able to do everything but run any scripts in virtual-environments/images/linux/scripts/installers related to installing powershell modules due to an ssl ca cert issue trying to validate psg-prod-eastus.azureedge.net and www.powershellgallery.com. The /etc/ssl/certs ca-certificates.crt file and maybe some of the digicerts in the repository files were not able to check via their local files those two sites and would fail to load the powershell modules. It is not clear to me how and where to update these in the setup. I tried to trick load them via the powershellcore.sh script by having it copy the ca-certificates.crt file that was in use in the cloudshell to /etc/ssl/certs, but it is not clear where powershell during the process goes to look for ca certs to validate the digital signatures of sites. I think if that was resolved, using the ubuntu 20.04 runner I stood up previously with cloudshell when I had public ip access could have even been used to make a new image. Something about the certs is not quite right.

Note that previously we were able to make the image with cloudshell by the way when we had public IP access not blocked by a policy.

[al-cheb]

packer, az cli and Az module works well on a linux vm.

[dashap]

One additional observation is that 20.04 ubuntu says there is zip and docker, but both of those packages were missing. I can certainly after the fact load them, but maybe they can get included? The docker one setting up as rootless was a bit tricky.

[catthehacker]

Docker and zip are included in all Ubuntu images.

Below output from last Ubuntu 20.04 build

- Docker Compose 1.29.2
- Docker-Buildx 0.6.0
- Docker-Moby Client 20.10.8+azure
- Docker-Moby Server 20.10.8+azure

| Name                   | Version                           |
| ---------------------- | --------------------------------- |
| zip                    | 3.0-11build1                      |