Monday, March 9, 2020

Bypassing on premises firewall to RDP or SSH into Azure VM

12 min to read.

Abstract

Believe it or not! On-premises firewalls port opening process is proving to be number one blocker for Azure adoption, till the time Site to Site VPN or Azure Express Route is setup.

Problem is not with changing the firewall rules to access Azure VM either by RDP or SSH; It is the “process” setup by an Enterprises to allow an outbound traffic from their on premises network; to Azure or internet in general.

I am not saying security teams working for on premises environment security is bad; it is their job to keep things secure and hence they will always hesitate/ seek approval/ seek clarifications to open 3389 or 22 port for outbound traffic from their network.

In this blog post I will illustrate a way by which you can access Azure VM either by RDP or SSH without asking your internal network/ security team to open any ports.

Let’s go!

Problem Statement

Refer to below diagram – [To get good view of the diagram, click on it]



       1.      As you can see in the diagram above; your laptop is sitting in your organizations network which is controlled environment with Firewall and proxy devices.
2.      You have azure VM with public IP x.x.x.x as shown in the diagram.
3.      To make RDP/ SSH into Azure VM you need to use standard ports like 3389 for RDP and 22 for SSH.
4.      There is no Site to Site or Express Route present between your on premises network and Azure environment.
5.     Your organizations on premises firewall device blocks any outbound/ outgoing internet traffic with port 3389/22.
6.      Hence your RDP/ SSH to Azure VM from your laptop fails.

So you may say that let me talk to my network/ security/ firewall team and see if they can open 3389/ 22 port outbound for my laptop IP?

Well you can do that but based on my experience I have seen, it takes weeks to get 3389/ 22 opened for internet or even for particular IP. Also organizations have a process set for making port opening and rule modification in firewall. As security teams and your firewall operations always go through stringent audit and compliance process; it will always be a big battle for you.

If you are financial organization then opening up 3389/ 22 over internet in on premises is never happening!

The most secured solution is to use either Site to Site VPN or Express route to connect to Azure VMs. However configuration of these also takes time.

So what is the solution to quickly get started with RDP/ SSH to Azure VM without opening on premises firewall rules?

Solution architecture – Have Azure VM behind Azure Load Balancer


Although all on premises firewall/ proxy servers will block 3389/ 22 ports; they will always have 443 port 
outbound openFor example from my laptop in above diagram if I plan to access https://google.com then this will be accessed over 443 port only as most of the trusted site works on https and default port for https is 443. 

So if I take RDP/ SSH from my laptop over 443 port instead of 3389/ 22 port then traffic originated from my laptop should cross on premises firewall and reach out side.

The question is how do I allow RDP/ SSH over 443? Because default port of RDP is 3389 and SSH is 22 and it 
can’t be changed. This is where I need “Inbound Network Address Translation [Inbound NAT]”.

Means my traffic should reach to Azure VM over 443 but the 443 port needs to be translated/ changed to 3389 or 22. However Azure VM can’t to inbound NAT and this is where we need Azure Load Balancer.

Azure load balancer is an awesome PaaS service. It takes care of inbuilt HA for you. It comes in 2 flavors –
       1.      Basic - Free
2.      Standard – Cost is associated with it.

For our case we can use Standard or Basic. I am poor guy and always love to avoid unnecessary spend. Because the contract of unnecessary spend I have given to my better half. So I will use Basic layer 4 Azure load balancer for this tutorial.

Refer to below diagram – [To get good view of the diagram, click on it]



       1.      From local laptop either for doing SSH or RDP use 443 as a port instead of standard ports for SSH and RDP.
2.      As the traffic to Azure VM is over 443; it is allowed in on premises firewall.
3.      The request for RDP/ SSH lands on Azure public Load balancer.
4.      The inbound NAT rules are processed, and incoming traffic translated to 3389/ 22 ports. Then forwarded to backend pool which is our VM we want to access.
5.      Similarly as session is established within on premises firewall; the return traffic flows back seamlessly to our laptop.

This way even if we don’t open any ports in on premises firewall we are able to reach to our VM on Azure.

Azure Public Load Balancer configuration

I am assuming you have Azure VM public IP, so follow the rest of the process.

First we will remove existing Public IP assigned to VM NIC; so that we can use same Public IP for Azure LB. As we are bringing Azure VM behind Azure LB; we don’t need Public IP for Azure VM.

Below screenshot depicts how can you disable the public IP for Azure VM NIC.



The IP that we assigned to Azure LB while creation will be configured automatically as “Frontend IP configuration” within Azure LB.

Create LB as shown. Attach same Public IP which we removed from Azure VM NIC. You can also create new to attach to LB but not required. Let us leverage existing public IP only.



Configure the existing VM as backend as shown below –



Then click on “Inbound NAT rules -> Add”. Provide the values as shown below –



In above NAT rule I am setting up custom port to 3389 as I am having windows VM to which I am doing RDP. If you have a linux VM, you should put custom port as 22.

After configuration of NAT rule in Azure LB if you see VM; you will still see same Public IP attached. Even if it shows public IP attached to Azure VM, it is actually attached to Azure Load Balancer. This can be confirmed with Public IP information blade from Azure portal as shown below –



RDP view

If you now try to make RDP using simply public IP as below it will not work. Because by default the system assumed the port number 3389.



Try the same RDP with 443 port and it should be successful as below –



In case this doesn’t work, make sure the NSG attached to VM is not blocking any traffic from Azure Load Balancer. By default rule in NSG allows the full traffic from load balancer to Azure VM and it’s weight is 65001 as shown below –



If you have added any other rule with lower weight value than 65001, then that rule might be blocking communication from load balancer to Azure VM. So kindly check NSG rules if this doesn’t work.

Benefits of this approach

Biggest benefit of this approach is you don’t have to inform anything to your security/ firewall teams. This works seamless. Biggest problem I see it as a security risk.

Bonus tip

Tip 1 –
Is this correct practice? Well not really. But if you want to get started immediately while your port opening in your organization is taking place or setup of Site to Site VPN, or Express Route is going to take time, then this is life saver.

Ultimately aim should be setting up Site to Site VNET or Express Route to access Azure resources from your on premises.

Tip 2 -
You can leverage azure offering called as Azure Bastion.

Conclusion

Hope this article helped you to get started on Azure VM quickly. Let me know your views in comments section below to improve and what are your thoughts on this approach.
Happy hacking!!

A humble request!

Internet is creating a lot of digital garbage. If you feel this a quality blog and someone will definitely get benefited, don't hesitate to hit share button present below. Your one share will save many precious hours of a developer. Thank you.



Next Related Posts



Thursday, January 30, 2020

Azure VM Disk Encryption - Storage Side Encryption vs Azure Disk Encryption

11 min to read.

Abstract

Encryption is a vast and complex topic. No matter how much you try to make it easy; it turns into more complex subject. The encryption is very close to heart for security teams in any organization.

Especially for security people working for a Bank or Financial institutes, the encryption will always be favorite topic. The audit, compliance and security teams always tend to evaluate encryption in and out.

With more and more organizations moving their workloads to Azure Cloud; encryption becomes hot topic. The most common service used on Azure is Azure VMs.

I have been part of the discussions where “Encryption of Azure VMs and its disk” took more than 6 months to satisfy all queries of security teams.

As always, I don’t want every discussion to continue that long and this is what this blog post targets. In this blog I have added crucial information about “Azure VM Disk Encryption” that will help you to make decision faster and move toward the Azure journey quickly for large scale adoption.

I will TRY to explain the Azure VM encryption scenarios and common questions which are not provided by Vast documentation of Azure VM Disk Encryption present – here, here, here, here and here.

Before we go forward it is always best to start with FAQ on Azure IaaS Disk Encryption.

Lets go!

What to expect out of Azure VM encryption?

The topic of encryption vastly moves around two things –

1.      Encryption at rest
2.      Encryption in transit

For Azure VMs when we talk about encryption for Azure VMs, it is mainly applicable to “Encryption at rest”. The data stored on Azure VM disks should be encrypted in the major requirement of organizations. Apart from that other requirements are –

      1.       I want to use my secret keys for encryption
2.      When I backup VM, the backup should also be encrypted
3.      When I restore VM from backup; it should be encrypted
4.      When I perform DR replication for VM; it should be encrypted
5.      My VM stored data should not be readable unless decryption keys are provided
6.      If I download the encrypted Azure VM from Azure and provision in my on premises then data on VM should not be readable
7.      What is difference between Azure managed disk “Server Side Encryption” (SSE) and “Azure Disk Encryption” (ADE)? When to use what?

These are the common questions/ requirements I have seen most of the companies demand for Azure VM 
encryption. Let us see how Azure VM encryption options resolve it.

Encryption methods for Azure VM

Azure VM managed disks can be encrypted using two methods –
1.     Server Side Encryption
2.      Azure Disk Encryption

Server Side Encryption of Azure VM Managed Disks

Server side encryption [SSE] is default offering. All of your Azure VMs managed disks are always encrypted by default when they are stored on underlying storage. This is encryption at rest by the Azure itself. 

You don’t need any additional efforts to perform Server Side Encryption of Azure VM Managed disk. More importantly you can disable it as well. Server side encryption is not optional, and always provided behind the scene.

Azure Disk Encryption of Azure VM Managed Disks

Azure Disk Encryption [ADE] is optional. This method provides an extra layer of security over SSE. This encryption is performed at OS level of VM and hence there are many conditions where ADE is supported/ not supported. Where as SSE is always performed at backend storage level and has nothing to do with OS of VM being encrypted. 

So there are no non-supported scenarios for Server Side Encryption of Azure VM Managed Disks.

For Windows VM ADE is configured using BitLocker.
For Linux VM ADE is configured using DMCrypt.

What is fundamental difference in Server Side Encryption and Azure Disk Encryption?

Differences are best explained by diagrams. Refer to below diagram where SSE and ADE is performed in the context of Azure VM.



As you can see in the diagram, ADE is performed at VM OS level whereas SSE is performed at the storage level. All Managed disks of Azure VM are backed by Azure page blobs and this is where SSE is performed. As ADE is performed at OS level, we use tools such as BitLocker and DMCrypt.

In next section we will talk about common questions/ requirements talked about Azure VM Managed Disk Encryption and at last we will discuss the hottest topic ADE vs SSE and when to use what.

Can I use my own Key for Server Side Encryption and Azure Disk Encryption of Azure VM Managed Disks?

This scenario is called as Bring your own Key [BYOK] scenario.

BYOK – Also known as – Customer Managed Keys [CMK] – Can be used for SSE and ADE both. For SSE as of today it is in preview in some regions. CMK can be leveraged only if you use Azure Key Vault. You can’t bypass Azure Key Vault for CMK.

BYOK process for SSE or ADE is as follows –
        You bring your key in Azure Key Vault. We call it as Key encryption key [KEK].
        Azure Key Vault uses KEK to encrypt the Data Encryption Key[DEK] while stored in Key Vault.
        DEK is actually generated automatically internally, and used for SSE or ADE, and for actual encryption of underlying data at rest of Azure VMs.
        This scenario of using your own key, using Azure Key Vault is called as Customer Managed Key [CMK].
        Remember in CMK, your key is never used for actual encryption of the Azure VM disk stored data. Rather it is used to encrypt the Data Encryption Key. The DEK is the real key which encrypts the data stored on Azure VM disks.

Can I disable SSE for Azure VM Managed Disks encryption?

No. SSE is provided by default and you can’t opt out of it or disable it. Even if you don’t use CMK, Azure will always keep data encrypted based on system/ platform generated keys and data at rest is always encrypted.

If my Azure VM is encrypted using SSE and I download the VHD. Then using this VHD if we create a VM will it be encrypted and data on it will be non readable?

No. As soon as the data leaves the boundary of underlying storage, it is decrypted. Hence if you provision VM vhd or data disk vhd the data will be readable.

If my Azure VM is encrypted using SSE + CMK and I download the VHD. Then using this VHD if we create a VM will it be encrypted and data on it will be non readable?

No. As soon as the data leaves the boundary of underlying storage, it is decrypted. Hence if you provision VM vhd or data disk vhd the data will be readable.

Note – If you want to use SSE only as it avoid lot of operational overheads and fastest way to compliance; but worried what if someone downloads the VHD; then you can create Azure Custom Role in such a way that it restricts the download VHD completely. 

Then assign the custom role created to those users at subscription level who work on daily VM operations using Azure portal, CLI or PowerShell.

Contact me if you need such a custom role built in your azure subscription.

Can I opt out of ADE?

Yes. ADE is completely optional.

If my VM is encrypted using ADE + CMK and I download VHD, will it be encrypted?

Yes. The Data disk and OS disks will not be readable.

If my VM is encrypted using ADE + CMK, and I take backup of VM using Azure Backup vault. Will that VM backup be encrypted?

Yes. VM backup is also encrypted. If you restore such a VM from backup then restored VM is also encrypted using existing keys.

What is the biggest risks customers should be aware of using Customer Managed Key [CMK] for SSE or ADE with Azure Key Vault?

CMK is where customer brings their own key. If you use your key for SSE or ADE then management and lifecycle of key is in your hand.

If key is lost, means data is lost. There won’t be any recovery point/ option available, if the KEK used for encryption of DEK, is lost.

As a failsafe mechanism always enable Soft Delete on Azure Key Vault and never perform Hard delete on Azure Key Vault secrets, Keys and certificates.

What type of VM images are supported for SSE?

OS
Windows
Linux
Gallery image
Yes
Yes
Marketplace, like CIS Benchmarked images
Yes
Yes
Custom built using Sysprep or generalized images
Yes
Yes

Note – As SSE is performed at the backend store of managed disks; the image type of VM really doesn’t matter.


What type of Azure VM images are supported for ADE?


OS
Windows
Linux
Gallery image
Yes
Yes
Marketplace, like CIS Benchmarked images
Yes
No
Custom built using Sysprep or generalized images
No*
No*

Note - Custom images and marketplace images for Linux OS are supported for encryption on case by case basis.


Can I create template generalized image from an encrypted VM and use it further?

ADE VMs must be encrypted one by one, the generalization process breaks ADE, that’s not a supported approach.

Customers could reduce some time using automation tools to trigger the encryption process as soon as the VM provisioning completes and any possible OS configuration details (orchestrate the process), but at this moment it is not possible to create images with OS already encrypted using ADE, as mentioned the VMs should be encrypted one by one. 

What is the recommended approach for ADE?

  • Deploy the VM using an endorsed supported gallery images
  • Encrypt the VM prior installing any apps or performing customizations
  • Once the VM is encrypted then you can install apps and perform any needed customizations like hardening, Antivirus install, monitoring agent install and so on (making sure those customization will not break ADE pre-requisites).

My VM is ADE encrypted, will I be able to take individual Folder backups?

No. When VM is ADE encrypted then individual folder backup is not supported. You need to first decrypt the VMs and then take individual folder backups.

Above point is valid for Windows and Linux both.

When VM is ADE encrypted, always take entire VM backup.

When I have VM encrypted using SSE, and I take backup. Is my backup is also encrypted?

Yes.

When I have VM encrypted using ADE, and I take backup. Is my backup is also encrypted?

Yes.

When I restore VM from backup, which was encrypted using SSE or ADE, my newly restored VM is always encrypted?

Yes.

In which scenarios the data disks are formatted for ADE?

If you have a Linux VM already provisioned with Data disk having data stored on it and if you perform ADE [DM-Crypt], data disks will be formatted. Therefore take the backup first if you are already using VM and then perform ADE.

If you have a Windows VM already provisioned with Data disk having data stored on it and if you perform ADE [BitLocker], data disks will NOT be formatted. However, recommended to always take the backup first if you are already using VM and then perform ADE.

In any scenario, OS drive is never formatted.

If we add new disks to already encrypted VM, will it format existing encrypted data disks?

Whether your current VM is Linux or windows  and has encrypted data disks, then while encryption of newly added data disks never cause any format operation on data disks which are already encrypted.

For linux, Any data disks that are not encrypted earlier and you do encryption using ADE then always Format happens for the first time. For Linux if you want any disks not to be encrypted, then unmount it always before encryption, remount after encryption is complete.

For Windows, Any data disks that are not encrypted earlier and you do encryption using ADE then also Format never happens, as for Windows BitLocker works in background.

What are operational overheads for ADE?

        As custom image based VMs are not supported, total provisioning time of VM increases. Also all hardening steps are recommended to be performed post VM ADE encryption completion. This adds to time required for ready VM.

        For Linux, the data disks are always formatted when first time ADE encryption happens. So if VMs are already in use this proposes challenge.

        If VM has 2 disks out of which 1 need to be encrypted and other need is not to be encrypted; you have unmount non encryption required disks and mount them back after encryption. This process will be required to be carried out everytime new disk is added to VM.

        Individual folders can’t be backed up if VM is encrypted. You have to first decrypt VM and then take backup of individual folders. In this case you are forced to take entire VM backup even if it may not be business requirement.

        ADE encrypted VMs can be restored from backup only in the same region and subscription.

        During encryption operation if OS +data encryption is happening the minimum RAM required is 8GB. Post encryption RAM can be reduced, if business/ application demands lower VM configuration.

        Only RHEL 7 images with PAYG are supported. Existing BYOL for RHEL is not supported for ADE.

        For key vault accessibility, if VM is not having internet access [which is the case in most of the org] aditional network requirements - https://docs.microsoft.com/en-us/azure/virtual-machines/linux/disk-encryption-overview#networking-requirements

Can I use 3rd party encryption service?

You can, but it may not integrate well with native azure services such as Azure backup vault and site recovery DR solution.

ADE and SSE are well integrated with all native Azure services and recommended approach for disk encryption.

I want to use 3rd party KMS such Thales/ Gemalto for individual folder encryption. Can I do that?

You can certainly do that however please confirm from the 3rd party provider that encryption used for individual folders will work with other azure services such as –

  • Azure Backup for VMs
  • Site recovery services
  • Performance is guaranteed for encryption and decryption process.

What is the cost associated with SSE or ADE?

Both are free. You are charged only for Azure Key Vault operations, key storage if you are using CMK. If platform managed keys are used then there won’t be any charge.

When to choose between SSE and ADE?

ADE
SSE
This is most convenient to use and provides solution to most of the  compliance needs.
If you are finance org and goes through extreme stringent compliance audit then opt for this. Always verify if your Security teams are okay with SSE; if not then use ADE on top of it.
Operational overheads are almost zero in this approach.
This approach increases operational overheads drastically.
The temp disks of Azure VM is not encrypted. So if you app running on VM uses temp disk for any operation then that data will not be encrypted.
Temp disks of Azure VM is also encrypted.


Conclusion

Hope this article helped you to get answers you are looking for. If you have any such questions but not answered; add your comments and I will try to give answers for them.

Happy questioning!!

A humble request!

Internet is creating a lot of digital garbage. If you feel this a quality blog and someone will definitely get benefited, don't hesitate to hit share button present below. Your one share will save many precious hours of a developer. Thank you.