Showing posts with label Azure Storage. Show all posts
Showing posts with label Azure Storage. Show all posts

Wednesday, September 4, 2019

Azure DevOps – Build and Release pipeline to Azure File Storage


Abstract

Azure File Storage is an ubiquitous service. It is so useful that I have hardly seen an enterprise not making use of it. Recently Premium tier of File storage is introduced with IOPS as 1,00,000 as opposed to standard File storage which used to offer 1000 IOPS only. This premium tier now even makes Azure File Storage natural choice for high performance demanding applications.

I have been architecting many scenarios and implementations where I used Azure File Storage for running web applications. In such a scenario your application binaries, DLLs, application files [or jar, war files in case Java] are present on Azure File Storage and this Azure File Storage is mapped as a drive to Azure VMs. This Azure VM then runs web servers likes Tomcat, IIS and maps their website path to the drive mapped using Azure File Storage. And this works awesome!

In today’s enterprise world DevOps has become a regular practice. Naturally when you plan to deploy web applications you use pipelines in Azure DevOps. In above scenario you would need to create Build and Release pipeline that will deploy your application to Azure File storage. Unfortunately, there is no default task exist in Azure DevOps that can publish the build output to Azure File Storage. This is what I am going to build in this post.

Let’s go!

Setting up pre-requisites

For this blog post I am going to use .NET Core language web application as my sample. It doesn’t matter if you use any other language-based example. So I already have a DevOps Organization created and in the same I have created one Project named as “MyNETCoreApp” as shown below –



It doesn’t matter which option you select for Agile process while creating project in Azure DevOps. Selection of Agile process while creation of project in Azure DevOps will matter in other important cases of setting up your business process. Not in Build and Release pipeline. But that is another blog on another day. Not today!

Next part is about creating the project n Visual studio [or in your favorite IDE] and pushing the same project in this current Azure DevOps Project Repo. I have it done already because I think this is straight forward and no step by step guide is required. So My project Repo looks as follows –




So we are all set for now to create Build and Release pipelines to release to Azure File Share.

Don’t believe on your eyes - Clearing up the confusion

When I asked my regular blog followers for the current topic most of them responded saying “Azure DevOps already offers a way/task to release the code to Azure Files.”. I said where? How and when did this happen? Then my dearest followers [most of them - not all] sent below screenshot to me [without Red and Green highlights]–



They were not wrong because they had just a given a quick look to the heading. Red highlighted Heading is perfectly fine and it will make you believe that Azure DevOps do offer a task to publish and release pipeline to Azure File Share. However if you focus on description of the task Green highlighted; you will see that Azure DevOps File Copy tasks do not publish to Azure Files but to Azure Blob and Virtual Machine only. Confusion because of names!

I am sure you know the difference between Azure Blob [https] storage and Azure File Share [SMB] storage. So moral of the story – we don’t have any tasks default provided in Azure DevOps to release the code to Azure Files share.

High Level Solution

Wednesday, July 10, 2019

Delete Azure File Storage Directory and files using PowerShell and secret sauce of recursion


Abstract


Come on guys!! Seriously?

After this successful post of Azure Virtual Machine – Frequently asked questions – not easily answered I got almost 54+ requests/ comments/ emails asking specifically for a blog post on sharing script for “Deleting Azure File Storage Directory using PowerShell”.

Not sure why people felt this is hard to write a PowerShell script. Well, this post is about the same topic and at the end of post you will know how to delete Azure file storage Directory and Files using PowerShell scripting.
You must have read the word in title “Secret sauce of Recursion”; about that later in the post below.

Lets go!

What is big deal in deleting Azure File Storage Directory and Files?


Well, you may think this as straight forward PowerShell. I also thought the same way until I actually tried for it.
As a best practice, before trying anything from PowerShell I always do the same task using Azure Portal. So for deleting Azure Files I went on to the portal. Created a storage account of type v2; and then created a File storage of 10GB as shown in the screenshot below. At this point the file share is empty.



Note – Focus on Blue box and blue underline; I wrote this blog at 2AM in the night. This is how I am getting attached to my blog after my wife.

Then I created a directory and uploaded few files within directory and few files directly in share and few directories inside the root directory and then again few files.

Note - If you have not understood above sentence; then blame 2AM time.

So my directory structure looks as below on Azure File storage share –



Yellow boxes are directories, blue boxes are files and green box is the actual file share inside which all of this is present. I tried to delete the directory wwwroot from the portal UI and I got below message.



This means I will have to delete each and every file present in the directory of my file share. If there is cascading of directories [directory within directory] then each of the directory will have to be made empty first and then delete each of the directory one by one and then the actual directory I want to delete.

This is big deal task and no wonder people are finding it difficult in PowerShell. As of now there is no single command by which you can delete directory including everything within it. So if I want to retain my file share but delete everything within it; I must empty all the directories.

What is my way out?


Well, you need to traverse through each of the directory and make it empty. So you might be thinking for for-each loop in PowerShell to do it. You are correct. But only for-each or for loop will not help you win the war. You need super power called as “RECURSION”. Write a recursive function in PowerShell and we should be able to win the war.

Why people find it difficult to write RECURSIVE functions?


People of my generation [people who started career between 2006 to 1012 year]who had started their programming career in C, C++, micro-controller programming will never find recursion difficult. Because the very first program we had build was Factorial number and it is product of all integers from 1 to n. For example factorial of 5 is 5 x 4 x 3 x 2 x 1  = 120.

This classic problem can be solved by recursion effectively.

Recursion is basic programming technique you can use in many languages including PowerShell, in which a function/ method calls itself to solve some problem. A method/ function that uses this technique is called as “Recursive function”. There are many problems that can be solved only by recursion.

To empty azure file directories we need recursion in PowerShell.

I have seen most of the new comers who starting their career in IT fields directly jump to Artificial Intelligence and Machine learning programming. More of less basics of programming may get missed like data structures, recursions etc.. I asked the year of experience to many people who demanded to write this PowerShell to delete Azure files directories and files and no surprise most of them were in the range of 4 to 6 years of experience.

Anyways no hard feelings!

Let us build PowerShell to delete Azure file storage directories and files recursively.

Define variable and create Azure Storage context


I am using latest Azure PowerShell module.

We need to define the variables for storage account details and then create the context for storage account. Context is an important steps as for every operation you perform against any of the service in Azure storage context should be passed in PowerShell commands.

#define varibales
$StorageAccountName = "YourStorageAccountName"
$StorageAccountKey = "YourStorageAccountPrimaryKey"
$AzShare = "kunalshare – you write your file share name here"
$AzDirectory = "LatestPublish"


#create primary region storage context
$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
$ctx.ToString()

#Check for Share Existence
$S = Get-AzStorageShare -Context $ctx -ErrorAction SilentlyContinue|Where-Object {$_.Name -eq $AzShare}

Get the directory name and call for delete function


Now as per the directory hierarchy in above diagram; you can see that inside File Share I have a root directory called as “LatestPublish” and all other files/ directories are present inside it.
So I want to delete “LatestPublish” folder. So let us retrieve the reference for the folder and then call delete recursive function.

# Check for directory
$d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name

if ($d.Name -notcontains $AzDirectory)
{
    # directory is not presetn; no action to be performed
   
}
else
{
    $dir = Get-AzStorageFile -Share $s -Path $AzDirectory   
    RemoveFileDir $dir $ctx #Calling function to remove directory.
}

Recursive function to remove directories


In above code of getting root directory we are calling a function named as RemoveFileDir. Let us write this function.

function RemoveFileDir ([Microsoft.Azure.Storage.File.CloudFileDirectory] $dir, [Microsoft.Azure.Commands.Common.Authentication.Abstractions.IStorageContext] $ctx)
{  
    $filelist = Get-AzStorageFile -Directory $dir
   
    foreach ($f in $filelist)
    {
       
        if ($f.GetType().Name -eq "CloudFileDirectory")
        {
            RemoveFileDir $f $ctx #Calling the same function again. This is recursion.

        }
        else
        {
            Remove-AzStorageFile -File $f           
        }
    }
    Remove-AzStorageDirectory -Directory $dir
   
}


In above function, we are receiving the root directory object to be deleted and storage context as parameter. Then we use for loop to traverse through the received directory object and see if there any more directories. If there is a directory found then we call the same function in recursion mode; until we reaches to last directory/ file present in the hierarchy. Once we reach to last item then we delete it.

Download entire source code

Download the entire source code of deleting Azure file directories from GitHub - https://github.com/kunalchandratre1/DeleteAzureFilesDirectoriesPowerShell

Bonus tip

If you delete file share from the azure portal; everything within that gets deleted automatically and you don’t have to indulge yourself in to PowerShell. However if you want to delete selected directories and files from File share then PowerShell is the only way or do it manually from portal.


Conclusion

Hope this blog helped you t get working Azure PowerShell script for deleting Azure file storage directory and files recursively. Please feel free to add/ update your comments. Thanks.


Happy recursion!! 

A humble request!

Internet is creating a lot of digital garbage. If you feel this a quality blog and someone will definitely get benefitted, don't hesitate to hit share button present below. Your one share will save many precious hours of a developer. Thank you.

Tuesday, March 7, 2017

How to download Azure blob storage contents in Azure Linux VM using Azure CLI

Abstract

I always get this question – how can I download Azure blob storage files in Azure Linux VM? When I say use Azure CLI (Command Line Interface) then next question asked is – Do you have step by step guide?
Well, this blog post is the answer to both questions.

The high level approach is outlined below –
  1. Provision Linux Azure VM in a subnet. [Of course this step is out of scope of this article. For detailed steps refer - this guide .]
  2. Install Azure CLI in Linux VM
  3. level Upload sample files to azure storage and then download them in a folder in Linux VM.

This article assumes you understand Azure storage and related concepts.

Wow, this is first blog post from me on Linux and Azure.

Prepare you Azure Linux VM

I have provisioned an Azure Linux VM with OS as CentOs 7.2. Added this VM in a Subnet, with NSG having only port 22 inbound open. Also I have attached a public IP to this VM so that I can make SSH to this VM from anywhere over port no 22. Step by step guide link is already shared above.
If you are having different OS than CentOS then commands in below steps will change however high level approach remains same.

Install CLI in Azure CentOS VM

First make SSH to your Linux VM and run command “sudo su” [without double quotes]. So in subsequent steps we will not face awesome “access denied” or “permission denied” errors. Or we don’t have to add “sudo” word in every command we run.

There are two versions of Azure CLI –
1.0    – This stuff is written in node.js and supports both Classic [old way of doing things on Azure] and ARM [new fancy way of doing things on Azure].
2.0    – To make this version impressive Microsoft calls it “Next generation CLI” and is written in python. Only supports ARM mode.

I will be using 2.0 version. Hence I need Python as well installed on the Linux Azure VM. So let’s first install python latest version on Azure CentOS VM.
Let’s make sure that yum is up to date by running below command –

sudo yum -y update

-y flag tells system that “relax, we are aware that we are making changes, hence do not prompt for confirmation and save our valuable time”. This command execution will take good amount of time.
Next install yum-utils using below command –

sudo yum -y install yum-utils

Now we need install IUS (Inline with upstream stable). Don’t get scared by name. This is community project which will ensure that whatever version we install for Python 3, we will get the most stable version. Run below command to install IUS –
sudo yum -y install https://centos7.iuscommunity.org/ius-release.rpm

After IUS now we can install recent version of Python. As of writing, the recent version is 3.6 but I will install 3.5 to be on safer side. In python 3.5 version I see 3.5.3 is the latest so let’s install it.
sudo yum -y install python35u-3.5.3 python35u-pip

To verify, simple run below command and it’s output should be 3.5.3.
python3.5 -V

Now install the required prerequisites on CentOS using below command –
sudo yum check-update; sudo yum install -y gcc libffi-devel python-devel openssl-devel

Finally, back to installation of CLI 2.0 -


curl -L https://aka.ms/InstallAzureCli | bash
This may prompt you to download CLI in which directory. Press enter to keep the default path of installation which would be “/root/lib/azure-cli”. Similarly keep pressing enter if more prompts are displayed.
Restart command shell to take changes effect –
Exec -l $SHELL
Just type “az”[without quotes] and it should you Azure cli commands information in CentOs. This means installation of Azure CLI 2.0 on Linux is successful.

Run below command to list storage related commands.
az storage -h

Upload sample files to Azure Storage

This step is straight forward. Use Azure portal and create one standard [not premium] ARM based storage account. Create container and upload 4 sample files in the container. It would look like below –






Add Azure account to CLI

Run command  as shown below. It will prompt you with a code and link to enter the code. After this you will be asked for login using existing azure related credentials. Successful login will show you the subscriptions associated to your account as below –



Set storage account and download the blob

Now set credentials for storage account.
export AZURE_STORAGE_ACCOUNT=YourStorageAccount
export AZURE_STORAGE ACCESS_KEY=YourStorageAccountKey

Create a directory named as test1 using the command. This is the directory in which we will download blob contents. -
mkdir test1/

After this run below command to download the blob file in test1 folder
az storage blob download -c sample -n File1.txt -f /test1/File1.txt
az storage blob download -c sample  /test1

Change directory to test by command cd test1 and run ls -l. This should list File1.txt as shown below.

Limitation

Using Azure CLI you can’t download all the blob from a container. You have to download each and every blob individually; bulk download of azure blobs is not supported.

Resolution to Bulk download

To download all blobs from a container instead of Azure CLI, we will need to use Azure XPlat CLI. Or we can also use Powershell as it is open source now [although I have not tried yet]. It’s common to refer many approaches to achieve one task when you are in open source. J
Azure XPlat CLI is a project that provides cross platform command line interface to manage Azure. Refer documentation here - https://github.com/Azure/azure-xplat-cli. But this is another blog on another day.

Conclusion

So I hope now you understand how easy it is to download azure blob storage contents in Linux Virtual Machine.
Please provide your valuable comments. Good news is its free!!
Keep Downloading!!

Friday, January 16, 2015

Azure Blob Snapshots using REST Api and Client Library


My article of Azure blob snapshots deep dive has been published. The article outlines various azure blob snapshots operations using client library and REST Api. The full source code is also available at bottom to download from GitHub. Here is the link –
http://www.dotnetcurry.com/showarticle.aspx?ID=1072

Cheers...
Happy Snapshotting!!

Monday, August 25, 2014

Using SAS, renew SAS and REST API to Upload large files to Azure blob storage in parallel and async


First of all thanks for overwhelming response to my earlier blog post related to upload of large files to azure block blob in parallel and async.
At the bottom of this post you will find the link for downloading the code sample.
In current post I will extend the same code library to support REST Api to perform azure blob upload. Also, as a best practice you should always use SAS (Shared Access Signature) for performing any operation on blob storage. Therefore in current post I will extend the code to support SAS as well.
Why we need to renew or extend Azure SAS –
Using SAS with blob storage is great option to provide one more level of security to azure storage operations however, anyone who gets access to SAS url can perform malicious operations. Therefore keeping SAS expiry time to minimum is always a good practice. But again, if we are performing upload of fairly large files then keeping SAS expiry to minimum will not help.
For example, let’s consider a scenario. Let’s say I will upload a file of size 100MB in azure blob storage using REST Api and SAS. As a best practice and already stated in my previous blog post of uploading large files to azure block blob in parallel, the files to be uploaded to azure need to be sliced in chunks.
Now if I have set the expiry time to 5 minutes. Of course upload of 100MB size will not be completed in 5 minutes and hence my block upload will fail in between.
Solution -
To continue to upload large files to azure block blob seamlessly after SAS expiry time, I need to renew the SAS token again. This is what exactly I am doing in this blog post and code sample.
So essentially, I check the response status code of every block upload. If response status code Created means current block is uploaded and continue to next. If the response status code is received as 403 Forbidden means the upload of block is failed due to invalid SAS. This is where I mark the particular failed block as Not Added.  Once the loop is over, I retrieve list of failed blocks and perform the same operation with new SAS. To perform same operation of upload with new SAS I am using popular and basic concept you also might know called as RECUSRSIVE FUNCTION.