Use PowerShell & azcopy to copy files from one Azure Storage account to another
I needed to build a script to copy files from one Azure storage account to another. The storage accounts were in different subscriptions - but using the access keys meant this didn't matter. All the files were in a single folder, and there were millions of files. I only wanted to pick up and copy the -video- files. In the script below I copied files from source > local folder > destination. I also spat out a listing of files at the end to both screen and CSV.
(NB: I could also have used azcopy to copy directly from source storage account to destination storage account - but there were a few 'page blob' type files for which that didn't work, so copying via local directory was the proposed solution.)
# COPY BLOB FILES
# FROM: AZURE SOURCE STORAGE ACCOUNT
# VIA: LOCAL FOLDER
# TO: AZURE DESTINATION STORAGE ACCOUNT
#PAGING PARAMETERS
$LimitRowsForTesting = 150
$MaxReturn = 100
$token = $Null
$total = 0
$bloblist = $null
#source storage account (e.g. https://mysource.blob.core.windows.net)
$source = "mysourcestorage"
$sourcekey = "bT8KLwsa2gYSDsSdt6sl8XPaQLm0rSieW4tNZlvpSgzpXC7AANXuMxIqyaXXkoOrgeg9wQ=="
#destination storage account (e.g. https://mydestination.blob.core.windows.net)
$destkey = "dOpX0PwYWjUf894WNWvd28ISGKxeplEbSiClqmwVrviOtmfIbqpZvNM0F8F4rtPI3PKIa52gSVQ=="
$dest = "mydestinationstorage"
$context = New-AzureStorageContext -StorageAccountName $source -StorageAccountKey $sourcekey
do
{ # Fetch first batch of files
$blobs = Get-AzureStorageBlob -Context $context -Container "video" -ContinuationToken $token -MaxCount $MaxReturn
$total += $blobs.Count
if($blobs.Length -le 0) { break;}
$token = $blobs[$blobs.Count -1].ContinuationToken;
#break out early for testing
if($total -ge $LimitRowsForTesting) { $token = $null }
Write-Host "Processed $total rows"
# filter the returned list to only include the file types we're looking for. Also showing example of looking for BlobType
#https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.blobtype.aspx
#$filteredBlobs = $Blobs | where {$_.BlobType -eq 'BlockBlob'}
#$filteredBlobs = $Blobs | where {$_.BlobType -eq 'PageBlob'}
$filteredBlobs = $Blobs | where {$_.ContentType -eq 'video/mp4'}
# add matching files to our overall list
if ($filteredBlobs.Count -ge 1) {
$bloblist += $filteredBlobs
}
# loop through this batch of files and copy them
foreach ($blob in $filteredBlobs){
#this code will perform an azcopy to local, then to dest
$sourcepath = "https://$source.blob.core.windows.net/video/"
$localpath = "d:\videotemp\"
$destpath = "https://$dest.blob.core.windows.net/video/"
$filename = $blob.Name
$localfile = $localpath + $filename
$logfile = "c:\azcopylogs\" + $blob.name + ".txt"
#copy to local temp folder
azcopy /V:$logfile /Source:$sourcepath /SourceKey:$sourcekey /Dest:$localpath /Pattern:$filename /XO /z:C:\azcopylogs\copylocal
#copy from local temp to remote storage
if (Test-Path $localfile){
azcopy /V:$logfile /Source:$localpath /Dest:$destpath /DestKey:$destkey /Pattern:$filename /XO /z:C:\azcopylogs\copyremote
}
#delete local copy
if (Test-Path $localfile){
Remove-Item $localfile
}
}
} while ($token -ne $null)
# when continuation/paging token is null there are no more batched rows to be returned
#output list of all files
$bloblist | Export-CSV c:\azcopylogs\file-listing.csv
$bloblist | Format-Table