We Keep Coding

How do I iterate CloudBlobDirectory to copy the data?

I have written below code to iterate through the Gen2 storage blob
CloudStorageAccount sourceAccount = CloudStorageAccount.Parse(sourceConnection);
CloudStorageAccount destAccount = CloudStorageAccount.Parse(destConnection);
CloudBlobClient sourceClient = sourceAccount.CreateCloudBlobClient();
CloudBlobClient destClient = destAccount.CreateCloudBlobClient();
CloudBlobContainer sourceBlobContainer = sourceClient.GetContainerReference(sourceContainer);
// Find all blobs that haven't changed since the specified date and time
IEnumerable<ICloudBlob> sourceBlobRefs = FindMatchingBlobsAsync(sourceBlobContainer, transferBlobsNotModifiedSince).Result;
private static async Task<IEnumerable<ICloudBlob>> FindMatchingBlobsAsync(CloudBlobContainer blobContainer, DateTime transferBlobsNotModifiedSince)
{
List<ICloudBlob> blobList = new List<ICloudBlob>();
BlobContinuationToken token = null;
// Iterate through the blobs in the source container
do
{
BlobResultSegment segment = await blobContainer.ListBlobsSegmentedAsync(prefix: "", currentToken: token);
foreach (CloudBlobDirectory VARIABLE in segment.Results)
{
BlobResultSegment segment2 = await VARIABLE.ListBlobsSegmentedAsync(currentToken: token);
foreach (CloudBlobDirectory VARIABLE2 in segment2.Results)//Bad coding
{
//how do I get children count ?
}
}
}while (token != null);
}
This will iterate only 2 levels but not dynamically till the inner levels. I have blob in below hierarchy
--Container
--FolderA
--FolderAA
--FolderAA1
--File1.txt
--File2.txt
--FolderAA2
--File1.txt
--File2.txt
--FolderAA3
--FolderAB
--File8.txt
--FolderAC
--File9.txt
This hierarchy is dynamic
How do I loop and copy the blob content.
Note: I do not want to use CLI commands to copy. Because I won't have any control once copy started.
Update
Found some samples here: https://csharp.hotexamples.com/examples/Microsoft.WindowsAzure.Storage.Blob/CloudBlobContainer/ListBlobsSegmented/php-cloudblobcontainer-listblobssegmented-method-examples.html

Please see the sample code below:
class Program
{
static void Main(string[] args)
{
var storageAccount = CloudStorageAccount.Parse("UseDevelopmentStorage=true");
var client = storageAccount.CreateCloudBlobClient();
var container = client.GetContainerReference("test");
var blobs = FindMatchingBlobsAsync(container).GetAwaiter().GetResult();
foreach (var blob in blobs)
{
Console.WriteLine(blob.Name);
}
Console.WriteLine("-------------------------------------");
Console.WriteLine("List of all blobs fetched. Press any key to terminate the application.");
Console.ReadKey();
}
private static async Task<IEnumerable<ICloudBlob>> FindMatchingBlobsAsync(CloudBlobContainer blobContainer)
{
List<ICloudBlob> blobList = new List<ICloudBlob>();
BlobContinuationToken token = null;
// Iterate through the blobs in the source container
do
{
BlobResultSegment segment = await blobContainer.ListBlobsSegmentedAsync(prefix: "", useFlatBlobListing: true, BlobListingDetails.None, 5000, token, new BlobRequestOptions(), new OperationContext());
token = segment.ContinuationToken;
foreach(var item in segment.Results)
{
blobList.Add((ICloudBlob)item);
}
} while (token != null);
return blobList;
}
}

How to get an image from Azure Blob Storage via web api as binary?

I have a webapi method so that users can upload their own profile picture, however in my entity which is saved in CosmosDB I save the profile picture URL.
public async Task<IHttpActionResult> UpdateSuperAdministrator(SuperAdministrator superadministrator)
{
var telemetry = new TelemetryClient();
try
{
var superAdministratorStore = CosmosStoreHolder.Instance.CosmosStoreSuperAdministrator;
//First we validate the model
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
//Then we validate the content type
if (!Request.Content.IsMimeMultipartContent("form-data"))
{
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}
//Initalize configuration settings
var accountName = ConfigurationManager.AppSettings["storage:account:name"];
var accountKey = ConfigurationManager.AppSettings["storage:account:key"];
var profilepicturecontainername = ConfigurationManager.AppSettings["storage:account:profilepicscontainername"];
//Instance objects needed to store the files
var storageAccount = new CloudStorageAccount(new StorageCredentials(accountName, accountKey), true);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer imagesContainer = blobClient.GetContainerReference(profilepicturecontainername);
var provider = new AzureStorageMultipartFormDataStreamProvider(imagesContainer);
foreach (MultipartFileData file in provider.FileData)
{
var fileName = file.Headers.ContentDisposition.FileName.Trim('\"').Trim();
if (fileName.EndsWith(".png"))
{
var img = Image.FromFile(file.LocalFileName);
if (img.Width != 200 && img.Height != 200)
{
string guid = Guid.NewGuid().ToString();
return BadRequest($"Error Lulo. Unsupported extension, only PNG is valid. Or unsuported image dimensions (200px x 200px)");
}
}
}
//Try to upload file
try
{
await Request.Content.ReadAsMultipartAsync(provider);
}
catch (Exception ex)
{
string guid = Guid.NewGuid().ToString();
var dt = new Dictionary<string, string>
{
{ "Error Lulo: ", guid }
};
telemetry.TrackException(ex, dt);
return BadRequest($"Error Lulo. An error has occured. Details: {guid} {ex.Message}: ");
}
// Retrieve the filename of the file you have uploaded
var filename = provider.FileData.FirstOrDefault()?.LocalFileName;
if (string.IsNullOrEmpty(filename))
{
string guid = Guid.NewGuid().ToString();
var dt = new Dictionary<string, string>
{
{ "Error Lulo: ", guid }
};
return BadRequest($"Error Lulo. An error has occured while uploading your file. Please try again.: {guid} ");
}
//Rename file
CloudBlockBlob blobCopy = imagesContainer.GetBlockBlobReference(superadministrator.Id + ".png");
if (!await blobCopy.ExistsAsync())
{
CloudBlockBlob blob = imagesContainer.GetBlockBlobReference(filename);
if (await blob.ExistsAsync())
{
await blobCopy.StartCopyAsync(blob);
await blob.DeleteIfExistsAsync();
}
}
superadministrator.ProfilePictureUrl = blobCopy.Name;
var result = await superAdministratorStore.UpdateAsync(superadministrator);
return Ok(result);
}
catch (Exception ex)
{
string guid = Guid.NewGuid().ToString();
var dt = new Dictionary<string, string>
{
{ "Error Lulo: ", guid }
};
telemetry.TrackException(ex, dt);
return BadRequest("Error Lulo: " + guid);
}
}
Now my question is about the endpoind GET. Because the profile picture needs to be shown in the navigation bar on the app, I need to get it in binary format, if that makes sense?
In other words, I cant return the ProfilePicture URL to the client (a React App), because its a client app which does not have access to the blob storage.
Here is my get.
[HttpGet]
public async Task<IHttpActionResult> GetSuperAdministrator(string email)
{
var telemetry = new TelemetryClient();
try
{
var superAdministratorStore = CosmosStoreHolder.Instance.CosmosStoreSuperAdministrator;
var superadministrator = await superAdministratorStore.Query().FirstOrDefaultAsync(x => x.EmailAddress == email);
if (superadministrator == null)
{
return NotFound();
}
return Ok(superadministrator);
}
catch (Exception ex)
{
string guid = Guid.NewGuid().ToString();
var dt = new Dictionary<string, string>
{
{ "Error Lulo: ", guid }
};
telemetry.TrackException(ex, dt);
return BadRequest("Error Lulo: " + guid);
}
}

Based on my knowledge, you React App will show a html page, which means that there will be a html <img src="url"> in the html. Then the browser will try to get the image from the url.
So, there would be 2 optional solutions:
1.Generate a storage blob url with SAS, and set it as the image's url in <img>. With SAS, a clinet will be able to access the image from storage directly.
2.The url would be the request path of yout Web API. For example: /image/{image_name}. And your web api will return a image:
[HttpGet]
public IActionResult GetImage()
{
// Get the image blob
CloudBlockBlob cloudBlockBlob = ********;
using(MemoryStream ms = new MemoryStream())
{
cloudBlockBlob.DownloadToStream(ms);
return File(ms.ToArray(), "image/jpeg");
}
}
Update:
My code:
public IActionResult Index()
{
string connString = "DefaultEndpointsProtocol=https;AccountName=storagetest789;AccountKey=G36mcmEthM****=;EndpointSuffix=core.windows.net";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connString);
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("pub");
CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference("p033tw9j.jpg");
using (MemoryStream ms = new MemoryStream())
{
cloudBlockBlob.DownloadToStream(ms);
return File(ms.ToArray(), "image/jpeg");
}
}
Screenshot:

failure deleting blobs from azure storage

I am trying to delete blobs from container. The DeleteIfExits returns true but nothing happens. I check the container using Azure's portal and I can still see the blobs.
What is wrong with my code?
private static void DeleteAllFilesWithSameName(String filePath, String filename, CloudBlobContainer container)
{
String filenameWidthoutExtension = Path.GetFileNameWithoutExtension(filename);
try
{
IEnumerable<IListBlobItem> blobs = container.ListBlobs(filenameWidthoutExtension, true);
if (blobs.Count<IListBlobItem>() > 0)
{
List<string> blobNames = blobs.OfType<CloudBlockBlob>().Select(b => b.Name).ToList();
foreach (String blobName in blobNames)
{
CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
bool isDeleted = blockBlob.DeleteIfExists();
}
}
}
catch (Exception e)
{
Console.Write(e.Data);
}
}

I believe the issue is with the logic in your code:
CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
Shouldn't the above line be?
CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobName);
Please try by changing your code to:
foreach (String blobName in blobNames)
{
CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobName);
bool isDeleted = blockBlob.DeleteIfExists();
}

How to get a list of all the blobs in a container in Azure?

I have the account name and account key of a storage account in Azure. I need to get a list of all the blobs in a container in that account. (The "$logs" container).
I am able to get the information of a specific blob using the CloudBlobClient class but can't figure out how to get a list of all the blobs within the $logs container.

There is a sample of how to list all of the blobs in a container at https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/#list-the-blobs-in-a-container:
// Retrieve the connection string for use with the application. The storage
// connection string is stored in an environment variable on the machine
// running the application called AZURE_STORAGE_CONNECTION_STRING. If the
// environment variable is created after the application is launched in a
// console or with Visual Studio, the shell or application needs to be closed
// and reloaded to take the environment variable into account.
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
// Create a BlobServiceClient object which will be used to create a container client
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
// Get the container client object
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("yourContainerName");
// List all blobs in the container
await foreach (BlobItem blobItem in containerClient.GetBlobsAsync())
{
Console.WriteLine("\t" + blobItem.Name);
}

Here's the updated API call for WindowsAzure.Storage v9.0:
private static CloudBlobClient _blobClient = CloudStorageAccount.Parse("connectionstring").CreateCloudBlobClient();
public async Task<IEnumerable<CloudAppendBlob>> GetBlobs()
{
var container = _blobClient.GetContainerReference("$logs");
BlobContinuationToken continuationToken = null;
//Use maxResultsPerQuery to limit the number of results per query as desired. `null` will have the query return the entire contents of the blob container
int? maxResultsPerQuery = null;
do
{
var response = await container.ListBlobsSegmentedAsync(string.Empty, true, BlobListingDetails.None, maxResultsPerQuery, continuationToken, null, null);
continuationToken = response.ContinuationToken;
foreach (var blob in response.Results.OfType<CloudAppendBlob>())
{
yield return blob;
}
} while (continuationToken != null);
}
Update for IAsyncEnumerable
IAsyncEnumerable is now available in .NET Standard 2.1 and .NET Core 3.0
private static CloudBlobClient _blobClient = CloudStorageAccount.Parse("connectionstring").CreateCloudBlobClient();
public async IAsyncEnumerable<CloudAppendBlob> GetBlobs()
{
var container = _blobClient.GetContainerReference("$logs");
BlobContinuationToken continuationToken = null;
//Use maxResultsPerQuery to limit the number of results per query as desired. `null` will have the query return the entire contents of the blob container
int? maxResultsPerQuery = null;
do
{
var response = await container.ListBlobsSegmentedAsync(string.Empty, true, BlobListingDetails.None, maxResultsPerQuery, continuationToken, null, null);
continuationToken = response.ContinuationToken;
foreach (var blob in response.Results.OfType<CloudAppendBlob>())
{
yield return blob;
}
} while (continuationToken != null);
}

Using the new package Azure.Storage.Blobs
BlobServiceClient blobServiceClient = new BlobServiceClient("YourStorageConnectionString");
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("YourContainerName");
var blobs = containerClient.GetBlobs();
foreach (var item in blobs){
Console.WriteLine(item.Name);
}

Since you container name is $logs, so i think your blob type is append blob. Here's a method to get all blobs and return IEnumerable:
private static CloudBlobClient _blobClient = CloudStorageAccount.Parse("connectionstring").CreateCloudBlobClient();
public IEnumerable<CloudAppendBlob> GetBlobs()
{
var container = _blobClient.GetContainerReference("$logs");
BlobContinuationToken continuationToken = null;
do
{
var response = container.ListBlobsSegmented(string.Empty, true, BlobListingDetails.None, new int?(), continuationToken, null, null);
continuationToken = response.ContinuationToken;
foreach (var blob in response.Results.OfType<CloudAppendBlob>())
{
yield return blob;
}
} while (continuationToken != null);
}
The method can be asynchronous, just use ListBlobsSegmentedAsync. One thing you need to note is the argument useFlatBlobListing need to be true which means that ListBlobs will return a flat list of files as opposed to a hierarchical list.

Use ListBlobsSegmentedAsync which returns a segment of the total result set and a continuation token.
ref:https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet?tabs=windows

In WebApI -> Swagger
[HttpGet(nameof(GetFileList))]
public async Task<IActionResult> GetFileList()
{
BlobServiceClient blobServiceClient = new BlobServiceClient(_configuration.GetValue<string>("BlobConnectionString"));
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(_configuration.GetValue<string>("BlobContainerName"));
var blobs = containerClient.GetBlobs();
return Ok(blobs);
}

how to search for a specific file in blob storage

I am kind of new to Blob storage and i need to access a specific file from blob storage.
i.e when i type in a specific folder it should list out all the blobs underneath it.
Can anyone help me with that
here's the code which i am trying to do.
if (AccountFileTransfer != null)
{
BlobClientFileTransfer = AccountFileTransfer.CreateCloudBlobClient();
ContainerFileTransfer = BlobClientFileTransfer.GetContainerReference(CONTAINER);
CloudBlob blob = ContainerFileTransfer.GetBlobReference(txtFileSearch.Text);
if (blob.Uri == null)
{
System.Windows.Forms.MessageBox.Show("Not a Valid blob search");
}
else
{
lvFileTransfer.Items.Add(blob.Uri);
}
}

Use Azure Search to index and search for files in Blob storage

Try this
if (AccountFileTransfer != null)
{
CloudBlobClient blobClient =
new CloudBlobClient(blobEndpoint,
new StorageCredentialsAccountAndKey(accountName, accountKey));
CloudBlobContainer container = blobClient.GetContainerReference(CONTAINER);
foreach (var blobItem in container .ListBlobs())
{
lvFileTransfer.Items.Add(blobItem .Uri);
}
}

Try this and If the blob in a Directory in container, in that case following format
container.GetBlobReference("Images/" + fileName);
public static bool BlobExists(CloudBlobContainer container, string fileName)
{
var blob = container.GetBlobReference(fileName);
try
{
blob.FetchAttributes();
return true;
}
catch (StorageException e)
{
if (e.RequestInformation.HttpStatusCode == (int)HttpStatusCode.NotFound)
{
return false;
}
}
return false;
}