We Keep Coding

Problems generating a SAS token in C# for Blob paths with special characters

We are implementing a file store in our application and we store all the files in private containers in Azure Blob Storage. We have a virtual folder system which we replicate in our Blob storage.
For example, Let's say i work for Company A, and i upload file_1.txt to Folder #1, it will reside in /vault/Company A/Folder #1/file_1.txt in the Blob storage.
We generate SAS tokens using the following code:
public static Uri GetServiceSasUriForCloudBlockBlob(CloudBlockBlob cloudBlockBlob, string permissions = "r")
{
var sasBuilder = new SharedAccessBlobPolicy()
{
SharedAccessStartTime = DateTimeOffset.UtcNow.AddMinutes(-5),
SharedAccessExpiryTime = DateTimeOffset.UtcNow.AddMinutes(5),
Permissions = SharedAccessBlobPolicy.PermissionsFromString(permissions)
};
var sasUri = cloudBlockBlob.GetSharedAccessSignature(sasBuilder);
return new Uri(cloudBlockBlob.Uri + sasUri);
}
However, this does not work. The error we get is:
<Error>
<script type="text/javascript"/>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:f82118d1-101e-002a-1381-97ac16000000 Time:2022-07-14T12:55:34.6370028Z</Message>
<AuthenticationErrorDetail>Signature did not match. String to sign used was r 2022-07-14T12:50:27Z 2022-07-14T13:00:27Z /blob/[blobname]/vault/Company A/Folder #1/file_1.txt 2019-07-07 b </AuthenticationErrorDetail>
</Error>
When generating a SAS token from the Azure Portal or the Azure Storage Explorer there is no problem
It seems to be an issue with the special characters in the path to the file in the Blob. So we tried escaping all spaces and special characters manually to fix this issue, however when doing this the CloudBlockBlob encodes it again (e.g.: it escapes My%20File.txt to My%2520File.txt).
Currently the only operation we use is Read on Objects, but this may be expanded in the future.
We could disallow spaces and special character in folders/files but this doesn't feel like solving the issue but working around it. How can we fix this without implementing naming policies?
EDIT: Turns out this was a design issue, and while the SDK docs never explicitly disencourages the use unescaped blob paths it does disallow container names with anything other than alphanumerics and dashes.

For anyone having the same issue (whether on SDK version 11 or 12), i can highly recommend not using spaces/special characters without encoding them part by part,
var fileName = "file.txt"
// Note that the order here matters
var folderNames = ["Folder #1", "Folder #1.1"]
// becomes: Folder+%25231/Folder+%25231.1
var encodedPath = folderNames.Select(WebUtility.UrlEncode).Aggregate((x, y) => x + "/" + y);
// becomes: Folder+%25231/Folder+%25231.1/file.txt
var blobPath = ${encodedPath}/{fileName}"
This looks worse in Azure Storage Explorer but this does circumvent issues with encoding string programmatically

Using AWS SDK on .net with localstack (TransferUtility/S3 - setting endpoint)

I have localstack (https://github.com/localstack/localstack) running and am able to use the aws s3 cli to upload files to it.
What I want to be able to do is use the .NET AWS ADK with localstack. I'd like the following code to upload a file into localstack:
using (var tfu = new TransferUtility())
{
await tfu.UploadAsync(new TransferUtilityUploadRequest
{
Key = key,
BucketName = bucketName,
ContentType = document.ContentType,
Headers = { ["Content-Disposition"] = "attachment; filename=\"test.txt\"" },
InputStream = stream
});
}
My problem is I don't know how to set the endpoints so that localstack is used by the SDK rather than aws. Apparently you can set the AWSEndpointDefinition in appSettings.config as mentioned in the AWS SDK documentation, e.g:
<add key="AWSEndpointDefinition" value="C:\Dev\localstack\endpoints.json"/>
However I have no idea what this endpoints.json config should look like. I tried using this file:
https://raw.githubusercontent.com/aws/aws-sdk-net/master/sdk/src/Core/endpoints.json
When I do this, as soon as I new up a TransferUtility class I get a null reference exception - this is before I point anything to my localstack setup.
The version of AWS ASK is 3.3.0.
Another thing to note is that in some places in the documentation it is implied that the config should be an xml file rather than a json, however, when I try to use an xml file instead I get a different exception when newing up TransferUtility: 'Invalid character '<' in input string'

You can easily override it by creating an S3 client and passing it to TransferUtility constructor.
var config = new AmazonS3Config { ServiceURL = "http://localhost:4572" };
var s3client = new AmazonS3Client(config);
Do not forget to replace URL if your localstack is using different port for S3.
Hope this helps.

c# Azure Cannot set the blob tier

CloudBlockBlob doesn't have any method to set the blob tier to hot/cool/archive. I have also checked the other blob types and they do not have a method that allows this either.
I.E this method: https://learn.microsoft.com/en-us/rest/api/storageservices/set-blob-tier
Is their any way to change the blob tier in code from hot to cold in C# with azure storage?

I think the method is exactly what you need: CloudBlockBlob.SetStandardBlobTier. Maybe you were not checking the latest version of Azure Storage Client Library?

Is their any way to change the blob tier in code from hot to cold in C# with azure storage?
As ZhaoXing Lu mentioned that we could use CloudBlockBlob.SetStandardBlobTier.
Note: The operation is allowed on a page blob in a premium storage account and on a block blob in a blob storage account
The following code works correctly on my side. I use the library WindowsAzure.Storage 9.1.1
var cloudBlobClient = storageAccount.CreateCloudBlobClient();
var container = cloudBlobClient.GetContainerReference("container");
var blob = container.GetBlockBlobReference("blob name");
blob.SetStandardBlobTier(StandardBlobTier.Cool);
blob.FetchAttributes();
var tier = blob.Properties.StandardBlobTier;

Using Azure Blob storage client library v12 for .NET, replace myaccount with the name of your storage account, mycontainer with your container name and myblob with the blob name for which the tier is to be changed:
var sharedKeyCredential = new StorageSharedKeyCredential("myaccount", storageAccountKey);
var baseBlobContainerUrl = string.Format("{0}.blob.core.windows.net", "myaccount");
var blobServiceClient = new BlobServiceClient(new Uri($"https://{baseBlobContainerUrl}"), sharedKeyCredential);
var containerClient = blobServiceClient.GetBlobContainerClient("mycontainer");
BlobClient blobClient = containerClient.GetBlobClient("myblob");
// Set access tier to cool.
await blobClient.SetAccessTierAsync(AccessTier.Cool);
If you are working with Azure Gov, use this url insteadl "{0}.blob.core.usgovcloudapi.net"
Keep in mind, your storage account should support Cool Storage.

How to get all files from a directory in Azure BLOB

We are storing the application logs in Azure BLOB storage. We are currently downloading the files using the complete URI to the file. Every time when the settings of Azure BLOB is changed, the file name also gets changed. So we are getting error as file not exists.
Suggest a way to get the files using file extension from a directory without having file name in UI.

Use the ListBlobs() method to retrieve all blobs for a specific container and then you could use the static .NET Path.GetExtension() method to retrieve the file extension so you can filter them. Example:
var storageAccount = CloudStorageAccount.Parse("yourCS");
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(container);
var list = container.ListBlobs();
var blobs = list.OfType<CloudBlockBlob>()
.Where(b => Path.GetExtension(b.Name).Equals("yourextension"))

Get a reference of Azure blob by the full Uri not the blob name?

I 'm saving the uri of the file in the database in this form:
https://app.blob.core.windows.net/container/Accounts/Images/acc.jpg
But to delete it I need to pass only the blob name,
when I try this
CloudBlockBlob blockBlob = Container.GetBlockBlobReference(uri);
The blob's full uri becomes:
https://app.blob.core.windows.net/container/https://app.blob.core.windows.net/container/Accounts/Images/acc.jpg
So I get 404 error (not found),
I can do some trimming to the uri but that doesn't seem efficient.
so is there a way to delete a blob/ get reference by its full URI?

I did face similar issue , since i already had valid container reference this worked for me :
CloudBlockBlob blockblob = container.GetBlockBlobReference(new CloudBlockBlob(blobUri).Name);

It is possible to do this creating the CloudBlockBlob with this constructor:
public CloudBlockBlob(Uri blobAbsoluteUri)
In your case, assuming uri is of type Uri and not just a string:
CloudBlockBlob blob = new CloudBlockBlob(uri);
You might need to use your credentials if the blob isn't public or the uri doesn't contain SAS credentials (like to one you included). In that case you will need this constructor:
public CloudBlockBlob(Uri blobAbsoluteUri, StorageCredentials credentials)
As stated by Zhaoxing Lu - Microsoft on the comments,
Public access is read only access, you need to specify the storage
account key or Shared Access Signature for deleting operation.

You can now use the BlobUriBuilder class to safely get the blob name from the URI without the need for string parsing or other methods.
BlobUriBuilder blobUriBuilder = new BlobUriBuilder(new Uri(blobUri));
var sourceBlob = container.GetBlobClient(blobUriBuilder.BlobName);

Your problem is that you're putting the URI string inside the blob name with this function GetBlockBlobReference as defined: public virtual CloudBlockBlob GetBlockBlobReference(string blobName);. You will use moondaisy solution's.