azure function binding causing multiple events fired - c#

Why are we getting duplicate events for the same BlobCreated?
I'm using the following binding in my function app:
[Blob("%Detach:Output%/{file}", Write)] CloudBlockBlob #out,
The only time when I am writing to this output binding is here:
await #out.UploadTextAsync(xml);
I have an event defined like so:
Where Detach:Output env variable is xml-output-with-links-container.
I am consistently getting 2 events for every execution of this function:
The eventTime between the two events are slightly different:
2019-08-05T22:27:06.5279893Z
2019-08-05T22:27:06.5019647Z
We know that they are firing for the same blob because the subject of the event identifies which blob it is:
"subject": "/blobServices/default/containers/xml-output-with-links-container/blobs/tobossthe_awesome_blob.xml",
I've tested this manually by uploading a payload to xml-output-with-links-container and have gotten just 1 event to fire. Yet, when the function is executed, two events are created.
Why are we getting duplicate events?
Here's the entire function:
{
[FunctionName("Detach")]
[StorageAccount("Global:Connection")]
public static async Task Run(
[QueueTrigger("%Detach:Trigger%")] DetachJob detach,
[Blob("%Detach:Input%/{file}", Read)] CloudBlockBlob #in,
[Blob("%Detach:Output%/{file}", Write)] CloudBlockBlob #out,
[Blob("%Detach:Error%/{file}", Write)] CloudBlockBlob fail,
[Blob("%Detach:Error%/{file}_error", Write)] CloudBlockBlob error,
[Blob("%Detach:Attachments%", Write)] CloudBlobContainer attachments)
{
try
{
var xml = await #in.DownloadTextAsync();
var size = ToInt32(GetEnvironmentVariable("Detach:BytesThreshold"));
var bigNodes = GetNodesGreaterThan(size, xml);
foreach (var node in bigNodes)
{
var ext = GetExtension(node.Value);
var path = $"{detach.file}/{node.Key}{ext}.base64";
var attachment = attachments.GetBlockBlobReference(path);
await attachment.UploadTextAsync(node.Value);
xml = xml.Replace(node.Value, path);
}
await #out.UploadTextAsync(xml);
}
catch (Exception e)
{
await error.UploadTextAsync(e.ToString());
await fail.StartCopyAsync(#in);
}
}
}
I was thinking that perhaps the CloudBlockBlob was triggering twice. So I changed the binding to be CloudBlobContainer:
[Blob("%Detach:Output%", Write)] CloudBlobContainer #out,
And updated the respective code:
var shrink = #out.GetBlockBlobReference(file);
await shrink.UploadTextAsync(xml);
Yet the result stayed the same: I still got 2 events.

I was triggering the Detach function by dropping a payload into blob storage using a logic app with the following step:
This step was generating 2 BlobCreated events!
After turning off thunking, the issue has been resolved, and only 1 BlobCreated event is now generating:

Related

Add BackgroundImage with EPPlus only allows path but cannot get path in Blazor WASM

This may not be 100% an EPPlus issue, but since it is Blazor WASM it appears I cannot get the file path to a static image in the wwwroot/images folder. I can get the url and paste it into a browser and that works, even adding that same path to the src attribute of an img works, neither of those helps me.
FYI "background" in this context means a watermark.
It appears that the EPPlus dev team only wants a drive path the file (ex. C:\SomeFolder\SomeFile.png), and I am not seeing how to get that within Blazor WASM. I can get the bytes of the file in c# and even a stream, but no direct path.
My code is the following:
using (var package = new ExcelPackage(fileName))
{
var sheet = package.Workbook.Worksheets.Add(exportModel.OSCode);
sheet.BackgroundImage.SetFromFile("https://localhost:44303/images/Draft.png");
...
}
This returns an exception:
Unhandled exception rendering component: Can't find file /https:/localhost:44303/images/Draft.png
Noticing that leading / I even tried:
sheet.BackgroundImage.SetFromFile("images/Draft.png");
Which returned the same error:
Unhandled exception rendering component: Can't find file /images/Draft.png
So, I am perhaps needing 1 of 2 possible answers:
A way to get a local drive path to the file so the .SetFromFile method is not going to error.
To have a way to set that BackgroundImage property with a byte array or stream of the image. There is this property BackgroundImage.Image but it is readonly.
Thanks to a slap in the face from #Panagiotis-Kanavos I wound up taking the processing out of the client and moving it to the server. With that, I was able to use Static Files to add the watermark with relatively little pain.
In case anyone may need the full solution (which I always find helpful) here it is:
Here is the code within the button click on the Blazor component or page:
private async Task GenerateFile(bool isFinal)
{
...
var fileStream = await excelExportService.ProgramMap(exportModel);
var fileName = "SomeFileName.xlsx";
using var streamRef = new DotNetStreamReference(stream: fileStream);
await jsRuntime.InvokeVoidAsync("downloadFileFromStream", fileName, streamRef);
}
That calls a client-side service that really just passes control over to the server:
public class ExcelExportService : IExcelExportService
{
private const string baseUri = "api/excel-export";
private readonly IHttpService httpService;
public ExcelExportService(IHttpService httpService)
{
this.httpService = httpService;
}
public async Task<Stream> ProgramMap(ProgramMapExportModel exportModel)
{
return await httpService.PostAsJsonForStreamAsync<ProgramMapExportModel>($"{baseUri}/program-map", exportModel);
}
}
Here is the server-side controller that catches the call from the client:
[Route("api/excel-export")]
[ApiController]
public class ExcelExportController : ControllerBase
{
private readonly ExcelExportService excelExportService;
public ExcelExportController(ExcelExportService excelExportService)
{
this.excelExportService = excelExportService;
}
[HttpPost]
[Route("program-map")]
public async Task<Stream> ProgramMap([FromBody] ProgramMapExportModel exportModel)
{
return await excelExportService.ProgramMap(exportModel);
}
}
And that in-turn calls the server-side service where the magic happens:
public async Task<Stream> ProgramMap(ProgramMapExportModel exportModel)
{
var result = new MemoryStream();
ExcelPackage.LicenseContext = LicenseContext.Commercial;
var fileName = #$"Gets Overwritten";
using (var package = new ExcelPackage(fileName))
{
var sheet = package.Workbook.Worksheets.Add(exportModel.OSCode);
if (!exportModel.IsFinal)
{
var pathToDraftImage = #$"{Directory.GetCurrentDirectory()}\StaticFiles\Images\Draft.png";
sheet.BackgroundImage.SetFromFile(pathToDraftImage);
}
...
sheet.Cells.AutoFitColumns();
package.SaveAs(result);
}
result.Position = 0; // Without this, data does not get written
return result;
}
For some reason, this next method was not needed when doing this on the client-side but now that it is back here, I had to add a method that returned a stream specifically and used the ReadAsStreamAsync instead of ReadAsJsonAsync:
public async Task<Stream> PostAsJsonForStreamAsync<TValue>(string requestUri, TValue value, CancellationToken cancellationToken = default)
{
Stream result = default;
var responseMessage = await httpClient.PostAsJsonAsync(requestUri, value, cancellationToken);
try
{
result = await responseMessage.Content.ReadAsStreamAsync(cancellationToken: cancellationToken);
}
catch (HttpRequestException e)
{
...
}
return result;
}
Lastly, in order for it to give the end-user a download link, this was used (taken from the Microsoft Docs):
window.downloadFileFromStream = async (fileName, contentStreamReference) => {
const arrayBuffer = await contentStreamReference.arrayBuffer();
const blob = new Blob([arrayBuffer]);
const url = URL.createObjectURL(blob);
const anchorElement = document.createElement("a");
anchorElement.href = url;
anchorElement.download = fileName ?? "";
anchorElement.click();
anchorElement.remove();
URL.revokeObjectURL(url);
}

Creating a container in azure exits the method

I'm learning Azure storage development and I'm trying to create a container and then upload a file to it. I created a console app and am calling the upload method from Main. Here is the first half of the upload method that creates the container.
public async static Task<bool> UploadToAzure()
{
string containerName = "sample-"+Guid.NewGuid().ToString();
var connectionString = "XXXXXXXXXXXXXXXXXXXXX";
var blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
try
{
var test = blobServiceClient.GetBlobContainers();
foreach (var container in test)
{
Console.WriteLine(container.Name);
}
var result = await containerClient.CreateIfNotExistsAsync(Azure.Storage.Blobs.Models.PublicAccessType.BlobContainer);
var containerProp = await containerClient.GetPropertiesAsync();
Console.WriteLine(containerProp.Value);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message+"\n}");
}
I started with just calling the containerclient.CreateIfNotExistsAnsync() and it wouldn't create the container. On a whim, I added the blobServiceClient.GetBlobContainers() and now it will create the container. I can comment out blobServiceClient.GetBlobContainers() and it will not create the container. I can put it after the create method and it still won't work. That's the first issue.
The second one is, it exits the code when inside the containerclient.CreateIfNotExistsAnsync() method. It will create the folder but never hit the next line of code. I get a message in the console that it "exited with code 0. Press any key to close this window...". It doesn't hit the Catch or display any kind of error, it just exists.
I got this pattern from my Udemy course and I've read a number of articles online and these seem to be the right steps. Any suggestions are appreciated.
Thanks
After reproducing from our end, The code you were using was working for us until we found that you have missed using await while calling UploadToAzure().
Below is the code that we used
namespace ConsoleApplication
{
class Program
{
static async Task Main(string[] args)
{
await UploadToAzure();
}
public async static Task<bool> UploadToAzure()
{
string containerName = "sample-" + Guid.NewGuid().ToString();
var connectionString = "<YOUR_COONECTION_STRING>";
var blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
await containerClient.CreateIfNotExistsAsync(Azure.Storage.Blobs.Models.PublicAccessType.BlobContainer);
return true;
}
}
}
else you can completely remove async and have
namespace CreateContainer
{
class Program
{
static async Task Main(string[] args)
{
UploadToAzure();
}
public static bool UploadToAzure()
{
string containerName = "sample-" + Guid.NewGuid().ToString();
var connectionString = "<YOUR_COONECTION_STRING>";
var blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
containerClient.CreateIfNotExists(Azure.Storage.Blobs.Models.PublicAccessType.BlobContainer);
return true;
}
}
}
NOTE: It is preferable to use the async method since you might face some issues if you are dealing with large volumes of data.

Error when attaching file to System.Net.Mail. How do I fix this error?

Trying to send an email with an attachment. However I am getting an error:
"Cannot convert from
'System.Threading.Tasks.Task' to
'System.Net.Mail.Attachment'
My error occurs in the line Attachments.Add(GetAttachment(attachmentFileName));
I have tried various conversions (see code) but I dont quite see what the issue is. I know the solution is right in front of me but I dont see it.
public class NonFERosterEmail : BaseNotificationEmail<OfferViewModel>
{
public NonFERosterEmail(OfferViewModel vm, string emailList, string attachmentFileName) : base(vm)
{
To.AddRange(GetTo(emailList));
Body = GetBody();
Subject = GetSubject();
//Attachments.Add(new Attachment(GetAttachment(attachmentFileName)));
Attachments.Add(GetAttachment(attachmentFileName));
From = new MailAddress(ConfigurationManager.AppSettings["RedirectEmailTo"]);
}
//public async Task<List<Attachment>> GetAttachment(string attachmentFileName)
public async Task<Attachment> GetAttachment(string attachmentFileName)
{
//var ret = new List<Attachment>();
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["azureStorageAccount"]);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("attachments");
CloudBlockBlob blob = container.GetBlockBlobReference(attachmentFileName);
var contentType = MimeMapping.GetMimeMapping(attachmentFileName);
Stream target = new MemoryStream();
await blob.DownloadToStreamAsync(target);
target.Position = 0;
//ret.Add(new Attachment(target, attachmentFileName, contentType));
Attachment ret = new Attachment(target, attachmentFileName, contentType);
return ret;
}
//remainder of code left out for brevity
}
I expect the GetAttachment to return a correct Attachment object which would be added to the Mail object and sent successfully.
I believe the answers from #SLaks and #Roman Marusyk are correct, but it looks like you are calling GetAttachment from the constructor, which is not asynchronous. As such you would not be able to use await without using an async method. Try using the result property of GetAttachment as shown below.
Attachments.Add(GetAttachment(attachmentFileName).Result);
A better solution would be to use .GetAwaiter().GetResult(), which as #Roman Marusyk pointed out and shown in this post, if the method fails it will throw the exception directly rather than throwing an AggregateException.
A better solution would be to use the following
Attachments.Add(GetAttachment(attachmentFileName).GetAwaiter().GetResult());
To get the value from a Task<T>, you must make your method async and await the task.
You need to await when call method that returns Task, so instead of this
Attachments.Add(GetAttachment(attachmentFileName));
Use:
Attachments.Add(await GetAttachment(attachmentFileName));
or
Attachments.Add(GetAttachment(attachmentFileName).GetAwaiter().GetResult());

How to trigger a ServiceBusTrigger?

I have an Azure WebJob which has a similar code inside:
public class Functions
{
public static void GenerateImagesForViewer(
[QueueTrigger("resize-images-queue")] BlobInformation blobInfo,
[Blob("unprocessed-pdf-storage-container/{BlobName}", FileAccess.Read)] Stream input,
[Blob("unprocessed-pdf-storage-container/{BlobNameWithoutExtention}-pdf.jpg")] CloudBlockBlob outputPdf)
{
//Do something here
string connectionString = "myConnectionString";
TopicClient Client =
TopicClient.CreateFromConnectionString(connectionString, "resize-
images-topic");
var topicMessage = new BrokeredMessage(blobInfo);
Client.Send(topicMessage);
}
public static void GenerateImagesForViewerW80(
[ServiceBusTrigger("resize-images-topic", "SizeW80")] BlobInformation blobInfo,
[Blob("unprocessed-pdf-storage-container/{BlobNameWithoutExtention}-pdf.jpg", FileAccess.Read)] Stream input,
[Blob("processed-image-storage-container/{BlobNameWithoutExtention}-h0-w80.jpg")] CloudBlockBlob outputBlob_0_80)
{
// It never comes here
//Do something here
}
}
After uploading data (BlobInformation object) to my Queue there is no problem triggering the first method (GenerateImagesForViewer). But when I try to send data (BlobInformation object) to the topic it never triggers any of the subscribers(GenerateImagesForViewerW80). Is there something wrong in the code, or there is a required configuration in Azure?
In Program.cs, config.UseServiceBus(); is necessary for usage of ServiceBus trigger. We won't see warning if there are other trigger or bindings in Functions, like your case.
See code sample below and check official guidance for more details.
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
config.UseServiceBus();
var host = new JobHost(config);
host.RunAndBlock();
Besides, I see some suspicious blank in your input and output blob path. If it's the same as your original code, just remove them otherwise the trigger won't execute code related to blob operation correctly.

Checking if a blob exists in Azure Storage

I've got a very simple question (I hope!) - I just want to find out if a blob (with a name I've defined) exists in a particular container. I'll be downloading it if it does exist, and if it doesn't then I'll do something else.
I've done some searching on the intertubes and apparently there used to be a function called DoesExist or something similar... but as with so many of the Azure APIs, this no longer seems to be there (or if it is, has a very cleverly disguised name).
The new API has the .Exists() function call. Just make sure that you use the GetBlockBlobReference, which doesn't perform the call to the server. It makes the function as easy as:
public static bool BlobExistsOnCloud(CloudBlobClient client,
string containerName, string key)
{
return client.GetContainerReference(containerName)
.GetBlockBlobReference(key)
.Exists();
}
Note: This answer is out of date now. Please see Richard's answer for an easy way to check for existence
No, you're not missing something simple... we did a good job of hiding this method in the new StorageClient library. :)
I just wrote a blog post to answer your question: http://blog.smarx.com/posts/testing-existence-of-a-windows-azure-blob.
The short answer is: use CloudBlob.FetchAttributes(), which does a HEAD request against the blob.
Seem lame that you need to catch an exception to test it the blob exists.
public static bool Exists(this CloudBlob blob)
{
try
{
blob.FetchAttributes();
return true;
}
catch (StorageClientException e)
{
if (e.ErrorCode == StorageErrorCode.ResourceNotFound)
{
return false;
}
else
{
throw;
}
}
}
If the blob is public you can, of course, just send an HTTP HEAD request -- from any of the zillions of languages/environments/platforms that know how do that -- and check the response.
The core Azure APIs are RESTful XML-based HTTP interfaces. The StorageClient library is one of many possible wrappers around them. Here's another that Sriram Krishnan did in Python:
http://www.sriramkrishnan.com/blog/2008/11/python-wrapper-for-windows-azure.html
It also shows how to authenticate at the HTTP level.
I've done a similar thing for myself in C#, because I prefer to see Azure through the lens of HTTP/REST rather than through the lens of the StorageClient library. For quite a while I hadn't even bothered to implement an ExistsBlob method. All my blobs were public, and it was trivial to do HTTP HEAD.
The new Windows Azure Storage Library already contains the Exist() method.
It´s in the Microsoft.WindowsAzure.Storage.dll.
Available as NuGet Package
Created by: Microsoft
Id: WindowsAzure.Storage
Version: 2.0.5.1
See also msdn
Here's a different solution if you don't like the other solutions:
I am using version 12.4.1 of the Azure.Storage.Blobs NuGet Package.
I get an Azure.Pageable object which is a list of all of the blobs in a container. I then check if the name of the BlobItem equals to the Name property of each blob inside the container utilizing LINQ. (If everything is valid, of course)
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using System.Linq;
using System.Text.RegularExpressions;
public class AzureBlobStorage
{
private BlobServiceClient _blobServiceClient;
public AzureBlobStorage(string connectionString)
{
this.ConnectionString = connectionString;
_blobServiceClient = new BlobServiceClient(this.ConnectionString);
}
public bool IsContainerNameValid(string name)
{
return Regex.IsMatch(name, "^[a-z0-9](?!.*--)[a-z0-9-]{1,61}[a-z0-9]$", RegexOptions.Singleline | RegexOptions.CultureInvariant);
}
public bool ContainerExists(string name)
{
return (IsContainerNameValid(name) ? _blobServiceClient.GetBlobContainerClient(name).Exists() : false);
}
public Azure.Pageable<BlobItem> GetBlobs(string containerName, string prefix = null)
{
try
{
return (ContainerExists(containerName) ?
_blobServiceClient.GetBlobContainerClient(containerName).GetBlobs(BlobTraits.All, BlobStates.All, prefix, default(System.Threading.CancellationToken))
: null);
}
catch
{
throw;
}
}
public bool BlobExists(string containerName, string blobName)
{
try
{
return (from b in GetBlobs(containerName)
where b.Name == blobName
select b).FirstOrDefault() != null;
}
catch
{
throw;
}
}
}
Hopefully this helps someone in the future.
This is the way I do it. Showing full code for those who need it.
// Parse the connection string and return a reference to the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("AzureBlobConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("ContainerName");
// Retrieve reference to a blob named "test.csv"
CloudBlockBlob blockBlob = container.GetBlockBlobReference("test.csv");
if (blockBlob.Exists())
{
//Do your logic here.
}
If your blob is public and you need just metadata:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "HEAD";
string code = "";
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
code = response.StatusCode.ToString();
}
catch
{
}
return code; // if "OK" blob exists
If you don't like using the exception method then the basic c# version of what judell suggests is below. Beware though that you really ought to handle other possible responses too.
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create(url);
myReq.Method = "HEAD";
HttpWebResponse myResp = (HttpWebResponse)myReq.GetResponse();
if (myResp.StatusCode == HttpStatusCode.OK)
{
return true;
}
else
{
return false;
}
With the updated SDK, once you have the CloudBlobReference you can call Exists() on your reference.
See http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.cloudblockblob.exists.aspx
Although most answers here are technically correct, most code samples are making synchronous/blocking calls. Unless you're bound by a very old platform or code base, HTTP calls should always be done asynchonously, and the SDK fully supports it in this case. Just use ExistsAsync() instead of Exists().
bool exists = await client.GetContainerReference(containerName)
.GetBlockBlobReference(key)
.ExistsAsync();
With Azure Blob storage library v12, you can use BlobBaseClient.Exists()/BlobBaseClient.ExistsAsync()
Answered on another similar question: https://stackoverflow.com/a/63293998/4865541
Java version for the same ( using the new v12 SDK )
This uses the Shared Key Credential authorization (account access key)
public void downloadBlobIfExists(String accountName, String accountKey, String containerName, String blobName) {
// create a storage client using creds
StorageSharedKeyCredential credential = new StorageSharedKeyCredential(accountName, accountKey);
String endpoint = String.format(Locale.ROOT, "https://%s.blob.core.windows.net", accountName);
BlobServiceClient storageClient = new BlobServiceClientBuilder().credential(credential).endpoint(endpoint).buildClient();
BlobContainerClient container = storageClient.getBlobContainerClient(containerName);
BlobClient blob = container.getBlobClient(blobName);
if (blob.exists()) {
// download blob
} else {
// do something else
}
}

Categories