There is an Azure function that is triggered when HTML files are placed into Azure blob storage.The function opens the HTML file, and transforms it into JSON. There is a small percentage of triggered files (less than 1%), that result in the following exception:
Microsoft.WindowsAzure.Storage.StorageException
There does happen to be a second function triggered by the placement of the blob that changes the files content type, but I am not sure if this is effecting the first function's ability to also open the file.
What can be done to allow the Azure functions to correctly process the HTML files without throwing this type of exception?
Exception properties:
Message: Exception while executing function: [Function name here] The condition specified using HTTP conditional header(s) is not met.
Exception type: Microsoft.WindowsAzure.Storage.StorageException
Failed method: HtmlAgilityPack.HtmlDocument.Load
Exception type: Microsoft.WindowsAzure.Storage.StorageException
Function 1 (supporting methods, class, and namespace omitted for brevity):
using System;
using System.Collections.Generic;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using HtmlAgilityPack;
using System.Threading.Tasks;
[FunctionName("Function name")]
public static async Task Run([BlobTrigger("container-name/html/{name}", Connection = "ConnectionString")]Stream myBlob, ILogger log, Binder binder)
{
var doc = new HtmlDocument();
doc.Load(myBlob);
var form = doc.DocumentNode.SelectSingleNode("//form");
var elements = form.SelectNodes("//input");
CustomType MyObject = BuildObject(elements);
var attributes = new Attribute[]
{
new BlobAttribute("container-name/json/" + MyObject.ID + ".json"),
new StorageAccountAttribute("ConnectionString")
};
using (var writer = await binder.BindAsync<TextWriter>(attributes))
{
writer.Write(BuildJSON(MyObject));
}
}
Function 2 same trigger but in a different function and it's own .cs file. Class and namespace omitted for brevity:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage.Blob;
[FunctionName("Function name")]
public static async Task Run([BlobTrigger("container-name/html/{name}", Connection = "ConnectionString")]ICloudBlob myBlob)
{
if (myBlob.Properties.ContentType == "text/html; charset=utf-8")
return;
myBlob.Properties.ContentType = "text/html; charset=utf-8";
await myBlob.SetPropertiesAsync();
}
I think your error should appear like this: Funtion1 retrieves the blob, and then function2's operation on the blob causes the change of Etag. Then function1 tries to load the retrieved blob, but finds that the Etag has changed, so it returns to you abnormal.
If the resource is accessed or changed by multiple apps, please have a try to make sure the orginal files are not changed. Otherwise the Etag of the blob will be changed automatically.
Azure Storage blob use 'strong Etag validation'. the content of the two resource representations must be byte-for-byte identical and that all other entity fields (such as Content-Language) are also unchanged.
Please refer to this:https://www.microsoftpressstore.com/articles/article.aspx?p=2224058&seqNum=12
Related
I'm using azure blobstorage in c#, is there a way, a method to get the list of files from a given specific folder?
like get all file names inside this url https://prueba.blob.core.windows.net/simem/UAL/Dato%20de%20archivo%20prueba%20No1/2022/1/16
i know that using container.GetBlobs() i would get all files but not from a specific folder
Just use
var results = await container.ListBlobsSegmentedAsync(prefix, true, BlobListingDetails.None, null, null, null, null);
You can get file names from a specific folder using BlobServiceClient and GetBlobs and by using below code in C# Console App and I followed Microsoft-Document and #Cindy Pau's answer:
using Azure.Storage.Blobs;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp4
{
class Program
{
static void Main(string[] args)
{
string cs= "Connection String of Storage Account";
string f = "test";
BlobServiceClient b = new BlobServiceClient(cs);
string c = "pool";
BlobContainerClient containerClient =b.GetBlobContainerClient(c);
var bs= containerClient.GetBlobs(prefix: f);
foreach (var x in bs)
{
Console.WriteLine(x.Name);
Console.ReadLine();
}
}
}
}
In Storage Account of pool Container:
Now inside test Folder:
Output:
Press Enter after every line to get File names one by one.
When configuring the output blob storage container for an Azure function, is it somehow possible to run some code in order to generate the path where the BLOB will be stored? To be more precise, I would like to use a new GUID within the path, every time this function would be triggered. Something like this (code does not work):
[FunctionName("BlobTriggered")]
public static void BlobTriggered(
[BlobTrigger("myContainer/{name}.{extension}")] Stream myBlob,
[Blob("myContainer/{Guid.NewGuid()}", FileAccess.Write)] Stream outputContainer,
string name,
string extension,
TraceWriter log)
{
...
}
In the code above, I am trying to generate the GUID by using Guid.NewGuid(), which doesn't work. Is there a similar way to achieve this?
You can set the variable in {} and set the corresponding parameter in the declaration section to get this value in the attribute. But because the parameters of the function declaration part must be fixed at compile time, I think your idea cannot be completed using binding. But you can still achieve what you want, please have a look of the below code, I am using Storage Blob SDK:
using System;
using System.IO;
using Azure.Storage.Blobs;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
namespace FunctionApp53
{
public static class Function1
{
[FunctionName("Function1")]
public static void Run([BlobTrigger("samples-workitems/{name}.{extension}", Connection = "str")]Stream myBlob,
string name, ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
string connectionString = "DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;xxx;EndpointSuffix=core.windows.net";
BlobServiceClient myClient = new BlobServiceClient(connectionString);
var container = myClient.GetBlobContainerClient("samples-workitems");
string a = Guid.NewGuid().ToString();
var blockBlob = container.GetBlobClient(a);
blockBlob.Upload(myBlob);
}
}
}
I am writing a Azure Function for PDF conversion with dependencies on DataLogics PDF conversion and a Nuget package (mlkpwgen) for password generation.
Functions are
using System.IO;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json;
using System;
using MlkPwgen;
using Datalogics.PDFL;
using System.Diagnostics;
namespace FunctionApp1
{
public static class Function1
{
[FunctionName("Function1")]
public static IActionResult Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequest req, TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
string name = req.Query["name"];
PDFConversion();
string requestBody = new StreamReader(req.Body).ReadToEnd();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
public static string PDFConversion()
{
using (Library lib = new Library())
{
String sInput = #"C:\Users\Kunal\Downloads\Indian Management.pdf";
String sOutput = #"C:\Users\Kunal\Downloads\WatermarkedOutput.pdf";
Document doc = new Document(sInput);
string ownerPassword = PasswordGenerator.Generate(length: 32);
string userPassword = PasswordGenerator.Generate(length: 32);
doc.Secure(PermissionFlags.Print | PermissionFlags.HighPrint, ownerPassword, userPassword);
WatermarkParams watermarkParams = new WatermarkParams();
watermarkParams.Rotation = 45.3f;
watermarkParams.Opacity = 0.15f;
watermarkParams.TargetRange.PageSpec = PageSpec.AllPages;
WatermarkTextParams watermarkTextParams = new WatermarkTextParams();
Color color = new Color(0.0f / 255.0f, 0.0f / 255.0f, 0.0f / 255.0f);
watermarkTextParams.Color = color;
watermarkTextParams.Text = "Centre Code - Unit - 0101";
Font f = new Font("Arial", FontCreateFlags.Embedded | FontCreateFlags.Subset);
watermarkTextParams.Font = f;
watermarkTextParams.FontSize = 80f;
watermarkTextParams.TextAlign = HorizontalAlignment.Center;
doc.Watermark(watermarkTextParams, watermarkParams);
doc.EmbedFonts();
doc.Save(SaveFlags.Full | SaveFlags.Linearized, sOutput);
Process.Start(#"C:\Users\Kunal\Downloads\WatermarkedOutput.pdf");
return sInput;
}
}
}
}
I am getting the following Exception
"System.Private.CoreLib: Exception while executing function:
Function1. Datalogics.PDFL: The type initializer for
'Datalogics.PDFL.PDFLPINVOKE' threw an exception. Datalogics.PDFL: The
type initializer for 'SWIGExceptionHelper' threw an exception.
Datalogics.PDFL: Unable to load DLL 'DL150PDFLPINVOKE': The specified
module could not be found. (Exception from HRESULT: 0x8007007E)."
The same code works fine as a Console application. What am I missing here?
If fixing the hard-coded file names still doesn't help, the error sounds like a permission exception.
Azure Functions run on App Service, which has a sandbox for all the code, where some calls are not allowed. E.g. GDI32 which is used extensively by PDF generation libraries.
Read more in Azure Web App sandbox.
Thanks for reading through the question and trying to answer.
I found that even after adding reference to the Datalogics.PDFL.dll, the code was failing.
So i copied all the other dll's into the bin\debug folder and now the code works fine
DL150ACE.dll
DL150AdobeXMP.dll
DL150AGM.dll
DL150ARE.dll
DL150AXE8SharedExpat.dll
DL150BIB.dll
DL150BIBUtils.dll
DL150CoolType.dll
DL150JP2KLib.dll
DL150PDFL.dll
DL150PDFLPINVOKE.dll
DL150pdfport.dll
DL150pdfsettings.dll
DotNETViewerComponent.dll
Per this MS Forums post:
Azure Functions does not provide support for loading native binaries in its current release. Even if we were able to install this package, you may still encounter errors when those native dlls are loaded during runtime.
So this is expected behavior when trying to call native binaries. Please contact our Support department if you have any more questions about getting started using the PDF Library.
I have been using edge.js to call a C# function from within my Node.js app, however when I go to execute the C# code I get for example:
Metadata file 'System.Collections.Generic.dll' could not be found
Metadata file 'System.Text.dll' could not be found
...
My code is this below, basically wanting to run a SSIS package using a stored procedure which I am calling from C#. Basically all my referenced dll's can't be found? Where should I put the dlls for edge to find them?
var executeSQL = edge.func(function() {
/*
#r "System.Data.dll"
#r "System.Collections.Generic.dll"
#r "System.Linq.dll"
#r "System.Text.dll"
using System.Linq;
using System.Text;
using System.Data;
using System.Collections.Generic;
using System.Threading.Tasks;
public class StartUp
{
public async Task<object> Invoke(object input)
{
string result = string.Empty;
string packagePath = #"\SSISDB\test\package.dtsx";
string spName = "storedProcName";
using (var conn = new System.Data.SqlClient.SqlConnection("connectionString"))
using (var command = new System.Data.SqlClient.SqlCommand(spName, conn)
{
CommandType = System.Data.CommandType.StoredProcedure
})
{
conn.Open();
command.Parameters.AddWithValue("#PackagePath", packagePath);
command.ExecuteNonQuery();
Console.WriteLine("Finished");
};
return null;
}
}
*/
});
I know I can do this without C# and just use a module within node like mssql to execute the stored procedure but this was just an example test to get used to using edge.js
The comment from stuartd was correct in the sense to put the dlls under the same directory as the script (which I had tried) but I was still having the same issue. I solved my problem by having my C# code as a separate file and then referenced that file as below as part of the executeSSIS function. payload is just the object that gets passed from my node.js script to my C# script. Doing it this way solved my issue.
var payload = {
filePath: 'C:/temp/xlsx/' + req.file.filename,
path: req.packageName,
server: req.server
};
var executeSSIS = edge.func({
source: __dirname + '/cs/Program.cs',
references: [
__dirname + '/cs/System.Data.dll'
]
});
executeSSIS(payload);
i want to make console c# program to download and execute a program.exe file from web.
so i get a "Webexception unhandle" error when i run this program.can any one help me. thanks
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Net;
using System.Diagnostics;
using System.IO;
namespace Downloader{
class Program
{
static void Main(string[] args)
{
Uri uri = new Uri("http://www.pendrivelinux.com/downloads/Universal-USB-Installer/Universal-USB-Installer-1.9.5.1.exe");
string filename = #"C:\bootable.exe";
using (var wc = new WebClient())
{
wc.Credentials = CredentialCache.DefaultCredentials;
wc.Headers.Add(HttpRequestHeader.UserAgent, "anything");
wc.DownloadFile(uri, filename);
}
}
}
}
can any one help me.
I'm guessing your WebException occurs on the last line (please in future include that in your question - we can't really help you based on guesswork).
wc.DownloadFile(uri, filename);
From the documentation on this method call, it will throw a WebException if one of the following is true:
The URI formed by combining BaseAddress and address is invalid.
filename is null or Empty.
The file does not exist.
An error occurred while downloading data.
We can rule out #2 immediately, as you've set that. To find out which of these situations you've got, try looking at the exception's message property (or the message property of the inner exception if there is one).