I have an http trigger that calls a support function to perform a SQL update on some JSON. I made a mistake in the support function and used command.ExecuteNonQueryAsync() by accident. This code ran just fine locally in VS Studio, 100% of the time. But, when I published to Azure the function would work once. The only way to make it work again was start/stop the function. I performed a memory dump and saw there was one thread running. The HTTP request always returned 200, even though it only changed the data once.
Now my question. The Azure HTTP function does not wait for the threads to finish before exiting? I am just trying to understand better Azure threading.
Thanks
public static class SetNotificationConfig
{
[FunctionName("SetNotificationConfig")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
try
{
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject<dynamic>(requestBody);
var tem = data["NotificationConfig"];
string notificationConfig = JsonConvert.SerializeObject(tem);
SchedulerSupport.SetNotificationConfigJSON(1, notificationConfig);
}
catch (Exception ex)
{
return new BadRequestObjectResult("Error updating notification configuration: " + ex.Message);
}
return new OkResult();
}
}
public static void SetNotificationConfigJSON(int storeId, string notificationConfig)
{
if (notificationConfig.Length > 0)
{
using SqlConnection connection = new SqlConnection(SchedulerSupport.ConnectionStr());
connection.Open();
var queryString = "Update SchedulerConfig Set ConfigInfo = JSON_MODIFY(ConfigInfo, '$.NotificationConfig', JSON_QUERY(#notificationConfig)) Where StoreId = #StoreId";
using SqlCommand command = new SqlCommand(queryString, connection);
try
{
command.Parameters.Add(new SqlParameter("#StoreId", System.Data.SqlDbType.Int));
command.Parameters["#StoreId"].Value = storeId;
command.Parameters.AddWithValue("#NotificationConfig", notificationConfig);
command.ExecuteNonQuery();
}
catch
{
throw;
}
}
}
Related
I have an Azure Function in C# which I have almost built. It is setup to receive data from SendGrid, verify the data and then write the data to Cosmos. Here is the code:
namespace SendGrid.CosmosLogging
{
public static class SendGrid_Logging
{
[FunctionName("SendGrid_Logging")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]
HttpRequest req,
[CosmosDB(
databaseName: "portal-dev",
collectionName: "sendgrid",
ConnectionStringSetting = "CosmosDbConnectionString")]
IAsyncCollector<dynamic> documentsOut,
ILogger log)
{
try
{
log.LogInformation("SendGrid C# HTTP trigger function started processing a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
var headers = req.Headers;
var sgbody = req.Body;
//Check validity
CheckValidity(log, documentsOut, sgbody); //req.Body - has to be Stream object
}
catch (Exception e)
{
log.LogInformation("Error validating SendGrid event, error is: "+ e.ToString());
throw;
}
}
private static async void CheckValidity(ILogger log, IAsyncCollector<dynamic> documentsOut, Stream sgBody)
{
// Add a JSON document to the output container.
await documentsOut.AddAsync( new
{
//Write the sendgrid body only to the Cosmos container
sgBody
//Body = "This is something"
//name = name
});
responseMessage =
"The SendGrid HTTP triggered function executed successfully processing Id: ";//${doc_id}
log.LogInformation(responseMessage);
}
}
}
The problem is the line writing the sgBody during the CheckValidity method. This never comes through to the Cosmos container. However if it is replaced with something like:
// Add a JSON document to the output container.
await documentsOut.AddAsync( new
{
//Write the sendgrid body only to the Cosmos container
//sgBody
Body = "This is something"
//name = name
});
Then I get an output that looks like this in Cosmos:
{
"Body": "This is something",
"id": "e65c2efe-79d2-4997-b1b2-33833dbf14ce",
"_rid": "2WwUAKT2MsvdKgAAAAAAAA==",
"_self": "dbs/2WwUAA==/colls/2WwUAKT2Mss=/docs/2WwUAKT2MsvdKgAAAAAAAA==/",
"_etag": "\"12015ff1-0000-1a00-0000-61679d8c0000\"",
"_attachments": "attachments/",
"_ts": 1634180492
}
So my question is, how do I get the entire contents of the sgBody variable to write? There is something I am clearly missing here (I aren't a C# developer, but am trying my best!).
EDIT:
The format I am wanting to get as output to the Cosmos container is a properly formatted JSON body, something like:
{
"body":{
"timestamp":1631141801,
"sg_template_name":"Receiver - NonProd",
"ip":"149.72.246.163",
"tls":1,
"smtp-id":"<2MhvBnobRN-KQNdFF9cO4w#geopod-ismtpd-5-0>",
"email":"somewhere#something.com",
"response":"250 2.6.0 <2MhvBnobRN-KQNdFF9cO4w#geopod-ismtpd-5-0> [InternalId=300647724419, Hostname=SY4P282MB3809.AUSP282.PROD.OUTLOOK.COM] 289934 bytes in 0.131, 2154.683 KB/sec Queued mail for delivery",
"sg_message_id":"2MhvBnobRN-KQNdFF9cO4w.filterdrecv-75ff7b5ffb-vm78l-1-61393FA4-43.0",
"event":"delivered",
"sg_event_id":"ZGVsaXZlcmVkLTAtMjI4MDI0MTAtMk1odkJub2JSTi1LUU5kRkY5Y080dy0w",
"sg_template_id":"d-04096fb423674bdf8870dfc92eec944f",
"category":[
"develop",
"Something"
]
},
"id":"0c4143fa-d5a2-43e8-864f-82f333ace3cd",
"_rid":"SL81APS-4Q21KQAAAAAAAA==",
"_self":"dbs/SL81AA==/colls/SL81APS-4Q0=/docs/SL81APS-4Q21KQAAAAAAAA==/",
"_etag":"\"7700726c-0000-1a00-0000-61393fb20000\"",
"_attachments":"attachments/",
"_ts":1631141810
}
OK, I managed to do this myself. This will be particularly helpful to anyone using an Azure function to consume SendGrid events, which are then written to a Cosmos database, but obviously useful in other situations too. I have also added in headers, which come through as an iHeaderDictionary type, which is a bit hard to work with when you want standard JSON.
namespace SendGrid.CosmosLogging
{
public static class SendGrid_Logging
{
[FunctionName("SendGrid_Logging")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]
HttpRequest req,
[CosmosDB(
databaseName: "portal-dev",
collectionName: "sendgrid",
ConnectionStringSetting = "CosmosDbConnectionString")]
IAsyncCollector<dynamic> documentsOut,
ILogger log)
{
try
{
log.LogInformation("SendGrid C# HTTP trigger function started processing a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
var headers = req.Headers;
var sgbody = req.Body;
//Create a new dictionary and fill it with the items in the existing headers. This is done due to needing to
//convert the headers to a dynamic object for the Cosmos write. Can only easily stringify a Dictionary object,
//but not the specific HttpRequest header object, so have to convert.
Dictionary<string,string> ihHeaders = new Dictionary<string,string>
{};
foreach (var item in req.Headers)
{
ihHeaders.Add(item.Key,item.Value);
}
string jsonHeaders = JsonConvert.SerializeObject(ihHeaders,Formatting.Indented);
//Use these dynamic objects for writing to Cosmos
dynamic sendgriddata = JsonConvert.DeserializeObject(requestBody);
dynamic sendgridheaders = JsonConvert.DeserializeObject(jsonHeaders);
//Check validity
CheckValidity(log, documentsOut, sendgriddata, sendgridheaders); //req.Body - has to be Stream object
}
catch (Exception e)
{
log.LogInformation("Error validating SendGrid event, error is: "+ e.ToString());
throw;
}
}
private static async void CheckValidity(ILogger log, IAsyncCollector<dynamic> documentsOut, dynamic sgBody, dynamic sgHeaders)
{
// Add a JSON document to the output container.
await documentsOut.AddAsync( new
{
//Write the sendgrid body only to the Cosmos container
headers = sgHeaders
body = sgBody
});
responseMessage =
"The SendGrid HTTP triggered function executed successfully processing Id: ";//${doc_id}
log.LogInformation(responseMessage);
}
}
This gives a nicely formatted JSON output in Cosmos:
I'm getting this weird issue when using Durable Azure functions to submit messages to azure service bus.
My code is a simple Fan-Out implementation
REST trigger get the number of messages to be submitted and hands that an orchestrator.
Orchestrator stores the calls activity which will create and submit the message to Service bus.
The issue is when I send the REST parameter asking to add 3000 messages, more than 3000 get added.
Worse, it's not the same number either - 3104, 3100, 3286 anything...
See code below:
[FunctionName("Function1_HttpStart")]
//public static async Task<HttpResponseMessage> HttpStart(
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
String type = req.Query["type"];
if(!long.TryParse(req.Query["count"], out var count))
{
return new ObjectResult($"Parse failed for parameter 'count' ({req.Query["count"]}) to Int.") { StatusCode = 400};
}
var restInputs = new RestInputs()
{ Type = type, Count = count };
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync
("EmailQueueSubmitter_OrchestratorSingleton"
, restInputs);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
[FunctionName("EmailQueueSubmitter_OrchestratorSingleton")]
public static async Task<List<string>> EmailQueueSubmitter_OrchestratorSingleton(
[OrchestrationTrigger] IDurableOrchestrationContext context, ILogger log)
{
var outputs = new List<string>();
try
{
var restInputs = context.GetInput<RestInputs>();
var parallelTasks = new List<Task>();
long runBatchLen;
long i_batch, i_iter, batchCount = 0;
for (i_batch = 0; i_batch < restInputs.Count; i_batch++)
{
parallelTasks.Add(context.CallActivityAsync("EmailQueueSubmitter_ActivitySendMessageBatchSingleton", i_batch.ToString()));
log.LogWarning($"Message {i_batch} Added");
}
log.LogWarning($"Awaiting {parallelTasks.Count} tasks");
await Task.WhenAll(parallelTasks);
var doneTaskCount = parallelTasks.Where(t => t.IsCompleted).ToList().Count;
var successTaskCount = parallelTasks.Where(t => t.IsCompletedSuccessfully).ToList().Count;
var faultedTaskCount = parallelTasks.Where(t => t.IsFaulted).ToList().Count;
var exceptionTaskCount = parallelTasks.Where(t => t.Exception != null).ToList().Count;
log.LogWarning($"Done:{doneTaskCount}, Success: {successTaskCount}, Fault:{faultedTaskCount}, Exception:{exceptionTaskCount}");
log.LogWarning($"Achieved completion.");
}
catch (Exception ex)
{
log.LogError(ex.Message);
throw new InvalidOperationException(ex.Message);
}
return outputs;
}
[FunctionName("EmailQueueSubmitter_ActivitySendMessageBatchSingleton")]
public static async Task EmailQueueSubmitter_ActivitySendMessageBatchSingleton([ActivityTrigger] IDurableActivityContext activityContext, ILogger log)
{
log.LogWarning($"Starting Activity.");
var payload = activityContext.GetInput<String>();
await ServiceBus_Sender.SendMessageBatch(payload);
log.LogWarning($"Finished Activity.");
}
public static ServiceBusMessage CreateMessage(String Payload)
{
try
{
var sbMsg = new ServiceBusMessage(Payload)
{
MessageId = Guid.NewGuid().ToString(),
ContentType = "text/plain"
};
//sbMsg.ApplicationProperties.Add("RequestType", "Publish");
return sbMsg;
}
catch (Exception ex)
{
throw new InvalidOperationException(ex.Message, ex);
}
}
Thanks #Camilo Terevinto for the information, I am converting this to an answer so that may help other community members:
As suggested in the comments, to run a duplicate check you could generate a Guid and send it together with the data, and then check that the Guid wasn't handled already before. Hopefully this resolved your issue.
OP Edit: Duplicates check was enabled by changing the service bus queue to be session enabled and have de-duplication turned on. The submitted messages' MessageId was set to be unique in each session. This is the only way I can think of to deal with the at-least-once guarantees...
I am creating a microservice where one app is sending selected filters to other app using azure service bus queue.
I am able to send and receive the message however unable to use received message in my SQL Query.
The API is getting hit by one application (frontend).
.../api/User
Our controller
public class UserController : ControllerBase
{
public IEnumerable<dynamic> Get()
{
return userRepository.GetAll();
}
}
GetAll method
public IEnumerable<dynamic> GetAll()
{
ReceiveMsg().GetAwaiter().GetResult(); // We have called receiveMsg from here
startdate = content1[0];
enddate = content1[1];
using (IDbConnection dbConnection = connection)
{
var result = connection.Query("select * from [User] where DateofBirth between '" + startdate + "' and'" + enddate + "'");
return result;
}
}
Receive Message method`
public static async Task ReceiveMsg()
{
//
string sbConnectionString = <connection string for Service Bus namespace>;
string sbQueueName = <Queue name>;
try
{
queueClient = new QueueClient(sbConnectionString, sbQueueName);
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
queueClient.RegisterMessageHandler(ReceiveMessagesAsync, messageHandlerOptions);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
Console.ReadKey();
await queueClient.CloseAsync();
}
}
public static async Task ReceiveMessagesAsync(Message message, CancellationToken token)
{
Debug.WriteLine($"Received message: {Encoding.UTF8.GetString(message.Body)}");
var receivedmsg = Encoding.UTF8.GetString(message.Body);
ServiceBusMessage DeserializeMsg = JsonConvert.DeserializeObject<ServiceBusMessage>(receivedmsg);
content1 = DeserializeMsg.Content;
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
Console.WriteLine(exceptionReceivedEventArgs.Exception);
return Task.CompletedTask;
}`
You're mixing two paradigms here - a message pump and receive messages on demand. With a message pump, when using .ReceiveMessageHandler() you're requesting the Service Bus SDK to run a continuous loop, aka "message pump", to receive messages as they arrive. That doesn't align with an explicit message retrieval when making a request to the Web API (via controller). You need to redesign your application and retrieve the message(s) upon demand rather than running a message pump.
I've a function app. I wrote a unit test project(xunit) to test my function app code. In unit test method I'm calling Run method of my function app. When request is 'Get' my function app returns a simple json object. While this is working fine in my machine, in all my colleague's machines following line is throwing StackOverFlowException. When I checked the exception details, StackTrace is null.
request.CreateResponse(HttpStatusCode.OK, jObject)
In debug window I see the error as "An unhandled exception occurred in System.private.corlib.dll"
Function App:
public static async Task<HttpResponseMessage> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, ILogger log)
{
if (req.Method.Method.Equals(HttpMethod.Get.Method))
{
NameValueCollection queryString = System.Web.HttpUtility.ParseQueryString(req.RequestUri.Query);
operation = queryString.Get("operation");
if (operation == "GetVersion")
{
version = VersionHistory.GetVersion("ABC");// returns
// {"ABC" : "2.1"}
return req.CreateResponse(HttpStatusCode.OK, version);
//Above line causes StackOverFlowException
}
else
{
return 'Error message'
}
}
}
VersionHistory in above code is just a static class I'm using. It simply returns a json object something like {{"ABC": "0.1.2"}}. It has nothing to do with .net framework version.
public static JObject GetVersion(string key)
{
// some logic. This logic is hit only once. I'm sure this is not
// causing any exception
return JObject.Parse("{\"" + key + "\":\"0.1.2\"}");
}
unit test:
public async Task Run_WhenGetRequest_ShouldReturnAppropriateFunctionAppVersion()
{
const string QUERYSTRING_KEY = "operation";
const string QUERYSTRING_VALUE = "GetVersion";
const string FUNCTION_APP_NAME = "ABC";
const string FUNCTION_APP_VERSION = "2.1";
_request.Method = HttpMethod.Get;
NameValueCollection queryList = HttpUtility.ParseQueryString(_request.RequestUri.Query);
if (queryList.Get(QUERYSTRING_KEY) == null)
{
string queryString = QueryHelpers.AddQueryString(_request.RequestUri.ToString(), QUERYSTRING_KEY, QUERYSTRING_VALUE);
_request.RequestUri = new System.Uri(queryString);
}
HttpResponseMessage response = await FunctionAppClassName.Run(_request, _logger);
// Assert code
}
I've constructed request object in a Fixture class as following and mocked Logger instance.
Request = new HttpRequestMessage
{
Content = new StringContent("", Encoding.UTF8, "application/json"),
RequestUri = new Uri("http://localhost/")
};
var services = new ServiceCollection().AddMvc().AddWebApiConventions().Services.BuildServiceProvider();
Request.Properties.Add(nameof(HttpContext), new DefaultHttpContext { RequestServices = services });
Any idea how to fix this?
Finally we found the issue. Problem was as we are calling Http triggered function app from test case, it was not able to find the MediaType. After adding media type in response like below it worked.
return req.CreateResponse(HttpStatusCode.OK, jObject, "application/json");
I'm still not sure why this issue occurred only in Visual Studio 2019 16.6.x and 16.7.x but not in 16.4.x. We would appreciate if someone can throw some light on this
Thank you those who gave your time.
Using VS 2017 Community. Azure.
I have Azure setup, I have a blank webapp created just for test purpose.
My actual site is an Angular2 MVC5 site, currently run locally.
The following is the code that should... Contact azure providing secret key(the site is registered in azure Active directory).
From this i get a token i then can use to contact azure api and get list of sites.
WARNING: code is all Sausage code/prototype.
Controller
public ActionResult Index()
{
try
{
MainAsync().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
return View();
}
static async System.Threading.Tasks.Task MainAsync()
{
string tenantId = ConfigurationManager.AppSettings["AzureTenantId"];
string clientId = ConfigurationManager.AppSettings["AzureClientId"];
string clientSecret = ConfigurationManager.AppSettings["AzureClientSecret"];
string token = await AuthenticationHelpers.AcquireTokenBySPN(tenantId, clientId, clientSecret).ConfigureAwait(false);
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", "Bearer " + token);
client.BaseAddress = new Uri("https://management.azure.com/");
await MakeARMRequests(client);
}
}
static async System.Threading.Tasks.Task MakeARMRequests(HttpClient client)
{
const string ResourceGroup = "ProtoTSresGrp1";
// Create the resource group
// List the Web Apps and their host names
using (var response = await client.GetAsync(
$"/subscriptions/{Subscription}/resourceGroups/{ResourceGroup}/providers/Microsoft.Web/sites?api-version=2015-08-01"))
{
response.EnsureSuccessStatusCode();
var json = await response.Content.ReadAsAsync<dynamic>().ConfigureAwait(false);
foreach (var app in json.value)
{
Console.WriteLine(app.name);
foreach (var hostname in app.properties.enabledHostNames)
{
Console.WriteLine(" " + hostname);
}
}
}
}
Controller class uses a static helper class that gets the token from Azure...
public static class AuthenticationHelpers
{
const string ARMResource = "https://management.core.windows.net/";
const string TokenEndpoint = "https://login.windows.net/{0}/oauth2/token";
const string SPNPayload = "resource={0}&client_id={1}&grant_type=client_credentials&client_secret={2}";
public static async Task<string> AcquireTokenBySPN(string tenantId, string clientId, string clientSecret)
{
var payload = String.Format(SPNPayload,
WebUtility.UrlEncode(ARMResource),
WebUtility.UrlEncode(clientId),
WebUtility.UrlEncode(clientSecret));
var body = await HttpPost(tenantId, payload).ConfigureAwait(false);
return body.access_token;
}
static async Task<dynamic> HttpPost(string tenantId, string payload)
{
using (var client = new HttpClient())
{
var address = String.Format(TokenEndpoint, tenantId);
var content = new StringContent(payload, Encoding.UTF8, "application/x-www-form-urlencoded");
using (var response = await client.PostAsync(address, content).ConfigureAwait(false))
{
if (!response.IsSuccessStatusCode)
{
Console.WriteLine("Status: {0}", response.StatusCode);
Console.WriteLine("Content: {0}", await response.Content.ReadAsStringAsync());
}
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsAsync<dynamic>().ConfigureAwait(false);
}
}
}
}
ISSUE:
Ok so the issue I was faced with was Async Deadlocks in my code. So i looked at this stack post stack post here
I fixed the issues by putting in .ConfigureAwait(false) on most of the await declarations.
Code runs and gets all the way back to the controller with a token etc and runs through the MakeARMRequests(HttpClient client) method, however the json only returns 1 result "{[]}" when i debug and as such ignores the loops.
My question is, is my code the culprit here? or would this point to a configuration setting in azure?
Not sure if this is the issue you are facing now BUT you never wait for a result from your async action in the first method Index in your code. MainAsync().ConfigureAwait(false); will immediately return and continue to the next block while the task MainAsync() will start in the background. The catch handler also does nothing because you dont wait f or a result.
Option 1 (recommended)
public async Task<ActionResult> Index()
{
try
{
await MainAsync().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
return View();
}
Option 2 if you can't use async/await for some reason
public ActionResult Index()
{
try
{
MainAsync().GetAwaiter().GetResult();
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
return View();
}
The Code looks OK and runs fine, Anyone who could help verify would be good, but one can assume this is OK.
The issue for this was configuration in azure, When you register an app you must set a certain number of Access controls via the subscription.
In this case I set some more specific things for the web api , for now set the app as owner and made reference to service management api.
Probably don't need half the "IAM" added in the subscription to the registered app, I simply went through adding the relevant ones and debugging each time until finally i got the results expected.