We Keep Coding

Logging and viewing custom events in Azure Blob / Table

I'd like to log some business events from my WebApp (e.g. an invoice was issued or user updated some record and i'd like to see what was changed) and use some already existing tools to view them. My first approach was to use Azure Table Storage along with Azure Storage Explorer
My example event object looks like this:
public abstract class ApplicationEvent : TableEntity
{
public string ClientIP { get; set; }
public string User { get; set; }
public string Message { get; set; }
public string Type { get; set; }
public string Details { get; set; } // JSON document with audit
public ApplicationEvent()
{
var now = DateTime.UtcNow;
PartitionKey = string.Format("{0:yyyy-MM}", now);
RowKey = string.Format("{0:dd HH:mm:ss.fff}-{1}", now, Guid.NewGuid());
}
}
At small events volume it seems reasonable solution. But then again - as the log will grow viewing/searching logs can become very inefficient. I've thought of creating one table per month/week but it doesn't feel right.
I've read about using blobs to store the log but i can't find any existing tool that user can use to browse/filter these events in my custom format (i want to be able to see all my ApplicationEvent properties). So the question is:
- which is better: table or blob? Or maybe something else?
- if blob is the way to go - how to efficiently create/write custom events to Blob Storage and how to view those events in human readable form?

Dotnet core MVC xml parameter binding always give null

I've read a few questions on SO but the solutions are all for ASP.NET webApi not dotnet core.
I've added xml support in my Startup.cs
services
.AddMvc()
.AddXmlSerializerFormatters();
Here's my controller method:
[Route("main")]
[HttpPost]
public string MainPost([FromBody]MessageModel msg)
{
_log.LogInformation("body msg ="+msg.Content);
return "test";
}
Here's my XML
<xml>
<ToUserName>toUser</ToUserName>
<FromUserName>FromUser</FromUserName>
<CreateTime>1348831860</CreateTime>
<MsgType>text</MsgType>
<Content>test content</Content>
<MsgId>1234567890123456</MsgId>
</xml>
Here's my Model class:
[XmlRoot("xml")]
public class MessageModel
{
public string ToUserName { get; set; }
public string FromUserName { get; set; }
public DateTime CreateTime { get; set; }
public string MessageType { get; set; }
public string Content { get; set; }
public int MsgId { get; set; }
}
When I send post request(with header "application/xml") to this route, it always gives me null for MessageModel msg
What am I missing here?
I know there's a DataContract thing but I can't control the xml format sending
to my server so I need a way to bind xml in the format stated above to my object.
Any help or lead to any document will be much appreciated.

I've found the problem thanks to #jdweng 's comment which gives me a idea on what might go wrong.
The problem is that there are some elements in the XML cannot be serialized to the designated type in my model class. They're DateTime CreateTime and int MsgId.
So changing all the fields to string solved problem.
I don't know whether why isn't dotnet core tells us about this instead of just giving us null if the binding fails due to this kind of reason, this is definitely something they can improve on.

Azure Table Storage, ExecuteBatchAsync, Unexpected response code for operation : 0

By leveraging the Windows Azure SDK, I try to insert entities using CloudTable.ExecuteBatchAsync and TableBatchOperations.
The entity serialized as Json:
{
"LastAccessDate":"2015-02-27T00:00:00Z",
"Title":"Google open-sources HTTP/2-based RPC framework",
"PublicationDate":"0001-01-01T00:00:00",
"Id":"tag:theregister.co.uk,2005:story/2015/02/27/google_opensources_http2based_rpc_framework/",
"LastUpdatedDate":"2015-02-27T00:00:00Z",
"FeedUrl":"http://www.theregister.co.uk/software/developer/headlines.atom",
"Url":"http://go.theregister.com/feed/www.theregister.co.uk/2015/02/27/google_opensources_http2based_rpc_framework/",
"PartitionKey":"http%3a%2f%2fwww.theregister.co.uk%2fsoftware%2fdeveloper%2fheadlines.atom",
"RowKey":"http%3a%2f%2fgo.theregister.com%2ffeed%2fwww.theregister.co.uk%2f2015%2f02%2f27%2fgoogle_opensources_http2based_rpc_framework%2f",
"Timestamp":"0001-01-01T00:00:00+00:00",
"ETag":null
}
Represented by this POCO entity:
public class SyndicationFeedArticle : TableEntity
{
public virtual DateTime LastAccessDate { get; set; }
public virtual string Title { get; set; }
public virtual DateTime PublicationDate { get; set; }
public virtual string Id { get; set; }
public virtual DateTime LastUpdatedDate { get; set; }
public virtual string FeedUrl { get; set; }
public virtual string Url { get; set; }
}
The problem arises when the entity is constructed from RSS xml processing. ExecuteBatch of inserts throws Unexpected response code for operation : 0. I do understand that it means that the batch operation at index 0 failed. Usually it's a problem with either Partition or Row keys being incorrect such as containing invalid characters (which is not the case above) or exceeding 1kb in size which in this case doesn't apply.
Here's what puzzling me:
Both unit test and business code entities (although constructed differently) generate the exact same Json serialization data
Both processes (manual new() creation and RSS business code) works well in a Unit test
There is no issue at all when storing on the Emulator, it fails in Azure Cloud only.
What I'm looking for are pointers on how I could troubleshoot this issue. I've recreated various scenarios in unit tests and made sure my entities don't break constraints on keys but no luck. My main issue is why can't I get constant behaviours between the emulator and the cloud. This would really help me out or at least, it would point me toward another way to fix this problem.
Thanks!

Ok, got it figured out.
What happens in this particular scenario:
PublicationDate is instantiated using a DateTimeOffset.Date as source value which may be set at MinValue. The table storage doesn't support MinValue for a DateTime column.
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<code>OutOfRangeInput</code>
<message xml:lang="en-US">One of the request inputs is out of range. RequestId:9e74bf7a-0002-004e-1142-16dabb000000 Time:2015-02-28T02:01:42.2124803Z</message>
</error>
I changed the entity property for a DateTime? and manages this as a business rule. Of course, I'm left wondering why locally on the emulator this is supported...

MVC3 Date in UK Format not received on Controller

Given this VM
public class ApplicationDTO : BaseDTO
{
public DateTime Date { get; set; }
public int JobId {get;set;}
public int Status {get;set;}
[Required]
public string Message { get; set; }
public string ExpertCode { get; set; }
}
I have a hidden field thusly
#Html.Hidden("Date", DateTime.Now)
Which fiddler shows me is sent to the server as I would expect (UK format, I'm in the UK!)
But on the controller the date shows as being the default min. date
Is it just the UK format? If so, what is my best way round it? Whilst currently I am setting it to the current date, potentially it could be set to any given date i.e.
#Html.HiddenFor(x => x.Date)
I am using AJAX to submit the form, if that makes a difference.

If it is a Get MVC uses culture invariant format (Basically US format) by default during model binding. So it doesn't accept UK date format. I believe the design reasons are that a querystring could be passed around so therefore it needs to be culture invariant (I wasn't 100% convinced by that logic).
This article covers a possible solution http://xhalent.wordpress.com/2011/05/14/localization-of-dates-in-asp-net-mvc/
We just make sure we never do a Get with a UK date format
This questions also covers the issue Globalization problem with DateTime and ASP.NET MVC 3 Model Binding

you should use Data Annotation for Formatting the date on your Model or View model
public class ApplicationDTO : BaseDTO
{
[DisplayFormat(DataFormatString="{0:dd/MM/yyyy}", ApplyFormatInEditMode=true)]
public DateTime Date { get; set; }
public int JobId {get;set;}
public int Status {get;set;}
[Required]
public string Message { get; set; }
public string ExpertCode { get; set; }
}
Apply the format according you.
As i have seen you are not providing any format to your property

I had a similar issue with dd.MM.yyyy format and had to set globalization element in web.config to appropriate culture and uiCulture attribute settings.

If you use #Html.Hidden (not the For helper) it won't be able to match the parameter in the request to the property in your model.
If you use #Html.HiddenFor(x=>x.Date, "28/2/2013") you will see the date populated appropriately.
Also you can verify the Request.Params collection. Put a break point in the Action. You'll see that one of the Params has your date.... JQuery does not affect this in any way.

Tips on architecture for consuming/wrapping large webservice

I am looking for good practices for implementing a smart architecture and way to handle integration against a system with many different wdsl webservices.
I have been hobby developing with C# for 2 years~, by that I am not always using the correct terminology, but I'll try to describe what I am looking for.
The main reason I'm posting this, is to get ideas of areas that I should read up on, design patterns to implement and how to manage API calls in a good way.
I'm integrating against a system that offers lots of different API's. Each API have 20-30 methods. Each API takes a number of parameters in XML format.
To give you an idea, I have taken the following API as an example, its the DeviceManager API and the Create method to create a device in the system.
The API takes two parameters, a string key and a XML parameter string.
Example
DeviceManager
public string Create(string sKey, string sXmlParameters)
**Parameters:**
Name: sKey Type: string
Description:
The API security key. A valid API security key is required for every API call.
Name: sXmlParameters Type: string
Description: Values needed to create a new device. See the Additional Information >section for XML format.
Return Value: Type: string
Description: Returns XML containing the status of the creation and if the status is
Constants.ExternalServiceReturnCodes.SUCCEEDED, the ID of the new device.
XMLParameters:
<PARAMETERS>
<NAME>string - up to 64 characters. Non-blank name of the device.
Must be unique within a gateway.
</NAME>
<DESCRIPTION>string - up to 1024 characters. The description of the new device.
</DESCRIPTION> (optional)
<TYPEID>string of DeviceType. The type of device.
</TYPEID>
<GATEWAYID>string - 32-character GUID. The ID of the gateway to associate with the
device. If this node is included, it must contain an ID of
a gateway that exists in the solution.
</GATEWAYID> (optional)
<INSTALLATIONDATETIME>
date time in UTC - greater than or equal to
1753-01-01 00:00:00.000 and less than or equal to
9999- 12-31 23:59:59.998. The date time that the device was installed.
</INSTALLATIONDATETIME> (optional - if not included, the installation
date time is set to the date time in UTC when the device is
created in the solution)
<SERIALNUMBER>string - up to 64 characters. The serial number of the device
</SERIALNUMBER>
<TIMEZONEID>string - time zone ID. The time zone ID of the device.
Call the TimeZoneManager.RetrieveList API to get a list of valid time zone IDs
</TIMEZONEID> (required for device type 'meter')
<HARDWAREVERSION>string - up to 32 characters. The hardware version of the device.
</HARDWAREVERSION> (optional)
<GEOGRAPHICCOORDINATES>
<LATITUDE>decimal - greater than or equal to -90 and less than or
equal to 90. The latitude geographical coordinate of the
device in decimal degrees. The value must be from the
WGS84 spatial reference system.
If more than 6 digits after the decimal point are included,
the value will be rounded to 6 digits.
</LATITUDE>
<LONGITUDE>
decimal - greater than or equal to -180 and less than or
equal to 180. The longitude geographical coordinate of the
device in decimal degrees. The value must be from the
WGS84 spatial reference system.
If more than 6 digits after the decimal point are included,
the value will be rounded to 6 digits.
</LONGITUDE>
</GEOGRAPHICCOORDINATES> (optional)
<METER>
<ID>string - 12 hexadecimal characters.</ID>
<TRANSFORMERID>string - up to 128 characters.</TRANSFORMERID>
<DOWNLIMIT>integer - greater than or equal to 0 and less than or
equal to 65535.
</DOWNLIMIT> (optional)
<METER>
</PARAMETERS>
Return API payload format:
<DEVICEID>string - 32-character GUID. The ID of the new device.</DEVICEID>
The API's always return the data in the following format:
<RETURNS>
<STATUS>
String from Constants.ExternalServiceReturnCodes class.
</STATUS>
<APIPAYLOAD>
Additional information from the API call. Node may be empty. For
APIs that return information, that information will be shown in
the Return section.
</APIPAYLOAD>
</RETURNS>
So in the example above, the payload would look something like:
<RETURNS>
<STATUS>
SUCCEEDED
</STATUS>
<APIPAYLOAD>
<DEVICEID>string - 32-character GUID. The ID of the new device.</DEVICEID>
</APIPAYLOAD>
</RETURNS>
As for now, I have been working on designing classes for all XMLparameters to be able to serialize and deserialize the parameters and payloads from the APIs, but it is a very time consuming job to define these. Example given below for the create API parameters and payload. (simple example using get set)
DeviceManager Create Parameters
[XmlRoot(ElementName = "PARAMETERS")]
public class Create
{
[XmlElement(ElementName = "NAME")]
public string Name { get; set; }
[XmlElement(ElementName = "DESCRIPTION")]
public string Description { get; set; }
[XmlElement(ElementName = "TYPEID")]
public string TypeId { get; set; }
[XmlElement(ElementName = "GATEWAYID")]
public string GatewayId { get; set; }
[XmlElement(ElementName = "INSTALLATIONDATETIME")]
public string InstallationDateTime { get; set; }
[XmlElement(ElementName = "SERIALNUMBER")]
public string SerialNumber { get; set; }
[XmlElement(ElementName = "TIMEZONEID")]
public string TimeZoneId { get; set; }
[XmlElement(ElementName = "HARDWAREVERSION")]
public string HardWareVersion { get; set; }
[XmlElement(ElementName = "GEOGRAPHICCOORDINATES")]
public CreateParametersGeoGraphicCoordinates GeographicCoordinates { get; set; }
[XmlElement(ElementName = "METER")]
public CreateMeter Meter { get; set; }
}
public class CreateMeter
{
[XmlElement(ElementName = "NEURONID")]
public string NeuronId { get; set; }
[XmlElement(ElementName = "TRANSFORMERID")]
public string TransformerId { get; set; }
[XmlElement(ElementName = "UTILITYID")]
public string UtilityId { get; set; }
[XmlElement(ElementName = "LONTALKKEY")]
public string LonTalkKey { get; set; }
[XmlElement(ElementName = "DOWNLIMIT")]
public string DownLimit { get; set; }
}
public class CreateParametersGeoGraphicCoordinates
{
[XmlElement(ElementName = "LATITUDE")]
public string Latitude { get; set; }
[XmlElement(ElementName = "LONGITUDE")]
public string Longitude { get; set; }
}
And for PayLoads I have the following generic class and DeviceManager.Create Payload specific class:
Create PayLoad
public class CreatePayLoad
{
[XmlElement(ElementName = "DEVICEID")]
public string DeviceId { get; set; }
}
PayLoad
[XmlRoot(ElementName = "RETURNS")]
public class PayLoad<T> where T : new()
{
public PayLoad()
{
ApiPayLoad = new T();
}
/// <summary>
/// Contains the payload from the command.
/// </summary>
[XmlElement(ElementName = "APIPAYLOAD")]
public T ApiPayLoad { get; set; }
/// <summary>
/// Status of the call
/// </summary>
[XmlElement(ElementName = "STATUS")]
public string Status { get; set; }
}
So in my code i can do the call in the following way:
Example use
//Create the parameters
var parameters = new Create
{
Description = "Description",
GatewayId = "GatewayId",
Name = "NameOfDevice",
GeographicCoordinates = new CreateParametersGeoGraphicCoordinates
{
Latitude = "Lat",
Longitude = "Long"
},
Meter = new CreateMeter
{
TransformerId = "ID",
DownLimit = "120"
}
};
//Serialize the parameters to xml
string sXmlParameters = Helper.SerializeToXml<Create>(parameters);
//API call
string apiPayLoad = DeviceManager.Create(apiKey, sXmlParameters);
//Deserialize payload
var payLoad = Helper.DeserializeFromXml<PayLoad<CreatePayLoad>>(apiPayLoad);
Can someone please contribute with ideas and better ways to manage this?
Please bear in mind that there are about 300 methods in the system, some with very complicated xml parameters where some properties are optional, some mandatory if property A, B or C are used and so on.
I have been looking at XSD.exe, but the generated code is not neat to the eye and it doesn't handle collections very well.
A friend also proposed T4 templates, where one could generate classes based on templates, but I'm not really finding any good examples out there.
I'm not sure if I have explained myself in a good way, If I'm being unclear - please let me know and I will try to elaborate.
Thank you,
amr-it

Stupid question time: so are you saying that none of the APIs are strongly typed? Meaning, all of the signatures are:
public string Create(string sKey, string sXmlParameters)
And the schema of the input "sXmlParameters" parameter is only defined by textual documentation? If that's the case, then I think you're overthinking things, and it will be just as efficient to write parsing methods for responses as it will be to create classes and use templating and casting techniques to convert responses into objects.
However, you mention XSD.exe, so maybe there are defined schemas for inputs and outputs? If that's the case, I would create those class definitions using the XSD tool, and not worry that they are not "neat to the eye" - you never look at those c# files, you just use the objects. In terms of their being weak with collections - if you're trying to filter/sort/inspect/traverse complex nested collection data, I would suggest you look at LINQ query expressions; they will let you quickly grab the objects you need. Of course, they don't help much with object instantiation, but I don't think there's a quick fix for that.

They are not strongly typed, all the API's take the sXmlParameters and are only defined by textual documentation. With XSD.exe, I've made examples of XML, then generated xsd's and from that I've generated some .cs files. However, it is a very time consuming task.
I was recommended to look into DSL fluent interfaces, that could help me in constructing the sXmlParameters.
var result = DeviceManager.Create(apiKey,
Parameters.
.Type(Parameters.Create)
.Name("Name of Device")
.Description("Description of Device")
.Coordinates("Lat","Long")
.Validate()
.asXML()
);
Using this, I can construct parameters from code, and have a base class validating the required fields before returning the parameters as XML.
However, mapping the PayLoads for each result will still be a time consuming task. I was thinking to use T4 templates to generate the classes based on XML or similair.