We Keep Coding

How to implement observability (as described by Charity Majors) in an Asp.Net Core application?

In https://charity.wtf/tag/observability/ in the section HOW TO GATHER AND STRUCTURE YOUR DATA Charity Majors presents the following scheme:
Early in the pipeline initialize a "blob" accessible everywhere in code with every request having its own copy (in effect tied to the HttpContext)
Any code that wishes to record some information adds key-value pairs to this "blob"
When the request exits the "blob" is delivered to the observability server as a structured event.
This scheme makes a lot of sense and is quite elegant. It avoids verbal diarrhea of the regular logs. However, it is also oversimplified, because it ignores the batch processing requests, where a request has to process N records which is likely to produce N sets of similar structured events nested within the parent structured event corresponding to the request itself. In this case we have a choice of either sending the batch record event as soon as a record is processed or accumulate the nested events within the "blob" to be sent when the request is over. In any case this complication is not discussed in the post.
Before potentially inventing the wheel I am trying to figure out whether we have something ready to use this observability scheme. Here is what I have so far:
We are using Azure and its Application Insights TrackEvent API seems to be perfect as the low level API. It supports events of the form <Event Name, string-to-string map, string-to-float map> which seems to be rich enough. The collected events can be then queried across all the dimensions anyway we like it. So, the observability server is not a problem for me.
The "blob" implementation is a problem. I was looking for Honeycomb implementation for .Net Core and came across this page - https://docs.honeycomb.io/getting-data-in/dotnet/. However the 2 options presented there do not seem mature enough:
HoneyComb .NET leaves it up to us to figure out how to deal with nested events.
OpenTelemetry .NET does not seem to be about observability at all, but about the "3 pillars of observability" - logging, metrics and tracing. Which do not give observability according to https://www.infoq.com/news/2019/02/rethinking-observability/
The Microsoft.Extensions.Logging.ILogger interface seems to present an adequate interface. It looks like the BeginScope method exists explicitly to delimit the different batches. But it is just an interface.
So, my question is there an implementation for Asp.Net Core for observability as structured events that also takes into account the nested events in one way or another and that can be customized to deliver to the Application Insights bucket?
I could elaborate on the challenges of the batch processing, but this question is already too long, so I stop here.

Intercept and modify Ack response message BizTalk 2013 R2

I have written a custom pipeline component assembler to modify the response ACK HL7 message.
I have invoked Assemble(pContext) of Microsoft.Solutions.BTAHL7.Pipelines.HL72fAsm in the implemented method Assemble(pContext) of IAssemblerComponent interface
gives me result IBaseMessage
which is an HL7, then I do my manipulations on it to fix one of the fields and return that modified IBaseMessage.
All these works just fine, I tried EvenLogger to verify it.
But still the Sender application doesn't receive the modified message, it receives the auto-generated message.
Is there something I'm missing out, why do I not get the custom assembler result out from the SendPipeline of 2 way receive port
Note : BTAHL7 Configuration explorer is configured for original mode. The send pipeline on RequestResponse receive port is set to my custom pipeline

My suggestion is after all the more important points.
The first thing you're employer or customer should say is NO. That is invalid HL7 and you cannot support that.
But, if they are unable to unwilling to comply, the next thing you need to do is inform your management that their non-compliance will cost you a lot of extra time and money to accommodate. To fully support this change will likely cost more then implementing the business messages, I am totally serious. This is not a problem with BizTalk Server, you app or you.
Depending on the relationship, your management can legitimately ask them how they are going to pay for this customization. It's going to cost your side a lot more to break HL7 to comply with them than it will for them to fix it.
Next, and perhaps most important, due to the nature of it's message content, HL7 has very strict completeness requirements, which they are fundamentally breaking. The Trading Partner needs to fully document this requirement to take ownership of it because there is a huge consequence, they are breaking tracing/tracking on you end.
This means that it will be substantially more difficult to investigate and resolve messaging issues for you, not them. This might raise legal or compliance issues your side needs to be aware of.
So, provided you technical, medical and legal teams are all satisfied, the first thing I would try is a Pipeline Component that simply swaps the two values, MSH10 and MSA02. That way, they will receive both values.
Finally, here's a novel solution. Since this is their problem, and a problem for every one of their trading partners, what if you offer help them fix it. All then need to do is what I suggested, swap MSH10 and MSA02 on the received message.

Send Email in Global Exception Handler?

I want to add global exception handling code to C# WPF apps and, although it seems rather rakish (nobody else seems to do it), I want to send email to the developer with exception info. The main problem I can think of happening here is if the developer ends up changing his email address after the software has been deployed. Perhaps an email to the department (such as a listserv type of email broadcast address) would be more appropriate? Has anybody used this sort of methodology, and if so, what solution have you come up with to make sure that somebody gets the exception-generated email?
Is this the best solution:
// in exception code (pseudocode)
try
SendEmailToTheCoder();
catch
on EmailAddressNotValid:
try
SendEmailToTheListServ()
catch
on EmailNotSent:
LogExceptionDataToLog("Bla");
...or has my brain gone pear-shaped again (I don't know what that means, but it's British, so it must be wickedly funny)

The best thing to do is to keep the details of the messaging outside of the application.
For instance, you may log errors to a text file or some other kind of log, then have an external application or Windows service monitor that log and decide what to do -- such as sending an email, or creating a digest of all messages of the day and emailing it, or a similar action.
This way, you can optimize and modify what happens in case of these errors, without having to change your program code. You can also reuse that system with other applications that also just log errors to a text file, which has a lower probability of error than connecting to an SMTP server and sending a message.

I would just create a distribution group, something like developer#yourcompany.com and add people responsible for the program part of that distribution group. If one developer leaves the company, nothing in your code needs to change and no trying one thing first and then another one.
Better yet, use a logging framework such as log4net (nlog is also popular); you can configure it to log to different places (xml, database, email, etc). If you do log to email, I'd always send it to a distribution group, anyway, even if that distribution group is composed of only one member.

Web Service Client Architecture - c#

Morning all,
I have been tasked with developing a client tool for a cloud web service API (A simple WSDL). I am not a seasoned or even qualified developer, I have an intermediate knowledge of C# and enough I believe to make this work, but I don't want a solution that just works, I want to build something clean and well coded which another dev can read and understand and which is intuitive.
You may want to stop me there and say "That is something you can only learn through experience." if that is the case then I can accept that and move on, but if you do have some advice the rest of the details are below.
The solution will be a C# Console application. I have produced a spec for this, it is below:
1.) Create a console application in .NET which has the following
capabilities:
2.) Consume CSV file containing Processed Data OR ODBC
Connection to staging SQL database and read records directly out of
load table
3.) Make the following calls to Zuora Webservice (Asynchronous) ·
SubscribeWithExisitingAccount() · Create() ·
Login() · Subscribe() · Update() · Delete()
(*) Calls marked with this are possibly avoidable,
*it is possible to create a subscription, account and contact with a
single call (Subscribe())
*Create() may be the exception as a scenario may occur where we need
to create an instance of an object with no corresponding subscription.
4.) Report back the success and errors of every record into a CSV
file.
Mappings will be done on a 1 to 1 basis, where the input file
will have the same column names as the target
Where I lack knowledge is following a design which will make this app make sense and work efficiently. I am not looking for someone to do this for me, what I am looking for is tips on how I can improve on what I am already doing
Currently I am just organically building the solution due to a lack of foresight on jobs like this, so I am also interested in things I can do post development.
ALL Advice and criticism is welcomed.
Thanks in advance,
Matt

Design principles are a big subject, and how to apply them correctly is only something that comes with experience. There's a lot more of them out there then you'd ever use in a given project, and in some cases using them correctly means not using them at all (or only choosing specific ones that suit the project). The first step is wanting to write good code though, so you're starting in the right place. :) A couple of things did stand out to me:
2.) Consume CSV file containing Processed Data OR ODBC Connection to
staging SQL database and read records directly out of load table
What you're going to want to do here is only build the logic that does something with this data once. The most direct way to achieve that is to have your logic expect the data in a certain format (probably business classes that hold the parsed data and that your logic an use).
So what you'll do is take the input data (CSV/SQL Table/Whatever) and first parse it into your internal business classes. Then you feed the parsed data to your logic that does whatever your app does with it. The advantage here is that you can change the logic once and it will work with both data types, AND if someone comes along later and says "now we need it to read this Excel file" all you'll have to do is add another parser to get the Excel data into your internal format. No changes to the logic will be required.
4.) Report back the success and errors of every record into a CSV
file.
Mappings will be done on a 1 to 1 basis, where the input file
will have the same column names as the target
Same as above. Don't assume that you'll be exporting to CSV forever, make a simple "ReportError" class or some such that holds error details and stick it into a List while doing your processing. At the end when it's time to output your errors, you can convert that into a CSV. So if this requirement changes and you instead report errors to a web service, you only have to change a small part of the code (and none of it is your processing logic).
There's a theme here. :) Try to encapsulate logical bits so that if something changes it's easy to find where that something is in code. If you can learn to do that, you'll wind up with maintainable code even if you don't follow any other process or pattern (particularly since as one person you won't be making huge projects).
3.) Make the following calls to Zuora Webservice (Asynchronous) ·
SubscribeWithExisitingAccount() · Create() · Login() · Subscribe() ·
Update() · Delete()
As a console app, I'm going to question if you actually need these to be asynchronous or not. What do you hope to gain from an async call to Login()? Can your program do anything while waiting for Login() to return?
It's not that async is terribly difficult, but it IS more to manage then synchronous calls. For a console app from someone whose not that experienced in the technology yet, I'm not sure what benefit you're gaining to weigh against the extra effort it requires of you.

I would recomend you read a book on webservices (this is a good one) They arent really something you can just pick up from playing about and can be quite frustraiting if you dont know what your doing.
As for development, I recomend you prototype it first. Hammer something out thats messy but lets you get an idea of how to do things. You can then use that as a reference for when your actually building your app.

How can I make a "greenscreen" web app?

In informal conversations with our customer service department, they have expressed dissatisfaction with our web-based CSA (customer service application). In a callcenter, calls per hour are critical, and lots of time is wasted mousing around, clicking buttons, selecting values in dropdown lists, etc. What the dirrector of customer service has wistfully asked for is a return to the good old days of keyboard-driven applications with very little visual detail, just what's necessary to present data to the CSR and process the call.
I can't help but be reminded of the greenscreen apps we all used to use (and the more seasoned among us used to make). Not only would such an application be more productive, but healthier for the reps to use, as they must be risking injury doing data entry through a web app all day.
I'd like to keep the convenience of browser-based deployment and preserve our existing investment in the Microsoft stack, but how can I deliver this keyboard-driven ultra-simple greenscreen concept to the web?
Good answers will link to libraries, other web applications with a similar style, best practices for organizing and prioritizing keyboard shortcut data (not how to add them, but how to store and maintain the shortcuts and automatically resolve conflicts, etc.
EDIT: accepted answers will not be mini-lectures on how to do UI on the web. I do not want any links, buttons or anything to click on whatsoever.
EDIT2: this application has 500 users, spread out in call centers around North America. I cannot retrain them all to use the TAB key

I make web based CSR apps. What your manager is forgetting is now the application is MUCH more complex. We are asking more from our reps than we did 15 years ago. We collect more information and record more data than before.
Instead of a "greenscreen" application, you should focus on making the web application behave better. For example,dont have a dropdown for year when it can be a input field. Make sure the taborder is correct and sane, you can even put little numbers next to each field grouping to indicate tab order. Assign different screens/tabs to F keys and denote them on the screen.
You should be able to use your web app without a mouse at all with no loss of productivity if done correctly.
Leverage the use of AJAX so a round trip to the server doesn't change the focus of their cursor.
On a CSR app, you often have several defaults. you should assign each default a button and allow the csr to push 1 button to get the default they want. this will reduce the amount of clicking and mousing around.
Also very important You need to sit with the CSR's and watch them for a while to get a feel for how they use the app. if you haven't done this, you are probably overlooking simple changes that will greatly enhance their productivity.

body { background: #000; color: #0F0; }
More seriously, it's entirely possible to bind keyboard shortcuts to actions in a web app.
You might consider teaching your users to just use the Tab key - that's how I fill out most web forms. Tab to a select list and type out the first few letters of the option I'm attempting to select. If the page doesn't do goofy things with structure and tabindexes, I can usually fill out most web forms with just the keyboard.

As I had to use some of those apps over time, will give my feedback as a user, FWIW, and maybe it helps you to help your users :-) Sorry it's a bit long but the topic is rather close to my heart - as I had myself to prototype the "improved" interface for such a system (which, according to our calculations, saves very nontrivial amounts of money and avoids the user dissatisfaction) and then lead the team that implemented it.
There is one common issue that I noticed with quite a few of CRMs: there is 20+ fields on the screen, of which typically one uses 4-5 for performing of 90% of operations. But one needs to click through the unnecessary fields anyway.
I might be wrong with this assumption, of course (as in my case there was a wide variety of users with different functions using the system). But do try to sit down with the users and see how they are using the application and see if you can optimize something UI-wise - or, if really it's a matter of not knowing how to use "TAB" (and they really need to use each and every of those 20 fields each time) - you will be able to coach a few of them and check whether this is something sufficient for them - and then roll out the training for the entire organization. Ensure you have the intuitive hotkey support, and that if a list contains 2000 items, the users do not have to scroll it manually to find the right one, but rather can use FF's feature to select the item by typing the start of its text.
You might learn a lot by looking at the usage patterns of the application and then optimizing the UI accordingly. If you have multiple organizational functions that use the system - then the "ideal UI" for each of them might be different, so the question of which to implement, and if, becomes a business decision.
There are also some other little details that matter for the users - sometimes what you'd thought would be the main input field for them in reality is not - and they have an empty textarea eating up half of the screen, while they have to enter the really important data into a small text field somewhere in the corner. Or that in their screen resolution they need the horizontal scrolling (or, scrolling at all).
Again, sitting down with the users and observing should reveal this.
One more issue: "Too fast developer hardware" phenomenon: A lot of the web developers tend to use large displays with high resolution, showing the output of a very powerful PCs. When the result is shown on the CSR's laptop screen at 1024x768 of a year-old laptop, the layout looks quite different from what was anticipated, as well as the rendering performance. Tune, tune, tune.
And, finally - if your organization is geographically disperse, always test with the longest-latency/smallest bandwidth link equivalent. These issues are not seen when doing the testing locally, but add a lot of annoyance when using the system over the WAN. In short - try to use the worst-case scenario when doing any testing/development of your application - then this will become annoying to you and you will optimize its use - so then the users that are in better situation will jump in joy over the apps performance.
If you are in for the "green screen app" - then maybe for the power users provide a single long text input field where they could type all the information in the CLI-type fashion and just hit "submit" or the ENTER key (though this design decision is not something to be taken lightly as it is a lot of work). But everyone needs to realize that "green-screen" applications have a rather steep learning curve - this is another factor to consider from the business point of view, along with the attrition rate, etc. Ask the boss how long does the typical agent stay at the same place and how would the productivity be affected if they needed a 3-month term to come to full speed. :) There's a balance that is not decided by the programmers alone, nor by the management alone, but requires a joint effort.
And finally a side note in case you have "power users": you might want to take a look at conkeror as a browser - though fairly slow in itself, it looks quite flexible in what it can offer from the keyboard-only control perspective.

I can't agree with the others more when they say the first priority of the redesign should be going and talking to / observing your users and see where they have problems. I think you would see far more ROI if you find out the most common tasks and the most common errors your users make and streamline those within the bounds of your existing UI. I realize this isn't an easy thing to do, but if you can pull it off you'll have much happier users (since you've solved their workflow issues) and much happier bosses (since you saved the company money by not having to re-train all the users on a completely new UI).
After reading everyone else's answers and comments, I wanted to address a few other things:
EDIT: accepted answers will not be mini-lectures on how to do UI on the web. I do not want any links, buttons or anything to click on whatsoever.
I don't mean to be argumentative, but this sounds like you've already made up your mind without having thought of the implications on the users. I can immediately see a couple pitfalls with this approach:
A greenscreen-esque UI may not be
more productive for your users. For
example, what's the average age of
your users? Most people 25 and
younger have had little to no
exposure to these types of UIs.
Suddenly imposing this sort of
interface on them could cause a
major backlash from your users. As an example, look at what happened
when Facebook decided to change its
UI to the "stream" concept - huge
outrage from the users!
The web wasn't really designed with this sort of interface in mind. What I mean is that people are not used to having command-line-like interfaces when they visit a website. They expect visual medium (images, buttons, links, etc.) in addition to text. Changing too drastically from this could confuse your users.
Programming this type of interface will be tough. As in my last point, the web doesn't play well with command-line-like or text-only interfaces. Things like function keys, keyboard shortcuts (like ctrl- and alt-) are all poorly and inconsistently supported which means you'll have to come up with your own ways of accessing standard things like help (since F1 will map to the web browser's help, not your app's).
EDIT2: this application has 500 users, spread out in call centers around North America. I cannot retrain them all to use the TAB key
I think this argument is really just a strawman. If you are introducing a wholly new UI, you're going to have to train your users on it. Really, it should be assumed that any change to your UI will require training in one form or another. Something simple like adding tab-navigation to the UI is actually comparatively small in the training department. If you did this it would be very easy to send out a "handy new feature in the UI" email, or even better, have some sort of "tip of the day" (that users can toggle off, of course) which tells them about cool timesaving features like tab navigation.
I can't speak for the other posters here, but I did want to say that I hope you don't think we're being too argumentative here as that's not our (well OK, my) intent. Rather the reaction comes from us hearing the idea for your UI and not being convinced that it is necessarily the best thing for your users. You are fully welcome to say I'm wrong and that this is what your users will benefit most from; but before you do, just remember that at the end of the day it's your users who matter most and if they don't buy in to your new UI, no one will.

It's really more of a keyboard-centric mentality when developing. I use the keyboard for as much as possible and the apps I build tend to show that (so I can quickly go through my use cases).
Something as simple as getting the tab order correct could be all your app needs (I guess I'm not sure if you can set this in ASP.NET...). A lot of controls will auto-complete for the rest.