We Keep Coding

How to put a TPL Dataflow TranformBlock or ActionBlock in a separate file?

I want to use the TPL Dataflow for my .NET Core application and followed the example from the docs.
Instead of having all the logic in one file I would like to separate each TransformBlock and ActionBlock (I don't need the other ones yet) into their own files. A small TransformBlock example converting integers to strings
class IntToStringTransformer : TransformBlock<int, string>
{
public IntToStringTransformer() : base(number => number.ToString()) { }
}
and a small ActionBlock example writing strings to the console
class StringWriter : ActionBlock<string>
{
public StringWriter() : base(Console.WriteLine) { }
}
Unfortunately this won't work because the block classes are sealed. Is there a way I can organize those blocks into their own files?

Dataflow steps/blocks/goroutines are fundamentally functional in nature and best organized as modules of factory functions, not separate classes. A TPL DataFlow pipeline is quite similar to a pipeline of function calls in F#, or any other language. In fact, one could look at it as a PowerShell pipeline, except it's easier to write.
There's no need to create a class or implement an interface to add a new function to that pipeline, you just add it and redirect the output to the next function.
TPL Dataflow blocks provide the primitives to construct a pipeline already and only require a transformation function. That's why they are sealed, to prevent misuse.
The natural way to organize dataflows is similar to F# too - create libraries with the functions that perform each job, putting them in modules of related functions. Those functions are stateless, so they can easily go into a static library, just like extension methods.
For example, there could be one module for database related functions that perform bulk inserts or read data, another to handle exports to various file formats, separate classes to call external web services, another to parse specific message formats.
A real Example
For the last 7 years I'm working with several complex pipelines for an Online Travel Agency (OTA). One of them calls several GDSs (the intermediaries between OTAs and airlines) to retrieve transaction information - ticket issues, refunds, cancellations etc. Next step retrieves the ticket records, the detailed ticket informations. Finally, the records are inserted into the database.
GDSs are too big to bother with standards, so their "SOAP" web services aren't even SOAP-compliant, much less follow WS-* standards. So each GDS needs a separate class library to call the services and parse the outputs. No dataflows there yet, the project is already complex enough
Writing the data to the database is pretty much the same always, so there's a separate project with methods that take eg an IEnumerable<T> and write it to the database with SqlBulkCopy.
It's not enough to load new data though, things often go wrong so I need to be able to load already stored ticket information.
Organisation
To preserve sanity :
Each pipeline gets its own file:
A Daily pipeline to load new data,
A Reload pipeline to load all stored data
A "Rerun" pipeline to use the existing data and ask again for any missing data.
Static classes are used to hold the worker functions and separately factory methods that produce Dataflow blocks based on configuration. Eg, a CreateLogger(path,level) creates an ActionBlock<Message> that logs specific messages.
Common dataflow extension methods - since DataFlow blocks follow the same basic patterns, it's easy to create a logged block by combining eg a Func<TIn,TOut> and a logger block. Or create a LinkTo overload that redirects bad records to a logger or database. Those are common enough they can become extension methods.
If those were in the same file, it would be very hard to edit one pipeline without affecting another. Besides, there's a lot more to a pipeline than the core tasks, eg:
Logging
Handling bad records and partial results (can't stop a 100K import for 10 errors)
error handling (which isn't the same as handling bad records)
monitoring - what's this monster doing for the last 15 minutes? Did a DOP=10 improve performance at all?
Don't create a parent pipeline class.
Some of the steps are common, so at first, I created a parent class with common steps that got overloaded, or simply replaced in child classes. VERY BAD IDEA. Each pipeline is similar but not quite, and inheritance means that modifying one step or one connection risks breaking everything. After about 1 year things became unbearable, so I split the parent class into separate classes.

As #Panagiotis explained, I think you have to put aside the OOP Mindset a little.
What you have with DataFlow are Buildingblocks that you configure to execute what you need. I'll try to create a little example of what I mean by that:
// Interface and impl. are in separate files. Actually, they could
// even be in a different project ...
public interface IMyComplicatedTransform
{
Task<string> TransformFunction(int input);
}
public class MyComplicatedTransform : IMyComplicatedTransform
{
public Task<string> IMyComplicatedTransform.TransformFunction(int input)
{
// Some complex logic
}
}
class DataFlowUsingClass{
private readonly IMyComplicatedTransform myTransformer;
private readonly TransformBlock<int , string> myTransform;
// ... some more blocks ...
public DataFlowUsingClass()
{
myTransformer = new MyComplicatedTransform(); // maybe use ctor injection?
CreatePipeline();
}
private void CreatePipeline()
{
// create blocks
myTransform = new TransformBlock<int, string>(myTransformer.TransformFunction);
// ... init some more blocks
// TODO link blocks
}
}
I think this is the closest to what you are looking for to do.
What you end up with is a set of interfaces and implementations which can be tested independently. The client basically boils down to "gluecode".
Edit: As #Panagiotis correctly states, the interfaces are even superfluent. You could do without.

Adding entities to Stanford NLP NER Classifier

I have a very simple method to extract Names, Organisations and Locations from a string. I am using the .NET Nuget Libraries for Stanford NLP. It looks like this.
CRFClassifier Classifier = CRFClassifier.getClassifier(StanfordNLPConfig.NER.ClassifierModel);
List<IndexViewModel> ivms = new List<IndexViewModel>();
try
{
foreach (List sentence in Classifier.classify(content).toArray())
{
NLPTranslator translator = new NLPTranslator();
ivms.AddRange(translator.NERTranslate(sentence));
}
}
catch (Exception ex)
{
throw ex;
// Error silently
}
return ivms;
The model is the 3class jar file it came with - english.all.3class.distsim.crf.ser.gz.
This is working really well for me, but would I'd like to do is interface with the model to be able to add in my own entities should I need to, this seems very American Centric and I'd like to be able to put my own UK companies, locations etc.
Is there any way I can just add in these entities as I have been reading about training it but that you possibly can't extend the model, if this is the case can I combine Classifiers and run it through a UK one/US one etc. If that's possible, how can I actually make my own Classifier as I would like to make these in .NET if possible.

If you want to change the way the model works you'll need to provide training data and train your own model. For licensing reasons Stanford doesn't distribute the data the public models are trained on, but you can use the same features.
You can read about how to train a model here, though unfortunately the first sentence of the instructions is this:
The documentation for training your own classifier is somewhere between bad and non-existent.
If this is your first time working with a CRF there will many things to learn but it is manageable. It may be helpful to look at documentation for other packages such as CRFSuite and CRF++ - they generally all use basically the same training data format and are similar in many ways.
Also note that the existing models cannot be extended by training them just on new input, the system just isn't set up that way.

Is there an equivalent of Java's Spring MVC style url mapping for C#/.NET?

I am looking for the C# equivalent of Spring MVC's url mapping using annotations, i.e in Java I can write:
#Controller
#RequestMapping("/some-friendly-url/")
class MyController
{
#RequestMapping(value = "/{type}/more-seo-stuff/{color}", method = RequestMethod.GET)
public List<SomeDTO> get(#PathVariable String type,
#PathVariable String color,
int perPage) {
...
}
#RequestMapping(method = RequestMethod.POST)
public String post(#RequestBody SomeDTO somethingNew) {
...
}
}
It's actually much more powerful than this simple example as anyone familiar the the concept knows.
I've tried to search on how to achieve the same with either ASP.MVC 3 or with MonoRail and both frameworks seem to be based on RoR's convention-over-configuration "//" philosophy and it would be hard to achieve the above with them and require a lot of bespoke routing entries outside the controller class with only a small subset of the functionality available via attributes. Spring.NET does not seem to address this either stating that ASP.MVC's routing functionality is sufficient.
Is there anything out there in the C# world that provides this type of functionality? I was just about to start looking into writing something of my own to address this, but I was hoping not to have to do that.
Edit: Finally found the "AttributeRouting" project which is available on NuGet as well: https://github.com/mccalltd/AttributeRouting/wiki/1.-Getting-Started. Works perfectly. Doesn't support to full range of features that Spring MVC does, but supports most of it.
Also Akos Lukacs pointed to another good library below by ITCloud. However that one unfortunately is not available on NuGet.

Sure, you can use Spring.NET:
http://www.springframework.net/

I Eventually used https://github.com/mccalltd/AttributeRouting/wiki/1.-Getting-Started. Posting this only now for the sake of keeping the question complete.

What is a real-life use for the builder design pattern?

I've read about it, I understand it's basic function--I'd like to know an example of a common, real-life use for this pattern.
For reference, I work mostly with business applications, web and windows, using the Microsoft stack.

Think of an Itinerary builder. There are lots of things you can add to you Itinerary like hotels, rental cars, airline flights and the cardinality of each is 0 to *. Alice might have a car and hotel while Bob might have two flights, no car and three hotels.
It would be very hard to create an concrete factory or even an abstract factory to spit out an Itinerary. What you need is a factory where you can have different steps, certain steps happen, others don't and generally produce very different types of objects as a result of the creation process.
In general, you should start with factory and go to builder only if you need higher grain control over the process.
Also, there is a good description, code examples and UML at Data & Object Factory.

Key use cases:
When the end result is immutable, but
doing it all with a constructor would
be too complicated
When I want to partially build
something and reuse that partially
built thing, but customize it at
the end each time
When you start with the factory pattern, but the thing being built by
the factory has too many permutations
In summary, builder keeps your constructors simple, yet permits immutability.
You said C#, but here's a trivial Java example:
StringBuilder sb = new StringBuilder();
sb.append("Hello");
sb.append(" ");
sb.append("World!");
System.out.println(sb.toString());
As opposed to:
String msg = "";
msg += "Hello";
msg += " ";
msg += "World!";
System.out.println(msg);

EDIT: You will see in my comments that I may have rushed into answering this question, and confused myself in the process. I will go ahead and edit this to work with the Abstract Factory, as I think I originally intended, but please note that this is mainly for reference, not necessarily as a response to the original question.
The most common example I've seen described deals with how GUI components are built.
For example, if you were designing a form for your application, whose GUI components could take on multiple representations (perhaps based on which platform you were running on), you would design an abstract factory to handle the creation of those components.
In order to add new controls to the form, the code might look something like this:
public MyForm ()
{
GuiFactory factory = new Win32Factory ();
Button btn = factory.CreateButton ();
btn.Text = "Go!"
btn.Location = new Point (15, 50);
this.Controls.Add (btn);
}
This satisfies the Abstract Factory pattern because you can create different instances of the factory object to create different representations of your created objects without changing the client code (this is a rudimentary example, but I think normally you wouldn't create the Win32Factory using new, it would be acquired via some other abstraction).

A common example that we used to see all the time was a "sysgen" of an operating system. you had a process that selected all the modules you needed, configured them, and returned a bootable image that had been customized.

One use case that I've encountered is when having multiple data sources. The particular case involved a cache and a database. The majority of the data was pulled from cache, or not. The second loader looked at the data to see whether or not it was loaded from the cache. It would query the database to finish populating the data.

Sometimes it can be helpful to think of non-software related example of design patterns in order to understand them.
I have a system I am working on now that uses a builder to create an order. The order is a class composed of several other classes. My builder creates and validates the associated classes, and if all are valid it then creates an instance of the order class. This way I can be sure that I never have an instance of an order that is missing data.

Report handler architecture question

I am attempting to have a ReportHandler service to handle report creation. Reports can have multiple, differing number of parameters that could be set. In the system currently there are several different methods of creating reports (MS reporting services, html reports, etc) and the way the data is generated for each report is different. I am trying to consolidate everything into ActiveReports. I can't alter the system and change the parameters, so in some cases I will essentially get a where clause to generate the results, and in another case I will get key/value pairs that I must use to generate the results. I thought about using the factory pattern, but because of the different number of query filters this won't work.
I would love to have a single ReportHandler that would take my varied inputs and spit out report. At this point I'm not seeing any other way than to use a big switch statement to handle each report based on the reportName. Any suggestions how I could solve this better?

From your description, if you're looking for a pattern that matches better than Factory, try Strategy:
Strategy Pattern
Your context could be a custom class which encapsulates and abstracts the different report inputs (you could use the AbstractFactory pattern for this part)
Your strategy could implement any number of different query filters or additional logic needed. And if you ever need to change the system in the future, you can switch between report tools by simply creating a new strategy.
Hope that helps!

In addition to the strategy pattern, you can also create one adaptor for each of your underlying solutions. Then use strategy to vary them. I've built similar with each report solution being supported by what I called engines, In addition to the variable report solution we have variable storage solution as well - output can be stored in SQL server or file system.
I would suggest using a container then initializing it with the correct engine, e.g.:
public class ReportContainer{
public ReportContainer ( IReportEngine reportEngine, IStorageEngine storage, IDeliveryEngine delivery...)
}
}
/// In your service layer you resolve which engines to use
// Either with a bunch of if statements / Factory / config ...
IReportEngine rptEngine = EngineFactory.GetEngine<IReportEngine>( pass in some values)
IStorageEngine stgEngine = EngineFactory.GetEngine<IStorageEngien>(pass in some values)
IDeliverEngine delEngine = EngineFactory.GetEngine<IDeliverEngine>(pass in some values)
ReportContainer currentContext = new ReportContainer (rptEngine, stgEngine,delEngine);
then ReportContainer delegates work to the dependent engines...

We had a similar problem and went with the concept of "connectors" that are interfaces between the main report generator application and the different report engines. By doing this, we were able to create a "universal report server" application. You should check it out at www.versareports.com.