In my scenario, I have a Winforms client that connects to WebApi2. The data is stored in a SQL Server database.
To speed up performance, I am researching if storing data in local cache is a viable solution. Preferably, the local cache should be stored in files instead of kept in-memory as RAM might be an issue. The data is all POCO classes, some being much more complex than others, and most classes being related to each other.
I have made a shortlist of which frameworks might be viable:
MemoryCache
MemCached
CacheManager
StackExchange.Redis
Local Database
Using MemoryCache, I would need to implement my own solution, but it will fit my initial requirements.
However, one common problem that I am seeing is the updating of related classes. For example, I have a relationship between CustomerAddress and PostCode. If I change some properties in a postcode object, I can easily update its local cache. But how is it possible to update/invalidate any other classes that use this postcode, in this case CustomerAddress?
Does any of the frameworks above have methods that help in this kind of situation, or is it totally dependent on the developer to handle such cache invalidation?
The CachingFramework.Redis library provides a mechanism to relate tags to keys and hashes so you can then invalidate them in a single operation.
I'm assuming that you will:
Store the Customer Addresses in Redis with keys like "Address:{AddressId}".
Store the Post Codes in Redis with keys like "PostCode:{PostCodeId}".
And that your model is something like this:
public class CustomerAddress
{
public int CustomerAddressId { get; set; }
public int CustomerId { get; set; }
public int PostCodeId { get; set; }
}
public class PostCode
{
public int PostCodeId { get; set; }
public string Code { get; set; }
}
My suggestion is to:
Mark the Customer Addresses objects on Redis with tags like "Tag-PostCode:{PostCodeId}".
Use a cache-aside pattern to retrieve the Customer Addresses and Post Codes from cache/database.
Invalidate the cache objects by tag when a Post Code is changed.
Something like this should probably work:
public class DataAccess
{
private Context _cacheContext = new CachingFramework.Redis.Context("localhost:6379");
private string FormatPostCodeKey(int postCodeId)
{
return string.Format("PostCode:{0}", postCodeId);
}
private string FormatPostCodeTag(int postCodeId)
{
return string.Format("Tag-PostCode:{0}", postCodeId);
}
private string FormatAddressKey(int customerAddressId)
{
return string.Format("Address:{0}", customerAddressId);
}
public void InsertPostCode(PostCode postCode)
{
Sql.InsertPostCode(postCode);
}
public void UpdatePostCode(PostCode postCode)
{
Sql.UpdatePostCode(postCode);
//Invalidate cache: remove CustomerAddresses and PostCode related
_cacheContext.Cache.InvalidateKeysByTag(FormatPostCodeTag(postCode.PostCodeId));
}
public void DeletePostCode(int postCodeId)
{
Sql.DeletePostCode(postCodeId);
_cacheContext.Cache.InvalidateKeysByTag(FormatPostCodeTag(postCodeId));
}
public PostCode GetPostCode(int postCodeId)
{
// Get/Insert the postcode from/into Cache with key = PostCode{PostCodeId}.
// Mark the object with tag = Tag-PostCode:{PostCodeId}
return _cacheContext.Cache.FetchObject(
FormatPostCodeKey(postCodeId), // Redis Key to use
() => Sql.GetPostCode(postCodeId), // Delegate to get the value from database
new[] { FormatPostCodeTag(postCodeId) }); // Tags related
}
public void InsertCustomerAddress(CustomerAddress customerAddress)
{
Sql.InsertCustomerAddress(customerAddress);
}
public void UpdateCustomerAddress(CustomerAddress customerAddress)
{
var updated = Sql.UpdateCustomerAddress(customerAddress);
if (updated.PostCodeId != customerAddress.PostCodeId)
{
var addressKey = FormatAddressKey(customerAddress.CustomerAddressId);
_cacheContext.Cache.RenameTagForKey(addressKey, FormatPostCodeTag(customerAddress.PostCodeId), FormatPostCodeTag(updated.PostCodeId));
}
}
public void DeleteCustomerAddress(CustomerAddress customerAddress)
{
Sql.DeleteCustomerAddress(customerAddress.CustomerAddressId);
//Clean-up, remove the postcode tag from the CustomerAddress:
_cacheContext.Cache.RemoveTagsFromKey(FormatAddressKey(customerAddress.CustomerAddressId), new [] { FormatPostCodeTag(customerAddress.PostCodeId) });
}
public CustomerAddress GetCustomerAddress(int customerAddressId)
{
// Get/Insert the address from/into Cache with key = Address:{CustomerAddressId}.
// Mark the object with tag = Tag-PostCode:{PostCodeId}
return _cacheContext.Cache.FetchObject(
FormatAddressKey(customerAddressId),
() => Sql.GetCustomerAddress(customerAddressId),
a => new[] { FormatPostCodeTag(a.PostCodeId) });
}
}
To speed up performance, I am researching if storing data in local
cache is a viable solution. Preferably, the local cache should be
stored in files instead of kept in-memory as RAM might be an issue
The whole issue is to avoid storing it in files, to avoid DISK operations which are slow, thus Redis is RAM based memory.
Does any of the frameworks above have methods that help in this kind
of situation, or is it totally dependent on the developer to handle
such cache invalidation?
You can save the entire object as JSON instead of applying logic and disassembles the objects, which will be also slow and error prone when applying changes.
Related
I'm new to NoSql and MongoDB. I'm using the MongoDB C# driver inside Visual Studio.
I've read in different places that it's preferable to have a single instance of your database class that maintains the connection(s) to keep everything thread safe and that it's generally a bad idea to use static classes for database CRUD operations.
At the start of my program I instantiate my database class which opens a connection. Within that class and also in derived classes I can perform CRUD operations. But now I'm in a different part of my solution (same namespace, different class) and I need to do read operations to check if a user exists. I also need to compose a new document that I then want to insert.
Now I'm in a situation where that's only possible by creating a new instance of the database class to access its CRUD methods. I want to avoid static CRUD methods (that could be accessed from other classes) because then the base class of my database connection also needs to be static. I cannot figure out how to approach this and what would be the recommended way.
From the MongoDB website:
The MongoClient instance actually represents a pool of connections to the database; you will only need one instance of class MongoClient even with multiple threads.
http://mongodb.github.io/mongo-csharp-driver/2.2/getting_started/quick_tour/
Does this mean I should create a new MongoClient everytime I need to acces the database in others parts of my program?
UPDATE
It seems I was a bit mistaken about the static properties and how they can be used. I now have it setup like this:
class Database
{
const string MongoConnection = "mongodb+srv://user:password#cluster.mongodb.net";
public static MongoClient Client { get; set; }
public static IMongoDatabase Directory { get; set; }
public static IMongoCollection<User> Collection { get; set; }
public Database()
{
Client = new MongoClient(MongoConnection);
Directory= Client.GetDatabase("studentDB");
Collection = Directory.GetCollection<User>("users");
}
public static void InsertNewUser(User user)
{
Collection.InsertOne(user);
}
public static bool EmailHasAccount(string email)
{
return Collection.Find(x => x.Email == email).FirstOrDefault() == null ? false : true;
}
public static User RetrieveUserAccount(string email)
{
return Collection.Find(x => x.Email == email).FirstOrDefault();
}
}
public class User
{
public Guid Id { get; private set; }
public string Name { get; set; }
public string Email { get; set; }
public User(string name, string email)
{
Id = Guid.NewGuid();
Name = name;
Email = email;
}
}
And in my main program I can use it like this:
var db = new Database();
var user = new User("myName", "email#address");
Database.InsertNewUser(user);
Console.WriteLine(Database.EmailHasAccount("email#address")); // returns true
Console.WriteLine(Database.RetrieveUserAccount("email#address").Name); // returns "myName"
That's exactly what I was looking for. What would be the best way to handle multiple collections? Would it be safe to change the Collection property or is it better to create separate properties? Is a Generic even possible?
How can I add items to my list SearchedVideos?
I would like to have these items on the list until the end of my application.
Now I have error like this:
NullReferenceException: Object reference not set to an instance of an object.
I create context with prop as Singleton like this:
public List<QueryViewModel> SearchedVideos { get; set; }
In startup
services.AddSingleton<YtContext>();
My model
public class ExecutedQuery
{
public Query Query { get; }
public string Title { get; set; }
public IReadOnlyList<Video> Videos { get; set; }
public ExecutedQuery(Query query, string title, IReadOnlyList<Video> videos)
{
Query = query;
Title = title;
Videos = videos;
}
}
My service
public async Task<ExecutedQuery> ExecuteQueryAsync(Query query)
{
// Search
if (query.Type == QueryType.Search)
{
var videos = await _youtubeClient.SearchVideosAsync(query.Value);
var title = $"Search: {query.Value}";
var executedQueries = new ExecutedQuery(query, title, videos);
var qw = new QueryViewModel
{
ExecutedQueries = executedQueries,
};
_ytcontext.SearchedVideos.Add(qw);
return executedQueries;
}
}
My QueryViewModel
public ExecutedQuery ExecutedQueries { get; set; }
My Controller
[HttpGet("Search/all")]
public async Task<IActionResult> ListAllQueriesAsync(string query)
{
var req = _queryService.ParseQuery(query);
var res = await _queryService.ExecuteQueryAsync(req);
return View(res);
}
If you are wanting to edit this list from one instance to another then you'll need to use some kind of datasource. If a database is not an option then a text file will have to do. Use a Json string and serialize/deserialize to your object. https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/how-to-serialize-and-deserialize-json-data. I've used this method to mockup an application but if you are going to be doing alot of writing to the file you may run into issues.
If you can hard code the list in the application then a Singleton will work. Read up here. https://learn.microsoft.com/en-us/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-2.2
Each request is its own thing, unaffected by anything that's happened before or since. As such, you pretty much start from a blank slate. The typical means for persisting state between one or more additional requests is the session. Sessions are essentially fake state, through a combination of server-side (some persistent store) and client-side components (cookies), something that appears like persistence of state can be achieved. However, particularly on the server-side, you still need some sort of store, which is generally a database of some sort, be it relational (SQL Server, etc.) or NoSQL (Redis, etc.). The default session store will be in-memory, which may suffice for your needs, but as memory is volatile, any sort of app restart will take anything stored there along with it.
Alternatively, there's statics and objects with singleton lifetimes. In either case, they're virtually the same as in-memory storage - they'll persist the life of the application and no more.
Statics are just members with a static keyword on them. It's probably the simplest and most straight-forward approach, but also the most fragile. It's virtually impossible to test statics, so you're basically creating black-holes in your code where anything could happen.
A better approach is to simply use an object with a singleton lifetime. These can be create via the AddSingleton<T> method on the service collection. For example, you could create a class like:
public class MySingleton
{
public ICollection<IReadOnlyList<Video>> SearchedVideo { get; set; }
}
And then register it as a singleton in ConfigureServices:
services.AddSingleton<MySingleton>();
Then, in your controllers, views, and such, you can inject MySingleton to access the SearchedVideos property. As a singleton, the data there will persist for the life of the application.
The chief difference between sessions, particularly in-memory sessions, and either statics or singletons is one of breadth. Sessions will always be tied to a particular client, whereas statics and singletons will be scoped to the application. That means that if you use statics or singletons, all clients will see the same data and will potentially manipulate the same data. If you need something that is client-specific, you must use sessions, instead.
#natsukiss i guess you are trying to call Add() method from null property. Even you create a list you should set an initial instance for SearchedVideo Property. Because if you dont create an instance, it means that property will not have address in memory. Because of that sometimes we are using string TestVal = "". That means we sets initial value on Common Language Runtime(CLR) to locate Address in Memory.
public List<QueryViewModel> SearchedVideos { get; set; } = new List<QueryViewModel>(); //<==
or if you are working with EntityFramework you should use
public ICollection<QueryViewModel> SearchedVideos { get; set; } = new HashSet<QueryViewModel>(); //<===
Suppose I have a model with 20 fields, and in my index page, I want to list all models that are stored in my database.
In index page, instead of listing all fields of the model, I only to list 3 fields.
So, I make two class:
class CompleteModel {
public int Id { get; set; }
public string Field01 { get; set; }
public string Field02 { get; set; }
public string Field03 { get; set; }
public string Field04 { get; set; }
public string Field05 { get; set; }
...
public string Field20 { get; set; }
}
now, in my Controller, I can use:
await _context.CompleteModel.ToListAsync();
but I feel that it does not seem to be the right way to do it, because I'm getting all fields and using only 3 fields.
So, I made this code:
class ViewModel {
public string Field02 { get; set; }
public string Field04 { get; set; }
public string Field08 { get; set; }
}
var result = _context.CompleteModel.Select(
x => new {
x.Field02,
x.Field04,
x.Field08
}).ToListAsync();
var listResults = new List<IndexViewModel>();
if (result != null)
{
listResults.AddRange(results.Select(x => new IndexViewModel
{
Field02 = x.Field02,
Field04 = x.Field04,
Field08 = x.Field08
}));
}
I think this is a lot of code to do this.
First, I selected all the fields that I want, then, copied everything to another object.
There's a "more directly" way to do the same thing?
Like:
_context.CompleteModel.Select(x => new IndexViewModel { Field02, Field04, Field08 });
You could use AutoMapper to reduce the boiler plate so you're not manually copying field values over.
If you include the AutoMapper NuGet package then you'd need to have the following in your startup somewhere to configure it for your classes:
Mapper.Initialize(cfg => cfg.CreateMap<CompleteModel, ViewModel>());
You could then do something like the following:
var results = await _context.CompleteModel.ToListAsync();
var viewModelResults = results.Select(Mapper.Map<ViewModel>).ToList();
There are a lot of configuration options for the package so do take a look at the documentation to see if it suits your needs and determine the best way to use it if it does.
In my view this is one of the weaknesses of over abstraction and layering. The VM contains the data that is valuable to your application within the context of use (screen, process etc). The data model contains all the data that could be stored that might be relevant. At some point you need to match the two.
Use EF Projection to fetch only the data you need from the database into projected data model classes (using the EF POCO layer to define the query, but not to store the resultant data).
Map the projected classes onto your VM, if there is a naieve mapping, using Automapper or similar. However unless you are just writing CRUD screens a simple field by field mapping is of little value; the data you fetch from your data store via EF is in its raw, probably relational form. The data required by your VM is probably not going to fit that form very neatly (again, unless you are doing a simple CRUD form), so you are going to need to add some value by coding the relationship between the data store and the View Model.
I think concentrating on the count of lines of code would lead to the wrong approach. I think you can look at that code and ask "is it adding any value". If you can delegate the task to Automapper, then great; but your VM isn't really pulling its weight other than adding some validation annotation if you can consistently delegate the task of data model to VM data copying.
I have this object:
public class Announcement
{
public int Id { get; set; }
public DateTime DateSent { get; set; }
private IList<string> _recipients;
public IList<string> Recipients
{
get { return _recipients; }
set { _recipients = value; }
}
public string RecipientsString
{
get { return String.Join("\n", _recipients); }
set { _recipients = value.Split('\n').ToList(); }
}
}
I can populate this object with the DateSent and RecipientString (a string of email addresses separated by \n) and save it to the database with no problems.
Now I want to move this to a web service so we can use it across multiple apps.
I created the exact same object in the webservice, and testing locally (on the service) everything works as expected.
But if I populate the object on the client and pass it to the service to be saved, the RecipientString is always empty (not null). The DateSent is fine.
I'm guessing the data is getting lost in serialization, but I don't know why, or how to solve this. I thought also, it could have something to do with the # in the email address, but I've ruled that out. Any suggestions?
This happens because de WSDL that is generated to describe your service can't describe the function that is used in your get and set functions. I suggest you keep RecipientsString as a common property, and create a private method GetRecipients on your class that processes the RecipientsString value and returns the list you need.
Use RecipientsString without backing field.
I'm still not yet sure on the best way to store selectlist options for front end display or db storage.
I've been using Enums at the moment, and also using description decorators (How do you create a dropdownlist from an enum in ASP.NET MVC?)
I'm now thinking that I might as well just create a full class for this stuff, so I can store the following information properly with full control:
Item Name
Full description
int for storage in db
order
Any methods to get information in anyway from the list.
Is it right I should be thinking about implementing all this myself by hand? I want a really solid way of doing this, and an enum doesn't really feel like it's going to cut it.
Is it right I should be thinking about implementing all this myself by
hand?
Yes. Enums are often leaky and insufficient abstractions that aren't always suitable for the complex domain model you actually wish to represent.
Rather than roll your own, you may want to consider Headspring's Enumeration class (via github, nuget). We use it all the time instead of enums because it's nearly as simple and is much more flexible.
An example of a "State" enumeration and using it as a select list:
public class State : Enumeration<State>
{
public static State Alabama = new State(1, "AL", "Alabama");
public static State Alaska = new State(2, "AK", "Alaska");
// .. many more
public static State Wyoming = new State(3, "WY", "Wyoming");
public State(int value, string displayName, string description) : base(value, displayName)
{
Description = description;
}
public string Description { get; private set; }
}
public IEnumerable<SelectListItem> Creating_a_select_list(State selected)
{
return State.GetAll().Select(
x => new SelectListItem
{
Selected = x == selected,
Text = x.Description,
Value = x.Value.ToString()
});
}
I'm not trying to sell you on this particular implementation, you could certainly hand code your own (the Enumeration class is only about 100 lines of code). But I definitely think you'd benefit from moving beyond basic enums. It is the right approach given the scenario you described in your question.
The first place where such information shoiuld be is the database...or any "virtual store" such as a web service that offers an interface to you db. In fact if there are other db entiies that use these values THEY MUST be represented in the database, otherwise you will run in big troubles. In fact, suppose one of such values is a string....if you don't define a table containing all possible values+a key and simply write the string as it is in other tables...it will be impossible for you to change the format of the string since it will be "spread" all over your db...On the contrary, if you just use an external key to refer to such strings...you can easily change them since the string is stored in just ONE place in your db.
Also the enumeration solution suffers of the problem that you cannot add or deleted values...so if such operations "conceptually" might make sense you cannot use an enumeration. You can use enumeration when all options "conceptually span" all possibilities, so you are sure you will never add/delete other options, such as in the case of the enumeration (yes, no, unknown).
That said, once you have your options in the db the remainder is easy...you will have DTO entities or Business entities representing them in exactly the same way you do for all other DB entities.
For visualization purposes you may have a ViewModel version of this options that might just contain key and description, and a "Repository method" that your controllers can call to have the list of all options.
Once retrieved you controllers put them in the overall page ViewViewModel...together with all other information to be shown on the page. From the ViewModel...you can access them to put them in a dropdown.
Summing up:
1) You need a DB representation of your options
2) Then you will have DTO, business layer, and View versions of this entities...as needed, exactly as for all other DB entities.
Are you looking for a one-size-fits-all solution for all your select list options? I personally advocate choosing the option that best fits the specific issue.
In a recent project I was introduced to a hybrid of a Smart Enum. Here's an example (I apologize for typos, I'm typing this cold):
public class Priority
{
public enum Types
{
High,
Medium,
Low
}
public Types Type { get; private set; }
public string Name { get { return this.Type.ToString(); } } // ToString() with no arguments is not deprecated
public string Description { get; private set; }
public static High = new Priority{ Type = Types.High, Description = "..."};
public static Medium = new Priority{ Type = Types.Medium, Description = "..."};
public static Low = new Priority{ Type = Types.Low, Description = "..."};
public static IEnumerable<Priority> All = new[]{High, Medium, Low};
public static Priority For(Types priorityType)
{
return All.Single(x => x.Type == priorityType);
}
}
So, in implementation, you could store the Enum value, but you would reference the object itself (Priority.For(entity.priority)) for the additional metadata when rendering your views.
Is that closer to what you're looking for?
Of course, one of the gotchas is if you need to write a query against the database that relies on the metadata on the lookup, this solution is going to create a few tears along the way.
You can use "repository pattern" for data access and use viewmodels between your controllers and views. Example:
//Model
public class CustomerViewModel
{
public Customer customer { get;set; }
public IEnumerable<Village> Villages { get; set; }
}
//Controller
public ActionResult Index()
{
var customerViewModel = new CustomerViewModel
{
Customer = new Customer(),
Villages = _villageService.GetAll()
};
return View(customerViewModel);
}
//View
#model ViewModel.RegisterViewModel
#Html.DropDownListFor(q => q.Customer.VillageId, new SelectList(Model.Villages, "Id", "Title"), "Please Select")
I have written a blog post about repository pattern, you may have a look.
I store my options in the View Models themselves:
public class ViewModel {
[Required]
public int SelectListValue { get; set; }
public IDictionary<String,String> SelectListOptions {
get {
return new Dictionary<String, String>{
{ "0", Resources.Option1},
{ "1", Resources.Option2},
{ "2", Resources.Option3}
};
}
}
}
Then I can just drop the following line into my view to render the select list:
<%= Html.DropDownListFor(m => m.SelectListValue, new SelectList(this.Model.SelectListOptions, "Key", "Value", "")) %>