I am new to Redis (using it at a hosted service) and want to use it as a demonstration / sandbox data storage for lists.
I use the following piece of code. It works - for me. But is it a valid (and not completely bad practice) usage for a small web site with several (up to 100) concurrent users (for a small amount of data - up to 1000 list items)?
I'm using static connection and a static redisclient typed list like this:
public class MyApp
{
private static ServiceStack.Redis.RedisClient redisClient;
public static IList<Person> Persons;
public static IRedisTypedClient<Person> PersonClient;
static MyApp()
{
redisClient = new RedisClient("nnn.redistogo.com", nnn) { Password = "nnn" };
PersonClient = redisClient.GetTypedClient<Person>();
Persons = PersonClient.Lists["urn:names:current"];
}
}
Doing this I have a very easy to use persistent list of data, which is exactly what I want when I'm building / demonstrating the basic blocks of my application.
foreach (var person in MyApp.Persons) ...
Adding a new person:
MyApp.Persons.Add(new Person { Id = MyApp.PersonClient.GetNextSequence(), Name = "My Name" });
My concern is (currently) not the fact that I am loading the complete list into memory at appstart, but rather the possibility that my connection to the redis host is not following good standards - or that there is some other issue that I'm not aware of.
Thanks
Actually when you use PersonClient.Lists["urn:names:current"] you're actually storing a reference to a RedisClient Connection which is not thread safe. It's ok if it's in a GUI or Console app, but not ideal in a multi-threaded web app. In most scenarios you want to be using a thread safe connection factory i.e.
var redisManager = new PooledRedisClientManager("localhost:6379");
Which acts very much like a database connection pool. So whenever you want to access the RedisClient works like:
using (var redis = redisManager.GetClient())
{
var allItems = redis.As<Person>().Lists["urn:names:current"].GetAll();
}
Note: .As<T> is a shorter alias for .GetTypedClient<T>
Another convenient short-cut to execute a typed client from a redisManager is:
var allItems = redisManager.ExecAs<Person>(r => r.Lists["urn:names:current"].GetAll());
I usually prefer to pass around IRedisClientsManager in my code so it doesn't hold a RedisClient connection but can access it whenever it needs to.
There is an example project here
Related
I have a requirement where we need a plugin to retrieve a session id from an external system and cache it for a certain time. I use a field on the entity to test if the session is actually being cached. When I refresh the CRM form a couple of times, from the output, it appears there are four versions (at any time consistently) of the same key. I have tried clearing the cache and testing again, but still the same results.
Any help appreciated, thanks in advance.
Output on each refresh of the page:
20170511_125342:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125358:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125410:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125342:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125437:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125358:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125358:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
20170511_125437:1:55a4f7e6-a1d7-e611-8100-c4346bc582c0
To accomplish this, I have implemented the following code:
public class SessionPlugin : IPlugin
{
public static readonly ObjectCache Cache = MemoryCache.Default;
private static readonly string _sessionField = "new_sessionid";
#endregion
public void Execute(IServiceProvider serviceProvider)
{
var context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
try
{
if (context.MessageName.ToLower() != "retrieve" && context.Stage != 40)
return;
var userId = context.InitiatingUserId.ToString();
// Use the userid as key for the cache
var sessionId = CacheSessionId(userId, GetSessionId(userId));
sessionId = $"{sessionId}:{Cache.Select(kvp => kvp.Key == userId).ToList().Count}:{userId}";
// Assign session id to entity
var entity = (Entity)context.OutputParameters["BusinessEntity"];
if (entity.Contains(_sessionField))
entity[_sessionField] = sessionId;
else
entity.Attributes.Add(new KeyValuePair<string, object>(_sessionField, sessionId));
}
catch (Exception e)
{
throw new InvalidPluginExecutionException(e.Message);
}
}
private string CacheSessionId(string key, string sessionId)
{
// If value is in cache, return it
if (Cache.Contains(key))
return Cache.Get(key).ToString();
var cacheItemPolicy = new CacheItemPolicy()
{
AbsoluteExpiration = ObjectCache.InfiniteAbsoluteExpiration,
Priority = CacheItemPriority.Default
};
Cache.Add(key, sessionId, cacheItemPolicy);
return sessionId;
}
private string GetSessionId(string user)
{
// this will be replaced with the actual call to the external service for the session id
return DateTime.Now.ToString("yyyyMMdd_hhmmss");
}
}
This has been greatly explained by Daryl here: https://stackoverflow.com/a/35643860/7708157
Basically you are not having one MemoryCache instance per whole CRM system, your code simply proves that there are multiple app domains for every plugin, so even static variables stored in such plugin can have multiple values, which you cannot rely on. There is no documentation on MSDN that would explain how the sanboxing works (especially app domains in this case), but certainly using static variables is not a good idea.Of course if you are dealing with online, you cannot be sure if there is only single front-end server or many of them (which will also result in such behaviour)
Class level variables should be limited to configuration information. Using a class level variable as you are doing is not supported. In CRM Online, because of multiple web front ends, a specific request may be executed on a different server by a different instance of the plugin class than another request. Overall, assume CRM is stateless and that unless persisted and retrieved nothing should be assumed to be continuous between plugin executions.
Per the SDK:
The plug-in's Execute method should be written to be stateless because
the constructor is not called for every invocation of the plug-in.
Also, multiple system threads could execute the plug-in at the same
time. All per invocation state information is stored in the context,
so you should not use global variables or attempt to store any data in
member variables for use during the next plug-in invocation unless
that data was obtained from the configuration parameter provided to
the constructor.
Reference: https://msdn.microsoft.com/en-us/library/gg328263.aspx
Code Details:
// Singleton class CollectionObject
public class CollectionObject
{
private static CollectionObject instance = null;
// GetInstance() is not called from multiple threads
public static CollectionObject GetInstance()
{
if (CollectionObject.instance == null)
CollectionObject.instance = new CollectionObject();
return CollectionObject.instance;
}
// Dictionary object contains Service ID (int) as key and Service object as the value
// Dictionary is filled up during initiation, before the method call ReadServiceMatrix detailed underneath
public Dictionary<int, Service> serviceCollectionDictionary = new Dictionary<int,Service>();
public Service GetServiceByIDFromDictionary(int servID)
{
if (this.serviceCollectionDictionary.ContainsKey(servID))
return this.serviceCollectionDictionary[servID];
else
return null;
}
}
DataTable serviceMatrix = new DataTable();
// Fill serviceMatrix data table from the database
private int ReadServiceMatrix()
{
// Access the Singleton class object
CollectionObject collectionObject = CollectionObject.GetInstance();
// Parallel processing of the data table rows
Parallel.ForEach<DataRow>(serviceMatrix.AsEnumerable(), row =>
{
//Access Service ID from the Data table
string servIDStr = row["ServID"].ToString().Trim();
// Access other column details for each row of the data table
string currLocIDStr = row["CurrLocId"].ToString().Trim();
string CurrLocLoadFlagStr = row["CurrLocLoadFlag"].ToString().Trim();
string nextLocIDStr = row["NextLocId"].ToString().Trim();
string nextLocBreakFlagStr = row["NextLocBreakFlag"].ToString().Trim();
string seqStr = row["Seq"].ToString().Trim();
int servID = Int32.Parse(servIDStr);
int currLocID = Int32.Parse(currLocIDStr);
int nextLocID = Int32.Parse(nextLocIDStr);
bool nextLocBreakFlag = Int32.Parse(nextLocBreakFlagStr) > 0 ? true : false;
bool currLocBreakFlag = Int32.Parse(CurrLocLoadFlagStr) > 0 ? true : false;
int seq = Int32.Parse(seqStr);
// Method call leading to the issue (definition in Collection Object class)
// Fetch service object using the Service ID from the DB
Service service = collectionObject.GetServiceByIDFromDictionary(servID);
// Call a Service class method
service.InitLanes.Add(new Service.LaneNode(currLoc.SequentialID, currLocBreakFlag, nextLoc.SequentialID, nextLocBreakFlag, seq));
}
Issue that happens is:
In the code above for all the Service objects in the dictionary, the subsequent method call is not made, leading to issues in further processing. It has to o with fetching the Service object from the dictionary in parallel mode
The db an dictionary contains all the Ids /Service objects, but my understanding is when processing in Parallel mode for the Singleton class, few of the objects are skipped leading to the issue.
In my understanding the service id passed and service object created is local to a thread, so there shouldn't be an issue that I am facing. This kind of issue is only possible, when for a given method call one thread replace service id value of another thread by its, thus both end up with Service object and few are thus skipped, which is strange in my view until and unless I do not understand the Multi threading in this case correctly
Currently I am able to run the same code in non threaded mode by using the foreach loop instead of Parallel.ForEach / Parallel.Invoke
Please review and let me know your view or any pointer that can help me resolve the issue
In my understanding the service id passed and service object created
is local to a thread
Your understanding is incorrect, if two threads request the same service id the two threads will be both working on the same singular object. If you wanted separate objects you would need to put some kind of new Service() call in GetServiceByIDFromDictionary instead of a dictionary of existing values.
Because multiple threads could be using the same service objects I think your problem lies from the fact that service.InitLanes.Add is likely not thread safe.
The easiest fix is to just lock on that single step
//...SNIP...
Service service = collectionObject.GetServiceByIDFromDictionary(servID);
// Call a Service class method, only let one thread do it for this specific service instance,
// other threads locking on other instances will not block, only other threads using the same instance will block
lock(service)
{
service.InitLanes.Add(new Service.LaneNode(currLoc.SequentialID, currLocBreakFlag, nextLoc.SequentialID, nextLocBreakFlag, seq));
}
}
This assumes that this Parallel.Foreach is the only location collectionObject.GetServiceByIDFromDictionary is used concurrently. If it is not, any other locations that could potentially be calling any methods on returned services must also lock on service.
However if Service is under your control and you can somehow modify service.InitLanes.Add to be thread safe (perhaps change InitLanes out with a thread safe collection from the System.Collections.Concurrent namespace) that would be a better solution than locking.
1.Implementing singleton always think about using of it in mulithreaded way. Always use multithreaded singleton pattern variant, one of them - lazy singleton. Use Lazy singleton using System.Lazy with appropriate LazyThreadSafeMode consturctor argument:
public class LazySingleton3
{
// static holder for instance, need to use lambda to enter code here
//construct since constructor private
private static readonly Lazy<LazySingleton3> _instance
= new Lazy<LazySingleton3>(() => new LazySingleton3(),
LazyThreadSafeMode.PublicationOnly);
// private to prevent direct instantiation.
private LazySingleton3()
{
}
// accessor for instance
public static LazySingleton3 Instance
{
get
{
return _instance.Value;
}
}
}
Read about it here
2.Use lock-ing of your service variable in parallel loop body
// Method call leading to the issue (definition in Collection Object class)
// Fetch service object using the Service ID from the DB
Service service = collectionObject.GetServiceByIDFromDictionary(servID);
lock (service)
{
// Call a Service class method
service.InitLanes.Add(new Service.LaneNode(currLoc.SequentialID,
currLocBreakFlag, nextLoc.SequentialID,
nextLocBreakFlag, seq));
}
3.Consider to use multithreading here. Using lock-ing code make your code not so perfomant as synchronous. So make sure you multithreaded/paralelised code gives you advantages
4.Use appropriate concurrent collections instead of reinventing wheel - System.Collections.Concurrent Namespace
I basically have created a class which when a user logs into a website it then queries the database and stores some settings in a List (So I have key/pair values).
The reason for this is because I want to always be able to access these settings without going to the database again.
I put these in a class and loop through the fields via a SQL query and add them to the list.
How can I then access these variables from another part of the application? or is there a better way to do this? I'm talking server side and not really client side.
Here is an example of what I had at the moment:
public static void createSystemMetaData()
{
string constring = ConfigurationManager.ConnectionStrings["Test"].ConnectionString;
SqlConnection sql = new SqlConnection(constring);
sql.Open();
SqlCommand systemMetaData = new SqlCommand("SELECT * FROM SD_TABLES", sql);
//Set Modules
using (SqlDataReader systemMetaDataReader = systemMetaData.ExecuteReader())
{
while (systemMetaDataReader.Read())
{
var name = systemMetaDataReader.GetOrdinal("Sequence").ToString();
var value = systemMetaDataReader.GetOrdinal("Property").ToString();
var Modules = new List<KeyValuePair<string, string>>();
Modules.Add(new KeyValuePair<string, string>(name, value));
}
}
}
Thanks
Any static properties of a class will be preserved for the lifetime of the application pool, assuming you're using ASP.NET under IIS.
So a very simple class might look like:
public static class MyConfigClass
{
public static Lazy<Something> MyConfig = new Lazy<Something>(() => GetSomethings());
public static Something GetSomethings()
{
// this will only be called once in your web application
}
}
You can then consume this by simply calling
MyConfigClass.MyConfig.Value
For less users you can go with the SessionState as Bob suggested,however with more users you might need to move to a state server or load it from Data Base each time.
As others have pointed out, the risk of holding these values in global memory is that the values might change. Also, global variables are a bad design decision as you can end up with various parts of your application reading and writing to these values, which makes debugging problems harder than it need be.
A commonly adopted solution is to wrap your database access inside a facade class. This class can then cache the values if you wish to avoid hitting the database for each request. In addition, as changes are routed through the facade too, it knows when the data has changed and can empty its cache (forcing a database re-read) when this occurs. As an added bonus, it becomes possible to mock the facade in order to test code without touching the database (database access is notoriously difficult to unit test).
From the looks of things you are using universal values irrespective of users so an SqlCacheDependency would be useful here:
Make sure you setup a database dependency in web.config for the name Test
public static class CacheData {
public static List<KeyValuePair<string,string>> GetData() {
var cache = System.Web.HttpContext.Current.Cache;
SqlCacheDependency SqlDep = null;
var modules = Cache["Modules"] as List<KeyValuePair<string,string>>;
if (modules == null) {
// Because of possible exceptions thrown when this
// code runs, use Try...Catch...Finally syntax.
try {
// Instantiate SqlDep using the SqlCacheDependency constructor.
SqlDep = new SqlCacheDependency("Test", "SD_TABLES");
}
// Handle the DatabaseNotEnabledForNotificationException with
// a call to the SqlCacheDependencyAdmin.EnableNotifications method.
catch (DatabaseNotEnabledForNotificationException exDBDis) {
SqlCacheDependencyAdmin.EnableNotifications("Test");
}
// Handle the TableNotEnabledForNotificationException with
// a call to the SqlCacheDependencyAdmin.EnableTableForNotifications method.
catch (TableNotEnabledForNotificationException exTabDis) {
SqlCacheDependencyAdmin.EnableTableForNotifications("Test", "SD_TABLES");
}
finally {
// Assign a value to modules here before calling the next line
Cache.Insert("Modules", modules, SqlDep);
}
}
return modules;
}
I've got an MVC3 project and one of the models is built as a separate class library project, for re-use in other applications.
I'm using mini-profiler and would like to find a way to profile the database connections and queries that are made from this class library and return the results to the MVC3 applciation.
Currently, in my MVC3 app, the existing models grab a connection using the following helper class:
public class SqlConnectionHelper
{
public static DbConnection GetConnection()
{
var dbconn = new SqlConnection(ConfigurationManager.ConnectionStrings["db"].ToString());
return new StackExchange.Profiling.Data.ProfiledDbConnection(dbconn, MiniProfiler.Current);
}
}
The external model can't call this function though, because it knows nothing of the MVC3 application, or of mini-profiler.
One way I thought of would be to have an IDbConnection Connection field on the external model and then pass in a ProfiledDbConnection object to this field before I call any of the model's methods. The model would then use whatever's in this field for database connections, and I should get some profiled results in the MVC3 frontend.
However, I'm not sure if this would work, or whether it's the best way of doing this. Is there a better way I'm missing?
ProfiledDbConnection isn't dapper: it is mini-profiler. We don't provide any magic that can take over all connection creation; the only thing I can suggest is to maybe expose an event in your library that can be subscribed externally - so the creation code in the library might look a bit like:
public static event SomeEventType ConnectionCreated;
static DbConnection CreateConnection() {
var conn = ExistingDbCreationCode();
var hadler = ConnectionCreated;
if(handler != null) {
var args = new SomeEventArgsType { Connection = conn };
handler(typeof(YourType), args);
conn = args.Connection;
}
return conn;
}
which could give external code the chance to do whatever they want, for example:
YourType.ConnectionCreated += (s,a) => {
a.Connection = new StackExchange.Profiling.Data.ProfiledDbConnection(
a.Connection, MiniProfiler.Current);
};
I'm creating a WCF Service Library and I have a question regarding thread-safety consuming a method inside this library, here is the full implementation that I have until now.
namespace WCFConfiguration
{
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall, ConcurrencyMode = ConcurrencyMode.Single)]
public class ConfigurationService : IConfigurationService
{
ConcurrentDictionary<Tuple<string,string>, string> configurationDictionary = new ConcurrentDictionary<Tuple<string,string>, string>();
public void Configuration(IEnumerable<Configuration> configurationSet)
{
Tuple<string, string> lookupStrings;
foreach (var config in configurationSet)
{
lookupStrings = new Tuple<string, string>(config.BoxType, config.Size);
configurationDictionary.TryAdd(lookupStrings, config.RowNumber);
}
}
public void ScanReceived(string boxType, string size, string packerId = null)
{
}
}
}
Imagine that I have a 10 values in my configurationDictionary and many people want to query this dictionary consuming ScanReceived method, are those 10 values be shared for each of the clients that request ScanReceived? Do I need to change my ServiceBehavior?
The Configuration method is only consumed by one person by the way.
With an InstanceContextMode of PerCall you should get a new instance of your service and your dictionary every time a remote call comes in (which I'm assuming will be either Configuration() or ScanReceived()). This won't share configurationDictionary between multiple clients.
With a ConcurrencyMode of Single you'll only ever have a single thread running your code at any one time. So any issues of concurrency are moot.
Two ways to share configurationDictionary:
Make it static: private readonly static ConcurrentDictionary<...> ...
Change your InstanceContextMode, and possibly your ConcurrencyMode.
I'd recommend the first option if you're not expecting thousands of calls to your service per second.
MSDN has some information about InstanceContextMode and instancing of service classes.
If you change to InstanceContextMode.Single and ConcurrencyMode.Multiple you can get many threads executing at once and you will need to worry about synchronised access.
The main thing to watch out for is that when you're querying / iterating over your dictionary, someone else might modify it. Grabbing a snapshot of the dictionary and then querying the snapshot should get around that problem:
foreach (var x in configurationDictionary.ToArray())
{
....
}
Or, if you're using LINQ:
var someResult = configurationDictionary.ToArray().Where(...).Select(...)