My MVC based application, hooks some web services which send back lots of data!
Using the same, I render my views. The web services are slow and out of my control.
So I would like to store this info per session, but I am afraid that, this will bring down my web server to its knees. With few hundred users, the web server will run out of memory.
Is there a way that I can store this session data in a file per session? I am more looking at some out of the box open source solutions.
I welcome, new suggestion as well!
You can store pretty much any object in the Session storage, with a few exceptions which are generally related to running on a server farm. I'm going to ignore those cases here however.
If you're dealing with only a few MB of data, storing it in the Session object (or a Cache, as #Rick suggests) isn't necessarily a major problem. Once the data has been returned from the web service and parsed into your own internal data structures, simply place the data structure's root object into the Session. I use this method fairly often to store the results of database queries that take a long time to run, especially when the query criteria are unlikely to change frequently.
For larger data sets you should probably use a database to store the information. Create tables that match the structure of the data you're returning and tag the data in some way to indicate how old it is and what criteria were used when fetching it. Update as required, and query the database for records on each client request.
There are plenty of other options, including creating temporary files to store the data using the SessionID to identify them, but I recommend investigating the database option first.
Caching is your friend. And since you use MS technology you might want to take a look at the Cache Class
You could just serialize the result collection and save it on files as xml (even process it using linq/XPath directly from XML) , or use any .net native xml database to store and persist data on a file.
Related
Here I want to store my data somewhere temporarily before saving to actual DB,
I looked for some articles and suggects memorycache that stores data at server site which i want to avoid,
so is there any other way to implement this in .net core?
You can use TempData.Peek() or TempData.Keep() to store data temporarily before saving to database.
Refer this: https://www.c-sharpcorner.com/UploadFile/ansh06031982/using-tempdata-peek-and-keep-in-Asp-Net-mvc/
It is in MVC but also applicable to .NET MVC Core.
You have several choices, all with their own advantages and disadvantages.
As has been suggested, you can keep data on the client - that's in the HTML page.
You can also keep data at the client in cookies and other client caches. This is often used for authorisation data when a user logs into a web site.
There is also Session on the server, depending on how stateless you feel.
Lastly, you can store data in SQL Server in some table that acts as a temporary store until a user posts the final data to the main table. I assume here that a user logs in and you can identify them.
TempData may be a way to go for you.
Check this on how to use it.
Though I assume you have your reasons to do that. A lot better solution would be to collect all your data at the client-side and submit it in one go.
When working with ASP.NET MVC and SQL Server we are wondering if caching to XML is still something to think about or are their other possibilities for this?
Like for instance we have a table called Customers. If you call this db table everytime you click on Customers or do sorting or filtering in the app why not store this info in a xml file.
Then you work only with the xml file and not the db and you update the xml after adding changes to the customers table.
It is an absolutely brilliant idea.
If:
You only have 1 client
Or you have multiple client but they don't mind seeing old data
You have a database system that doesn't provide caching possibilities
You do not use database access frameworks that can handle caching for you
In short, no, it actually is almost never a good idea.
Databases are made to be used. Most of them can handle a much higher load than programmers think they can, as long as you treat them well. If necessary, a lot of them provide perfectly fine caching possibilities to improve performance if needed.
Any useful type of caching in your application should involving refreshing that cache when anything changes. Implementing that by yourself is usually not a good idea. If you do want a very simple cache of data that was just on the screen before the user clicked away, memory would be the place for it, not a file system. Unless you need centralised session cache, but that goes way beyond "let's write some xml".
Caching to xml file is bad choice. Database system can handle load of 100 users in 5 seconds if you have 50000 records in your table. If you want more speed than this then try using In-memory sql which stores data in RAM for fast access. But for it you need high RAM capacity on server.
The company I work for is running a C# project that crawling data from around 100 websites, saving it to the DB and running some procedures and calculations on that data.
Each one of those 100 websites is having around 10,000 events, and each event is saved to the DB.
After that, the data that was saved is being generated and aggregated to 1 big xml file, so each one of those 10,000 events that were saved, is now presented as a XML file in the DB.
This design looks like that:
1) crawling 100 websites to collects the data and save it the DB.
2) collect the data that was saved to the DB and generate XML files for each event
3) XML files are saved to the DB
The main issue for this post, is the selection of the saved XML files.
Each XML is about 1MB, and considering the fact that there are around 10,000 events, I am not sure SQL Server 2008 R2 is the right option.
I tried to use Redis, and the save is working very well (and fast!), but the query to get those XMLs works very slow (even locally, so network traffic wont be an issue).
I was wondering what are your thoughts? please take into consideration that it is a real-time system, so caching is not an option here.
Any idea will be welcomed.
Thanks.
Instead of using DB you could try a cloud-base system (Azure blobs or Amazon S3), it seems to be a perfect solution. See this post: azure blob storage effectiveness, same situation, except you have XML files instead of images. You can use a DB for storing the metadata, i.e. source and event type of the XML, the path in the cloud, but not the data itself.
You may also zip the files. I don't know the exact method, but it can surely be handled on client-side. Static data is often sent in zipped format to the client by default.
Your question is missing some details such as how long does your data need to remain in the database and such…
I’d avoid storing XML in database if you already have the raw data. Why not have an application that will query the database and generate XML reports on demand? This will save you a lot of space.
10GBs of data per day is something SQL Server 2008 R2 can handle with the right hardware and good structure optimization. You’ll need to investigate if standard edition will be enough or you’ll have to use enterprise or data center licenses.
In any case answer is yes – SQL Server is capable of handling this amount of data but I’d check other solutions as well to see if it’s possible to reduce the costs in any way.
Your basic arch doesn't seem to be at fault, its the way you've perceived the redis, basically if you design your key=>value right there is no way that the retrieval from redis could be slow.
for ex- lets say I have to store 1 mil objects in redis, and say there is an id against which I am storing my objects, this key is nothing but a guid, the save will be really quick, but when it comes to retrieval, do I know the "key" if i KNOW the key it'll be fast, but if I don't know it or I am trying to retrieve my data not on the basis of key but on the basis of some Value in my objects, then off course it'll be slow.
The point is - when it comes to retrieval you should just work against the "Key" and nothing else, so design your key like a pre-calculated value in itself; so when I need to get some data from redis/memcahce, I could make the KEY, and just do a single hit to get the data.
If you could put more details, we'll be able to help you better.
I wonder if there are any cache engines or if someone has a good solution that can handle these requirements:
It should store plain HTML fragments of a page like the standard Output cache in asp.net
The HTML may contain dynamic content from a database
When an object is updated in the database all the cached HTML fragment containing that particular object should be destroyed and re-cached next time it will be requested.
There is a separate admin tool to handle all data in the database so I can easy store the Id’s in a cachetable when an object is invalid. I can also make a request to a page that destroy all cached HTML fragment for that object.
But when I write the markup, how could I do to store and retrieve a particular segment from the cache? Of cause I could do this in code behind and have the markup in a string but I don’t want that. I want to have the markup as intact as possible.
I'm assuming this is all within an ASP .Net page. I think your situation is simple enough to where you can write this caching mechanism yourself:
Make your SQL query as usual
Use the results to output your aspx page
Store the results in a static variable or static "Cache" class, this should persist from request to request
Future requests would use your stored results, unless they are invalidated as you mention they would be. (In my experience, a 1 minute time expiration can work well).
If you want to get more complicated, and preserve the cache even if IIS is restarted: You could easily use an XmlSerializer and write your results to a file instead of using static variables.
You have two problems to consider:
Number 1: Cache invalidation. This can be solved using cache dependencies (see http://msdn.microsoft.com/en-us/library/system.web.caching.cachedependency.aspx ) which will allow things to be invalidated and recached as data changes (automatically!).
Number 2: Cache storage. For this, just use the standard ASP.Net cache API using HttpContext.Current.Cache (or a suitable abstraction). That supports the dependency system as mentioned above.
The ASP.Net cache implementation is pluggable so you can change the storage mechanism between in memory, files, SQL databases, Memcached (via Enyim) or Microsoft Velocity for example. Each cache store has different reliability that you can pick and choose based on your requirements.
We have an application (rules engine) that has a lot of tables in memory to perform certain business rules. This engine is also used for writing back to the database when needed.
The DB structure is denormalized, and we have 5 transactional tables, that also sometimes need to be queried for reporting.
The issue here is, we want to cache the data inside the app, so it loads on App startup, and then only changes if the DB changed.
Any recommendations?
We are leaning towards creating a DB service, that will handle all Inserts, Updates and Deletes, and queue them to decrease load on the DB server (the transactional tables have loads of indexes also). Also, we are thinking of enabling the DB service to sit on top and serve all reports / other apps that need direct DB access.
The aim here ofcourse is to decrease DB hits for Select queries per request, and prioritize transactions. Also to ensure people accessing apps dont bring the DB server down.
Rules Engine is a C# desktop app, reporting and other apps are web based.
What would be the best way to go about this? I did also think of removing all indexes from my transactional table, and having a trigger insert into a new table which would be a copy, but indexed for report retrieval.
You should perhaps look at distributed caching solutions (from both performance and scalability point of view). In short, I am taking about scalable DB Services backed by distributed cache (so that multiple DB services get served by same cache).
Here's the article that discusses distributed caching including various approaches for database synchronization. And here is the blog post that list few options in .NET for distributed caching.
I've done something similar with an obscenely complex rules engine. Ultimately, I set it up so that the data was serialized centrally (with a process to release new changes, causing a new copy to be serialized and the blob stored somewhere accessible). During load, each app-server would check whether they have the up to date version of the blob, and if not fetch it (and store it locally).
Then all it has to do is deserialize the data into memory. No db hit, except for occasionally grabbing the new blob. It also means the app-server can work while the db server is offline (as long as it has a cached copy of the blob). It also polled periodically for new updates while running, of course - but only to the "is there a new blob" code (it still didn't need to hit the main tables).
You may be interested in this article It uses xml to store a readonly copy of the database (in memory). And XPath to query. Nowadays you'd prefer to query with linq, of course.