C# Where to cache big data tables [closed] - c#

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
I have to execute a long running PL/SQL query and need to store the result set somewhere and serve to UI request. After some time need to refresh it automatically / manually.
there will be multiple result sets need to be handled in the same way. Each and every result set will have millions of records.
My project is using AngularJS with Web API.
I am using ADO.net with Oracle Client, not entity framework.
What I feel, .net MemoryCache is not suitable because the no.of resultset and size will be keep on growing.
I am planning to cache it in MongoDb. is there any other solution you suggest?
Thanks

Your question says "Each and every result set will have millions of records"
In that context,
Caching in server is not a good idea. Even distributed caching may result in performance issue.
Create dedicated physical tables in oracle to store your results
Load & refresh the result table whenever required
Fetch and return the result from results table
If you can't insert/delete records from your oracle database you may need to go with a dedicated application database.
Even if you return millions or records from web API, the angular application should process all your data. That might result in long running JavaScript and client side performance.

Redis is a good solution. Check out http://redis.io/

Related

how to auto refresh div value when add/edit value in Sql table? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I'm using MVC 5 .net with Entity Framework.
I want to design client dashboard with list out latest data from sql database.
so how to call ajax method or refresh particular table/div value whenever any value add/edit from other source in database.
I want to call function/ajax method only when any value add/edit in database from any other source without unnecessary auto refresh using setInterval function.
please help, thanks in advance.
There are several ways of accomplishing this.
Easiest is to simply poll the data at set intervals from the server. And when it changes update. To the user this will look like changes are getting pushed out. This will also not require you to do anything special besides calling what you are already doing except in a setInteraval
You could implement websockets. This enables you to literally push data from the server to a javascript function. The support for this is pretty widespread, and there are many great tutorials for this.
Implement a framework like SignalR. This is your most complete and most bakward compatible solution. If there is support for websockets it will be used otherwise it will fall back to simple ajax calls.
Personally Id opt for option 1. unless you have some major traffic or the queries are super heavy to compute on the server. You could even implement a second API that you can call to check for updates very frequently, the second api might return an integer, if that integer is higher than last time we know there is an update and we should make the big heavy query. If this dosent work go for signalR or something similar. https://www.reddit.com/r/dotnet/comments/68rgxq/signalr_alternatives/

Performance evaluation of Web API calls with Database LINQ queries [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have a UI which calls WebAPIs (WebAPI 2.0), Web API are basically LINQ queries (to MS SQL database) and some processing logic for the data. I want to do performance evaluation of the complete flow (click from UI to API, back to UI to display data) upon a huge DB with 30K - 60K records in it.
How can it be done? Let me know the methods/tools used for this.
currently I am tracking time-taken in chrome debug window, which shows the total time for each network call.
Wow. This is a subject in its own right but here's an approach:
The bits are independent so you break it down. You measure your LINQ queries without any of the logic or web api stuff getting in the way. If LINQ is against stored procedures then measure those first. Then you measure the cost of the logic, then you measure the cost of sending X rows of data using WebAPI. You should avoid including the cost of actually retrieving the rows from the database so you're checking just the connectivity. I'd also consider writing a browserless test client (i.e. GETS/POSTS or whatever) to eliminate the browser as a variable.
Now you've got a fairly good picture of where the time gets spent. You know if you've got DB issues, query issues, network issues or application server issues.
Assuming it all goes well, now add a bunch of instances to your test harness so you're testing concurrent access, load testing and the like. Often if you get something wrong you can't surface that with a single user so this is important.
Break it down into chunks and have a data set that you can consistently bring back to a known state.
As for tools, it really depends on what you use. VS comes with a bunch of useful things but there are tons of third party ones too. If you have a dedicated test team this might be part of their setup. SQL Server has a huge chunk of monitoring capability. Ask your DBAs. If you've got to find your own way, just keep in mind that you want to be able to do this by pressing a button, not by setting up a complex environment.

How to store common data inside the application [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
We have configuration data stored within the tables like casestatustypes with columns such as statusid, name, description. How to store these kind of data in the web application, instead of hitting the database each time for retrieving the statusid.
You use a technique known as caching. Basically, you build an in-memory copy of the data that you use for retrieval purposes. When you start the application, you pull from the database to create this cache. When you do an insert, update, or delete; you do it to both the cache and database.
Its easy enough to implement yourself, and there are several good libraries out there (Microsoft even has one in Enterprise Library http://msdn.microsoft.com/library/cc467894.aspx).
Gotchas:
If the data set is large, you'll want to implement a caching strategy that doesn't hold the entire dataset in memory (libraries are useful for this).
Since its a web-app, you need to make sure the cache isn't going to be re-created for each session.

Does frequent INSERT/UPDATE query compromise the database safety? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
Hi, I am relatively new to using databases in a web application and I am trying to develop a dynamic application for my users.
So, I was wondering how safe is it to have your application frequently (say every 2 seconds) execute an INSERT/UPDATE query from the same user.
I'm aware that INSERT/UPDATE queries are quite slow to execute (I've read its 12 INSERT per second) but that's not my question.My question is concerning the safety of my database.
The frequency of executing INSERTs or UPDATEs to your database is in no way related to the safety of of your database. But it might impact its performance, that's possible.
I think the concern of the OP is more about the robustness of the database's ability to handle load and not become corrupt, rather than issues of SQL injection. (Valuable comments to take note of though). The volumes suggested per user should no concern.
Database integrity for any DB should be checked with regular maintenance. Make sure you are doing the following and you should have no problems with performance and reliability.
Back up your DB. Full backup and transaction log backups.
Re-build and/or Re-Organize indexes
Delete old data backups and check free space often.
Check the maintenance logs for issues on your DB.
Then monitor performance for sufficient Memory, DISK I/O, and CPU.
wihout "with(nolock)" segment after the table name, the table being insert/update is always locked.
if so, how can it performance well

make application responsive and fast [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I created a wpf browser application ,and I hosted it in a live server ,it is a Inventory Management application ,which uses database heavily for saving, updating and selecting data.On save and select operation it takes very long time to save and fetch data to and from database.In the meantime (when save or select data from database) the application is irresponsive .How to make the application fast when dealing with database and how to make the application responsive during that operation .I am totally messed up ,its very kind if anyone has the answer .
Is using stored procedures instead of raw sql bring performance or not?
Edit
My application interact with database by using raw sql statements.
Responsive Application
In order to make your application responsive you need to make every interaction with the database in a new thread or task. You could also use the new C# 5.0 features await and async.
Your current application is blocking because the main thread is handling database operations as well as user interface, so what you need to do is to handle these two separate functionalities in separate threads.
Fast Database Access
I don't know how are you dealing with the database right now, but you could definitely use some kind of ORM like Entity Framework or NHibernate.
For better performance, but much less readable and mantainable code, which is really important, you could use raw sql.
try this
to access DB faster in .NEt i am sure it not problem with c# but for querying and retireving DB u need to have better mechanism

Categories