I'm currently working in a chat API, and I receive multiple requests at the same time, from different sessions, so its almost impossible to track each
conversation separately, because it mixes with all the others logs from other conversations.
So I want to create a separated file for each session(conversation) dynamically, with the filename as the sessionId, but if I create multiple loggers, my application just freeze, because I can have more than 100 sessions simultaneously.
I have also tried to change the file path (programmatically) for each request with its id on it, but it also freezes the application after 1-2 hours.
Is there any solution for this problem?
If these conversation files are so important, consider other options than logging. A database might be appropriate.
Another solution might be to parse the log files and split them into conversation files in a separate (logical?) process (perhaps later, after the session has ended.) This way the program doesn't need to keep track of many files at the same time and parsing can be done faster/more efficiently.
Related
I am using NLog as my logging framework. I envisage I will have logs coming in from multiple sources (20, 30+)
I want to be able to live monitor at will.
What Viewers (commercial or free) are the best to use?
I am currently rolling over my days and using C:\Logging as my "base" logging directory.
NLog FileName for trace is as follows:
C:\Logging\${appdomain:format={1\}}\${shortdate}\MyType.xml
I have Trace/Debug/Info,Warn/Error/Fatal all going into their own separate files (Debug.xml/Info.xml/Error.xml etc), all in the above file name format.
I also have a UDP target setup, and that is currently going to Sentinel. This works fine, and would be a great solution for me if sentinel could setup multiple apps/tabs/receiveds. But on the surface, I can only have one it would seem. The other problem is that I have millions of logs pumping through. Last time i left it running for a while, it killed all the memory in my system.
Ideally, What i would like, is an application that i simply add the "C:\Logging" folder to, like a "watch folder" and it keeps pumping out my logs, including detecting when a new file is created (example Fatal.xml), which would also handle date rollovers. Also the addition of multiple receiver types eg UDP
Not sure if Amazon is an option for you, but I ran across an AWS NLog Target that I was looking implement. I am not capturing as many logs as you, but do have logs coming from multiple servers. This would send the items written to the logs to Amazon CloudWatch Logs target and searchable in the console.
I am not sure on the bandwidth required to duplicate log items to AWS but it would put them in one place. CloudWatch Retention has been increased but if you do find issues, you could always go back to the text log files for more details past the retention period.
You could also setup CloudWatch Alarms to let you know if there are issues.
I'm currently working on a C# project of an application we'd like to develop. We're brainstorming over the question of sharing the data between users. We'd like to be able to specify a folder where all the files of the application are going to be saved and we'd like to be able to save them on a shared folder (server, different PC or Mac, Nas, etc.).
The deployment would be like so :
Installation on the first PC, we choose a network drive, share, whatever and create all the files for the application in this location.
On the second PC we install the application and we choose the same location (on the network), the application doesn't create anything, it sees that it's already existing and it uses these files as the application's data
Same thing on the other clients
The application's files are going to be documents (most likely XML formatted documents) and when opening the application we want to show all the existing documents. The thing is, we don't only want to have the list of documents and be able to edit their content, we also would like to be able to edit the document's property, so in a way we'd like a file (Sqlite, XML, whatever) representing the list of all the documents and their attributes. Same thing for a list of addresses.
I know all that looks exactly like a client / server with database solution, but this solution is out of the question. I was first looking at SQLite for my data files, but I know concurrency can be a real problem and file lock doesn't work well. The thing is, I would have the same problem with simple XML files (refreshing the content when several users are working, accessing locked files).
So I guess my final question is : Is it feasable? Is there an alternative I didn't see which would allow us to do that more easily?
EDIT :
OK I'm not responding to every post or comment, because I'm currently testing concurrency with SQLite. What I did, and please correct me if the way I test this is wrong, is launch X BackgroundWorker which are all going to insert record in a sample database (which is recreated everytime I start the application). I tried launching 100 iterations of INSERT in the database via these backgroundWorkers.
Of course concurrency is working with one application running, it's simply waiting for the last BackgroundWorker to do it's job and then writing the next record. I also tried inserting at (almost) the same time, meaning I put a loop in every BackgroundWorker waiting for a modulo 5 timestamp (every 5 seconds, every BackgroundWorker runs). Again, it's waiting for the previous insert query to end before doing the next and everything's working fine. I even tried it with 500 BackgroundWorkers and it worked fine.
I then tried launching my app several times and running them simultaneously. When doing this I did have some issue. With two instances of my app it was still working fine, but when trying this with 4-5 instances, it got really buggy and I got two types of error : 1. database is locked 2. disk I/O failure. But mostyle locked databases.
What I did was pretty intensive, in the scenario of my application, it will never ever come to 5 processes trying to simultaneously insert 500 hunded rows at the same time (maybe I'll get a concurrency of two or three connections). But what really bugged me and what makes me think my testing method is not really a good one, is that I got these errors trying to work on a database on a shared network, on a NAS AND on my own HDD. Everytime it worked for maybe 30-40 queries then throwing me "database is locked" error.
Am I testing it wrong? Maybe I shouldn't be trying so hard to make this work, but I'm still not convinced that SQLite is not a good alternative to what I'm trying to do, since the concurrency is going to be really small.
With your optimistic/pessimistic locking, you are ultimately trying to build a database. Also, you WILL have issues with consistency while trying to keep multiple files in sync with each other. Think about if you update the "metadata" file, and the write fails half-way through because of a network blip. File corruption will ensue, and you will be left trying to reconstruct things from backups.
I would suggest a couple of likely solutions:
1) Host the content yourselves, and let them be pure clients (cloud based deployments are ideal for this). Most network/firewall issues can be circumvented by using HTTP as your transport (web services).
2) Have one of the workstations be the "server", which keeps it data files on the NFS. This will give you transactional integrity, incremental backups, etc. There are lots of good embedded database managements systems to help you manage this complexity. MS SQL Server even has some great options for this.
You right, Sqlite uses file locks on database file, so storing all data files in database would bring write-starvation problem for editing your documents.
May be it's better choice to implement simple optimistic/pessimistic locking by yourself on particular-file level? For example, in case of using pessimistic lock you just don't allow anyone to edit particular file, if somebody already in process of editing it. In this case you will hold lock just on one file, but not on the entire database. If possibility of conflict(editing particular file at the same time) is pretty low, it is better to go with optimistic locking.
Simple optimistic locking implementation:
When user get file for reading - it's OK, no problem here. If user get file for editing, you could calculate hash for this file(or get timestamp of last updated time of the file), and then, when user tries to save edited file, compare current(at the moment of saving) hash/timestamp to make sure that file has not been changed by somebody else. If file has not been changed then it's ok to save it. IF file has been changed, then current user is out of luck, you need to inform him about it. This optimistic scenario is nice when possibility of this "out of luck" is pretty low. Otherwise it's better to stick with pessimistic locking, when you do not allow user even to start file editing if somebody else is doing it.
Windows Service - C# - VS2010
I have multiple instances of a FileWatcher Service. Each one looks for a different extension in the directory. I have a separate Router service that monitors the directory for zip files and renames the extensions to one of the values that the services look at.
Example:
Directory in question (all FileWatcher Services monitor this directory) contains the following files:
a.zip, b.zip, c.zip
FileWatcher1 looks for extensions of *.000, FileWatcher2 looks for extensions of *.001, FileWatcher3 looks for extensions of *.002
The Router will see the .zip files and change the file extensions on the zip files, but it should keep in sequence in order to delegate the same amount of work to each FileWatcher.
Also, if there are two zip files dropped, it would change a.zip -> a.000, and b.zip -> b.001, but if 5 minutes go by and another batch of zip files are dropped, it should know to rename the next file to *.002.
I have everything working fine, but now I need to implement the sequential part to the Router and am not sure the best way of implementation (currently router is changing every extension to *.000 thus only one FileWatcher is getting the work). I know this might be considered a cheap way of doing this but it's all we really need at the moment. Any help would be appreciated.
Maybe a different way of looking at it. Have you thought about having a single watcher and then using a thread pool? The reason why I am suggesting this is that you will have to start looking at the sizes and complexities of the fields to adequately distribute the work. You might start pushing more work to .000 because it's next in line when it is still busy processing a large amount of data from the first job whereas .001 could be free as it was processing a small file.
If you really want to get around the problem of the next extension in line, why not just keep a static variable with the next extension number. I am not 100% sure if the Router Filewatcher will run multiple threads when it sees new files one after the other but I don't think so. If that does happen then you will need to put some thread safety code when accessing the static variable.
Can the Router just keep a counter and do a mod 3 (or N, where N is the number of watchers) operation for every new file?
I've got an application receiving messages from different sources (chat rooms and private chats). Multiple instances of the application can be opened, and the final result should be something similar to the following scenario:
Currently, each application saves logs in a directory which name is the account used to log on the chat server; while it's not a problem for private chat sources (unique for each application instance), it's useless to have the same logs saved multiple times, concerning common chat rooms. Logs are saved in plain text format, so they can be accessed and read without being routed through the application.
If I don't save the logs in separate folders, I might get I/O exceptions due to accessing the same file simultaneously from multiple processes, and I'd need to verify whether the line about to be saved hasn't already been written by other applications. I need to optimize the whole operation and try to maintain code readability.
Besides, my current approach for writing lines is the following:
public void Write(string message)
{
using (var writer = new StreamWriter(_fileName, File.Exists(_fileName)))
writer.WriteLine(message);
}
Which, considering that logs are constantly written, might not be the most efficient solution.
Summed up, my questions are:
How do I create an unique log folder/database, maintaining their format (plain text) but solving the aforementioned duplicates/access problem?
How do I improve, if possible, the writing method? Remember that logs need to be constantly written, but that closing the StreamWriter when the application exits would not be a proper solution, as the application is meant to run for a long time.
Thank you.
I would come up with a simple solution, which might be appropriate for your needs, not entirely sure though.
My approach would be to use a single file for each chat session/room. If such a session is started, the application tries to create/open that file and creates a write lock for that file. If it gets an IOException (because the file is locked) it can simply skip logging completely.
To be honest, if I were you I would be looking at already exising open source frameworks e.g. NLog. It's fast enough and supports asynchronous logging so it should do exactly what your looking for.
Not sure, if I should write this as an answer or a comment, but might need the room:
You mentioned your sketch showing the desired result, but as I said this will prevent you from deduping if you don't couple the instances. So here is what I would suggest:
You create two applications: LogWriter, which is a singleton and sits at the bottom of your sketch
LogProcessor, which is the application instances in your sketch.
Upon startup of a LogProcessor instance, it spawns a LogWriter or connects to it, if it is already running.
LogProcessor handles the incoming log requests, maybe preprocesses them if you need to do so, then sends them on to the LogWriter as a tuple of Timestamp, ChatroomID, UserID (need not be unique), text, and maybe a hash for easier deduping. Calculating the hash in the instances makes better use of multiple cores
The LogWriter keeps a hanging list sorted by timestamp, containing the hashes, so it is able to quickly discard the duplicate items
For the rest of the items, LogWriter determines the logfile path. If a stream is already open to that path, it writes the item out, updates the LastUsed Timestamp on that stream and is done
If no stream is open, LogWriter opens one, then writes.
If the max number of streams is reached, it closes the oldest stream (as by the above mentioned LastUsed Timestamp) and opens the needed new stream instead.
Perhaps change your application design in such a way that a logger is attached to a chat room not to a user.
When users enter a chat room, the chatroom will pass a logger object to the users.
This way all users will use the same logger. The problem then becomes: 1 consumer (the logger), and multiple producers (all those users who want to log).
See my reply to this post:
Write to FileOutputStream from multiple threads in Java
here
https://stackoverflow.com/a/8422621/1007845
I have several processes running concurrently that I want to log to the same file.
We have been using Enterprise Library 4.1 Logging Application Block (with a RollingFlatFileTraceListener), and it works fine, apart from the fact that it prepends a GUID to the log file name when two processes try to write to the log file at the same time (a quirk of System.Diagnostics.TextWriterTraceListener I believe).
I've tried various things, including calling Logger.Writer.Dispose() after writing to the log file, but it's not ideal to do a blocking call each time a log entry is being written.
The EntLib forums suggest using MSMQ with a Distributor Service, but that is not an option as MSMQ is not allowed at my company.
Is there another way I can quickly and easily log from multiple threads/processes to the same file?
Sorry to say but the answer is no. The File TraceListeners lock the output file so only one TraceListener can log to a file.
You can try other Trace Listeners that are not file based (e.g. Database, Event Log).
Another option I can think of would be to write your own logging service (out of process) that would log to the file and accepts LogEntries. Then create a custom trace listener that sends a message to your service.
It might not be a good idea since you would have a bit of custom development plus it could impact performance since it is an out of process call. Basically you are setting up your own simplified-pseudo-distributor-service.
EntLib locks the log file when it writes to it. Therefore, 2 processes cannot write to the same log file.
When we have had this problem, that we needed to log from many difference places, to the same place, we have used database logging.
If you are 100% stuck logging to a text file, then you could log to individual log files, and then write a program to merge these files.
I know this is old, but if you are still curious. log4net supports this:
http://logging.apache.org/log4net/release/faq.html#How do I get multiple process to log to the same file?
The problem occurs when the App Pool Recycles and allows for Overlapping Threads. The closing thread has it still open, and the new thread gets the error. Try disabling the overlapping recycling behavior in IIS, or create your own version of the text writer.