I created a static ip and log class.
the ip class find out the users ip address and log class log it into a text file.
every thing work just fine but i wonder what happens if so many requests came at a same time?
i mean the both classes are static and base on static classes it causes problem.
how can i managed them?
here is my ip class:
public static class IP
{
public static string IP()
{
System.Web.HttpContext context = System.Web.HttpContext.Current;
string ipAddress = context.Request.ServerVariables["HTTP_X_FORWARDED_FOR"];
if (!string.IsNullOrEmpty(ipAddress))
{
string[] addresses = ipAddress.Split(',');
if (addresses.Length != 0)
{
return addresses[0];
}
}
return context.Request.ServerVariables["REMOTE_ADDR"];
}
}
}
and here part of my log class which write into the text file:
private static void WriteLine(string message)
{
string filePath = FilePath();
CreateFile(filePath);
try
{
using (StreamWriter log = File.AppendText(filePath))
log.WriteLine(message);
}
catch (Exception)
{
//If can not access to file do nothing
//throw;
}
}
You aren't going to run into contention problems due to your classes being static. Your IP.IP() method class is pure (i.e. it does not change the state of anything) and contains no locks, so there is no chance of there being any contention there.
You do potentially have problems in WriteLine due to the fact that you are probably writing your log file on the same thread as you are doing your work. That means the file write is acting as a lock since only one write can occur at any one time.
What you want is to log to a queue and then to write that queue on a separate thread; that is a classic producer-consumer pattern.
Alternatively you could avoid reinventing the wheel and use an existing logging framework that will handle these things for you like log4net
Streamwriter has a default of 4kb buffer which can be modified if needed as defined by:
public StreamWriter(
Stream stream,
Encoding encoding,
int bufferSize
)
More than likely, your computer (including disk access) is most likely a lot faster than your internet access.
It will work fine, because you don't have any public variable which will be kept in memory and changing on every time class is accessed.
So as the method ends, the scope of your variables will be finished.
However if they are in memory, they will not be effected by how many users use it at the same time, and there will be no mess.
Related
For context, I am creating a networking library and have internally implemented object pooling - instances of byte[] get reused in order to avoid creating constant garbage.
For users of my library, I have exposed a public ref struct Packet which allows the user to perform write operations to the pooled byte[]. Here is the code:
public ref struct Packet
{
private readonly byte[] _buffer;
public Packet()
{
// Here we take a byte-array from the pool.
_buffer = Pool.Get();
}
// Some write methods here...
public void Return()
{
// Here byte-array returns to the pool so it can be reused later.
Pool.Return(_buffer);
}
}
While all of this is great and it works, there are some problems that I have with this approach:
I am leaking implementation details: user shouldn't really know or care how packets get their byte[], they only want to get the packet, write some data to it and send it.
I am forcing my users to call Return method when they are done using the packet.
If user forgets to call Return, byte[] will not get returned and leak occurs.
To somewhat remedy said problems, here is what I tried. Whenever user sends the packet, that send method will call Return method for the user.
public class Client
{
public void Send(Packet packet)
{
// Performs socket operations here...
// Returns byte-array to the pool so users don't have to.
packet.Return();
}
}
While this works most of the time, what if user creates a packet, but does not send it? Again, leak will occur. There seems to be no way to actually hide that library is internally using object pooling and there must exist public Return method so user can manually return byte[] in case they don't actually send the packet.
So, what I want to achieve is:
User doesn't even have access to Return method, it is completely hidden from the user.
byte[] instances get reused (pooled) and are always returned to the pool, even if user does not send the packet or does anything with it.
Any suggestions?
I would consider implementing IDisposable. Nevertheless, it could be beneficial to only get from the Pool in case a Send request is invoked:
public void Send(Action<Packet> modifyPacket)
{
// Performs socket operations here...
var packet = new Packet();
try
{
modifyPacket(packet);
// send...
}
finally
{
packet.Return();
}
}
Thus the payload would be out of the Pool for the shortest amount of time. As I do not know the methods on Packet I do not know in how far you want to hide it. But I hope you get the idea.
Context:
I am implementing a logging mechanism for a Web API project that writes serialized objects to a file from multiple methods which in turn is read by an external process (nxLog to be more accurate). The application is hosted on IIS and uses 18 worker processes. The App pool is recycled once a day. The expected load on the services that will incorporate the logging methods is 10,000 req/s. In short this is a classic produces/consumer problem with multiple producers (the methods that produce logs) and one consumer (the external process who reads from the log files). Update: Each process uses multiple threads as well.
I used BlockingCollection to store data (and solve the race condition) and a long running task that writes the data from the collection to the disk.
To write to the disk I am using a StreamWriter and a FileStream.
Because the write frequency is almost constant ( as I said 10,000 write per second) I decided to keep the streams open for the entire lifetime of the application pool and periodically write logs to the disk. I rely on the App Pool recycle and my DI framework to dispose my logger daily. Also note that this class will be singleton, because I didn't want to have more than one thread dedicated to writing from my thread pool.
Apparently the FileStream object will not write to the disk until it is disposed. Now I don't want the FileStream to wait for an entire day until it writes to the disk. The memory it will require to hold all that serialized object will be tremendous, not to mention that any crash on the application or the server will cause data loss or corrupted file.
Now my question:
How can I have the underlying streams (FileStream and StreamWriter) write to the disk periodically without disposing them? My initial assumption was that it will write to the disk once FileSteam exceeds its buffer size which is 4K by default.
UPDATE: The inconsistencies mentioned in the answer have been fixed.
Code:
public class EventLogger: IDisposable, ILogger
{
private readonly BlockingCollection<List<string>> _queue;
private readonly Task _consumerTask;
private FileStream _fs;
private StreamWriter _sw;
public EventLogger()
{
OpenFile();
_queue = new BlockingCollection<List<string>>(50);
_consumerTask = Task.Factory.StartNew(Write, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
}
private void OpenFile()
{
_fs?.Dispose();
_sw?.Dispose();
_logFilePath = $"D:\Log\log{DateTime.Now.ToString(yyyyMMdd)}{System.Diagnostic.Process.GetCurrentProcess().Id}.txt";
_fs = new FileStream(_logFilePath, FileMode.Append, FileAccess.Write, FileShare.ReadWrite);
_sw = new StreamWriter(_fs);
}
public void Dispose()
{
_queue?.CompleteAdding();
_consumerTask?.Wait();
_sw?.Dispose();
_fs?.Dispose();
_queue?.Dispose();
}
public void Log(List<string> list)
{
try
{
_queue.TryAdd(list, 100);
}
catch (Exception e)
{
LogError(LogLevel.Error, e);
}
}
private void Write()
{
foreach (List<string> items in _queue.GetConsumingEnumerable())
{
items.ForEach(item =>
{
_sw?.WriteLine(item);
});
}
}
}
There are a few "inconsistencies" with your question.
The application is hosted on IIS and uses 18 worker processes
.
_logFilePath = $"D:\Log\log{DateTime.Now.ToString(yyyyMMdd)}{System.Diagnostic.Process.GetCurrentProcess().Id}.txt";
.
writes serialized objects to a file from multiple methods
Putting all of this together, you seem to have a single threaded situation as opposed to a multi-threaded one. And since there is a separate log per process, there is no contention problem or need for synchronization. What I mean to say is, I don't see why the BlockingCollection is needed at all. It's possible that you forgot to mention that there are multiple threads within your web process. I will make that assumption here.
Another problems is that your code does not compile
class name is Logger but the EventLogger function looks like a constructor.
some more incorrect syntax with string, etc
Putting all that aside, if you really have a contention situation and want to write to the same log via multiple threads or processes, your class seems to have most of what you need. I have modified your class to do some more things. Chief to note are the below items
Fixed all the syntax errors making assumptions
Added a timer, which will call the flush periodically. This will need a lock object so as to not interrupt the write operation
Used an explicit buffer size in the StreamWriter constructor. You should heuristically determine what size works best for you. Also, you should disable AutoFlush from StreamWriter so you can have your writes hit the buffer instead of the file, providing better performance.
Below is the code with the changes
public class EventLogger : IDisposable, ILogger {
private readonly BlockingCollection<List<string>> _queue;
private readonly Task _consumerTask;
private FileStream _fs;
private StreamWriter _sw;
private System.Timers.Timer _timer;
private object streamLock = new object();
private const int MAX_BUFFER = 16 * 1024; // 16K
private const int FLUSH_INTERVAL = 10 * 1000; // 10 seconds
public EventLogger() {
OpenFile();
_queue = new BlockingCollection<List<string>>(50);
_consumerTask = Task.Factory.StartNew(Write, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
}
void SetupFlushTimer() {
_timer = new System.Timers.Timer(FLUSH_INTERVAL);
_timer.AutoReset = true;
_timer.Elapsed += TimedFlush;
}
void TimedFlush(Object source, System.Timers.ElapsedEventArgs e) {
_sw?.Flush();
}
private void OpenFile() {
_fs?.Dispose();
_sw?.Dispose();
var _logFilePath = $"D:\\Log\\log{DateTime.Now.ToString("yyyyMMdd")}{System.Diagnostics.Process.GetCurrentProcess().Id}.txt";
_fs = new FileStream(_logFilePath, FileMode.Append, FileAccess.Write, FileShare.ReadWrite);
_sw = new StreamWriter(_fs, Encoding.Default, MAX_BUFFER); // TODO: use the correct encoding here
_sw.AutoFlush = false;
}
public void Dispose() {
_timer.Elapsed -= TimedFlush;
_timer.Dispose();
_queue?.CompleteAdding();
_consumerTask?.Wait();
_sw?.Dispose();
_fs?.Dispose();
_queue?.Dispose();
}
public void Log(List<string> list) {
try {
_queue.TryAdd(list, 100);
} catch (Exception e) {
LogError(LogLevel.Error, e);
}
}
private void Write() {
foreach (List<string> items in _queue.GetConsumingEnumerable()) {
lock (streamLock) {
items.ForEach(item => {
_sw?.WriteLine(item);
});
}
}
}
}
EDIT:
There are 4 factors controlling the performance of this mechanism, and it is important to understand their relationship. Below example will hopefully make it clear
Let's say
average size of List<string> is 50 Bytes
Calls/sec is 10,000
MAX_BUFFER is 1024 * 1024 Bytes (1 Meg)
You are producing 500,000 Bytes of data per second, so a 1 Meg buffer can hold only 2 seconds worth of data. i.e. Even if FLUSH_INTERVAL is set to 10 seconds the buffer will AutoFlush every 2 seconds (on an average) when it runs out of buffer space.
Also remember that increasing the MAX_BUFFER blindly will not help, since the actual flush operation will take longer due to the bigger buffer size.
The main thing to understand is that when there is a difference in incoming data rates (to your EventLog class) and outgoing data rates (to the disk), you will either need an infinite sized buffer (assuming continuously running process) or you will have to slow down your avg. incoming rate to match avg. outgoing rate
Maybe my answer won't address your concrete concern, but I believe that your scenario could be a good use case for memory-mapped files.
Persisted files are memory-mapped files that are associated with a
source file on a disk. When the last process has finished working with
the file, the data is saved to the source file on the disk. These
memory-mapped files are suitable for working with extremely large
source files.
This could be very interesting because you'll be able to do logging from different processes (i.e. IIS worker processes) without locking issues. See MemoryMappedFile.OpenExisting method.
Also, you can log to a non-persistent shared memory-mapped file and, using a task scheduler or a Windows service, you can take pending logs to their final destination using a persistable memory-mapped file.
I see a lot of potential on using this approach because of your multi/inter-process scenario.
Approach #2
If you don't want to re-invent the wheel, I would go for a reliable message queue like MSMQ (very basic, but still useful in your scenario) or RabbitMQ. Enqueue logs in persistent queues, and a background process may consume these log queues to write logs to the file system.
This way, you can create log files once, twice a day, or whenever you want, and you're not tied to the file system when logging actions within your system.
Use the FileStream.Flush() method - you might do this after each call to .Write. It will clear buffers for the stream and causes any buffered data to be written to the file.
https://msdn.microsoft.com/en-us/library/2bw4h516(v=vs.110).aspx
I have a windows service to get data from remote Database to update local. For each get operation i am writing a log file with information (errors, success) for review.
However the data is so huge that sometimes, Service is using up RAM on the server and hence crashing it :(. I understand there is a memory leak, I am unable to spot it though. However, in my initial analysis I figured out that the memory leak might be in how I am writing to the log. Here is a short explanation of what i am doing.
I have a Utility "STATIC" class with a method to write entries to log.
private static readonly object syncObj = new object(); //Class scope property
private static void Log()
{
lock (syncObj)
{
using (var sw = new StreamWriter("c:/test.txt", true))
{
sw.WriteLine("message");
}
}
}
This method is called number of times to log file entries. I am unable to understand where the memory is leaking and how to fill the leak.
Is it the Static class with static method in it (where i cannot use the IDisposable) [this i am not sure as i am already writing the using clause to limit the scope which internally does that anyways.]
Is it the streamwriter which i am not sure, should i replace it with File.AppendAllText(path, message)?
Is it the lock which i am using inefficiently?
Any help is truly appreciated.
I am developing in C# two simple applications, running in the same local machine without network requirements.
The first application initializes an DLL (Class1) and set a variable. The second application just read it the data which was previously stored. Both applications instanciates the same Class1.
Code:
DLL (Class1):
public class Class1
{
private string variableName;
public string MyProperty
{
get { return variableName; }
set { variableName = value; }
}
}
Application A:
class Program
{
static void Main(string[] args)
{
Class1 class1 = new Class1();
string localReadVariable = Console.ReadLine();
class1.MyProperty = localReadVariable;
}
}
Application B:
class Program
{
static void Main(string[] args)
{
ClassLibraryA.Class1 localClass = new ClassLibraryA.Class1();
string z = localClass.MyProperty;
Console.WriteLine(z);
}
}
My problem is that I do not know how to read a variable from another thread.
Application B must read the "variableName" set by application B
Thank you
You need some sort of mechanism to communicate between the applications.
This can be through the registry, files, memory mapped files etc...
If both applications are expected to do write, you need to add synchronization logic to your code.
There is no simple way for Application B to read data created in Application A. Each application has its own address space and thus do not know of the others existence.
But, there are ways to do this!
See this question for one method..
I've successfully used two methods:
Use a database table to contain your common data. If you wrap your
calls to it in transactions then you also protection from
concurrency issues.
Use PersistentDictionary to store your data, protected by a mutex. You must have some interprocess locking since PersistentDictionary can only be open by one process at a time.
You can use .net Remoting to communicate between your two application.
Remoting also does not require a network address to communicate.
I'm taking over a C# project, and when testing it out I'm getting errors. The error is that the log file cannot be written to because it is in use by another process. Here's the code:
public void WriteToLog(string msg)
{
if (!_LogExists)
{
this.VerifyOrCreateLogFile(); // Creates log file if it does not already exist.
}
// do the actual writing on its own thread so execution control can immediately return to the calling routine.
Thread t = new Thread(new ParameterizedThreadStart(WriteToLog));
t.Start((object)msg);
}
private void WriteToLog(object msg)
{
lock (_LogLock)
{
string message = msg as string;
using (StreamWriter sw = File.AppendText(LogFile))
{
sw.Write(message);
sw.Close();
}
}
}
_LogLock is defined as a class variable:
private object _LogLock = 0;
Based on my research and the fact that this has been working fine in a production system for a few years now, I don't know what the problem could be. The lock should prevent another thread from attempting to write to the log file.
The changes I've made that need to be tested are a lot more log usage. We're basically adding a debug mode to save much more info to the log than used to be saved.
Thanks for any help!
EDIT:
Thanks for the quick answers! The code for VerifyOrCreateLogFile() does use the _LogLock, so that shouldn't be an issue. It does do some writing to the log before it errors out, so it gets past creating the file just fine.
What seems to be the problem is that previously only one class created an instance of the log class, and now I've added instances to other classes. It makes sense that this would create problems. Changing the _LogLock field to be static fixes the issue.
Thanks again!
The lock should prevent another thread from attempting to write to the log file.
This is only true if you're using a single instance of this class.
If each (or even some) of the log requests use a separate instance, then the lock will not protect you.
You can easily "correct" this by making the _LogLock field static:
private static object _LogLock = 0;
This way, all instances will share the same lock.
I see 2 problems with the code:
Lock must be the same among all "users" of ths Log class, easiest solution is to make either _LogLock or the complete class static
VerifyOrCreateLogFile could pose a problem if 2 or more parallel threads call WriteToLog when _LogExists is false...
One possibility is that the OS isn't releasing the file lock quickly enough before you exit the lock in WriteToLog and another thread that was blocked waiting for the lock tried to open it before the OS finished releasing the file lock. Yes, it can happen. You either need to sleep for a little before trying to open the file, centralize the writing to the log to a dedicated object (so that he and only he has access to this file and you don't have to worry about file lock contentions).
Another possibility is that you need to lock around
if (!_LogExists) {
this.VerifyOrCreateLogFile(); // Creates log file if it does not already exist.
}
The third possibility is that you have multiple instances of whatever class is housing these methods. The lock object won't be shared across instances (make it static to solve this).
At the end of the day, unless you're an expert in writing safe multi-threaded code, just let someone else worry about this stuff for you. Use a framework that handles these issues for you (log4net?).
you can do the code executable by simply
removing sw.Close(); from your code ...
do it....
it will work fine.....