I have an application that works with physical memory snapshots (for example, VMware VMEM files). Among other things, it can read processes/modules out of the snapshot by virtual rather than physical address. This involves reconstructing the module 4KB at a time through the page table, which, in turn, means a lot of calls to the Stream's Seek() method.
For reasons I'm not sure of, these calls to Seek() bog things down dramatically. As a result, I'm looking for a way around them -- or at least, a way around the managed Seek() implementation. My best guess is to PInvoke SetFilePointer and work with that directly, but in order to do that I need to get the IntPtr/SafeFileHandle for the Stream. I have a few restrictions:
The API I'm working with is limited to .NET 3.5, so unfortunately MemoryMappedFile isn't an option.
I can't use a FileStream (which already has a private SafeFileHandle field that can be accessed with reflection) or PInvoke CreateFile() to get at the snapshot another way -- the API includes a BinaryReader that has an exclusive lock on the snapshot.
Of course, unlike FileStream, neither BinaryReader nor its underlying Stream have a reference to the file handle. But certainly one must exist? In which case, how do I go about acquiring it?
There's no file handle for Stream because it's an abstract class. A class implementing Stream may or may not use a file handle -- FileStream does, since it reads data from a file, but MemoryStream, for example, does not.
To get the underlying file handle (in this case a SafeFileHandle) of a BinaryReader whose Stream is a FileStream, use reflection to access private SafeFileHandle _handle, like so:
SafeFileHandle sfh = (SafeFileHandle)typeof(FileStream).GetField("_handle", BindingFlags.NonPublic | BindingFlags.Instance).GetValue((FileStream)YOUR_BINARY_READER.BaseStream)
On a side note: neither direct calls to SetFilePointer() nor MemoryMappedFile helped in this case. It seems there's no fast way to handle random disk access at the volume I'm using it (millions of consecutive calls).
Since you have BinaryReader over FileStream you can access the BaseStream of the reader, cast that to FileStream and then use its public SafeFileHandle property to access the handle. Something like this:
FileStream stream = (FileStream)(reader.BaseStream);
//use stream.SafeFileHandle
Related
how to read from a stream - a BufferedStream without using heavy locks? I'm reading audio data from a file using NAudio. Since NAudio stream have lock on Read methods, I used BufferedStream as a wrapper. I need only read data. However, I lost thread-safety. How to make stream threadsafe without using locks?
Unfortunately I cannot read data in large blocks, because interface which I use contains method float GetSample(long) which read just 1 sample at once.
I've never used that library, but if I understand you correctly, your disk file is locking because you are reading it, so no other application can read it, is that correct?
In this case, it's not your app that's become multi-threaded, and you shouldn't need locks within your application. You do however have to make sure that the file is opened for read only puroposes allowing readonly access to other applications.
If NAudio offers the full set of parameters when opening a file, you'll be able to open the file with sharing allowed, otherwise if NAudio can be passed a stream to read from, you'll be able to open the file however you want and then pass the stream to NAudio.
https://msdn.microsoft.com/en-us/library/y973b725%28v=vs.110%29.aspx
i want to transfer data over sockets and currently i am creating a memory stream.
i can also use a network stream.
Can anyone please help me understand the difference between c# network stream and memory stream?
A NetworkStream is directly related to a socket; it does not know it's own length, you cannot seek, and the read/write functions are directly bound to the receive/send APIs (and therefore, read and write are entirely unrelated to eachother). It can timeout, and a read can take a considerable time if waiting for more data.
A MemoryStream is basically a wrapper over a local byte[]. It has a known length (which can change), you can seek, and read/write are directly related: both increment the same position cursor, and you can write something, rewind, and then read it. All operations are very timely.
It might be easier to ask "what are the similarities", which would be simply: both have a read/write API, by virtue of being subclasses of Stream.
both streams are derive of Stream, this classes are warper for different purpose
According to my understanding, Network Stream reads from the network interface, where if you use a Memory Stream (I mean, in the same scenario), all the data will be loaded to memory first (I assume it reads to the end of the actual stream), then the read operations will read from memory.
The first read operation to occur on the Memory Stream, all the data needs to be loaded in to memory.
Where network stream, you can read the data as they arrive.
Question:
What is different between FileStream and StreamWriter in .Net?
What context are you supposed to use it? What is their advantage and disadvantage?
Is it possible to combine these two into one?
What is different between FileStream and StreamWriter in dotnet?
A FileStream is a Stream. Like all Streams it only deals with byte[] data.
A StreamWriter : TextWriter is a Stream-decorator. A TextWriter encodes the primitive type like string, int and char to byte[] and then writes hat to the linked Stream.
What context are you supposed to use it? What is their advantage and disadvantage?
You use a bare FileStream when you have byte[] data. You add a StreamWriter when you want to write text. Use a Formatter or a Serializer to write more complex data.
Is it possible to combine these two into one?
Yes. You always need a Stream to create a StreamWriter. The helper method System.IO.File.CreateText("path") will create them in combination and then you only have to Dispose() the outer writer.
FileStream writes bytes, StreamWriter writes text. That's all.
A FileStream is explicitly intended for working files.
A StreamWriter can be used to stream to any type of Stream - network sockets, files, etc.
ScottGu explains the different Stream objects quite nicely here: http://www.codeguru.com/Csharp/Csharp/cs_data/streaming/article.php/c4223
They are two different levels used in outputting information to known data sources.
A FileStream is a type of Stream, which is conceptually a mechanism that points to some location and can handle incoming and/or outgoing data to and from that location. Streams exist for reading/writing to files, network connections, memory, pipes, the console, debug and trace listeners, and a few other types of data sources. Specifically, a FileStream exists to perform reads and writes to the file system. Most streams are pretty low-level in their usage, and deal with data as bytes.
A StreamWriter is a wrapper for a Stream that simplifies using that stream to output plain text. It exposes methods that take strings instead of bytes, and performs the necessary conversions to and from byte arrays. There are other Writers; the other main one you'd use is the XmlTextWriter, which facilitates writing data in XML format. There are also Reader counterparts to the Writers that similarly wrap a Stream and facilitate getting the data back out.
Well, from the MSDN for FileStream:
Exposes a Stream around a file, supporting both synchronous and asynchronous read and write operations.
and the MSDN for StreamWriter:
Implements a TextWriter for writing characters to a stream in a particular encoding.
The most obvious difference is that FileStream allows read/write operations, while StreamWriter is write only.
The StreamWriter page goes on to add:
StreamWriter is designed for character output in a particular encoding, whereas classes derived from Stream are designed for byte input and output.
So a second difference is that FileStream is for bytes, while StreamWriter is for text.
One key difference (in addition to the above comments), could be that FileStream supports random disk access read and writes to any specified FileStream.Position. For large file modifications, that can be invaluable.
A decompression API that I am using has the following API:
Decode(Stream inStream,Stream outStream)
I'd like to create a wrapper around this API, such that I can create my own Stream class which offers up the decoded data.
Stream decodedStream=new BlaDecodeStream(inStream);
So that I can than use this stream as a parameter to the XmlReader constructor in the same way one might use the System.IO.Compression.GZipStream. As far as I can tell, the only other option is set outStream stream to a MemoryStream or to a FileStream and go in two hops. The files I am dealing with are enormous, so neither of these options are particularly attractive.
Before I go reinventing the wheel, is there any prior art that I might be able to draw from, or something in the BCL I might have missed? The CircularStream implementation here would go some of the way to helping, but I'm really looking for something similar that would block (as opposed to over/underrun) when the Stream's internal buffer is 'empty' when reading from it and block when the internal buffer is full when writing to it.
In this way it could serve as parameter outStream and simultaneously (i.e. from another thread) could be read from by the XmlReader.
I asked about a blocking stream reader a while ago. I implemented one of the suggestions and it works fine.
I have an application on the Compact Framework that has some large embedded resources (some of them are several megabytes). I am calling assembly.GetManifestResourceStream(...) which returns a Stream object. However, I noticed that on some devices this call not only takes quite a while but causes the device to run out of available memory. Eventually I used reflector to look at the code for this method on the compact framework and it uses an internal method to get a byte[] of the resource data. It then returns this data wrapped in a MemoryStream.
Is there any way to retrieve a resource without using this call since it will always read everything into memory? Ideally I'd like to work with a Stream that I can get random access to without having to read the whole thing into memory (similar to how a FileStream works). It would be pretty neat if I could simply open a FileStream on the assembly and start reading at the appropriate offset, but I doubt this is how resources are embedded.
Don't use an embedded resource. Add it as a content file and open it off disk with a file stream.
I found an open source tool that exposes a lot of the assemblies meta meta and that allowed me to peak into the resource manually:
http://www.jbrowse.com/products/asmex/