A decompression API that I am using has the following API:
Decode(Stream inStream,Stream outStream)
I'd like to create a wrapper around this API, such that I can create my own Stream class which offers up the decoded data.
Stream decodedStream=new BlaDecodeStream(inStream);
So that I can than use this stream as a parameter to the XmlReader constructor in the same way one might use the System.IO.Compression.GZipStream. As far as I can tell, the only other option is set outStream stream to a MemoryStream or to a FileStream and go in two hops. The files I am dealing with are enormous, so neither of these options are particularly attractive.
Before I go reinventing the wheel, is there any prior art that I might be able to draw from, or something in the BCL I might have missed? The CircularStream implementation here would go some of the way to helping, but I'm really looking for something similar that would block (as opposed to over/underrun) when the Stream's internal buffer is 'empty' when reading from it and block when the internal buffer is full when writing to it.
In this way it could serve as parameter outStream and simultaneously (i.e. from another thread) could be read from by the XmlReader.
I asked about a blocking stream reader a while ago. I implemented one of the suggestions and it works fine.
Related
how to read from a stream - a BufferedStream without using heavy locks? I'm reading audio data from a file using NAudio. Since NAudio stream have lock on Read methods, I used BufferedStream as a wrapper. I need only read data. However, I lost thread-safety. How to make stream threadsafe without using locks?
Unfortunately I cannot read data in large blocks, because interface which I use contains method float GetSample(long) which read just 1 sample at once.
I've never used that library, but if I understand you correctly, your disk file is locking because you are reading it, so no other application can read it, is that correct?
In this case, it's not your app that's become multi-threaded, and you shouldn't need locks within your application. You do however have to make sure that the file is opened for read only puroposes allowing readonly access to other applications.
If NAudio offers the full set of parameters when opening a file, you'll be able to open the file with sharing allowed, otherwise if NAudio can be passed a stream to read from, you'll be able to open the file however you want and then pass the stream to NAudio.
https://msdn.microsoft.com/en-us/library/y973b725%28v=vs.110%29.aspx
I have an application that works with physical memory snapshots (for example, VMware VMEM files). Among other things, it can read processes/modules out of the snapshot by virtual rather than physical address. This involves reconstructing the module 4KB at a time through the page table, which, in turn, means a lot of calls to the Stream's Seek() method.
For reasons I'm not sure of, these calls to Seek() bog things down dramatically. As a result, I'm looking for a way around them -- or at least, a way around the managed Seek() implementation. My best guess is to PInvoke SetFilePointer and work with that directly, but in order to do that I need to get the IntPtr/SafeFileHandle for the Stream. I have a few restrictions:
The API I'm working with is limited to .NET 3.5, so unfortunately MemoryMappedFile isn't an option.
I can't use a FileStream (which already has a private SafeFileHandle field that can be accessed with reflection) or PInvoke CreateFile() to get at the snapshot another way -- the API includes a BinaryReader that has an exclusive lock on the snapshot.
Of course, unlike FileStream, neither BinaryReader nor its underlying Stream have a reference to the file handle. But certainly one must exist? In which case, how do I go about acquiring it?
There's no file handle for Stream because it's an abstract class. A class implementing Stream may or may not use a file handle -- FileStream does, since it reads data from a file, but MemoryStream, for example, does not.
To get the underlying file handle (in this case a SafeFileHandle) of a BinaryReader whose Stream is a FileStream, use reflection to access private SafeFileHandle _handle, like so:
SafeFileHandle sfh = (SafeFileHandle)typeof(FileStream).GetField("_handle", BindingFlags.NonPublic | BindingFlags.Instance).GetValue((FileStream)YOUR_BINARY_READER.BaseStream)
On a side note: neither direct calls to SetFilePointer() nor MemoryMappedFile helped in this case. It seems there's no fast way to handle random disk access at the volume I'm using it (millions of consecutive calls).
Since you have BinaryReader over FileStream you can access the BaseStream of the reader, cast that to FileStream and then use its public SafeFileHandle property to access the handle. Something like this:
FileStream stream = (FileStream)(reader.BaseStream);
//use stream.SafeFileHandle
I am writting a mp3 player which streams songs from Internet. The problem is that I can't use Seek() method in ConnectStream.
I thinking maybe it's possible to copy ConnectStream to MemoryStream asynchronously during streaming and use MemoryStream's Seek() method.
Does anyone has idea is it possible? Or maybe someone knows other way how to do this?
Thanks.
If you are looking to allow the user to seek back and forth through the song, you are probably on the right track. Allocate a MemoryStream, and use the CopyTo method (assuming that you are on .NET 4), as follows:
var ms = new MemoryStream();
myConnectStream.CopyTo(ms);
If you want to download and seek at the same time (like a youtube video for instance), you need to use slightly more complex techniques. Rather than describe them in detail, I'll just refer you to the source:
http://msdn.microsoft.com/en-us/magazine/cc337900.aspx
Yes, this is possible with System.IO.MemoryStream. You are then buffering some of the stream in the local memory. Consider using System.IO.BufferedStream
Background:
I'm trying to write a simple SoapExtension class to log inbound/outbound Soap messages from an asmx web service. Following this article on msdn, I have been able to get things working. However I'd really like to understand why/how it's working rather than just copy & pasting code.
The question:
What I'm stuggling to grasp specifically is the handling of the IO streams in the example. All other articles I've read on the web handle the streams in an identical way... first getting a reference to the original stream, creating an in memory "working" stream, and then swapping the contents as necessary.
First question is, what is meant by "stream chaining" in this context? My understaning of streams is that writing to any stream will automatically write to the 'inner' streams in a pipeline. If that's the case, why is it necessary to manually copy contents from one stream to another?
Second question is, in the examples Copy method they're creating a StreamReader and StreamWriter each time, without disposing them - is this not putting extra pressure on the GC? Doesn't seem like something you'd want on a high traffic web service... I tried wrapping both in using statements, but disposing the reader/writer also closed the stream which led to more serious errors. .NET 4 has new Stream.CopyTo(Stream) methods, but what would be a better approach for .NET 3.5?
Well, by chaining streams you can basically have different streams that do different things, in a chained sequence. For instance, you can have one stream that compresses the data, and then another stream that encrypts the data (or the opposite if we are moving in the other direction).
As for ChainStream itself, well... There are lots of things to say about this one. I really recommend this article called Inside of Chainstream, which is extremely in-depth and also covers most of the questions you have.
The chaining is done in the framework. You get the original stream and return the stream where you put your modified result. The framework will chain this new stream into any other extensions.
It is implemented this way because the chaining works "backwards". Normally you add new functionality on top of streams but in this case you want to deal with the information fed into the original stream.
Calling close on stream is the same as Dispose.
Question:
What is different between FileStream and StreamWriter in .Net?
What context are you supposed to use it? What is their advantage and disadvantage?
Is it possible to combine these two into one?
What is different between FileStream and StreamWriter in dotnet?
A FileStream is a Stream. Like all Streams it only deals with byte[] data.
A StreamWriter : TextWriter is a Stream-decorator. A TextWriter encodes the primitive type like string, int and char to byte[] and then writes hat to the linked Stream.
What context are you supposed to use it? What is their advantage and disadvantage?
You use a bare FileStream when you have byte[] data. You add a StreamWriter when you want to write text. Use a Formatter or a Serializer to write more complex data.
Is it possible to combine these two into one?
Yes. You always need a Stream to create a StreamWriter. The helper method System.IO.File.CreateText("path") will create them in combination and then you only have to Dispose() the outer writer.
FileStream writes bytes, StreamWriter writes text. That's all.
A FileStream is explicitly intended for working files.
A StreamWriter can be used to stream to any type of Stream - network sockets, files, etc.
ScottGu explains the different Stream objects quite nicely here: http://www.codeguru.com/Csharp/Csharp/cs_data/streaming/article.php/c4223
They are two different levels used in outputting information to known data sources.
A FileStream is a type of Stream, which is conceptually a mechanism that points to some location and can handle incoming and/or outgoing data to and from that location. Streams exist for reading/writing to files, network connections, memory, pipes, the console, debug and trace listeners, and a few other types of data sources. Specifically, a FileStream exists to perform reads and writes to the file system. Most streams are pretty low-level in their usage, and deal with data as bytes.
A StreamWriter is a wrapper for a Stream that simplifies using that stream to output plain text. It exposes methods that take strings instead of bytes, and performs the necessary conversions to and from byte arrays. There are other Writers; the other main one you'd use is the XmlTextWriter, which facilitates writing data in XML format. There are also Reader counterparts to the Writers that similarly wrap a Stream and facilitate getting the data back out.
Well, from the MSDN for FileStream:
Exposes a Stream around a file, supporting both synchronous and asynchronous read and write operations.
and the MSDN for StreamWriter:
Implements a TextWriter for writing characters to a stream in a particular encoding.
The most obvious difference is that FileStream allows read/write operations, while StreamWriter is write only.
The StreamWriter page goes on to add:
StreamWriter is designed for character output in a particular encoding, whereas classes derived from Stream are designed for byte input and output.
So a second difference is that FileStream is for bytes, while StreamWriter is for text.
One key difference (in addition to the above comments), could be that FileStream supports random disk access read and writes to any specified FileStream.Position. For large file modifications, that can be invaluable.