ByteStream source in SharpDX.MediaFoundation hanging - c#

I have been trying to get a custom audio stream to work with SharpDX.MediaFoundation.
To this end I have wrapped my audio object in a class that implements System.IO.Stream as follows:
public class AudioReaderWaveStream : System.IO.Stream
{
byte[] waveHeader = new byte[44];
AudioCore.IAudioReader reader = null;
ulong readHandle = 0xffffffff;
long readPosition = 0;
public AudioReaderWaveStream(AudioCore.CEditedAudio content)
{
reader = content as AudioCore.IAudioReader;
readHandle = reader.OpenDevice();
int sampleRate = 0;
short channels = 0;
content.GetFormat(out sampleRate, out channels);
System.IO.MemoryStream memStream = new System.IO.MemoryStream(waveHeader);
using (System.IO.BinaryWriter bw = new System.IO.BinaryWriter(memStream))
{
bw.Write("RIFF".ToCharArray());
bw.Write((Int32)Length - 8);
bw.Write("WAVE".ToCharArray());
bw.Write("fmt ".ToCharArray());
bw.Write((Int32)16);
bw.Write((Int16)3);
bw.Write((Int16)1);
bw.Write((Int32)sampleRate);
bw.Write((Int32)sampleRate * 4);
bw.Write((Int16)4);
bw.Write((Int16)32);
bw.Write("data".ToCharArray());
bw.Write((Int32)reader.GetSampleCount() * 4);
}
}
protected override void Dispose(bool disposing)
{
if (readHandle != 0xffffffff)
{
reader.CloseDevice(readHandle);
readHandle = 0xfffffffff;
}
base.Dispose(disposing);
}
~AudioReaderWaveStream()
{
Dispose();
}
public override bool CanRead
{
get
{
return true;
}
}
public override bool CanSeek
{
get
{
return true;
}
}
public override bool CanWrite
{
get
{
return false;
}
}
public override long Length
{
get
{
// Number of float samples + header of 44 bytes.
return (reader.GetSampleCount() * 4) + 44;
}
}
public override long Position
{
get
{
return readPosition;
}
set
{
readPosition = value;
}
}
public override void Flush()
{
//throw new NotImplementedException();
}
public override int Read(byte[] buffer, int offset, int count)
{
if (count <= 0)
return 0;
int retCount = count;
if (Position < 44)
{
int headerCount = count;
if ( Position + count >= 44 )
{
headerCount = 44 - (int)Position;
}
Array.Copy(waveHeader, Position, buffer, offset, headerCount);
offset += headerCount;
Position += headerCount;
count -= headerCount;
}
if (count > 0)
{
float[] readBuffer = new float[count/4];
reader.Seek(readHandle, Position - 44);
reader.ReadAudio(readHandle, readBuffer);
Array.Copy(readBuffer, 0, buffer, offset, count);
}
return retCount;
}
public override long Seek(long offset, System.IO.SeekOrigin origin)
{
if (origin == System.IO.SeekOrigin.Begin)
{
readPosition = offset;
}
else if (origin == System.IO.SeekOrigin.Current)
{
readPosition += offset;
}
else
{
readPosition = Length - offset;
}
return readPosition;
}
public override void SetLength(long value)
{
throw new NotImplementedException();
}
public override void Write(byte[] buffer, int offset, int count)
{
throw new NotImplementedException();
}
}
I then take this object and create a source resolver using it as follows:
// Create a source resolver.
SharpDX.MediaFoundation.ByteStream sdxByteStream = new ByteStream( ARWS );
SharpDX.MediaFoundation.SourceResolver resolver = new SharpDX.MediaFoundation.SourceResolver();
ComObject source = (ComObject)resolver.CreateObjectFromStream( sdxByteStream, "File.wav", SourceResolverFlags.MediaSource );
However every time I'm doing this it hangs on the CreateObjectFromStream call. I've had a look inside SharpDX to see whats going on and it seems the actual hang occurs when it makes the call to the underlying interface through CreateObjectFromByteStream. I've also looked to see what data is actually read from the byte stream. It reads the first 16 bytes which includes the 'RIFF', the RIFF size, the 'WAVE' and the 'fmt '. Then nothing else.
Has anyone got any ideas what I could be doing wrong. I've tried all sorts of combinations of the SourceResolverFlags but nothing seems to make any difference. It just hangs.
It does remind me somewhat of interthread marshalling but all the media foundation calls are made from the same thread so I don't think its that. I'm also fairly sure that MediaFoundation uses free threading so this shouldn't be a problem anyway.
Has anyone any idea what I could possibly be doing wrong?
Thanks!

Ok I have come up with a solution to this. It looks like I may be having a COM threading issue. The read happens in a thread and that thread was calling back to the main thread which the function was called from.
So I used the async version of the call and perform an Application.DoEvents() to hand across control where necessary.
Callback cb = new Callback( resolver );
IUnknown cancel = null;
resolver.BeginCreateObjectFromByteStream( sdxByteStream, "File.wav", (int)(SourceResolverFlags.MediaSource | SourceResolverFlags.ByteStream), null, out cancel, cb, null );
if ( cancel != null )
{
cancel.Dispose();
}
while( cb.MediaSource == null )
{
System.Windows.Forms.Application.DoEvents();
}
SharpDX.MediaFoundation.MediaSource mediaSource = cb.MediaSource;
I really hate COM's threading model ...

Related

VST host playback timing issues (VST.NET + NAudio)

I'm just getting started in trying to work out VST plugin hosting for a small music program that I've been working on for a while. I've now reached the point where I'm able to take melodies stored within my program and send the midi data to a hosted plugin (using VST.NET) and outputting the audio to WaveOut (NAudio). The problem is that the audio output is playing far too quickly and also not in time.
Here's the code that I'm using for playback, parts of which are based on the example host within the VST.NET sample projects:
public class PhraseEditorWaveProvider : VstWaveProvider
{
public PhraseEditor PhraseEditor { get; private set; }
public Rational PlaybackBeat { get; private set; }
public PhraseEditorWaveProvider(PhraseEditor phraseEditor, string pluginPath, WaveFormat waveFormat = null)
: base(pluginPath, waveFormat)
{
PhraseEditor = phraseEditor;
}
public override int Read(byte[] buffer, int offset, int count)
{
decimal framesPerBeat = (60 / (decimal)PhraseEditor.Phrase.Tempo) * WaveFormat.SampleRate;
Rational startBeat = PlaybackBeat;
Rational endBeat = startBeat + Rational.FromDecimal(count / framesPerBeat);
//Get list of note starts and note ends that occur within the beat range
List<VstEvent> vstEvents = new List<VstEvent>();
foreach(Note note in PhraseEditor.Phrase.Notes)
{
if(note.StartBeat >= startBeat && note.StartBeat < endBeat)
vstEvents.Add(NoteOnEvent(1, (byte)note.Pitch.Value, 100, (int)(note.Duration * framesPerBeat), (int)((note.StartBeat - startBeat) * framesPerBeat)));
if(note.EndBeat >= startBeat && note.EndBeat < endBeat)
vstEvents.Add(NoteOffEvent(1, (byte)note.Pitch.Value, (int)((note.EndBeat - startBeat) * framesPerBeat)));
}
foreach(Chord chord in PhraseEditor.Phrase.Chords)
{
if(chord.StartBeat >= startBeat && chord.StartBeat < endBeat)
{
//Play each note within a chord in the 4th octave, with velocity 70
foreach (Pitch pitch in chord.Pitches)
vstEvents.Add(NoteOnEvent(1, (byte)((pitch.Value % 12) + 48), 70, (int)(chord.Duration * framesPerBeat), (int)((chord.StartBeat - startBeat) * framesPerBeat)));
}
if(chord.EndBeat >= startBeat && chord.EndBeat < endBeat)
{
foreach(Pitch pitch in chord.Pitches)
vstEvents.Add(NoteOffEvent(1, (byte)((pitch.Value % 12) + 48), (int)((chord.EndBeat - startBeat) * framesPerBeat)));
}
}
PlaybackBeat = endBeat;
return base.Read(vstEvents.OrderBy(x => x.DeltaFrames).ToArray(), buffer, offset, count);
}
}
public abstract class VstWaveProvider : IWaveProvider
{
private WaveFormat _waveFormat;
public WaveFormat WaveFormat
{
get
{
return _waveFormat;
}
set
{
_waveFormat = value;
BytesPerWaveSample = _waveFormat.BitsPerSample / 8;
}
}
public VstPluginContext VstContext { get; private set; }
public int BytesPerWaveSample { get; private set; }
public VstWaveProvider(VstPluginContext vstContext, WaveFormat waveFormat = null)
{
WaveFormat = (waveFormat == null) ? new WaveFormat(44100, 2) : waveFormat;
VstContext = vstContext;
}
public VstWaveProvider(string pluginPath, WaveFormat waveFormat = null)
{
WaveFormat = (waveFormat == null) ? new WaveFormat(44100, 2) : waveFormat;
VstContext = OpenPlugin(pluginPath);
}
public abstract int Read(byte[] buffer, int offset, int count);
protected int Read(VstEvent[] vstEvents, byte[] outputBuffer, int offset, int count)
{
VstAudioBufferManager inputBuffers = new VstAudioBufferManager(
VstContext.PluginInfo.AudioInputCount,
count / (Math.Max(1, VstContext.PluginInfo.AudioInputCount) * BytesPerWaveSample)
);
return Read(inputBuffers, vstEvents, outputBuffer, offset, count);
}
protected int Read(VstAudioBufferManager inputBuffers, VstEvent[] vstEvents, byte[] outputBuffer, int offset, int count)
{
VstAudioBufferManager outputBuffers = new VstAudioBufferManager(
VstContext.PluginInfo.AudioOutputCount,
count / (VstContext.PluginInfo.AudioOutputCount * BytesPerWaveSample)
);
VstContext.PluginCommandStub.StartProcess();
if(vstEvents.Length > 0)
VstContext.PluginCommandStub.ProcessEvents(vstEvents);
VstContext.PluginCommandStub.ProcessReplacing(inputBuffers.ToArray(), outputBuffers.ToArray());
VstContext.PluginCommandStub.StopProcess();
//Convert from multi-track to interleaved data
int bufferIndex = offset;
for (int i = 0; i < outputBuffers.BufferSize; i++)
{
foreach (VstAudioBuffer vstBuffer in outputBuffers)
{
Int16 waveValue = (Int16)((vstBuffer[i] + 1) * 128);
byte[] bytes = BitConverter.GetBytes(waveValue);
outputBuffer[bufferIndex] = bytes[0];
outputBuffer[bufferIndex + 1] = bytes[1];
bufferIndex += 2;
}
}
return count;
}
private VstPluginContext OpenPlugin(string pluginPath)
{
HostCommandStub hostCmdStub = new HostCommandStub();
hostCmdStub.PluginCalled += new EventHandler<PluginCalledEventArgs>(HostCmdStub_PluginCalled);
VstPluginContext ctx = VstPluginContext.Create(pluginPath, hostCmdStub);
ctx.Set("PluginPath", pluginPath);
ctx.Set("HostCmdStub", hostCmdStub);
ctx.PluginCommandStub.Open();
ctx.PluginCommandStub.MainsChanged(true);
return ctx;
}
private void HostCmdStub_PluginCalled(object sender, PluginCalledEventArgs e)
{
Debug.WriteLine(e.Message);
}
protected VstMidiEvent NoteOnEvent(byte channel, byte pitch, byte velocity, int noteLength, int deltaFrames = 0)
{
return new VstMidiEvent(deltaFrames, noteLength, 0, new byte[] { (byte)(144 + channel), pitch, velocity, 0 }, 0, 0);
}
protected VstMidiEvent NoteOffEvent(byte channel, byte pitch, int deltaFrames = 0)
{
return new VstMidiEvent(deltaFrames, 0, 0, new byte[] { (byte)(144 + channel), pitch, 0, 0 }, 0, 0);
}
}
Which would get called by the following:
WaveOut waveOut = new WaveOut(WaveCallbackInfo.FunctionCallback());
waveOut.Init(new PhraseEditorWaveProvider(this, #"C:\Users\james\Downloads\Cobalt\Cobalt\Cobalt 64bit\Cobalt.dll"));
waveOut.Play();
Where Cobalt is the current plugin that I'm using for testing.
For context, Rational is my own data type since other parts of my program are doing lots of manipulation of melodies and I found that doubles and decimals weren't giving me the precision that I required.
Also, both the VST plugin context and WaveOut are set to have sample rates of 44.1kHz, so there shouldn't need to be any up/down-sampling when passing the plugin output data into the WaveOut buffer.
I'm at a complete loss as to why the audio is playing back faster than expected. It seems to be roughly 4x faster than expected. If anyone can give any pointers what may be causing this I'd be hugely grateful.
With it playing out of time, I suspect that this is down to me not understanding correctly how the deltaFrame property works within VstMidiEvent. I've tried playing around with both deltaFrame and noteOffset, though don't seem to be having much luck with either, I'm currently working under the assumption that they measure the number of audio frames from the start of the current block of data, to the time of the event within that block. Unfortunately I've been struggling to find much useful documentation on this so it could be that I'm totally wrong about this though.
Look forward to any responses
Kind regards
James
Ok, I think I found what was causing the problem, it was in this section of code:
public override int Read(byte[] buffer, int offset, int count)
{
decimal framesPerBeat = (60 / (decimal)PhraseEditor.Phrase.Tempo) * WaveFormat.SampleRate;
Rational startBeat = PlaybackBeat;
Rational endBeat = startBeat + Rational.FromDecimal(count / framesPerBeat);
...
}
Which I just changed to this:
public override int Read(byte[] buffer, int offset, int count)
{
decimal framesPerBeat = (60 / (decimal)PhraseEditor.Phrase.Tempo) * WaveFormat.SampleRate;
int samplesRequired = count / (WaveFormat.Channels * (WaveFormat.BitsPerSample / 8));
Rational startBeat = PlaybackBeat;
Rational endBeat = startBeat + Rational.FromDecimal(samplesRequired / framesPerBeat);
...
}
Dumb mistake on my part, I'd been converting from bit-rate to sample-rate everywhere except in my method for getting upcoming midi events. My audio is now playing at a rate far closer to what I'd expect, and seems to be more reliable on the timing, though I haven't had a chance to fully test this yet.

Returning database results as a stream

I have a function that returns database query results. These results have got very large, and I now would like to pass them as a stream, so that the client can start to process them quicker, and memory usage is less. But I don't really know how to do this, the following function works, but what I want to know how to change it so that it starts to stream upon reading from the first table.
public Stream GetResults()
{
IFormatter formatter = new BinaryFormatter();
Stream stream = new MemoryStream();
formatter.Serialize(stream, GetItemsFromTable1());
formatter.Serialize(stream, GetItemsFromTable2());
formatter.Serialize(stream, GetItemsFromTable3());
formatter.Serialize(stream, GetItemsFromTable4());
stream.Position = 0;
return stream;
}
You could write a custom Stream implementation which functions as a pipe. If you then moved your GetItemsFromTable() method calls into a background task, the client could start reading results from the stream immediately.
In my solution below I'm using a circular buffer as a backing store for the pipe stream. Memory usage will be reduced only if the client consumes data fast enough. But even in the worst case scenario it shouldn't use more memory then your current solution. If memory usage is a bigger priority for you than execution speed then your stream could potentially block write calls until space is available. My solution below does not block writes; it expands the capacity of the circular buffer so that the background thread can continue filling data without delays.
The GetResults method might look like this:
public Stream GetResults()
{
// Begin filling the pipe with data on a background thread
var pipeStream = new CircularBufferPipeStream();
Task.Run(() => WriteResults(pipeStream));
// Return pipe stream for immediate usage by client
// Note: client is responsible for disposing of the stream after reading all data!
return pipeStream;
}
// Runs on background thread, filling circular buffer with data
void WriteResults(CircularBufferPipeStream stream)
{
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, GetItemsFromTable1());
formatter.Serialize(stream, GetItemsFromTable2());
formatter.Serialize(stream, GetItemsFromTable3());
formatter.Serialize(stream, GetItemsFromTable4());
// Indicate that there's no more data to write
stream.CloseWritePort();
}
And the circular buffer stream:
/// <summary>
/// Stream that acts as a pipe by supporting reading and writing simultaneously from different threads.
/// Read calls will block until data is available or the CloseWritePort() method has been called.
/// Read calls consume bytes in the circular buffer immediately so that more space is available for writes into the circular buffer.
/// Writes do not block; the capacity of the circular buffer will be expanded as needed to write the entire block of data at once.
/// </summary>
class CircularBufferPipeStream : Stream
{
const int DefaultCapacity = 1024;
byte[] _buffer;
bool _writePortClosed = false;
object _readWriteSyncRoot = new object();
int _length;
ManualResetEvent _dataAddedEvent;
int _start = 0;
public CircularBufferPipeStream(int initialCapacity = DefaultCapacity)
{
_buffer = new byte[initialCapacity];
_length = 0;
_dataAddedEvent = new ManualResetEvent(false);
}
public void CloseWritePort()
{
lock (_readWriteSyncRoot)
{
_writePortClosed = true;
_dataAddedEvent.Set();
}
}
public override bool CanRead { get { return true; } }
public override bool CanWrite { get { return true; } }
public override bool CanSeek { get { return false; } }
public override void Flush() { }
public override long Length { get { throw new NotImplementedException(); } }
public override long Position
{
get { throw new NotImplementedException(); }
set { throw new NotImplementedException(); }
}
public override long Seek(long offset, SeekOrigin origin) { throw new NotImplementedException(); }
public override void SetLength(long value) { throw new NotImplementedException(); }
public override int Read(byte[] buffer, int offset, int count)
{
int bytesRead = 0;
while (bytesRead == 0)
{
bool waitForData = false;
lock (_readWriteSyncRoot)
{
if (_length != 0)
bytesRead = ReadDirect(buffer, offset, count);
else if (_writePortClosed)
break;
else
{
_dataAddedEvent.Reset();
waitForData = true;
}
}
if (waitForData)
_dataAddedEvent.WaitOne();
}
return bytesRead;
}
private int ReadDirect(byte[] buffer, int offset, int count)
{
int readTailCount = Math.Min(Math.Min(_buffer.Length - _start, count), _length);
Array.Copy(_buffer, _start, buffer, offset, readTailCount);
_start += readTailCount;
_length -= readTailCount;
if (_start == _buffer.Length)
_start = 0;
int readHeadCount = Math.Min(Math.Min(_buffer.Length - _start, count - readTailCount), _length);
if (readHeadCount > 0)
{
Array.Copy(_buffer, _start, buffer, offset + readTailCount, readHeadCount);
_start += readHeadCount;
_length -= readHeadCount;
}
return readTailCount + readHeadCount;
}
public override void Write(byte[] buffer, int offset, int count)
{
lock (_readWriteSyncRoot)
{
// expand capacity as needed
if (count + _length > _buffer.Length)
{
var expandedBuffer = new byte[Math.Max(_buffer.Length * 2, count + _length)];
_length = ReadDirect(expandedBuffer, 0, _length);
_start = 0;
_buffer = expandedBuffer;
}
int startWrite = (_start + _length) % _buffer.Length;
int writeTailCount = Math.Min(_buffer.Length - startWrite, count);
Array.Copy(buffer, offset, _buffer, startWrite, writeTailCount);
startWrite += writeTailCount;
_length += writeTailCount;
if (startWrite == _buffer.Length)
startWrite = 0;
int writeHeadCount = count - writeTailCount;
if (writeHeadCount > 0)
{
Array.Copy(buffer, offset + writeTailCount, _buffer, startWrite, writeHeadCount);
_length += writeHeadCount;
}
}
_dataAddedEvent.Set();
}
protected override void Dispose(bool disposing)
{
if (disposing)
{
if (_dataAddedEvent != null)
{
_dataAddedEvent.Dispose();
_dataAddedEvent = null;
}
}
base.Dispose(disposing);
}
}
try
public Stream GetResults()
{
IFormatter formatter = new BinaryFormatter();
Stream stream = new MemoryStream();
formatter.Serialize(stream, GetItemsFromTable1());
formatter.Serialize(stream, GetItemsFromTable2());
formatter.Serialize(stream, GetItemsFromTable3());
formatter.Serialize(stream, GetItemsFromTable4());
stream.Seek(0L, SeekOrigin.Begin);
return stream;
}
why the changes?
remove using, because your stream gets disposed once it leaves the using-block. disposing the stream means you cannot use it anymore
seek to the beginning of the stream. if you start reading from the stream without seeking to its beginning, you would start to deserialize/ read from its end; but unfortunately there is no content behind the end of the stream
however, I don't see how using a MemoryStream reduces memory usage. I would suggest chaining it into a DeflateStream or a FileStream to reduce RAM-usage
hope this helps

AES OFB encryption for RijndaelManaged

I need to communicate from a C# application to another application via encrypted messages in OFB mode. I know that RijndaelManaged does not have support for AES OFB mode. Is there anybody more experienced than me aware of any other way to encrypt/decrypt using OFB mode?
The following stream implements OFB by using a key stream generated by a zero-fed CBC cipher stream.
public class OFBStream : Stream
{
private const int BLOCKS = 16;
private const int EOS = 0; // the goddess of dawn is found at the end of the stream
private Stream parent;
private CryptoStream cbcStream;
private CryptoStreamMode mode;
private byte[] keyStreamBuffer;
private int keyStreamBufferOffset;
private byte[] readWriteBuffer;
public OFBStream (Stream parent, SymmetricAlgorithm algo, CryptoStreamMode mode)
{
if (algo.Mode != CipherMode.CBC)
algo.Mode = CipherMode.CBC;
if (algo.Padding != PaddingMode.None)
algo.Padding = PaddingMode.None;
this.parent = parent;
this.cbcStream = new CryptoStream (new ZeroStream (), algo.CreateEncryptor (), CryptoStreamMode.Read);
this.mode = mode;
keyStreamBuffer = new byte[algo.BlockSize * BLOCKS];
readWriteBuffer = new byte[keyStreamBuffer.Length];
}
public override int Read (byte[] buffer, int offset, int count)
{
if (!CanRead) {
throw new NotSupportedException ();
}
int toRead = Math.Min (count, readWriteBuffer.Length);
int read = parent.Read (readWriteBuffer, 0, toRead);
if (read == EOS)
return EOS;
for (int i = 0; i < read; i++) {
// NOTE could be optimized (branches for each byte)
if (keyStreamBufferOffset % keyStreamBuffer.Length == 0) {
FillKeyStreamBuffer ();
keyStreamBufferOffset = 0;
}
buffer [offset + i] = (byte)(readWriteBuffer [i]
^ keyStreamBuffer [keyStreamBufferOffset++]);
}
return read;
}
public override void Write (byte[] buffer, int offset, int count)
{
if (!CanWrite) {
throw new NotSupportedException ();
}
int readWriteBufferOffset = 0;
for (int i = 0; i < count; i++) {
if (keyStreamBufferOffset % keyStreamBuffer.Length == 0) {
FillKeyStreamBuffer ();
keyStreamBufferOffset = 0;
}
if (readWriteBufferOffset % readWriteBuffer.Length == 0) {
parent.Write (readWriteBuffer, 0, readWriteBufferOffset);
readWriteBufferOffset = 0;
}
readWriteBuffer [readWriteBufferOffset++] = (byte)(buffer [offset + i]
^ keyStreamBuffer [keyStreamBufferOffset++]);
}
parent.Write (readWriteBuffer, 0, readWriteBufferOffset);
}
private void FillKeyStreamBuffer ()
{
int read = cbcStream.Read (keyStreamBuffer, 0, keyStreamBuffer.Length);
// NOTE undocumented feature
// only works if keyStreamBuffer.Length % blockSize == 0
if (read != keyStreamBuffer.Length)
throw new InvalidOperationException ("Implementation error: could not read all bytes from CBC stream");
}
public override bool CanRead {
get { return mode == CryptoStreamMode.Read; }
}
public override bool CanWrite {
get { return mode == CryptoStreamMode.Write; }
}
public override void Flush ()
{
// should never have to be flushed, implementation empty
}
public override bool CanSeek {
get { return false; }
}
public override long Seek (long offset, System.IO.SeekOrigin origin)
{
throw new NotSupportedException ();
}
public override long Position {
get { throw new NotSupportedException (); }
set { throw new NotSupportedException (); }
}
public override long Length {
get { throw new NotSupportedException (); }
}
public override void SetLength (long value)
{
throw new NotSupportedException ();
}
}
Additional class ZeroStream required by OFBStream
class ZeroStream : System.IO.Stream
{
public override int Read (byte[] buffer, int offset, int count)
{
for (int i = 0; i < count; i++) {
buffer [offset + i] = 0;
}
return count;
}
public override bool CanRead {
get { return true; }
}
... the rest is not implemented
}
And you can use it as I do for a test vector:
// NIST CAVP test vector F.4.1: OFB-AES128.Encrypt from NIST SP 800-38A
RijndaelManaged aes = new RijndaelManaged ();
aes.Key = FromHex ("2b7e151628aed2a6abf7158809cf4f3c");
aes.IV = FromHex ("000102030405060708090A0B0C0D0E0F");
MemoryStream testVectorStream = new MemoryStream (FromHex (
"6bc1bee22e409f96e93d7e117393172aae2d8a571e03ac9c9eb76fac45af8e5130c81c46a35ce411e5fbc1191a0a52eff69f2445df4f9b17ad2b417be66c3710"));
OFBStream testOFBStream = new OFBStream (testVectorStream, aes, CryptoStreamMode.Read);
MemoryStream cipherTextStream = new MemoryStream ();
testOFBStream.CopyTo (cipherTextStream);
Console.WriteLine (ToHex (cipherTextStream.ToArray ()));
Note that the stream handling has not been fully tested (yet).

XmlReader to read from fixed length buffer

The incoming stream comes in a fixed 1024 bytes buffer, the stream itself is a hug XML file which may take several rounds of read to finish. My goal is to read the buffer and figure out how many times an element occured in the big XML file.
My chanllenge is, since it is really a fixed length buffer, so it cannot gurantee the wellform of XML, if I wrap the stream in the XmlTextReader, I am always getting exception and cannot finish the read. for example, the element could be abcdef, while the 1st buffer could end at abc while second buffer start with def. I am really frustrated about this, anyone could advise a better way to archieve this using streaming fashion? (I do not want to load entire content in memory)
Thanks so much
Are your 1024-byte buffers coming from one of the standard, concrete implementations of System.IO.Stream? If t they are, you can just create your XmlTextReader around the base stream:
XmlTextReader tr = XmlTextReader.Create( myStreamInstance ) ;
If not -- say, for instance, you're "reading" the buffers from some sort of API -- you need to implement your own concrete Stream, something along these lines (all you should need to do is flesh out the ReadNextFrame() method and possibly implement your constructors):
public class MyStream : System.IO.Stream
{
public override bool CanRead { get { return true ; } }
public override bool CanSeek { get { return false ; } }
public override bool CanWrite { get { return false ; } }
public override long Length { get { throw new NotImplementedException(); } }
public override long Position {
get { throw new NotImplementedException(); }
set { throw new NotImplementedException(); }
}
public override int Read( byte[] buffer , int offset , int count )
{
int bytesRead = 0 ;
if ( !initialized )
{
Initialize() ;
}
for ( int bytesRemaining = count ; !atEOF && bytesRemaining > 0 ; )
{
int frameRemaining = frameLength - frameOffset ;
int chunkSize = ( bytesRemaining > frameRemaining ? frameRemaining : bytesRemaining ) ;
Array.Copy( frame , offset , frame , frameOffset , chunkSize ) ;
bytesRemaining -= chunkSize ;
offset += chunkSize ;
bytesRead += chunkSize ;
// read next frame if necessary
if ( frameOffset >= frameLength )
{
ReadNextFrame() ;
}
}
return bytesRead ;
}
public override long Seek( long offset , System.IO.SeekOrigin origin ) { throw new NotImplementedException(); }
public override void SetLength( long value ) { throw new NotImplementedException(); }
public override void Write( byte[] buffer , int offset , int count ) { throw new NotImplementedException(); }
public override void Flush() { throw new NotImplementedException(); }
private byte[] frame = null ;
private int frameLength = 0 ;
private int frameOffset = 0 ;
private bool atEOF = false ;
private bool initialized = false ;
private void Initialize()
{
if ( initialized ) throw new InvalidOperationException() ;
frame = new byte[1024] ;
frameLength = 0 ;
frameOffset = 0 ;
atEOF = false ;
initialized = true ;
ReadNextFrame() ;
return ;
}
private void ReadNextFrame()
{
//TODO: read the next (or first 1024-byte buffer
//TODO: set the frame length to the number of bytes actually returned (might be less than 1024 on the last read, right?
//TODO: set the frame offset to 0
//TODO: set the atEOF flag if we've exhausted the data source ;
return ;
}
}
Then instantiate your XmlReader as above:
System.IO.Stream s = new MyStream() ;
System.Xml.XmlReader xr = XmlTextReader.Create( s ) ;
Cheers!
That is sort of strange goal... Usually it is more like "count elements but not load whole XML to memory" which is trivial - write Stream derived class that represents you buffer as forward only stream (similar to NetworkStream) and read XML (i.e. using LINQ) normally using XmlReader, but do not construct XmlDocument.
If you clarify your goal it may be easier for others to advise.

How do I hash first N bytes of a file?

Using .net, I would like to be able to hash the first N bytes of potentially large files, but I can't seem to find a way of doing it.
The ComputeHash function (I'm using SHA1) takes a byte array or a stream, but a stream seems like the best way of doing it, since I would prefer not to load a potentially large file into memory.
To be clear: I don't want to load a potentially large piece of data into memory if I can help it. If the file is 2GB and I want to hash the first 1GB, that's a lot of RAM!
You can hash large volumes of data using a CryptoStream - something like this should work:
var sha1 = SHA1Managed.Create();
FileStream fs = \\whatever
using (var cs = new CryptoStream(fs, sha1, CryptoStreamMode.Read))
{
byte[] buf = new byte[16];
int bytesRead = cs.Read(buf, 0, buf.Length);
long totalBytesRead = bytesRead;
while (bytesRead > 0 && totalBytesRead <= maxBytesToHash)
{
bytesRead = cs.Read(buf, 0, buf.Length);
totalBytesRead += bytesRead;
}
}
byte[] hash = sha1.Hash;
fileStream.Read(array, 0, N);
http://msdn.microsoft.com/en-us/library/system.io.filestream.read.aspx
Open the file as a FileStream, copy the first n bytes into a MemoryStream, then hash the MemoryStream.
As others have pointed out, you should read the first few bytes into an array.
What should also be noted that you don't want to make a direct call to Read and assume that the bytes have been read.
Rather, you want to make sure that the number of bytes that are returned are the number of bytes that you requested, and make another call to Read in the event that the number of bytes returned doesn't equal the initial number requested.
Also, if you have rather large streams, you will want to create a proxy for the Stream class where you pass it the underlying stream (the FileStream in this case) and override the Read method to forward the call to the underlying stream until you read the number of bytes that you need to read. Then, when that number of bytes is returned, you would return -1 to indicate that there are no more bytes to be read.
If you are concerned about keeping too much data in memory, you can create a stream wrapper that throttles the maximum number of bytes read.
Without doing all the work, here's a sample boiler plate you could use to get started.
Edit: Please review comments for recommendations to improve this implementation. End edit
public class LimitedStream : Stream
{
private int current = 0;
private int limit;
private Stream stream;
public LimitedStream(Stream stream, int n)
{
this.limit = n;
this.stream = stream;
}
public override int ReadByte()
{
if (current >= limit)
return -1;
var numread = base.ReadByte();
if (numread >= 0)
current++;
return numread;
}
public override int Read(byte[] buffer, int offset, int count)
{
count = Math.Min(count, limit - current);
var numread = this.stream.Read(buffer, offset, count);
current += numread;
return numread;
}
public override long Seek(long offset, SeekOrigin origin)
{
throw new NotImplementedException();
}
public override void SetLength(long value)
{
throw new NotImplementedException();
}
public override void Write(byte[] buffer, int offset, int count)
{
throw new NotImplementedException();
}
public override bool CanRead
{
get { return true; }
}
public override bool CanSeek
{
get { return false; }
}
public override bool CanWrite
{
get { return false; }
}
public override void Flush()
{
throw new NotImplementedException();
}
public override long Length
{
get { throw new NotImplementedException(); }
}
public override long Position
{
get { throw new NotImplementedException(); }
set { throw new NotImplementedException(); }
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
if (this.stream != null)
{
this.stream.Dispose();
}
}
}
Here is an example of the stream in use, wrapping a file stream, but throttling the number of bytes read to the specified limit:
using (var stream = new LimitedStream(File.OpenRead(#".\test.xml"), 100))
{
var bytes = new byte[1024];
stream.Read(bytes, 0, bytes.Length);
}

Categories