We Keep Coding

Generalize deserializing from byte array

I am parsing a byte array containing different type values stored in a fixed format. For example first 4 bytes could be an int containing size of an array -
let's say the array of doubles so next 8 bytes represent a double - the first element of the array etc. It could in theory contain values of other types, but let's say we can only have
bool,int,uint,short,ushort,long,ulong,float,double and arrays of each of these. Simple approach:
public class FixedFormatParser
{
private byte[] _Contents;
private int _CurrentPos = 0;
public FixedFormatParser(byte[] contents)
{
_Contents = contents;
}
bool ReadBool()
{
bool res = BitConverter.ToBoolean(_Contents, _CurrentPos);
_CurrentPos += sizeof(bool);
return res;
}
int ReadInt()
{
int res = BitConverter.ToInt32(_Contents, _CurrentPos);
_CurrentPos += sizeof(int);
return res;
}
// etc. for uint, short, ushort, long, ulong, float, double
int[] ReadIntArray()
{
int size = ReadInt();
if (size == 0)
return null;
int[] res = new int[size];
for (int i = 0; i < size; i++)
res[i] = ReadInt();
return res;
}
// etc. for bool, uint, short, ushort, long, ulong, float, double
}
I can obviously write 18 methods to cover each case, but seems like there should be a way to generalize this.
bool val = Read<bool>();
long[] arr = ReadArray<long>(); // or ReadArray(Read<long>);
Obviously I don't mean write 2 wrappers in addition to the 18 methods to allow for this syntax. The syntax is not important, the code duplication is the issue. Another consideration is the performance. Ideally there would not be any (or much) of a performance hit. Thanks.
Update:
Regarding other questions that are supposedly duplicates. I disagree as none of them addressed the particular generalization I am after, but one came pretty close:
First answer in
C# Reading Byte Array
described wrapping BinaryReader. This would cover 9 of the 18 methods. So half of the problem is addressed. I still would need to write all of the various array reads.
public class FixedFormatParser2 : BinaryReader
{
public FixedFormatParser2(byte[] input) : base(new MemoryStream(input))
{
}
public override string ReadString()
{
//
}
public double[] ReadDoubleArray()
{
int size = ReadInt32();
if (size == 0)
return null;
double[] res = new double[size];
for (int i = 0; i < size; i++)
res[i] = ReadDouble();
return res;
}
}
How do I not write a separate ReadXXXArray for each of the types?
Nearest I got to it:
public void WriteCountedArray(dynamic[] input)
{
if (input == null || input.Length == 0)
Write((int)0);
else
{
Write(input.Length);
foreach (dynamic val in input)
Write(val);
}
}
This compiles but calling it is cumbersome and expensive :
using (FixedFormatWriter writer = new FixedFormatWriter())
{
double[] array = new double[3];
// ... assign values
writer.WriteCountedArray(array.Select(x=>(dynamic)x).ToArray());

I like doing like this
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data;
using System.Xml;
using System.Xml.Serialization;
using System.IO;
namespace ConsoleApplication50
{
class Program
{
static void Main(string[] args)
{
new Format();
}
}
public class Format
{
public enum TYPES
{
INT,
INT16,
LONG
}
public static List<Format> format = new List<Format>() {
new Format() { name = "AccountNumber", _type = TYPES.INT ,numberOfBytes = 4},
new Format() { name = "Age", _type = TYPES.INT16 ,numberOfBytes = 2},
new Format() { name = "AccountNumber", _type = TYPES.LONG ,numberOfBytes = 8}
};
public Dictionary<string, object> dict = new Dictionary<string, object>();
public string name { get; set; }
public TYPES _type { get; set; }
public int numberOfBytes { get; set; }
public Format() { }
public Format(byte[] contents)
{
MemoryStream stream = new MemoryStream(contents);
BinaryReader reader = new BinaryReader(stream);
foreach (Format item in format)
{
switch (item._type)
{
case TYPES.INT16 :
dict.Add(item.name, reader.ReadInt16());
break;
case TYPES.INT:
dict.Add(item.name, reader.ReadInt32());
break;
case TYPES.LONG:
dict.Add(item.name, reader.ReadInt64());
break;
}
}
}
}
}

clrobj(<class name>) does not have llvm when passing array of struct to GPU Kernel (ALEA Library)

I am getting the "Fody/Alea.CUDA: clrobj(cGPU) does not have llvm" build error for a code in which I try to pass an array of struct to the NVIDIA Kernel using ALEA library. Here is a simplified version of my code. I removed the output gathering functionality in order to keep the code simple. I just need to be able to send the array of struct to the GPU for the moment.
using Alea.CUDA;
using Alea.CUDA.Utilities;
using Alea.CUDA.IL;
namespace GPUProgramming
{
public class cGPU
{
public int Slice;
public float FloatValue;
}
[AOTCompile(AOTOnly = true)]
public class TestModule : ILGPUModule
{
public TestModule(GPUModuleTarget target) : base(target)
{
}
const int blockSize = 64;
[Kernel]
public void Kernel2(deviceptr<cGPU> Data, int n)
{
var start = blockIdx.x * blockDim.x + threadIdx.x;
int ind = threadIdx.x;
var sharedSlice = __shared__.Array<int>(64);
var sharedFloatValue = __shared__.Array<float>(64);
if (ind < n && start < n)
{
sharedSlice[ind] = Data[start].Slice;
sharedFloatValue[ind] = Data[start].FloatValue;
Intrinsic.__syncthreads();
}
}
public void Test2(deviceptr<cGPU> Data, int n, int NumOfBlocks)
{
var GridDim = new dim3(NumOfBlocks, 1);
var BlockDim = new dim3(64, 1);
try
{
var lp = new LaunchParam(GridDim, BlockDim);
GPULaunch(Kernel2, lp, Data, n);
}
catch (CUDAInterop.CUDAException x)
{
var code = x.Data0;
Console.WriteLine("ErrorCode = {0}", code);
}
}
public void Test2(cGPU[] Data)
{
int NumOfBlocks = Common.divup(Data.Length, blockSize);
using (var d_Slice = GPUWorker.Malloc(Data))
{
try
{
Test_Kernel2(d_Slice.Ptr, Data.Length, NumOfBlocks);
}
catch (CUDAInterop.CUDAException x)
{
var code = x.Data0;
Console.WriteLine("ErrorCode = {0}", x.Data0);
}
}
}
}
}

Your data is class, which is reference type. Try use struct. Reference type doesn't fit Gpu well, since it require of allocating small memory on the heap.

3rd party Pdf library significantly slower when running NUnit

I am evaluating Winnovative's PdfToText library and have run into something that concerns me.
Everything runs fine and I am able to extract the text content from a small 20k or less pdf immediately if I am running a console application. However, if I call the same code from the NUnit gui running it takes 15-25 seconds (I've verified it's PdfToText by putting a breakpoint on the line that extracts the text and hitting F10 to see how long it takes to advance to the next line).
This concerns me because I'm not sure where to lay blame since I don't know the cause. Is there a problem with NUnit or PdfToText? All I want to do is extract the text from a pdf, but 20 seconds is completely unreasonable if I'm going to see this behavior under certain conditions. If it's just when running NUnit, that's acceptable, but otherwise I'll have to look elsewhere.
It's easier to demonstrate the problem using a complete VS Solution (2010), so here's the link to make it easier to setup and run (no need to download NUnit or PdfToText or even a sample pdf):
http://dl.dropbox.com/u/273037/PdfToTextProblem.zip (You may have to change the reference to PdfToText to use the x86 dll if you're running on a 32-bit machine).
Just hit F5 and the NUnit Gui runner will load.
I'm not tied to this library, if you have suggestions, I've tried iTextSharp (way too expensive for 2 lines of code), and looked at Aspose (I didn't try it, but the SaaS license is $11k). But they either lack the required functionality or are way too expensive.

(comment turned into answer)
How complex are your PDFs? The 4.1.6 version of iText allows for a closed sourced solution. Although 4.1.6 doesn't directly have a text extractor it isn't too terribly hard to write one using the PdfReader and GetPageContent().

Below is the code I used to extract the text from the PDF using iTextSharp v4.1.6. If it seems overly verbose, it's related to how I'm using it and the flexibility required.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using iTextSharp.text.pdf;
namespace ClassLibrary1
{
public class PdfToken
{
private PdfToken(int type, string value)
{
Type = type;
Value = value;
}
public static PdfToken Create(PRTokeniser tokenizer)
{
return new PdfToken(tokenizer.TokenType, tokenizer.StringValue);
}
public int Type { get; private set; }
public string Value { get; private set; }
public bool IsOperand
{
get
{
return Type == PRTokeniser.TK_OTHER;
}
}
}
public class PdfOperation
{
public PdfOperation(PdfToken operationToken, IEnumerable<PdfToken> arguments)
{
Name = operationToken.Value;
Arguments = arguments;
}
public string Name { get; private set; }
public IEnumerable<PdfToken> Arguments { get; private set; }
}
public interface IPdfParsingStrategy
{
void Execute(PdfOperation op);
}
public class PlainTextParsingStrategy : IPdfParsingStrategy
{
StringBuilder text = new StringBuilder();
public PlainTextParsingStrategy()
{
}
public String GetText()
{
return text.ToString();
}
#region IPdfParsingStrategy Members
public void Execute(PdfOperation op)
{
// see Adobe PDF specs for additional operations
switch (op.Name)
{
case "TJ":
PrintText(op);
break;
case "Tm":
SetMatrix(op);
break;
case "Tf":
SetFont(op);
break;
case "S":
PrintSection(op);
break;
case "G":
case "g":
case "rg":
SetColor(op);
break;
}
}
#endregion
bool newSection = false;
private void PrintSection(PdfOperation op)
{
text.AppendLine("------------------------------------------------------------");
newSection = true;
}
private void PrintNewline(PdfOperation op)
{
text.AppendLine();
}
private void PrintText(PdfOperation op)
{
if (newSection)
{
newSection = false;
StringBuilder header = new StringBuilder();
PrintText(op, header);
}
PrintText(op, text);
}
private static void PrintText(PdfOperation op, StringBuilder text)
{
foreach (PdfToken t in op.Arguments)
{
switch (t.Type)
{
case PRTokeniser.TK_STRING:
text.Append(t.Value);
break;
case PRTokeniser.TK_NUMBER:
text.Append(" ");
break;
}
}
}
String lastFont = String.Empty;
String lastFontSize = String.Empty;
private void SetFont(PdfOperation op)
{
var args = op.Arguments.ToList();
string font = args[0].Value;
string size = args[1].Value;
//if (font != lastFont || size != lastFontSize)
// text.AppendLine();
lastFont = font;
lastFontSize = size;
}
String lastX = String.Empty;
String lastY = String.Empty;
private void SetMatrix(PdfOperation op)
{
var args = op.Arguments.ToList();
string x = args[4].Value;
string y = args[5].Value;
if (lastY != y)
text.AppendLine();
else if (lastX != x)
text.Append(" ");
lastX = x;
lastY = y;
}
String lastColor = String.Empty;
private void SetColor(PdfOperation op)
{
lastColor = PrintCommand(op).Replace(" ", "_");
}
private static string PrintCommand(PdfOperation op)
{
StringBuilder text = new StringBuilder();
foreach (PdfToken t in op.Arguments)
text.AppendFormat("{0} ", t.Value);
text.Append(op.Name);
return text.ToString();
}
}
}
And here's how I call it:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using iTextSharp.text.pdf;
namespace ClassLibrary1
{
public class PdfExtractor
{
public static string GetText(byte[] pdfBuffer)
{
PlainTextParsingStrategy strategy = new PlainTextParsingStrategy();
ParsePdf(pdfBuffer, strategy);
return strategy.GetText();
}
private static void ParsePdf(byte[] pdf, IPdfParsingStrategy strategy)
{
PdfReader reader = new PdfReader(pdf);
for (int i = 1; i <= reader.NumberOfPages; i++)
{
byte[] page = reader.GetPageContent(i);
if (page != null)
{
PRTokeniser tokenizer = new PRTokeniser(page);
List<PdfToken> parameters = new List<PdfToken>();
while (tokenizer.NextToken())
{
var token = PdfToken.Create(tokenizer);
if (token.IsOperand)
{
strategy.Execute(new PdfOperation(token, parameters));
parameters.Clear();
}
else
{
parameters.Add(token);
}
}
}
}
}
}
}

XML Serialization of List growing too large

I have a C# Windows forms application that runs a Trivia game on an IRC channel, and keeps the questions it asks, and the Leaderboard (scores) in Classes that I serialize to XML to save between sessions. The issue I have been having is best described with the flow, so here it is:
User X Gets entry in Leaderboard class with a score of 1. Class is saved to XML, XML contains one entry for user X.
User Y gets entry in Leaderboard class with a score of 1. Class is saved to XML, XML contains duplicate entries for User X, and one entry for User Y.
After running it for a week with under 20 users, I hoped to be able to write a web backend in PHP to help me use the scores. XML file is 2 megabytes.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Runtime.Serialization;
namespace IRCTriviaBot
{
[Serializable()]
public class LeaderBoard
{
[Serializable()]
public class Pair
{
public string user;
public int score;
public Pair(string usr, int scr)
{
user = usr;
score = scr;
}
public Pair() { }
}
private static List<Pair> pairs = null;
public List<Pair> Pairs
{
get
{
if (pairs==null)
{
pairs = new List<Pair>();
}
return pairs;
}
}
public LeaderBoard()
{
}
public void newScore(string usr)
{
bool found = false;
for (int i = 0; i < Pairs.Count && !found; ++i)
{
if (Pairs[i].user==usr)
{
found = true;
Pairs[i].score++;
}
}
if (!found)
{
Pairs.Add(new Pair(usr, 1));
}
}
public int getScore(string usr)
{
bool found = false;
for (int i = 0; i < Pairs.Count && !found; ++i)
{
if (Pairs[i].user == usr)
{
return Pairs[i].score;
}
}
if (!found)
{
return 0;
}
return 0;
}
}
}
Here's where the serialization and deserialization happens.
void parseMessage(string message, string user = "")
{
if (message == "-startgame-")
{
if (!gameStarted)
{
gameStarted = true;
openScores();
startGame();
}
}
else if (message == "-hint-")
{
if (!hintGiven && gameStarted)
{
sendMessage("Here's a better hint: " + Form2.qa.Answers[curQ].Trim());
hintGiven = true;
}
}
else if (message == "-myscore-")
{
sendMessage(user + ", your score is: " + leaderB.getScore(user));
}
else if (message.ToLower() == Form2.qa.Answers[curQ].ToLower())
{
if (gameStarted)
{
sendMessage(user + " got it right! Virtual pat on the back!");
leaderB.newScore(user);
saveScores();
System.Threading.Thread.Sleep(2000);
startGame();
}
}
else if (message == "-quit-")
{
if (gameStarted)
{
sendMessage("Sorry to see you go! Have fun without me :'(");
gameStarted = false;
}
else
{
sendMessage("A game is not running.");
}
}
else
{
if (gameStarted)
{
//sendMessage("Wrong.");
}
}
}
void saveScores()
{
//Opens a file and serializes the object into it in binary format.
Stream stream = System.IO.File.Open("scores.xml", FileMode.Open);
XmlSerializer xmlserializer = new XmlSerializer(typeof(LeaderBoard));
//BinaryFormatter formatter = new BinaryFormatter();
xmlserializer.Serialize(stream, leaderB);
stream.Close();
}
void openScores()
{
Stream stream = System.IO.File.OpenRead("scores.xml");
XmlSerializer xmlserializer = new XmlSerializer(typeof(LeaderBoard));
//BinaryFormatter formatter = new BinaryFormatter();
leaderB = (LeaderBoard)xmlserializer.Deserialize(stream);
stream.Close();
}

I think this has to do with pairs being marked static. I don't believe the XmlSerializer will clear a list before adding elements to it, so every time you call openScores() you will create duplicate entries rather than overwrite existing ones.
In general, I've observed that serialization and global variables don't play well together. For this purpose, "global variables" includes private statics, singletons, monostate classes like this, and thread-local variables.
It also looks like there's some waffle here between using XML and binary serialization. They are completely different beasts. XML serialization looks only at a class's public properties, while binary serialization looks only at a class's instance fields. Also, XML serialization ignores the Serializable attribute.

vhost.exe crashes when trying to make waveout work in C#

I tried to use waveout in C# to play more wav files simultaneously (at least the different ones). (SoundPlayer object only plays one in a time, and I don't want to use DirectSound and MediaPlayer objects or any other advenced technologies for this simple goal.)
I ported a working C++ code to C# and it works (after a research how to marshal and how to call native win32 dll-s and how to allocate and lock unmanaged memory) but it somehow make the vhost.exe crash and I have no clue why it does this. (It does not throw any exception only the standard windows error dialog appears and the program crashes and exit.)
Does anyone have any ideas?
Here is the source of that class:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using XiVo.PlayThrough;
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Timers;
namespace RTS
{
class WaveObject
{
protected IntPtr wo;
protected byte[] Adat=null;
protected WaveNative.WaveHdr woh;
protected int pgc;
public class NotAWaveformFileException : Exception
{
public NotAWaveformFileException(string str) : base(str) { }
public NotAWaveformFileException() { }
}
public class WaveFileIsCorruptedException : Exception
{
public WaveFileIsCorruptedException(string str) : base(str) { }
public WaveFileIsCorruptedException() { }
}
public class WaveMapperCouldNotBeOpenedException : Exception
{
public WaveMapperCouldNotBeOpenedException(string str) : base(str) { }
public WaveMapperCouldNotBeOpenedException() { }
}
public WaveObject() {}
public WaveObject(string Fn) : this() { Load(Fn); }
public void Load(string Fn)
{
IntPtr mmio;
NativeMMIO.mmckInfo Main, Sub;
WaveFormat wfx;
StringBuilder str=new StringBuilder(Fn);
int r;
mmio = NativeMMIO.mmioOpen(str,IntPtr.Zero,NativeMMIO.MMIO_READ);
if (mmio == IntPtr.Zero)
{
throw new FileNotFoundException(Fn + "not found!");
}
Main.fccType = NativeMMIO.mmioFOURCC('W', 'A', 'V', 'E');
if (NativeMMIO.mmioDescend(mmio, out Main, IntPtr.Zero, NativeMMIO.MMIO_FINDRIFF) != 0)
{
throw new NotAWaveformFileException();
}
Sub.ckid = NativeMMIO.mmioFOURCC('f', 'm', 't', ' ');
if (NativeMMIO.mmioDescend(mmio, out Sub, out Main, NativeMMIO.MMIO_FINDCHUNK) != 0)
{
throw new WaveFileIsCorruptedException("fmt chunk is not found!");
}
byte[] raw = new byte[Sub.cksize+2];
NativeMMIO.mmioRead(mmio, raw, (int)Sub.cksize);
GCHandle conv = GCHandle.Alloc(raw, GCHandleType.Pinned); // mapping a WaveFormat structure from the byte array
wfx = (WaveFormat)Marshal.PtrToStructure(conv.AddrOfPinnedObject(), typeof(WaveFormat));
conv.Free();
Sub.ckid = NativeMMIO.mmioFOURCC('d', 'a', 't', 'a');
if (NativeMMIO.mmioDescend(mmio, out Sub, out Main, NativeMMIO.MMIO_FINDCHUNK) != 0)
{
throw new WaveFileIsCorruptedException("data chunk is not found!");
}
Adat = new byte[Sub.cksize+2];
NativeMMIO.mmioRead(mmio, Adat, (int)Sub.cksize);
NativeMMIO.mmioClose(mmio, 0);
wfx.cbSize = (short)Marshal.SizeOf(wfx);
unchecked // WAVE_MAPPER is 0xFFFFFFFF and it does not let it convert to int otherwise
{
int res = WaveNative.waveOutOpen(out wo, (int)WaveNative.WAVE_MAPPER, wfx, null, 0, (int)WaveNative.CALLBACK_NULL);
if (res != WaveNative.MMSYSERR_NOERROR)
{
throw new WaveMapperCouldNotBeOpenedException();
}
}
woh.lpData = Marshal.AllocHGlobal((int)Sub.cksize); // alloc memory for the buffer
Marshal.Copy(Adat, 0, woh.lpData, (int)Sub.cksize);
woh.dwBufferLength = (int)Sub.cksize;
woh.dwBytesRecorded = 0;
woh.dwUser = IntPtr.Zero;
woh.dwFlags = 0;
woh.dwLoops = 1000000;
woh.reserved = 0;
woh.lpNext = IntPtr.Zero;
r = WaveNative.waveOutPrepareHeader(wo, ref woh, Marshal.SizeOf(woh));
pgc = System.Environment.TickCount;
}
public void Play()
{
if (System.Environment.TickCount - pgc > 50)
{
if (wo == null) throw new Exception("wo somehow became null.");
int res = WaveNative.waveOutReset(wo);
if (res != WaveNative.MMSYSERR_NOERROR) throw new Exception(string.Format("waveOutReset {0}",res));
res=WaveNative.waveOutWrite(wo, ref woh, Marshal.SizeOf(woh));
if ((res != WaveNative.MMSYSERR_NOERROR) && (res!=33)) throw new Exception(string.Format("waveOutWrite {0}",res));
}
}
~WaveObject()
{
Marshal.FreeHGlobal(woh.lpData); // release memory
}
}
}

After a lot of googling I realized that the bug is in the winmm.dll itself. Under some circumstances its helper thread crashes and bring down the host application too.
citation