We Keep Coding

How do I access the data in a Avro.snz file with C#

I have an Avro.snz file whose
avro.codecs is snappy
This can be opened with com.databricks.avro in Spark but it seems snappy is unsupported by Apache.Avro and Confluent.Avro, they only have deflate and null. Although they can get me the Schema, I cannot get at the data.
The next method gets and error. Ironsnappy is unable to decompress the file too, it says the input is
using (Avro.File.IFileReader<generic> reader = Avro.File.DataFileReader<generic>.OpenReader(avro_path))
{
schema = reader.GetSchema();
Console.WriteLine(reader.HasNext()); //true
var hi = reader.Next(); // error
Console.WriteLine(hi.ElementAt(0).ToString()); // error
}
I'm starting to wonder if there is anything in the Azure HDInsight library, but I cant seem to find the nuget package that gives me a way to read Avro with support for Snappy compression.
I'm open to any solution, even if that means downloading the source for Apache.Avro and adding in Snappy support manually, but to be honest, I'm sort of a newbie and have no idea how compression even works let alone add support to a library.
Can anyone help?
Update:
Just adding the snappy codec to Apache.Avro and changing the DeflateStream to Ironsnappy stream failed. It gave Corrupt input again. Is there anything anywhere that can open Snappy compressed Avro files with C#?
Or how do I determine what part of the Avro is snappy compressed and pass that to Ironsnappy.

Ok, so not even any comments on this. But I eventually solved my problem. Here is how I solved it.
I tried Apache.Avro and Confluent version as well, but their .net version has no snappy support darn. But I can get the schema as that is uncompressed apparently.
Since Parquet.Net uses IronSnappy, I built/added out the snappy codec in Apache.Avro by basically cloning its deflate code and changing a few names. Failed. Corrupt input Ironsnappy says.
I research Avro and see that it is seperated by an uncompressed Schema, followed by the name of the compression codec of the data, then the data itself, which are divided into blocks. Well, I have no idea where a block starts and ends. Somehow the binary in the file gives that info somehow, but I still have no idea, I couldn't get it with a hex editor even. I think Apache.Avro takes a long or a varint somehow, and the hex editor I used doesn't give me that info.
I found the avro-tools.jar tool inside Apache.Avro. To make it easier to use, I made it an executable with launch4j totally superfluous move but whatever. Then I used that cat my avro into 1 row, uncompressed and snappy. I used that as my base and followed the flow of Apache.Avro in the debugger. While also tracking the index of bytes and such with the hex editor and the debugger in C#.
With 1 row, it is guaranteed 1 block. So I ran a loop on the byte start index and end index. I found my Snappy block and was able to decompress it with IronSnappy. I modified the codec portion of my Apache.Avro snappy codec code to make it work with 1 block. (which was basically whatever block Apache.Avro took minus 4 bytes which I assume is the Snappy CRC check which I ignored.
It fails with multi blocks. I found its because Apache.Avro always throws the deflate codec a 4096 byte array after the first block. I reduced it to read size and did the minus 4 size thing again. It worked.
Success! So basically it was copy over deflate as a template for snappy, reduce block byte by 4, then make sure to resize the byte array to block byte size before getting Ironsnappy to decompress.
public override byte[] Decompress(byte[] compressedData)
{
int snappySize = compressedData.Length - 4;
byte[] compressedSnappy_Data = new byte[snappySize];
System.Array.Copy(compressedData, compressedSnappy_Data, snappySize);
byte[] result = IronSnappy.Snappy.Decode(compressedSnappy_Data);
return result;
}
if (_codec.GetHashCode() == DataFileConstants.SnappyCodecHash)
{
byte[] snappyBlock = new byte[(int)_currentBlock.BlockSize];
System.Array.Copy(_currentBlock.Data, snappyBlock, (int)_currentBlock.BlockSize);
_currentBlock.Data = snappyBlock;
}
I didn't bother with actually using the checksum as I don't know how or need to? At least not right now. And I totally ignored the compress function.
but if you really want my compress function here it is
public override byte[] Compress(byte[] uncompressedData)
{
return new byte[0];
}

The simplest solution would be to use:
ResultModel resultObject = AvroConvert.Deserialize<ResultModel>(byte[] avroObject);
From https://github.com/AdrianStrugala/AvroConvert
null
deflate
snappy
gzip
codes are supported

Appending bytes to an existing file C#

I am working on a steganography software in C#, more precisely for video files. My approach is to append the extra information at the end of a video file. However, I must read the whole video file in memory first. I used the File.ReadAllBytes() function in C# to read a video file (video around 200MB) into a byte array. Then I create a new array with the video's bytes and my data's bytes. But, this sometimes causes an OutOfMemoryException. And when it doesn't it is very slow. Is there a more efficient way to append bytes to an existing file in C# which will solve this issue? Thank you.

Open the file with FileMode.Append
var stream = new FileStream(path, FileMode.Append)
FileMode Enumeration
FileMode.Append:
Opens the file if it exists and seeks to the end of the file, or
creates a new file. This requires FileIOPermissionAccess.Append
permission. FileMode.Append can be used only in conjunction with
FileAccess.Write. Trying to seek to a position before the end of the
file throws an IOException exception, and any attempt to read fails
and throws a NotSupportedException exception.

Sure, it's easy:
using (var stream = File.Open(path, FileMode.Append))
{
stream.Write(extraData);
}
No need to read the file first.
I wouldn't class this as steganography though - that would involve making subtle changes to the video frames such that it's still a valid video and looks the same to the human eye, but the extra data is encoded within those frames so it can be extracted later.

attempt this method, I am unsure if it will yield faster results, but logically it should.
: https://stackoverflow.com/a/6862460/2835725

JpegBitmapDecoder decode jpeg failed?

I'm using the following code to Decode Jpge image
UnmanagedMemoryStream^ mStream = gcnew UnmanagedMemoryStream(_jpegDataBuff, _buffLength);
JpegBitmapDecoder^ decoder = gcnew JpegBitmapDecoder(mStream, BitmapCreateOptions::PreservePixelFormat, BitmapCacheOption::OnLoad);
BitmapSource^ bitmapSource = decoder->Frames[0];
I use the bitmapSource to display on control of WPF, but sometimes the the result as below
I have searched very much on internet, but I did not solve it yet!
Someone can help me?
Many Thanks,
T&T

From the looks of it I would say that either JPEG source is incomplete or invalid. The only exceptions JpegBitmapDecoder throws are ArgumentNullException (stream is null) and FileFormatException. The file format is most probably detected by analyzing file header. The JIF/JFIF file structure holds image description information in a relatively small header followed by a "raw" data.
I guess that decoder is built resilient to ignore invalid (rendered last before gray) and/or missing (rendered gray) blocks. I think the only way to detect is to either analyze final image (maybe checking that last 8x8px block doesn't have same color?) or source (maybe counting data blocks?).

.NET MessagingToolkit QRCode Encode Issue

I've been using the .NET MessagingToolkit.QRCode library for some time on my websites in order to generate QR codes with custom images on top and until now I have had no problems at all with this library.
Recently, the data I need to encode has gotten larger as a result of system growth and attempting to encode this data within a QR code has been causing internal "index out of range" exceptions within the Encode method of the QRCodeEncoder class.
Through testing I have noted that the maximum number of characters this method appears to accept is 86 and anything greater than this will throw the exception. Changing the encoding type of the string passed in does not help.
I did note that the version of the library I was using was out of date, however, updating to the latest version also did not solve the problem.
Has anyone else experienced this issue with the library and found a solution? Alternatively, is there another library I could use which does not have such problems?
The exception details are below:
2012-10-01 17:39:52,098 Error: System.IndexOutOfRangeException: Index was outside the bounds of the array.
at MessagingToolkit.QRCode.Codec.QRCodeEncoder.divideDataBy8Bits(Int32[] data, SByte[] bits, Int32 maxDataCodewords)
at MessagingToolkit.QRCode.Codec.QRCodeEncoder.calQrcode(Byte[] qrcodeData)
at MessagingToolkit.QRCode.Codec.QRCodeEncoder.Encode(String content, Encoding encoding)
at MessagingToolkit.QRCode.Codec.QRCodeEncoder.Encode(String content)

On the official page for that library there is a short description of solution for your problem: http://platform.twit88.com/news/60. I don't know if that would fix your problem though. I would find the source code of the library and debug through it to see the reason for the error.
hm... can't find the source code anywhere. So get in touch with the authors and ask them.
If that does not work out, I would change the library to something else. I have used QrCode.Net with MVC successfully and have some sample code how to integrate it to MVC, if you like.

I know this post is a little old but I ran into the same issue and discovered I had an illegal character in my string ( ” ) which the QR code did not recognize. Replaced this with a regular quote ( " ) and it worked fine.

I got same problem
QRCodeEncoder qrCE = new QRCodeEncoder();
qrCE.QRCodeEncodeMode = QRCodeEncoder.ENCODE_MODE.BYTE;
qrCE.QRCodeErrorCorrect = QRCodeEncoder.ERROR_CORRECTION.L;
qrCE.QRCodeVersion = 1;
picQRCode.Image = qrCE.Encode(memBarcodeDataForPrint.Text, System.Text.Encoding.UTF8);
I changed following line
qrCE.QRCodeVersion = 1;
like
qrCE.QRCodeVersion = 0;
It solved.

I experienced the same problem. However I found that creating a new instance of the encoder each time I called the Encode method solved the issue.

Same problem but when change version to 0 it working.

DotNetZip Creates corrupt archives (bad CRC)

There's a strange problem with DotNetZip that I can't seem to find a solution to.
I've searched for a few hours now and I just can't find anything on this, so here goes.
var ms = new MemoryStream();
using (var archive = new Ionic.Zip.ZipFile()) {
foreach (var file in files) {
// string byte[]
var entry = archive.AddEntry(file.Name, file.Data);
entry.ModifiedTime = DateTime.Now.AddYears(10); // Just for testing
}
archive.Save(ms);
}
return ms.GetBuffer();
I need to add the modified time, which is rather crucial, but right now I just have a dummy timestamp.
When I open the file with WinRAR, it says "Unexpected end of archive". Each individual file has checksum 00000000, and WinRAR says "The archive is either in unknown format or damaged". I can repair it, which brings it down 20% in size and makes everything OK. But that's not really useful..
When I make a breakpoint after adding all the entries, I can see in zip.Entries that all the entries have that same bad CRC, but all the data seems to be there.
So it shouldn't be the way I save the archive that's the problem.
I use my file collection elsewhere without problems, which adds to DotNetZip being weird. Well either that or I misunderstand something :)

GetBuffer is certainly wrong. It returns the internal buffer of the MemoryStream, which is often bigger than the actual content.
To return an array that only contains the actual content, use ToArray().
Or you could carefully handle the incompletely filled buffer in the consuming code. This would reduce GC pressure, since you don't need to allocate a whole new array for the return value.
If the zip-archive is large, I'd also consider saving to a file directly, instead of assembling the archive in-memory.