I found a C# source code that read from file which have 1000000 double number. The source code is below.
public void filereader()
{
using (BinaryReader b = new BinaryReader(File.Open("C:\\Users\\Hanieh\\Desktop\\nums.txt", FileMode.Open)))
{
int length = (int)b.BaseStream.Length;
byte[] fileBytes = b.ReadBytes(length);
for (int ii = 0; ii < fileBytes.Length - 32 ; ii++)
{
savg1[ii / 2] = (double)(BitConverter.ToInt16(fileBytes, ii) / 20000.0);// inja error index midee
ii++;
}
}
}
when I run source code to read from text file I have an error that related to savg1 index that is out of bound. I debug step by step and result shows size of length= 24000000 but savg1=1000000. my question is here: how this source code work and how I can fix this problem.
I suggest something like this (File.ReadAllBytes and BitConverter.ToDouble):
byte[] source = File.ReadAllBytes(#"C:\Users\Hanieh\Desktop\nums.txt");
double[] data = new double[source.Length / sizeof(double)];
for (int i = 0; i < data.Length; ++i)
data[i] = BitConverter.ToDouble(source, i * sizeof(double));
I would solve it like:
double[] data;
using (BinaryReader b = new BinaryReader(File.Open("C:\\Users\\Hanieh\\Desktop\\nums.txt", FileMode.Open)))
{
// create array/buffer for the doubles (filelength / bytes per double)
data = new double[b.BaseStream.Length / sizeof(double)];
// read the data from the binarystream
for (int i = 0; i < data.Length; i++)
data[i] = b.ReadDouble();
}
MessageBox.Show("doubles read: " + data.Length.ToString());
Although, your file nums.txt implies that it is a textfile. You might not read it as a binary file.
Related
I need to extract data from zip file that contains extra data.
When I opening it with 7-zip, it works fine, but there is warning in properties - "There are some data after the end of payload data"
But when I trying to unzip it with DotNetZip it gives me an error
using (var zip = ZipFile.Read("test.ef4"))
{
foreach (var zipEntry in zip)
{
Console.WriteLine(zipEntry.FileName);
var MemoryStream = new MemoryStream();
zipEntry.OpenReader().CopyTo(MemoryStream);
File.WriteAllBytes(zipEntry.FileName, MemoryStream.ToArray());
}
}
Exception:
Unhandled Exception: Ionic.Zip.ZipException: Cannot read that as a ZipFile
---> Ionic.Zip.BadReadException: Bad signature (0x0C000001) at position 0x00000060
How can I ignore this exception and unzip from file like 7-zip doing?
To reproduce it you can create archive with windows cmd.exe
Create archive.zip and extradata.txt with some random data. Then do this command -
copy /b archive.zip+extradata.txt
I found solution that worked for me:
All zip files have header like \x50\x4b\x01\x02 or \x50\x4b\x03\x04 in my case. (first \x50\x4b always the same. second half can be different)
So I wrote a code that founds offset of my ZIP file inside bytes. Footer of file is not important for any ziplib in C#
class Program
{
static void Main(string[] args)
{
var content = File.ReadAllBytes("qv.zip");
content = content.Skip(FirstPattern(content)).ToArray();
using (var zip = ZipFile.Read(new MemoryStream(content), new ReadOptions() {}))
{
foreach (var zipEntry in zip)
{
Console.WriteLine(zipEntry.FileName);
var MemoryStream = new MemoryStream();
zipEntry.OpenReader().CopyTo(MemoryStream);
File.WriteAllBytes(zipEntry.FileName, MemoryStream.ToArray());
}
}
}
private static int FirstPattern(byte[] bytes)
{
var header = StringToByteArray("504b0304"); // In my case all broken zip-files had 0304 ending
for (int i = 0; i < bytes.Length - header.Length; i++)
{
var j = 0;
while(j < header.Length && bytes[i + j] == header[j])
{
j++;
}
if(j == header.Length
return i;
}
return -1;
}
public static byte[] StringToByteArray(string hex)
{
int numberChars = hex.Length;
byte[] bytes = new byte[numberChars / 2];
for (int i = 0; i < numberChars; i += 2)
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
return bytes;
}
}
I'm using Digital Persona SDK to scan fingerprints in wsq format, for requeriment I need 512 x 512 image, the SDK only export 357 x 392 image.
The sdk provide a method to compress captured image from device in wsq format and return a byte array that I can write to disk.
-I've tried to allocate a buffer of 262144 for 512 x 512 image.
-Fill the new buffer with white pixel data each byte to value 255.
-Copy the original image buffer into the new image buffer. The original image doesn’t need to be centered but it's important to make sure to copy without corrupting the image data.
To summarize I've tried to copy the old image into the upper right corner of the new image.
DPUruNet.Compression.Start();
DPUruNet.Compression.SetWsqBitrate(95, 0);
Fid capturedImage = captureResult.Data;
//Fill the new buffer with white pixel data each byte to value 255.
byte[] bytesWSQ512 = new byte[262144];
for (int i = 0; i < bytesWSQ512.Length; i++)
{
bytesWSQ512[i] = 255;
}
//Compress capturedImage and get bytes (357 x 392)
byte[] bytesWSQ = DPUruNet.Compression.CompressRaw(capturedImage.Views[0].Width, capturedImage.Views[0].Height, 500, 8, capturedImage.Views[0].RawImage, CompressionAlgorithm.COMPRESSION_WSQ_NIST);
//Copy the original image buffer into the new image buffer
for (int i = 0; i < capturedImage.Views[0].Height; i++)
{
for (int j = 0; j < capturedImage.Views[0].Width; j++)
{
bytesWSQ512[i * bytesWSQ512.Length + j ] = bytesWSQ[i * capturedImage.Views[0].Width + j];
}
}
//Write bytes to disk
File.WriteAllBytes(#"C:\Users\Admin\Desktop\bytesWSQ512.wsq", bytesWSQ512);
DPUruNet.Compression.Finish();
When running that snippet I get IndexOutOfRangeException, I don't know if the loop or the calculation of indexes for new array are right.
Here is a representation of what I'm trying to do.
If someone is trying to achieve something like this or padding a raw image, I hope this will help.
DPUruNet.Compression.
DPUruNet.Compression.SetWsqBitrate(75, 0);
Fid ISOFid = captureResult.Data;
byte[] paddedImage = PadImage8BPP(captureResult.Data.Views[0].RawImage, captureResult.Data.Views[0].Width, captureResult.Data.Views[0].Height, 512, 512, 255);
byte[] bytesWSQ512 = Compression.CompressRaw(512, 512, 500, 8, paddedImage, CompressionAlgorithm.COMPRESSION_WSQ_NIST);
And the method to resize (pad) the image is:
public byte[] PadImage8BPP(byte[] original, int original_width, int original_height, int desired_width, int desired_height, byte pad_color)
{
byte[] canvas_8bpp = new byte[desired_width * desired_height];
for (int i = 0; i < canvas_8bpp.Length; i++)
canvas_8bpp[i] = pad_color; //Fill background. Note this type of fill will fail histogram checks.
int clamp_y_begin = 0;
int clamp_y_end = original_height;
int clamp_x_begin = 0;
int clamp_x_end = original_width;
int pad_y = 0;
int pad_x = 0;
if (original_height > desired_height)
{
int crop_distance = (int)Math.Ceiling((original_height - desired_height) / 2.0);
clamp_y_begin = crop_distance;
clamp_y_end = original_height - crop_distance;
}
else
{
pad_y = (desired_height - original_height) / 2;
}
if (original_width > desired_width)
{
int crop_distance = (int)Math.Ceiling((original_width - desired_width) / 2.0);
clamp_x_begin = crop_distance;
clamp_x_end = original_width - crop_distance;
}
else
{
pad_x = (desired_width - original_width) / 2;
}
//We traverse the captured image (either whole image or subset)
for (int y = clamp_y_begin; y < clamp_y_end; y++)
{
for (int x = clamp_x_begin; x < clamp_x_end; x++)
{
byte image_pixel = original[y * original_width + x];
canvas_8bpp[(pad_y + y - clamp_y_begin) * desired_width + pad_x + x - clamp_x_begin] = image_pixel;
}
}
return canvas_8bpp;
}
I'm trying to create a code to decompress an RLE Byte-Oriented image from a PostScript File I've already tried solutions found around the web and also tried to build my own ; but none of them produced the result i need.
After decompressing the rle image, i should have an RAW image i can open on photoshop (informing width, height and number of channels). However when i try to open the extracted image it doesn't work ; only a black output is show.
My inputs are an Binary ASCII Encoded file (encoded as a hexadecimal string) and a binary file ; both RLE Byte-Oriented compressed (in the hex file case, its just a question of converting it to bytes before trying the rle decompression).
https://drive.google.com/drive/u/0/folders/1Q476HB9SvOG_RDwK6J7PPycbw94zjPYU
I've posted samples here.
WorkingSample.raw -> Image Sample i got using another software, and its dimensions as well.
MySample.raw -> Image sample i built using my code, and its dimensions as well.
OriginalFile.ppf -> File containing the original image data and everything else.
ExtractedBinary.bin -> Only a binary portion from OriginalFile.ppf - makes it easier to read and work with the data.
This code was provided by the user nyerguds, he's part of the SO Community.
Original Source: http://www.shikadi.net/moddingwiki/RLE_Compression#Types_of_RLE
Its the one i tried to use but the results weren't correct. And to be honest i had difficulties understanding his code (he told me to change a few things in order to get it working for my case but i was unable to).
And here's what i tried to do following the PostScript Red Book:
Book: https://www.adobe.com/content/dam/acom/en/devnet/actionscript/articles/PLRM.pdf
The part:
"The RunLengthEncode filter encodes data in a simple-byte oriented format based on run length.
The compressed data format is a sequence of runs, where each run consists of a length byte followed by 1 to 128 bytes of data. If the length byte is in the range 0 to 127, the following length + 1 bytes (1 to 128 bytes) are to be copied literally upon decompression. If length is in the range of 129 to 255, the following single byte is to be replicated 257 - length times (2 to 128 times) upon decompression."
Page 142, RunLengthEncode Filter.
List<byte> final = new List<byte>();
var split01 = ArraySplit(bytefile, 2);
foreach (var binPart in split01)
{
try
{
if (binPart.ElementAt(0) <= 127)
{
int currLen = binPart[0] + 1;
for (int i = 0; i <= binPart[0]; i++)
{
final.Add(binPart[1]);
//Console.WriteLine(binPart[1]);
}
}
else if (binPart[0] >= 128)
{
int currLen = 257 - binPart[0];
for (int i = 0; i < currLen; i++)
{
final.Add(binPart[1]);
// Console.WriteLine(binPart[1]);
}
}
}
catch(Exception)
{
break;
}
}
File.WriteAllBytes(#"C:\test\again.raw", final.ToArray());
private static IEnumerable<byte[]> ArraySplit(byte[] bArray, int intBufforLengt)
{
int bArrayLenght = bArray.Length;
byte[] bReturn = null;
int i = 0;
for (; bArrayLenght > (i + 1) * intBufforLengt; i++)
{
bReturn = new byte[intBufforLengt];
Array.Copy(bArray, i * intBufforLengt, bReturn, 0, intBufforLengt);
yield return bReturn;
}
int intBufforLeft = bArrayLenght - i * intBufforLengt;
if (intBufforLeft > 0)
{
bReturn = new byte[intBufforLeft];
Array.Copy(bArray, i * intBufforLengt, bReturn, 0, intBufforLeft);
yield return bReturn;
}
}
private static byte[] StringToByteArray(String hex)
{
int iValue = 0;
int NumberChars = hex.Length;
if (NumberChars % 2 != 0)
{
string m = string.Empty;
}
byte[] bytes = new byte[NumberChars / 2];
try
{
for (int i = 0; i < NumberChars; i += 2)
{
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
iValue = i;
}
}
catch (Exception e)
{
var value = iValue;
Console.WriteLine(e.Message);
}
return bytes;
}
The desired output would be an TIFF Grayscale. However, i can deal with PNG''s also.
I've managed to extract uncompressed data from this kind of file already ; with Emgu(OpenCV Wrapper) i was able to create a viewable image and do my logic on it.
My actual results from RLE Compressed are only invalid RAW files that can't be viewed even on photoshop or IrfanViewer.
Any input is appreciated. Thanks.
EDIT1: stuck on this part
for(int i=0; i < bytefile.Length; i+=2)
{
try
{
var lengthByte = bytefile[i];
if (lengthByte <= 127)
{
int currLen = lengthByte + 1;
for (int j = 0; j < currLen; j++)
{
final.Add(bytefile[i]);
i++;
}
}
if (bytefile[i] >= 128)
{
int currLen = 257 - bytefile[i];
for (int k = 0; k < currLen; k++)
{
final.Add(bytefile[i + 1]);
}
}
}
catch(Exception)
{
break;
}
}
This is the logic i'm following. Before it was raising an Exception but i figured it out (it was because i forgot to add the ending byte ; makes no difference in the final result).
Try this basic outline:
int i = 0;
while (i < bytefile.length)
{
var lengthByte = bytefile[i++];
if (lengthByte <= 127)
{
int currLen = lengthByte + 1;
for (int j = 0; j < currLen; j++)
final.Add(bytefile[i++]);
}
else
{
int currLen = 257 - lengthByte;
byte byteToCopy = bytefile[i++];
for (int j = 0; j < currLen; j++)
final.Add(byteToCopy);
}
}
This is how I understand what's specified above, anyway.
Although not explicitly stated, I believe you are attempting to extract a RunLength Encoded image from a Postscript file and save that out as a grayscale TIFF.
As a starting point for something like this, have you tried simply saving out an uncompressed image from a Postscript file as a grayscale TIFF to ensure your application logic responsible for building up the TIFF image data indeed works as you expect it to? I'd caution that would a be a good first step before moving onto now supporting decompressing RLE data to then turn into a TIFF.
The reason I think that's important is because your problem may have nothing to do with how you're decompressing the RLE data but rather how you're creating your output TIFF from presumably correctly decoded data.
I have been experiencing some issues with imcompatiabilities between the difference implementations of zlib compression.
As a test case I thought to create test data with 10000 doubles ranging from 0 - 10000.
I created some test code to compress and decompress this data that uses the compress and uncompress in zlib.c
unsigned int Test(char* comparisonFile)
{
unsigned long partsSize = 0x80000;
const int arraySize = 10000;
Bytef doubleArray[sizeof(double) * arraySize];
Bytef outBuffer[sizeof(double) * arraySize];
for (int i = 0; i < arraySize; i++)
{
Bytef doubleBytes[sizeof(double)];
*(double*)(doubleBytes) = (double)i;
for (int x = 0; x < 8; x++)
doubleArray[(8 * i) + x] = doubleBytes[x];
}
compress(outBuffer, &partsSize, doubleArray, sizeof(double) * arraySize);
//create file of compressed data
char * filename = "zlibCompressed";
FILE * file = fopen(filename, "w+b");
int compressResult = et_int64(fwrite((char *)outBuffer, 1, size_t(partsSize), file));
fclose(file);
//load file of compressed data either from zlib or other
if (comparisonFile != NULL)
filename = comparisonFile;
FILE * compressedFile = fopen(filename, "r+b");
if (compressedFile == NULL)
return -1;
unsigned long outBufferSize = sizeof(double) * arraySize;
fseek(compressedFile, 0, SEEK_END);
long fsize = ftell(compressedFile);
fseek(compressedFile, 0, SEEK_SET); /* same as rewind(f); */
partsSize = int(fsize);
double * doubleResult = new double [arraySize];
Bytef* inBuffer = (Bytef*)malloc(sizeof(Bytef)*partsSize);
int readresult = et_int64(fread((char *)inBuffer, 1, partsSize , compressedFile));
if (readresult != partsSize)
return -1;
Bytef * uncompressedOutBuffer = static_cast<Bytef*>((void *)doubleResult);
int result = uncompress(uncompressedOutBuffer, &outBufferSize, inBuffer, partsSize);
for (int i = 0; i < arraySize; i++)
{
// uncompressed data does not match expectation
if ((int)doubleResult[i] != i)
return -2;
}
fclose(compressedFile);
return 0;
}
This allows me to test the internal compression and also substitute compression results from C#.
However, when I use ionic or standard deflate in the following manner I can only recover about 8150 of the expected 10000.
The uncompression returns a:
Z_DATA_ERROR
Given that these seem in theory interoperable, i'm not sure why C# compression results can only be partially unpacked with Adler's zlib? Any help out there?
public void ZlibTest()
{
byte[] buffer;
using (var ms = new MemoryStream())
{
for (int i = 0; i < 10000; i++)
ms.Write(BitConverter.GetBytes((double) i), 0, sizeof(double));
buffer = ms.ToArray();
}
var file = "dummy1";
if (File.Exists(file))
File.Delete(file);
using (Stream fs = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
using (var resultStream = new MemoryStream())
{
using (var compressionStream2 = new Ionic.Zlib.ZlibStream(resultStream, Ionic.Zlib.CompressionMode.Compress, CompressionLevel.Default))
{
compressionStream2.Write(buffer, 0, buffer.Length);
var packetLength = (int)resultStream.Length;
fs.Write(resultStream.ToArray(), 0, packetLength);
}
}
}
file = "dummy2";
using (Stream fs = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
using (var resultStream = new MemoryStream())
{
using (var compressionStream2 = new System.IO.Compression.DeflateStream(resultStream, System.IO.Compression.CompressionMode.Compress))
{
compressionStream2.Write(buffer, 0, buffer.Length);
var packetLength = (int)resultStream.Length;
fs.Write(BitConverter.GetBytes((ushort)40056), 0, sizeof(ushort));
fs.Write(resultStream.ToArray(), 0, packetLength);
}
}
}
}
They're both using the same zlib. There must be an error in your C# code. It is possible that you are not reading the file in binary mode in C#, which would cause occasional corruption.
I want to add some string in the middle of image metadata block. Under some specific marker. I have to do it on bytes level since .NET has no support for custom metadata fields.
The block is built like 1C 02 XX YY YY ZZ ZZ ZZ ... where XX is the ID of the field I need to append and YY YY is the size of it, ZZ = data.
I imagine it should be more or less possible to read all the image data up to this marker (1C 02 XX) then increase the size bytes (YY YY), add data at the end of ZZ and then add the rest of the original file? Is this correct?
How should I go on with it? It needs to work as fast as possible with 4-5 MB JPEG files.
In general there is no way to speed up this operation. You have to read at least portion that needs to be moved and write it again in updated file. Creating new file and copying content to it may be faster if you can parallelize read and write operations.
Note: In you particular case it may not be possible to just insert content in the middle of the file as most of file formats are not designed with such modifcations in mind. Often there are offsets to portions of the file that will be invalid when you shift part of the file. Specifying what file format you trying to work with may help other people to provide better approaches.
Solved the problem with this code:
List<byte> dataNew = new List<byte>();
byte[] data = File.ReadAllBytes(jpegFilePath);
int j = 0;
for (int i = 1; i < data.Length; i++)
{
if (data[i - 1] == (byte)0x1C) // 1C IPTC
{
if (data[i] == (byte)0x02) // 02 IPTC
{
if (data[i + 1] == (byte)fileByte) // IPTC field_number, i.e. 0x78 = IPTC_120
{
j = i;
break;
}
}
}
}
for (int i = 0; i < j + 2; i++) // add data from file before this field
dataNew.Add(data[i]);
int countOld = (data[j + 2] & 255) << 8 | (data[j + 3] & 255); // curr field length
int countNew = valueToAdd.Length; // new string length
int newfullSize = countOld + countNew; // sum
byte[] newSize = BitConverter.GetBytes((Int16)newfullSize); // Int16 on 2 bytes (to use 2 bytes as size)
Array.Reverse(newSize); // changes order 10 00 to 00 10
for (int i = 0; i < newSize.Length; i++) // add changed size
dataNew.Add(newSize[i]);
for (int i = j + 4; i < j + 4 + countOld; i++) // add old field value
dataNew.Add(data[i]);
byte[] newString = ASCIIEncoding.ASCII.GetBytes(valueToAdd);
for (int i = 0; i < newString.Length; i++) // append with new field value
dataNew.Add(newString[i]);
for (int i = j + 4 + newfullSize; i < data.Length; i++) // add rest of the file
dataNew.Add(data[i]);
byte[] finalArray = dataNew.ToArray();
File.WriteAllBytes(Path.Combine(Path.GetDirectoryName(jpegFilePath), "newfile.jpg"), finalArray);
Here is an easy and quite fast solution. It moves all bytes after given offset to their new position according to given extraBytes, so you can insert your data.
public void ExpandFile(FileStream stream, long offset, int extraBytes)
{
// http://stackoverflow.com/questions/3033771/file-io-with-streams-best-memory-buffer-size
const int SIZE = 4096;
var buffer = new byte[SIZE];
var length = stream.Length;
// Expand file
stream.SetLength(length + extraBytes);
var pos = length;
int to_read;
while (pos > offset)
{
to_read = pos - SIZE >= offset ? SIZE : (int)(pos - offset);
pos -= to_read;
stream.Position = pos;
stream.Read(buffer, 0, to_read);
stream.Position = pos + extraBytes;
stream.Write(buffer, 0, to_read);
}
Need to be checked, though...