We Keep Coding

C# - Read bytes from file from a specific string

I'm trying to parse a crg-file in C#. The file is mixed with plain text and binary data. The first section of the file contains plain text while the rest of the file is binary (lots of floats), here's an example:
$
$ROAD_CRG
reference_line_start_u = 100
reference_line_end_u = 120
$
$KD_DEFINITION
#:KRBI
U:reference line u,m,730.000,0.010
D:reference line phi,rad
D:long section 1,m
D:long section 2,m
D:long section 3,m
...
$
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
�#z����RA����\�l
...
I know I can read bytes starting at a specific offset but how do I find out which byte to start from? The last row before the binary section will always contain at least four dollar signs "$$$$". Here's what I've got so far:
using var fs = new FileStream(#"crg_sample.crg", FileMode.Open, FileAccess.Read);
var startByte = ??; // How to find out where to start?
using (BinaryReader reader = new BinaryReader(fs))
{
reader.BaseStream.Seek(startByte, SeekOrigin.Begin);
var f = reader.ReadSingle();
Debug.WriteLine(f);
}

When you have a mixture of text data and binary data, you need to treat everything as binary. This means you should be using raw Stream access, or something similar, and using binary APIs to look through the text data (often looking for cr/lf/crlf at bytes as sentinels, although it sounds like in your case you could just look for the $$$$ using binary APIs, then decode the entire block before, and scan forwards). When you think you have an entire line, then you can use Encoding to parse each line - the most convenient API being encoding.GetString(). When you've finished looking through the text data as binary, then you can continue parsing the binary data, again using the binary API. I would usually recommend against BinaryReader here too, because frankly it doesn't gain you much over more direct API. The other problem you might want to think about is CPU endianness, but assuming that isn't a problem: BitConverter.ToSingle() may be your friend.
If the data is modest in size, you may find it easiest to use byte[] for the data; either via File.ReadAllBytes, or by renting an oversized byte[] from the array-pool, and loading it from a FileStream. The Stream API is awkward for this kind of scenario, because once you've looked at data: it has gone - so you need to maintain your own back-buffers. The pipelines API is ideal for this, when dealing with large data, but is an advanced topic.

UPDATE: This code may not work as expected. Please review the valuable information in the comments.
using (var fs = new FileStream(#"crg_sample.crg", FileMode.Open, FileAccess.Read))
{
using (StreamReader sr = new StreamReader(fs, Encoding.ASCII, true, 1, true))
{
var line = sr.ReadLine();
while (!string.IsNullOrWhiteSpace(line) && !line.Contains("$$$$"))
{
line = sr.ReadLine();
}
}
using (BinaryReader reader = new BinaryReader(fs))
{
// TODO: Start reading the binary data
}
}

Solution
I know this is far from the most optimized solution but in my case it did the trick and since the plain text section of the file was known to be fairly small this didn't cause any noticable performance issues. Here's the code:
using var fileStream = new FileStream(#"crg_sample.crg", FileMode.Open, FileAccess.Read);
using var reader = new BinaryReader(fileStream);
var newLine = '\n';
var markerString = "$$$$";
var currentString = "";
var foundMarker = false;
var foundNewLine = false;
while (!foundNewLine)
{
var c = reader.ReadChar();
if (!foundMarker)
{
currentString += c;
if (currentString.Length > markerString.Length)
currentString = currentString.Substring(1);
if (currentString == markerString)
foundMarker = true;
}
else
{
if (c == newLine)
foundNewLine = true;
}
}
if (foundNewLine)
{
// Read binary
}
Note: If you're dealing with larger or more complex files you should probably take a look at Mark Gravell's answer and the comment sections.

Ruby ASCII-8BIT encoding equivalent in c#

I am trying to convert the following code from ruby to C#:
open(file_path, ’rb’) do |doc|
response = RestClient::Request.execute :method => :post,
:url => "http://something.com",
:payload => {
:document => doc
}
my understanding is this will read a file in binary in ASCII-8BIT as per the following documentation:
"b" Binary file mode
Suppresses EOL <-> CRLF conversion on Windows. And
sets external encoding to ASCII-8BIT unless explicitly
specified.
So trying to replicate this example code in C# I have written:
string myString;
Encoding enc = Encoding.GetEncoding(????);
using (FileStream fs = new FileStream("D:\\sample1page.pdf", FileMode.Open))
using (BinaryReader br = new BinaryReader(fs, enc))
{
byte[] bin = br.ReadBytes(System.Convert.ToInt32(fs.Length));
myString = enc.GetString(bin);
}
So the question is what encoding number do I need to do markerd by ???? is to do the equivalent ASCII-8BIT that works in ruby

How to convert NiceLabel files to text in c#

I'm trying to convert thousands of .lbl files to text to extract just the data I need to populate a database for a much cheaper, easier method of printing labels. However, I've tried many different approaches to convert the .lbl file to a text file:
see this question
and this question
But I just have had zero luck. Here's the code:
using (BinaryReader b = new BinaryReader(File.Open(#"stupid.lbl", FileMode.Open)))
{
int length = (int)b.BaseStream.Length;
byte[] allData = b.ReadBytes(length);
using (BinaryWriter w = new BinaryWriter(File.Open(#"test.txt", FileMode.Create)))
{
Encoding encEncoder = Encoding.ASCII;
string str = encEncoder.GetString(allData);
w.Write(str);
}
}
I do have the proper file paths and whatnot to be able to actually read/write and everything opens, but it's all gibberish...I just need to get the real, useful, human readable text out of it to push into an access db...Any ideas out there?

compressing a string in C# and uncompressing in python

I am trying to compress a large string on a client program in C# (.net 4) and send it to a server (django, python 2.7) using a PUT request.
Ideally I want to use the standard library at both ends, so I am trying to use gzip.
My C# code is:
public static string Compress(string s) {
var bytes = Encoding.Unicode.GetBytes(s);
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream()) {
using (var gs = new GZipStream(mso, CompressionMode.Compress)) {
msi.CopyTo(gs);
}
return Convert.ToBase64String(mso.ToArray());
}
}
The python code is:
s = base64.standard_b64decode(request)
buff = cStringIO.StringIO(s)
with gzip.GzipFile(fileobj=buff) as gz:
decompressed_data = gz.read()
It's almost working, but the output is: {▯"▯c▯h▯a▯n▯g▯e▯d▯"▯} when it should be {"changed"}, i.e. every other letter is something weird.
If I take out every other character by doing decompressed_data[::2], then it works, but it's a bit of a hack, and clearly there is something else wrong.
I'm wondering if I need to base64 encode it at all for a PUT request? Is this only necessary for POST?

I think the main problem might be C# uses UTF-16 encoded strings. This may yield a problem similar to yours. As any other encoding problem, we might need a little luck here but I guess you can solve this by doing:
decompressed_data = gz.read().decode('utf-16')
There, decompressed_data should be Unicode and you can treat it as such for further work.
UPDATE: This worked for me:
C Sharp
static void Main(string[] args)
{
FileStream f = new FileStream("test", FileMode.CreateNew);
using (StreamWriter w = new StreamWriter(f))
{
w.Write(Compress("hello"));
}
}
public static string Compress(string s)
{
var bytes = Encoding.Unicode.GetBytes(s);
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream())
{
using (var gs = new GZipStream(mso, CompressionMode.Compress))
{
msi.CopyTo(gs);
}
return Convert.ToBase64String(mso.ToArray());
}
}
Python
import base64
import cStringIO
import gzip
f = open('test','rb')
s = base64.standard_b64decode(f.read())
buff = cStringIO.StringIO(s)
with gzip.GzipFile(fileobj=buff) as gz:
decompressed_data = gz.read()
print decompressed_data.decode('utf-16')
Without decode('utf-16) it printed in the console:
>>>h e l l o
with it it did well:
>>>hello
Good luck, hope this helps!

It's almost working, but the output is: {▯"▯c▯h▯a▯n▯g▯e▯d▯"▯} when it should be {"changed"}
That's because you're using Encoding.Unicode to convert the string to bytes to start with.
If you can tell Python which encoding to use, you could do that - otherwise you need to use an encoding on the C# side which matches what Python expects.
If you can specify it on both sides, I'd suggest using UTF-8 rather than UTF-16. Even though you're compressing, it wouldn't hurt to make the data half the size (in many cases) to start with :)
I'm also somewhat suspicious of this line:
buff = cStringIO.StringIO(s)
s really isn't text data - it's compressed binary data, and should be treated as such. It may be okay - it's just worth checking whether there's a better way.

Why is http content different when sent from c# vs java?

I have an xml file that I need to send to a REST server as a post. When I read the exact same file from c# and java the bytes do not match when they arrive at the server. The java ones fail with a 500 Internal Server Error while the c# one works perfectly. The server is c#.
The file in c# is read as follows:
using (ms = new MemoryStream())
{
string fullPath = #"c:\pathtofile\datalast.xml";
using (FileStream outStream = File.OpenRead(fullPath))
{
outStream.CopyTo(ms);
outStream.Flush();
}
ms.Position = 0;
var xmlDoc = new XmlDocument();
xmlDoc.Load(ms);
content = xmlDoc.OuterXml;
}
content is then sent to a call that uses an HttpWebResponse
The java (Android) code reads the file like this:
FileInputStream fis = app.openFileInput(DATA_LAST_FILE_NAME);
byte[] buffer = new byte[1024];
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
int len;
while ((len = fis.read(buffer)) != -1)
{
outputStream.write(buffer, 0, len);
}
outputStream.close();
fis.close();
ByteArrayEntity data = new ByteArrayEntity(buffer);
data.setContentType("application/xml");
post.setEntity(data);
HttpResponse response = request.execute(post);
For the most part the arrays generated are identical. The only difference seems to be in the first 3 bytes. The c# byte array's first 3 values are:
239,187,191
The java ones are:
-17,-69,-65
What is happening here? What should I do?
Thanks,
\ ^ / i l l

Look at what you're doing here:
FileInputStream fis = app.openFileInput(DATA_LAST_FILE_NAME);
byte[] buffer = new byte[1024];
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
int len;
while ((len = fis.read(buffer)) != -1)
{
outputStream.write(buffer, 0, len);
}
outputStream.close();
fis.close();
ByteArrayEntity data = new ByteArrayEntity(buffer);
You're creating the ByteArrayEntity from the buffer that you've used when reading the data. It's almost certainly not the right length (it will always be length 1024), and it may well not have all the data either.
You should be using the ByteArrayOutputStream you've been writing into, e.g.
ByteArrayEntity data = new ByteArrayEntity(outputStream.toByteArray());
(You should be closing fis in a finally block, by the way.)
EDIT: The values you've printed to the console are indeed just showing the differences between signed and unsigned representations. They have nothing to do with the reason the Java code is failing, which is due to the above problem, I believe. You should look at what's being sent over the wire in Wireshark - that'll show you what's really going on.

Take a look at this: http://en.wikipedia.org/wiki/Byte_order_mark
EDIT: The reason why java and C# are different is that when reading the bytes, C# is unsigned, and java is signed. Same binary values, however.