We Keep Coding

encode and decode strange .shm file data to and from base64 c#

first a depressing fact: https://www.base64decode.org/ can do what i want to do.
i´m trying to encode and decode (to and from base64) a model file (.shm) generated by the image processing tool MVTec Halcon because i want to store it in a xml file.
If i open it, it has this strange form:
HSTF ÿÿÿÿ¿€ Q¿ÙG®záH?Üä4©±w?­Eè}‰#?ð ................
I´m using this methods to encode and decode it:
public static string Base64Encode(string text)
{
Byte[] textBytes = Encoding.Default.GetBytes(text);
return Convert.ToBase64String(textBytes);
}
public static string Base64Decode(string base64EncodedData)
{
Byte[] base64EncodedBytes = Convert.FromBase64String(base64EncodedData);
return Encoding.Default.GetString(base64EncodedBytes);
}
and calling the methods from a gui like this:
var model = File.ReadAllText(#"C:\Users\\Desktop\model_region_nut.txt");
var base64 = ImageConverter.Base64Encode(model);
File.WriteAllText(#"C:\Users\\Desktop\base64.txt", base64);
var modelneu = ImageConverter.Base64Decode(File.ReadAllText(#"C:\Users\\Desktop\base64.txt"));
File.WriteAllText(#"C:\Users\\Desktop\modelneu.txt", modelneu);
my result for modelneu is:
HSTF ?????? Q??G?z?H???4??w??E?}??#??
so you can see that there are lots of missing characters.. I guess the problem is caused by using .Default.
Thanks for your help,
Michel

If you're working with binary data, there is no reason at all to go through text decoding and encoding. Doing so only risks corrupting the data in various ways, even if you're using a consistent character encoding.
Just use File.ReadAllBytes() instead of File.ReadAllText() and skip the unnecessary Encoding step.

The problem is with reading file with unspecified encoding, check this question.
As mentioned there you can go with overload for ReadAllText to specify encoding and also for writing you must specofy encoding for WriteAllText I suggest using UTF-8 encoding so:
var model = File.ReadAllText(#"C:\Users\pichlerm\Desktop\model_region_nut.txt",Encoding.UTF8);
var base64 = ImageConverter.Base64Encode(model);
File.WriteAllText(#"C:\Users\\Desktop\base64.txt", base64,Encoding.UTF8);
var modelneu = ImageConverter.Base64Decode(File.ReadAllText(#"C:\Users\\Desktop\base64.txt"));
File.WriteAllText(#"C:\Users\pichlerm\Desktop\modelneu.txt", modelneu);

What is the efficient way to write stream of bytes into a file in c#?

I have big big data in form of bytes around 5GB.
I need to store this data in a file ServerData.xml. This data should be first converted into string and then should be saved to file so that we can perform operation on the file.
I used below code to convert stream of bytes to string and then to save the same in a file.
private const string fileName = "ServerData.xml";
public void ProcessBuffer(byte[] receiveBuffer, int bytes)
{
if (!File.Exists(fileName))
{
using (File.Create(fileName)) { };
}
TextWriter tw = new StreamWriter(fileName, true);
tw.Write(Encoding.UTF8.GetString(receiveBuffer).TrimEnd((Char)0));
tw.Close();
}
Is it the right way ?
or please suggest better way so that there should not be any memory issue if any in future ?

The code in your question can only work if ProcessBuffer is always called with a UTF-8 encoded text that is broken on code point boundaries. That seems pretty unlikely to me, so I would expect that you encounter errors when decoding to text.
However, decoding to text and then writing, is rather pointless and indeed counter-productive. The bytes are already UTF-8 encoded. Write them directly to file as they arrive from the socket. Don't perform any processing of them. When you come to read the XML using XmlReader, the parser will read the encoding as UTF-8 from the document's XML declaration, and be able to decode the rest of the document. I am assuming that the document's XML declaration specifies UTF-8 but that seems highly likely. You should check.
You should get rid of the text writer which is no use to you for writing bytes. Write the bytes directly to a file stream. And try to avoid opening and closing the file repeatedly. That's very inefficient. Open and close the file exactly once.

Why do you need to convert it to a string?
using System.IO;
public static void WriteBytes(byte[] bytes, string filename)
{
using (FileStream fs = new FileStream(filename, FileMode.OpenOrCreate))
using (BinaryWriter writer = new BinaryWriter(fs, Encoding.UTF8))
{
writer.Write(bytes);
}
}

You can simply write these bytes to a file using FileStream:
public void ProcessBuffer(byte[] receivedBuffer, int bytes)
{
using (var fileStream = new FileStream(fileName, FileMode.Create)) // overwrites file
{
fileStream.Write(receivedBuffer, 0, bytes);
}
}
Update: You won't be able to work with such a big XML document if you don't have enough resources. I would suggest reformatting this file. For example, I would parse this XML and insert data into a SQL database. Then, you can easily operate with such amounts of data.

I would prefer that I write all bytes to file. And when reading, convert it to string and then convert to XML using XDocument, XElement etc. By writing bytes in file you will save space, and it is efficient,
Instead of using FileStream, I will prefer File.WriteAllBytes method.
private const string fileName = "ServerData.xml";
public void ProcessBuffer(byte[] receiveBuffer, int bytes)
{
File.WriteAllBytes(filename, bytes);
// And when reading
var bytes = File.ReadAllBytes(filename);
var binaryReader = new BinaryReader(new MemoryStream(bytes));
// Parse strings and make xml,
binaryReader.ReadString();
}

Getting Json Output from Byte Array

I just started a new project on WCF and to be honest I'm very new at this with limited knowledge.
So what I'm trying to do is open a file that is stored in my computer (e.g. word, pdf, etc.) and display the contents in the webpage in JSon format. I converted the file in a byte array and tried to display the Stream. When I did that it asked me to open the file or save it. I don't want that - I just want the contents of the file to be displayed on my local host when i call the method.
Here's what I have:
public string GetRawFile()
{
string file = #"C:\.....\TestFile.pdf";
byte[] rawFile = File.ReadAllBytes(file);
//Stream stream = new MemoryStream(rawFile);
//DataContractJsonSerializer obj = newDataContractJsonSerializer(typeof(string));
//string result = obj.ReadObject(stream).ToString();
//Deserializing
MemoryStream stream = new MemoryStream();
BinaryFormatter binForm = new BinaryFormatter();
stream.Write(rawFile, 0, rawFile.Length);
stream.Seek(0, SeekOrigin.Begin);
Object obj = (Object) binForm.Deserialize(stream);
System.Web.Script.Serialization.JavaScriptSerializer xyz = new System.Web.Script.Serialization.JavaScriptSerializer();
string ejson = xyz.Serialize(obj);
WebOperationContext.Current.OutgoingRequest.ContentType = "text/json";
return ejson;
}
I'm trying to return a string and it's not working, but when I return just the stream it's popping up the "openwith" message.
Also should I use the GET or POST on my datacontract. I'm using REST in C#.

I'm assuming that your file actually contains json. If that is the case just do this;
string file = File.ReadAllText("C:\path\to\file.extension");
You're making the problem a lot more complicated than it needs to be. Just read the file and return it's data as a string. I think you want to use GET for the http method. Generally speaking, you all use post if you're adding new content. If for example the users request would cause the application to write some data to a file or data base then you would typically use POST for the http method. If they're just requesting data, you almost always use GET.

How to open a file as binary mode and replace hex strings?

I need open and edit a executable file in binary mode, to replace hex value as string.
In PHP, looks like this:
<?php
$fp = fopen('file.exe', 'r+');
$content = fread($fp, filesize('file.exe'));
fclose($fp);
print $content;
/* [...] This program cannot be run in DOS mode.[...] */
?>
How I get it in C#?

public void Manipulate()
{
byte[] data = File.ReadAllBytes("file.exe");
byte[] newData;
//walkthrough data and do what you need to do and move to newData
File.WriteAllBytes("new_file.exe", newData);
}

Use File.ReadAllBytes to read the bytes of a file as a byte array.
byte[] bytes = File.ReadAllBytes('file.exe');
If you want to convert this to a hex string (and I'd in general I'd advise against doing so - strings are immutable in C# so modifying even a single byte will require copying the rest of the string) you can for example use:
string hex = BitConverter.ToString(bytes);

You ask about writting to file, but you PHP code is for reading. For working with files you can use FileStream class
using(FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Write))
{
....
stream.WriteByte(byte);
....
}

Base64 decode in C# or Java

I have a Base64-encoded object with the following header:
application/x-xfdl;content-encoding="asc-gzip"
What is the best way to proceed in decoding the object? Do I need to strip the first line? Also, if I turn it into a byte array (byte[]), how do I un-gzip it?
Thanks!
I think I misspoke initially. By saying the header was
application/x-xfdl;content-encoding="asc-gzip"
I meant this was the first line of the file. So, in order to use the Java or C# libraries to decode the file, does this line need to be stripped?
If so, what would be the simplest way to strip the first line?

To decode the Base64 content in C# you can use the Convert Class static methods.
byte[] bytes = Convert.FromBase64String(base64Data);
You can also use the GZipStream Class to help deal with the GZipped stream.
Another option is SharpZipLib. This will allow you to extract the original data from the compressed data.

I was able to use the following code to convert an .xfdl document into a Java DOM Document.
I used iHarder's Base64 utility to do the Base64 Decode.
private static final String FILE_HEADER_BLOCK =
"application/vnd.xfdl;content-encoding=\"base64-gzip\"";
public static Document OpenXFDL(String inputFile)
throws IOException,
ParserConfigurationException,
SAXException
{
try{
//create file object
File f = new File(inputFile);
if(!f.exists()) {
throw new IOException("Specified File could not be found!");
}
//open file stream from file
FileInputStream fis = new FileInputStream(inputFile);
//Skip past the MIME header
fis.skip(FILE_HEADER_BLOCK.length());
//Decompress from base 64
Base64.InputStream bis = new Base64.InputStream(fis,
Base64.DECODE);
//UnZIP the resulting stream
GZIPInputStream gis = new GZIPInputStream(bis);
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(gis);
gis.close();
bis.close();
fis.close();
return doc;
}
catch (ParserConfigurationException pce) {
throw new ParserConfigurationException("Error parsing XFDL from file.");
}
catch (SAXException saxe) {
throw new SAXException("Error parsing XFDL into XML Document.");
}
}
Still working on successfully modifying and re-encoding the document.
Hope this helps.

In Java, you can use the Apache Commons Base64 class
String decodedString = new String(Base64.decodeBase64(encodedBytes));

It sounds like you're dealing with data that is both gzipped and Base 64 encoded. Once you strip off any mime headers, you should convert the Base64 data to a byte array using something like Apache commons codec. You can then wrap the byte[] in a ByteArrayInputStream object and pass that to a GZipInputStream which will let you read the uncompressed data.

For java, have you tried java's built in java.util.zip package? Alternately, Apache Commons has the Commons Compress library to work with zip, tar and other compressed file types. As to decoding Base 64, there are several open source libraries, or you can use Sun's sun.misc.BASE64Decoder class.

Copied from elsewhere, for Base64 I link to commons-codec-1.6.jar:
public static String decode(String input) throws Exception {
byte[] bytes = Base64.decodeBase64(input);
BufferedReader in = new BufferedReader(new InputStreamReader(
new GZIPInputStream(new ByteArrayInputStream(bytes))));
StringBuffer buffer = new StringBuffer();
char[] charBuffer = new char[1024];
while(in.read(charBuffer) != -1) {
buffer.append(charBuffer);
}
return buffer.toString();
}