We Keep Coding

How to find and replace a large section of bytes in a file?

I'm looking to find a large section of bytes within a file, remove them and then import a new large section of bytes starting where the old ones started.
Here's a video of the manual process that I'm trying to re-create in C#, it might explain it a little better: https://www.youtube.com/watch?v=_KNx8WTTcVA
I have only basic experience with C# so am learning as I go along, any help with this would be very appreciated!
Thanks.

Refer to this question:
C# Replace bytes in Byte[]
Use the following class:
public static class BytePatternUtilities
{
private static int FindBytes(byte[] src, byte[] find)
{
int index = -1;
int matchIndex = 0;
// handle the complete source array
for (int i = 0; i < src.Length; i++)
{
if (src[i] == find[matchIndex])
{
if (matchIndex == (find.Length - 1))
{
index = i - matchIndex;
break;
}
matchIndex++;
}
else
{
matchIndex = 0;
}
}
return index;
}
public static byte[] ReplaceBytes(byte[] src, byte[] search, byte[] repl)
{
byte[] dst = null;
byte[] temp = null;
int index = FindBytes(src, search);
while (index >= 0)
{
if (temp == null)
temp = src;
else
temp = dst;
dst = new byte[temp.Length - search.Length + repl.Length];
// before found array
Buffer.BlockCopy(temp, 0, dst, 0, index);
// repl copy
Buffer.BlockCopy(repl, 0, dst, index, repl.Length);
// rest of src array
Buffer.BlockCopy(
temp,
index + search.Length,
dst,
index + repl.Length,
temp.Length - (index + search.Length));
index = FindBytes(dst, search);
}
return dst;
}
}
Usage:
byte[] allBytes = File.ReadAllBytes(#"your source file path");
byte[] oldbytePattern = new byte[]{49, 50};
byte[] newBytePattern = new byte[]{48, 51, 52};
byte[] resultBytes = BytePatternUtilities.ReplaceBytes(allBytes, oldbytePattern, newBytePattern);
File.WriteAllBytes(#"your destination file path", resultBytes)
The problem is when the file is too large, then you require a "windowing" function. Don't load all the bytes in memory as it will take up much space.

C# - Convert Bytes Representing Char Numbers to Int16

If I have a byte array representing a number read from a file, how can the byte array be converted to an Int16/short?
byte[] bytes = new byte[]{45,49,54,50 } //Byte array representing "-162" from text file
short value = 0; //How to convert to -162 as a short here?
Tried using BitConverter.ToInt16(bytes, 0), but the value is not correct.
Edit: Looking for a solution that does NOT use string conversion.

This function performs some validations that you may be able to exclude. You could simplify it if you know your input array will always contain at least one element and that the value will be a valid Int16.
const byte Negative = (byte)'-';
const byte Zero = (byte)'0';
static Int16 BytesToInt16(byte[] bytes)
{
if (null == bytes || bytes.Length == 0)
return 0;
int result = 0;
bool isNegative = bytes[0] == Negative;
int index = isNegative ? 1 : 0;
for (; index < bytes.Length; index++)
{
result = 10 * result + (bytes[index] - Zero);
}
if (isNegative)
result *= -1;
if (result < Int16.MinValue)
return Int16.MinValue;
if (result > Int16.MaxValue)
return Int16.MaxValue;
return (Int16)result;
}

Like willaien said, you want to first convert your bytes to a string.
byte[] bytes = new byte[]{ 45,49,54,50 };
string numberString = Encoding.UTF8.GetString(bytes);
short value = Int16.Parse(numberString);
If you're not sure that your string can be parsed, I recommend using Int16.TryParse:
byte[] bytes = new byte[]{ 45,49,54,50 };
string numberString = Encoding.UTF8.GetString(bytes);
short value;
if (!Int16.TryParse(numberString, out value))
{
// Parsing failed
}
else
{
// Parsing worked, `value` now contains your value.
}

Convert Byte array to struct in VS2013 debugger

One of the simple tricks I like to use in C# is overriding ToString() so that the debugger displays a developer-specified, human-readable string in the watch window. It's really handy.
Currently, I'm debugging a 256 byte data packet that is a slight variant of ModBus/TCP. Rather than looking at 256 array indices in the watch window, I'd like to see something like "transaction_id_high", "transaction_id_low", etc., where the mapping is 1:1 as the fields are defined within the struct.
When I just tried putting (ModBusPacket)response_buffer in the watch window to see what would happen, it replied with Cannot convert type 'byte[]' to 'ModBusPacket'.
Has anyone ever tried to do this and succeeded?
ModBusPacket:
public struct ModBusPacket
{
char transaction_id_high;
char transaction_id_low;
char protocol_id_high;
char protocol_id_low;
char unit_id;
char function_code;
char sub_unit_id;
char[] data;
}
The byte array is merely
byte[] response_buffer = new byte[256];

If your packet is based on this, I would not suggest using char to represent bytes, since char in c# is a 16-bit numeric (ordinal) value. Instead I'd recommend using byte for 8-bit unsigned values, and UInt16 for 16-bit unsigned values. Then you can do:
[StructLayout(LayoutKind.Sequential)]
public struct ModBusPacket
{
// http://en.wikipedia.org/wiki/Modbus#Frame_format
// The byte order is Big-Endian (first byte contains MSB).
public const bool IsLittleEndian = false;
public UInt16 TransactionIdentifier;
public UInt16 ProtocolIdentifier;
public UInt16 Length;
public byte UnitIdentifier;
public byte FunctionCode;
public byte[] Data;
static int PostIncrement(ref int index, int inc)
{
int old = index;
index += inc;
return old;
}
static byte[] ElementArray(byte[] buffer, ref byte[] swapBuffer, ref int index, int size)
{
if (swapBuffer == null || swapBuffer.Length < size)
Array.Resize(ref swapBuffer, size);
Array.Copy(buffer, PostIncrement(ref index, size), swapBuffer, 0, size);
if (BitConverter.IsLittleEndian != IsLittleEndian)
Array.Reverse(swapBuffer);
return swapBuffer;
}
public ModBusPacket(byte[] buffer)
{
int pos = 0;
byte[] swapBuffer = null;
TransactionIdentifier = (buffer.Length >= pos + 2 ? BitConverter.ToUInt16(ElementArray(buffer, ref swapBuffer, ref pos, 2), 0) : (UInt16)0);
ProtocolIdentifier = (buffer.Length >= pos + 2 ? BitConverter.ToUInt16(ElementArray(buffer, ref swapBuffer, ref pos, 2), 0) : (UInt16)0);
Length = (buffer.Length >= pos + 2 ? BitConverter.ToUInt16(ElementArray(buffer, ref swapBuffer, ref pos, 2), 0) : (UInt16)0);
UnitIdentifier = (buffer.Length >= pos + 1 ? buffer[PostIncrement(ref pos, 1)] : (byte)0);
FunctionCode = (buffer.Length >= pos + 1 ? buffer[PostIncrement(ref pos, 1)] : (byte)0);
var length = Math.Max(buffer.Length - pos, 0);
Data = new byte[length];
if (length > 0)
Array.Copy(buffer, pos, Data, 0, length);
}
public override string ToString()
{
return ObjectExtensions.ToStringWithReflection(this);
}
}
public static class ObjectExtensions
{
public static string ToStringWithReflection<T>(this T obj)
{
if (obj == null)
return string.Empty;
var type = obj.GetType();
var fields = type.GetFields();
var properties = type.GetProperties().Where(p => p.GetIndexParameters().Length == 0 && p.GetGetMethod(true) != null);
var values = new List<KeyValuePair<string, object>>();
Array.ForEach(fields, (field) => values.Add(new KeyValuePair<string, object>(field.Name, field.GetValue(obj))));
foreach (var property in properties)
if (property.CanRead)
values.Add(new KeyValuePair<string, object>(property.Name, property.GetValue(obj, null)));
return values.Aggregate(new StringBuilder(), (s, pair) => (s.Length == 0 ? s.Append("{").Append(obj.GetType().Name).Append(": ") : s.Append("; ")).Append(pair)).Append("}").ToString();
}
}
Having done that, in the immediate window, you can type buffer.ToPacket() in the immediate window or watch window and see the formatted data. Or you could use a conversion operator to cast your byte array to a ModBusPacket if that would be more attractive.

IntPtr to String and long?

I am using C# and the .NET 2.0 framework.
I have this method here to get a string out of a IntPtr:
void* cfstring = __CFStringMakeConstantString(StringToCString(name));
IntPtr result = AMDeviceCopyValue_Int(device, unknown, cfstring);
if (result != IntPtr.Zero)
{
byte length = Marshal.ReadByte(result, 8);
if (length > 0)
{
string s = Marshal.PtrToStringAnsi(new IntPtr(result.ToInt64() + 9L), length);
return s;
}
}
return String.Empty;
It works and all data which is saved as string in my USB-Device (return data of method AMDeviceCopyValue).
Now I get this:
£\rº\0\0¨\t\0\0\0ˆ\0\0\0\0\0\0x>Ô\0\0\0\0Ũ\t\0\0\0€1ÔxÕ͸MÔ\0\0\0\0Ȩ\t\0\0\0€)\0fŒ\a\0Value\0\0˨\t\0\0\0€7fŒ\a\0Result\0Ψ\t\0\0\0ˆTÅfŒ\a\0\0Key\0\0\0\0ñ¨\t\0\0\0€+fŒ\a\0Port\0\0\0ô¨\t\0\0\0€%fŒ\a\0Key\0\0\0\0÷¨\t\0\0\0€:\0\0\0\0\0\0\0\0\0\0\0\0\0\0ú¨\t\0"
This is saved as long - so, how I can get this IntPtr into long and not string?

If you simply want to read a long instead of a string, use:
long l = Marshal.ReadInt64(result, 9);
return l;

Fastest way to convert a possibly-null-terminated ascii byte[] to a string?

I need to convert a (possibly) null terminated array of ascii bytes to a string in C# and the fastest way I've found to do it is by using my UnsafeAsciiBytesToString method shown below. This method uses the String.String(sbyte*) constructor which contains a warning in it's remarks:
"The value parameter is assumed to point to an array representing a string encoded using the default ANSI code page (that is, the encoding method specified by Encoding.Default).
Note: * Because the default ANSI code page is system-dependent, the string created by this constructor from identical signed byte arrays may differ on different systems. * ...
* If the specified array is not null-terminated, the behavior of this constructor is system dependent. For example, such a situation might cause an access violation. *
"
Now, I'm positive that the way the string is encoded will never change... but the default codepage on the system that my app is running on might change. So, is there any reason that I shouldn't run screaming from using String.String(sbyte*) for this purpose?
using System;
using System.Text;
namespace FastAsciiBytesToString
{
static class StringEx
{
public static string AsciiBytesToString(this byte[] buffer, int offset, int maxLength)
{
int maxIndex = offset + maxLength;
for( int i = offset; i < maxIndex; i++ )
{
/// Skip non-nulls.
if( buffer[i] != 0 ) continue;
/// First null we find, return the string.
return Encoding.ASCII.GetString(buffer, offset, i - offset);
}
/// Terminating null not found. Convert the entire section from offset to maxLength.
return Encoding.ASCII.GetString(buffer, offset, maxLength);
}
public static string UnsafeAsciiBytesToString(this byte[] buffer, int offset)
{
string result = null;
unsafe
{
fixed( byte* pAscii = &buffer[offset] )
{
result = new String((sbyte*)pAscii);
}
}
return result;
}
}
class Program
{
static void Main(string[] args)
{
byte[] asciiBytes = new byte[]{ 0, 0, 0, (byte)'a', (byte)'b', (byte)'c', 0, 0, 0 };
string result = asciiBytes.AsciiBytesToString(3, 6);
Console.WriteLine("AsciiBytesToString Result: \"{0}\"", result);
result = asciiBytes.UnsafeAsciiBytesToString(3);
Console.WriteLine("UnsafeAsciiBytesToString Result: \"{0}\"", result);
/// Non-null terminated test.
asciiBytes = new byte[]{ 0, 0, 0, (byte)'a', (byte)'b', (byte)'c' };
result = asciiBytes.UnsafeAsciiBytesToString(3);
Console.WriteLine("UnsafeAsciiBytesToString Result: \"{0}\"", result);
Console.ReadLine();
}
}
}

Oneliner (assuming the buffer actually contains ONE well formatted null terminated string):
String MyString = Encoding.ASCII.GetString(MyByteBuffer).TrimEnd((Char)0);

Any reason not to use the String(sbyte*, int, int) constructor? If you've worked out which portion of the buffer you need, the rest should be simple:
public static string UnsafeAsciiBytesToString(byte[] buffer, int offset, int length)
{
unsafe
{
fixed (byte* pAscii = buffer)
{
return new String((sbyte*)pAscii, offset, length);
}
}
}
If you need to look first:
public static string UnsafeAsciiBytesToString(byte[] buffer, int offset)
{
int end = offset;
while (end < buffer.Length && buffer[end] != 0)
{
end++;
}
unsafe
{
fixed (byte* pAscii = buffer)
{
return new String((sbyte*)pAscii, offset, end - offset);
}
}
}
If this truly is an ASCII string (i.e. all bytes are less than 128) then the codepage problem shouldn't be an issue unless you've got a particularly strange default codepage which isn't based on ASCII.
Out of interest, have you actually profiled your application to make sure that this is really the bottleneck? Do you definitely need the absolute fastest conversion, instead of one which is more readable (e.g. using Encoding.GetString for the appropriate encoding)?

I'm not sure of the speed, but I found it easiest to use LINQ to remove the nulls before encoding:
string s = myEncoding.GetString(bytes.TakeWhile(b => !b.Equals(0)).ToArray());

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace TestProject1
{
class Class1
{
static public string cstr_to_string( byte[] data, int code_page)
{
Encoding Enc = Encoding.GetEncoding(code_page);
int inx = Array.FindIndex(data, 0, (x) => x == 0);//search for 0
if (inx >= 0)
return (Enc.GetString(data, 0, inx));
else
return (Enc.GetString(data));
}
}
}

s = s.Substring(0, s.IndexOf((char) 0));

Just for completeness, you can also use built-in methods of the .NET framework to do this:
var handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
try
{
return Marshal.PtrToStringAnsi(handle.AddrOfPinnedObject());
}
finally
{
handle.Free();
}
Advantages:
It doesn't require unsafe code (i.e., you can also use this method for VB.NET) and
it also works for "wide" (UTF-16) strings, if you use Marshal.PtrToStringUni instead.

One possibility to consider: check that the default code-page is acceptable and use that information to select the conversion mechanism at run-time.
This could also take into account whether the string is in fact null-terminated, but once you've done that, of course, the speed gains my vanish.

An easy / safe / fast way to convert byte[] objects to strings containing their ASCII equivalent and vice versa using the .NET class System.Text.Encoding. The class has a static function that returns an ASCII encoder:
From String to byte[]:
string s = "Hello World!"
byte[] b = System.Text.Encoding.ASCII.GetBytes(s);
From byte[] to string:
byte[] byteArray = new byte[] {0x41, 0x42, 0x09, 0x00, 0x255};
string s = System.Text.Encoding.ASCII.GetString(byteArray);

This is a bit ugly but you don't have to use unsafe code:
string result = "";
for (int i = 0; i < data.Length && data[i] != 0; i++)
result += (char)data[i];