We Keep Coding

Converting HEX to (extended) ASCII 0-255 with NO UTF for DOS C#

Firstly, I don't want to be 'That Guy' when I ask this long question, even though I know it's been asked plenty of times in different way, but I'm having significant problems in getting a date format to store in a string correctly.
Some minor background.
I am using the DOS FileTime Date format that needs to be stored in an 8 character HEX format - as seen here: https://doubleblak.com/blogPosts.php?id=7
In short, the time and date are captured, then arranged in binary bits, and then converted to HEX.
What I need is to be able to do now, is store those HEX Values as a string, and be able to pass them to tagLib sharp to write a custom APE tag in an MP3 file. Easier said than done...
Writing the custom tags is easy, as it's basically just a matter of this:
TagLib.File file = TagLib.File.Create(filename);
TagLib.Ape.Tag ape_tag = (TagLib.Ape.Tag)file.GetTag(TagLib.TagTypes.Ape, true);
// Write - for my example
/* declarations:
public void SetValue(string key, string value);
public void SetValue(string key, uint number, uint count);
public void SetValue(string key, string[] value);
*/
ape_tag.SetValue("XLastPlayed", history );
So, on to the actual problem:
After the conversion of the date to the correct HEX Values, I get the following result:
928C9D51
However, to make this work and store it correctly, I need to convert it to ASCII values, so that it can then be stored by TagLibSharp.
If I convert this to ASCII, then I get the following: (which is wrong), as it should only be 4 ASCII characters long - even if they are unprintable, or sit in the > 127 character range.
"\u0092\u008c\u009dQ"
You can see in this image the extra HEX values that have been stored, which is incorrect.
This is a sample of the code of I've been trying to use, (in various forms) to get this to work.
string FirstHistory = "7D8C9D51";
String test1 = "";
for (int i = 0; i < FirstHistory.Length; i += 2)
{
string hs = FirstHistory.Substring(i, 2);
var enc = Encoding.GetEncoding("iso-8859-1"); //.ASCII;// .GetEncoding(437);
var bytes1 = enc.GetBytes(string.Format("{0:x1}", Convert.ToChar(Convert.ToUInt16(hs, 16))));
string unicodeString = enc.GetString(bytes1);
Console.WriteLine(unicodeString);
test1 = test1 + unicodeString;
}
// needs to be "00 00 00 21" for the standard date array for this file format.
byte[] bytesArray = { 0, 0, 0, 33 }; // A byte array containing non-printable characters
string s1 = "";
string history = "";
// Basically what the history will look like
// "???!???!???!???!???!???!???!???!???!???!???!???!???!???!???!???!???!"
for (int i =0; i < 18; i++)
{
if(i==0) {
history = test1; // Write the first value.
}
s1 = Encoding.UTF8.GetString(bytesArray); // encoding on this string won't effect the array date values
history = history + s1;
}
ape_tag.SetValue("XLastPlayed", history );
I am aware there are multiple encodings, and I've basically tried all that I can, and have read things, but I'm not getting anywhere.
Sometimes I think I've got it, but then when I look at the file I'm saving, it slips in a "C2" HEX value, when it shouldn't, and this is the unicode breaking everything. I've included an image of what it should be without these C2 Hex Values, and you can actually see the DOS Time and Date time appear correctly in the HxD Hex viewer.
I've tried various encodings such as 437, ios-8859-1, ASCII, and different methods such as using string builder, char, bytes, etc. and sometimes I get a date and time stamp where the values are correct, where the HEX values don't exceed in to the extended ASCII ranges, but then I run it again, and I'm back to square 1. It ALWAYS inserts those extended values as UTF8 entries and breaks regardless of what I do.
I'm sure there's not a bug in VS, but I'm running Microsoft Visual Studio Community 2019, Version 16.8.2 if that adds to the case.
I can not seem to find a way around this. Does anyone have any thoughts on this?
Thanks in advance.
*** UPDATE ***
This the update thanks to #xanatos
public static byte[] ConvertHexStringToByteArray(string str)
{
Dictionary<string, byte> hexindex = new Dictionary<string, byte>();
for (int i = 0; i <= 255; i++)
hexindex.Add(i.ToString("X2"), (byte)i);
List<byte> hexres = new List<byte>();
for (int i = 0; i < str.Length; i += 2)
hexres.Add(hexindex[str.Substring(i, 2)]);
return hexres.ToArray();
}
string FirstHistory = "7D8C9D51";
string s1 = "";
string history = "";
byte[] bytes = { 0, 0, 33, 0 }; // A byte array contains non-ASCII (or non-readable) characters
for (int i =0; i < 18; i++)
{
s1 = Encoding.UTF8.GetString(bytes); // ???
history = history + s1;
}
var theArray_SO = ConvertHexStringToByteArray(FirstHistory);
ape_tag.SetItem(new TagLib.Ape.Item("XLastPlayed", (new TagLib.ByteVector(theArray_SO)) + history));
*** UPDATE 2 - 30th Jan 2021 ***
After editing other values and resaving them, I ran into some trouble. It seems that there could be data corruption with TagLib and custom APE tags, specifically for this ByteVector data. If you just use the save method for editing other custom values then it's not a problem, but if you have custom values with these values with ByteVector values, you will most probably run in to trouble. This is what I still used for saving the files.
TagLib.File file = TagLib.File.Create(filename);
// changes
file.save();
However, to overcome this data corruption, I read (searched) the file first as a FileStream to locate the value I needed, and then put the values of 72 bytes after the found value to a new byte array, then save it back to the file.
I discovered that reading ByteVector data in through a string failed spectacularly and had results all over the place.
TagLib.Ape.Item item_Duration = ape_tag.GetItem("XLastScheduled");
While this can probably be rewritten a thousand ways, here's my code that works.
int foundlocation = 0;
int loop1 = 0;
byte[] sevenItems = new byte[80] { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };
string match = "XLastScheduled";
byte[] matchBytes = Encoding.ASCII.GetBytes(match);
{
using (var fs = new FileStream(filename, FileMode.Open))
{
int i = 0;
int readByte;
while ((readByte = fs.ReadByte()) != -1)
{
if (foundlocation == 0)
{
if (matchBytes[i] == readByte)
{
i++;
}
else
{
i = 0;
}
}
if (i == matchBytes.Length)
{
//Console.WriteLine("It found between {0} and {1}.", fs.Position - matchBytes.Length, fs.Position);
// set to true.
foundlocation = 1;
}
if (foundlocation==1)
{
//if (loop1 > 1)
{
// Start adding it at 2 bytes after it's found.
sevenItems[loop1] = (byte)readByte;
}
loop1++;
if(loop1 > 79)
{
fs.Close();
Console.WriteLine("Found the XLastScheduled data");
// 72/4 = 18 date/times
break;
}
}
// Then, I can save those values back as a vector byte array, instead of a string - hopefully...
}
fs.Close();
}
}
byte[] dst = new byte[sevenItems.Length - 8];
Array.Copy(sevenItems, 2, dst, 0, dst.Length);
TagLib.File file = TagLib.File.Create(filename);
// Get the APEv2 tag if it exists.
TagLib.Ape.Tag ape_tag = (TagLib.Ape.Tag)file.GetTag(TagLib.TagTypes.Ape, true);
// Save the new byteVector.
ape_tag.SetItem(new TagLib.Ape.Item("XLastScheduled", (new TagLib.ByteVector(dst))));
Console.WriteLine("XLastScheduled: set" );

There is another method for binary data:
var bytes = new byte[4] { 0xFF, 0, 0, 0xFF };
ape_tag.SetItem(new TagLib.Ape.Item("XLastPlayed", new ByteVector(bytes)));
Unclear if you need methods to convert from/to DOS FileTime:
public static uint ToDosFileTimeDate(DateTime dt)
{
ushort date = (ushort)(((dt.Year - 1980) << 9) | (dt.Month << 5) | (dt.Day));
ushort time = (ushort)((dt.Hour << 11) | (dt.Minute << 5) | (dt.Second >> 1));
uint dateTime = ((uint)date << 16) | time;
return dateTime;
}
public static DateTime FromDosFileTimeDate(uint ui)
{
ushort date = (ushort)(ui >> 16);
ushort time = (ushort)(ui & 0xFFFF);
var year = (date >> 9) + 1980;
var month = (date >> 5) & 0xF;
var day = date & 0x1F;
var hour = time >> 11;
var minute = (time >> 5) & 0x3F;
var second = (time & 0x1F) << 1;
return new DateTime(year, month, day, hour, minute, second, DateTimeKind.Local);
}
and to convert the uint to byte[4] arrays there are
uint ui = BitConverter.ToUInt32(bytes);
and
byte[] bytes = BitConverter.GetBytes(ui);

C# split byte array from file

Hello I'm doing an encryption algorithm which reads bytes from file (any type) and outputs them into a file. The problem is my encryption program takes only blocks of 16 bytes so if the file is bigger it has to be split into blocks of 16, or if there's a way to read 16 bytes from the file each time it's fine.
The algorithm is working fine with hard coded input of 16 bytes. The ciphered result has to be saved in a list or array because it has to be deciphered the same way later. I can't post all my program but here's what I do in main so far and cannot get results
static void Main(String[] args)
{
byte[] bytes = File.ReadAllBytes("path to file");
var stream = new StreamReader(new MemoryStream(bytes));
byte[] cipherText = new byte[16];
byte[] decipheredText = new byte[16];
Console.WriteLine("\nThe message is: ");
Console.WriteLine(stream.ReadToEnd());
AES a = new AES(keyInput);
var list1 = new List<byte[]>();
for (int i = 0; i < bytes.Length; i+=16)
{
a.Cipher(bytes, cipherText);
list1.Add(cipherText);
}
Console.WriteLine("\nThe resulting ciphertext is: ");
foreach (byte[] b in list1)
{
ToBytes(b);
}
}
I know that my loops always add the first 16 bytes from the byte array but I tried many ways and nothing work. It won't let me index the bytes array or copy an item to a temp variable like temp = bytes[i]. The ToBytes method is irrelevant, it just prints the elements as bytes.

I would like to recommend you to change the interface for your Cipher() method: instead of passing the entire array, it would be better to pass the source and destination arrays and offset - block by block encryption.
Pseudo-code is below.
void Cipher(byte[] source, int srcOffset, byte[] dest, int destOffset)
{
// Cipher these bytes from (source + offset) to (source + offset + 16),
// write the cipher to (dest + offset) to (dest + offset + 16)
// Also I'd recommend to check that the source and dest Length is less equal to (offset + 16)!
}
Usage:
For small files (one memory allocation for destination buffer, block by block encryption):
// You can allocate the entire destination buffer before encryption!
byte[] sourceBuffer = File.ReadAllBytes("path to file");
byte[] destBuffer = new byte[sourceBuffer.Length];
// Encrypt each block.
for (int offset = 0; i < sourceBuffer.Length; offset += 16)
{
Cipher(sourceBuffer, offset, destBuffer, offset);
}
So, the main advantage of this approach - it elimitates additional memory allocations: the destination array is allocated at once. There is also no copy-memory operations.
For files of any size (streams, block by block encryption):
byte[] inputBlock = new byte[16];
byte[] outputBlock = new byte[16];
using (var inputStream = File.OpenRead("input path"))
using (var outputStream = File.Create("output path"))
{
int bytesRead;
while ((bytesRead = inputStream.Read(inputBlock, 0, inputBlock.Length)) > 0)
{
if (bytesRead < 16)
{
// Throw or use padding technique.
throw new InvalidOperationException("Read block size is not equal to 16 bytes");
// Fill the remaining bytes of input block with some bytes.
// This operation for last block is called "padding".
// See http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Padding
}
Cipher(inputBlock, 0, outputBlock, 0);
outputStream.Write(outputBlock, 0, outputBlock.Length);
}
}

No need to read the whole mess into memory if you can only process it a bit at a time...
var filename = #"c:\temp\foo.bin";
using(var fileStream = new FileStream(filename, FileMode.Open))
{
var buffer = new byte[16];
var bytesRead = 0;
while((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) > 0)
{
// do whatever you need to with the next 16-byte block
Console.WriteLine("Read {0} bytes: {1}",
bytesRead,
string.Join(",", buffer));
}
}

You can use Array.Copy
byte[] temp = new byte[16];
Array.Copy(bytes, i, temp, 0, 16);

Windows Phone Encoding and Decoding audio using NSpeex. Having issue with decoding?

I am trying to encode a recorded audio using Nspeex and then transfer it over internet and decode on the other end. I am doing all this in Windows Phone 7/8. To encode and decode I am using following code. But while decoding I am not getting the result back correctly which I can play again. Can anyone provide me with encoding and decoding code which runs on WP7/8 recorded audio:
private static Microphone mic = Microphone.Default;
private static byte[] EncodeSpeech(byte[] buf, int len)
{
BandMode mode = GetBandMode(mic.SampleRate);
SpeexEncoder encoder = new SpeexEncoder(mode);
// set encoding quality to lowest (which will generate the smallest size in the fastest time)
encoder.Quality = 1;
int inDataSize = len / 2;
// convert to short array
short[] data = new short[inDataSize];
int sampleIndex = 0;
for (int index = 0; index < len; index += 2, sampleIndex++)
{
data[sampleIndex] = BitConverter.ToInt16(buf, index);
}
// note: the number of samples per frame must be a multiple of encoder.FrameSize
inDataSize = inDataSize - inDataSize % encoder.FrameSize;
var encodedData = new byte[len];
int encodedBytes = encoder.Encode(data, 0, inDataSize, encodedData, 0, len);
if (encodedBytes != 0)
{
// each chunk is laid out as follows:
// | 4-byte total chunk size | 4-byte encoded buffer size | <encoded-bytes> |
byte[] inDataSizeBuf = BitConverter.GetBytes(inDataSize);
byte[] sizeBuf = BitConverter.GetBytes(encodedBytes + inDataSizeBuf.Length);
byte[] returnBuf = new byte[encodedBytes + sizeBuf.Length + inDataSizeBuf.Length];
sizeBuf.CopyTo(returnBuf, 0);
inDataSizeBuf.CopyTo(returnBuf, sizeBuf.Length);
Array.Copy(encodedData, 0, returnBuf, sizeBuf.Length + inDataSizeBuf.Length, encodedBytes);
return returnBuf;
}
else
return buf;
}
private byte[] DecodeSpeech(byte[] buf)
{
BandMode mode = GetBandMode(mic.SampleRate);
SpeexDecoder decoder = new SpeexDecoder(mode);
byte[] inDataSizeBuf = new byte[4];
byte[] sizeBuf = new byte[4];
byte[] encodedBuf = new byte[buf.Length - 8];
Array.Copy(buf, 0, sizeBuf, 0, 4);
Array.Copy(buf, 4, inDataSizeBuf, 0, 4);
Array.Copy(buf, 8, encodedBuf, 0, buf.Length - 8);
int inDataSize = BitConverter.ToInt32(inDataSizeBuf, 0);
int size = BitConverter.ToInt32(sizeBuf, 0);
short[] decodedBuf = new short[inDataSize];
int decodedSize = decoder.Decode(encodedBuf, 0, encodedBuf.Length, decodedBuf, 0, false);
byte[] returnBuf = new byte[inDataSize * 2];
for (int index = 0; index < decodedBuf.Length; index++)
{
byte[] temp = BitConverter.GetBytes(decodedBuf[index]);
Array.Copy(temp, 0, returnBuf, index * 2, 2);
}
return returnBuf;
}
private static BandMode GetBandMode(int sampleRate)
{
if (sampleRate <= 8000)
return BandMode.Narrow;
if (sampleRate <= 16000)
return BandMode.Wide;
return BandMode.UltraWide;
}

I think your problem may be that you are newing up a new SpeexEncoder every time you want to encode audio. You should try making that a member for your class and re-use it.
I looked at the code for Nspeex I noticed that SpeexEncoder uses NbEncoder for the narrow band. In that class it looks like it keeps a history of some previous audio data in order perform the encoding. This should mean that the output for different instances of encoders would not go together.
private static Microphone mic = Microphone.Default;
private static SpeexEncoder encoder = CreateEncoder();
private static SpeexEncoder CreateEncoder()
{
BandMode mode = GetBandMode(mic.SampleRate);
SpeexEncoder encoder = new SpeexEncoder(mode);
// set encoding quality to lowest (which will generate the smallest size in the fastest time)
encoder.Quality = 1;
return encoder;
}
private static byte[] EncodeSpeech(byte[] buf, int len)
{
int inDataSize = len / 2;
...

How can I create semi dynamic byte array

How can generate a byte dynamically in c# code?
Format as below :
First byte is standard 88, second byte is the number of remaining byte, and the last one is the remaining bytes.
Example:
1 byte:
byte[] bytes = new byte[] { 0x88, 0x01, 0};
2 bytes:
byte[] bytes = new byte[] { 0x88, 0x02, 0, 0};
5 bytes:
byte[] bytes = new byte[] { 0x88, 0x05, 0, 0, 0, 0, 0};
Thank you.

Just this:
static byte[] Get(byte num)
{
byte[] a = new byte[num + 2];
a[0] = 0x88;
a[1] = num;
return a;
}
All the other bytes are initialized by default with 0.

If you have the data bytes in an array, you can create an array dynamically like this:
// data bytes
byte[] data = { 1, 2, 3 };
byte[] buffer = new byte[data.Length + 2];
buffer[0] = 0x88;
buffer[1] = (byte)data.Length;
data.CopyTo(buffer, 2);
If you create the data bytes in some other way, the first part of the code is the same, then just put the data in the rest of the array in the same way as the first two bytes. For example five zero bytes:
int len = 5;
byte[] buffer = new byte[len + 2];
buffer[0] = 0x88;
buffer[1] = (byte)len;
for (int i = 0; i < len; i++) {
buffer[i + 2] = 0;
}

File transfer issue in c#

I am transferring files using c#. I have used this code. The problem is small files like .txt files are transferred correctly but not big files like images, documents, pdf, ppt. Sometimes code works fine but most of the times it transfers less amount of data.
Server Code:
Socket clientSock = sock.Accept();
byte[] clientData = new byte[1024 * 50000];
int receivedBytesLen = clientSock.Receive(clientData);
int fileNameLen = BitConverter.ToInt32(clientData, 0);
string fileName = Encoding.ASCII.GetString(clientData, 4, fileNameLen);
BinaryWriter bWrite = new BinaryWriter(File.Open(receivedPath + "/" + fileName, FileMode.Append));
bWrite.Write(clientData, 4 + fileNameLen, receivedBytesLen - 4 - fileNameLen);
bWrite.Close();
clientSock.Close();
Client Code:
byte[] fileNameByte = Encoding.ASCII.GetBytes(fileName);
byte[] fileData = File.ReadAllBytes(filePath + fileName);
byte[] clientData = new byte[4 + fileNameByte.Length + fileData.Length];
byte[] fileNameLen = BitConverter.GetBytes(fileNameByte.Length);
fileNameLen.CopyTo(clientData, 0);
fileNameByte.CopyTo(clientData, 4);
fileData.CopyTo(clientData, 4 + fileNameByte.Length);
clientSock.Connect(ipEnd);
clientSock.Send(clientData);
clientSock.Close();
Complete code is given in the above link. I have also seen this post but this is not helpful.

Sometimes code works fine but most of the times it transfers less amount of data.
That's the nature of Socket.Receive(), it doesn't always return all data that gets sent to it.
You'll have to do a Receive(clientData, 4, 0) first to receive the bytes that indicate the size , then call Receive(clientData) in a loop until you've received size bytes. But beware that the Receive(buffer[], length, offset) overload just as easily as any other overload can return less than the expected amount of bytes. So you'll also have to call that in a loop:
Something like this:
// First receive the size
int sizeSize = 4; // Size of Int32 in bytes
int sizeOffset = 0;
var sizeBytes = new byte[sizeSize];
while (sizeOffset < sizeSize)
{
sizeOffset += clientSocket.Receive(sizeBytes, sizeSize - sizeOffset, sizeOffset);
}
var size = BitConverter.ToInt32(sizeBytes, 0);
// Then receive the data
byte[] fileData = new byte[size];
byte[] clientData = new byte[8192];
int totalBytes = 0;
while (totalBytes < size)
{
// This may return anything between 0 and 8192, even if not all sent data has been received yet. It may be in a buffer somewhere, waiting to be picked up. Check for 0, since that's when the client disconnects.
int bytesReceived = clientSocket.Receive(clientData);
// You now have received a chunk of data of bytesReceived length. Append it into the fileData array.
Buffer.BlockCopy(clientData, 0, fileData, totalBytes, bytesReceived);
totalBytes += bytesReceived;
}

As CodeCaster's answered that Socket.Receive(), it doesn't always return all data that gets sent to it. This is 100% correct and tested but the next step of sending the file size is not working, I found an easy and correct solution.
Socket.Receive() method takes the byte array in which received data will be copied and returns the number of bytes received. So we can easily loop it till bytes received are 0.
byte[] tempData = new byte[1024 * 5000];
BinaryWriter bWrite = null;
int bytes_received;
int fileNamelength = 0;
bool isFirstPacket = true;
do
{
bytes_received = clientSock.Receive(tempData);
if(isFirstPacket)
{
isFirstPacket = false;
int fileNameLen = BitConverter.ToInt32(tempData, 0);
string fileName = Encoding.ASCII.GetString(clientData, 4, fileNameLen);
bWrite = new BinaryWriter(File.Open(receivedPath + fileName, FileMode.Append))
bWrite.Write(tempData, 4 + fileNameLength, bytes_received - 4 - fileNamelength);
}
else
bWrite.Write(tempData, 0, bytes_received);
}
while (bytes_received != 0);