i have a list of chars on my class. Serialization and deserialization are works as expected. If my list contains which char is need to describe byte order mark. Example char code is 56256. So, created simple test to as this question is below.
[Test]
public void Utf8CharSerializeAndDeserializeShouldEqual()
{
UInt16 charCode = 56256;
char utfChar = (char)charCode;
using (MemoryStream ms = new MemoryStream())
{
using (StreamWriter writer = new StreamWriter(ms, Encoding.UTF8, 1024, true))
{
var serializer = new JsonSerializer();
serializer.Serialize(writer, utfChar);
}
ms.Position = 0;
using (StreamReader reader = new StreamReader(ms, true))
{
using (JsonTextReader jsonReader = new JsonTextReader(reader))
{
var serializer = new JsonSerializer();
char deserializedChar = serializer.Deserialize<char>(jsonReader);
Console.WriteLine($"{(int)utfChar}, {(int)deserializedChar}");
Assert.AreEqual(utfChar, deserializedChar);
Assert.AreEqual((int)utfChar, (int)deserializedChar);
}
}
}
}
Test works as fine when char code is not needed a BOM. For example 65(A) will pass this test.
Your problem is unrelated to Json.NET. Your problem is that U+DBC0 (decimal 56256) is an invalid unicode character, and, as explained in the documentation, the Encoding.UTF8 used by your StreamWriter will not encode such a character:
Encoding.UTF8 returns a UTF8Encoding object that uses replacement fallback to replace each string that it can't encode and each byte that it can't decode with a question mark ("?") character.
To confirm this, if you replace Encoding.UTF8 with new UTF8Encoding(true, true) in your test example, you will get the following exception:
EncoderFallbackException: Unable to translate Unicode character \uDBC0 at index 1 to specified code page.
If you are going to try to serialize invalid Unicode char values, you're going to need to manually encode them as, e.g., a byte array using the following:
public static partial class TextExtensions
{
static void ToBytesWithoutEncoding(char c, out byte lower, out byte upper)
{
var u = (uint)c;
lower = unchecked((byte)u);
upper = unchecked((byte)(u >> 8));
}
public static byte[] ToByteArrayWithoutEncoding(this char c)
{
byte lower, upper;
ToBytesWithoutEncoding(c, out lower, out upper);
return new byte[] { lower, upper };
}
public static byte[] ToByteArrayWithoutEncoding(this ICollection<char> list)
{
if (list == null)
return null;
var bytes = new byte[checked(list.Count * 2)];
int to = 0;
foreach (var c in list)
{
ToBytesWithoutEncoding(c, out bytes[to], out bytes[to + 1]);
to += 2;
}
return bytes;
}
public static char ToCharWithoutEncoding(this byte[] bytes)
{
return bytes.ToCharWithoutEncoding(0);
}
public static char ToCharWithoutEncoding(this byte[] bytes, int position)
{
if (bytes == null)
return default(char);
char c = default(char);
if (position < bytes.Length)
c += (char)bytes[position];
if (position + 1 < bytes.Length)
c += (char)((uint)bytes[position + 1] << 8);
return c;
}
public static List<char> ToCharListWithoutEncoding(this byte[] bytes)
{
if (bytes == null)
return null;
var chars = new List<char>(bytes.Length / 2 + bytes.Length % 2);
for (int from = 0; from < bytes.Length; from += 2)
{
chars.Add(bytes.ToCharWithoutEncoding(from));
}
return chars;
}
}
Then modify your test method as follows:
public void Utf8JsonCharSerializeAndDeserializeShouldEqualFixed()
{
Utf8JsonCharSerializeAndDeserializeShouldEqualFixed((char)56256);
}
public void Utf8JsonCharSerializeAndDeserializeShouldEqualFixed(char utfChar)
{
byte[] data;
using (MemoryStream ms = new MemoryStream())
{
using (StreamWriter writer = new StreamWriter(ms, new UTF8Encoding(true, true), 1024))
{
var serializer = new JsonSerializer();
serializer.Serialize(writer, utfChar.ToByteArrayWithoutEncoding());
}
data = ms.ToArray();
}
using (MemoryStream ms = new MemoryStream(data))
{
using (StreamReader reader = new StreamReader(ms, true))
{
using (JsonTextReader jsonReader = new JsonTextReader(reader))
{
var serializer = new JsonSerializer();
char deserializedChar = serializer.Deserialize<byte[]>(jsonReader).ToCharWithoutEncoding();
//Console.WriteLine(string.Format("{0}, {1}", utfChar, deserializedChar));
Assert.AreEqual(utfChar, deserializedChar);
Assert.AreEqual((int)utfChar, (int)deserializedChar);
}
}
}
}
Or, if you have a List<char> property in some container class, you can create the following converter:
public class CharListConverter : JsonConverter
{
public override bool CanConvert(Type objectType)
{
return objectType == typeof(List<char>);
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
if (reader.TokenType == JsonToken.Null)
return null;
var bytes = serializer.Deserialize<byte[]>(reader);
return bytes.ToCharListWithoutEncoding();
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
var list = (ICollection<char>)value;
var bytes = list.ToByteArrayWithoutEncoding();
serializer.Serialize(writer, bytes);
}
}
And apply it as follows:
public class RootObject
{
[JsonConverter(typeof(CharListConverter))]
public List<char> Characters { get; set; }
}
In both cases Json.NET will encode the byte array as Base64.
Related
I read from a stream that contains CRLF each X bytes (for me X is 2033 because the file is generated with bcp, I put X to 4 in sample code). I would like to transform the stream into another stream without this CRLF. The new stream will be deserialized as xml.
Likewise, I can do it easily and runs gracefully in this way:
using System.IO;
using System.Text;
using System.Xml.Linq;
public class CREliminator
{
public const int BcpChunkSize = 4;
public Stream Run(StreamReader reader)
{
var auxstream = new MemoryStream();
var auxwriter = new StreamWriter(auxstream);
var chunk = new char[BcpChunkSize];
do
{
var n_bytes =
reader
.ReadBlock(chunk, 0, BcpChunkSize);
auxwriter.Write(chunk[..n_bytes]);
auxwriter.Flush();
if (n_bytes == BcpChunkSize)
{
char[] chunk2 = new char[2];
n_bytes = reader.ReadBlock(chunk2, 0, 2);
}
} while (!reader.EndOfStream);
auxstream.Position = 0;
return auxstream;
}
}
public class UnitTest1
{
[Fact]
public void Test1()
{
var CRLF="\r\n";
var string_data = $"<doc{CRLF}umen{CRLF}t>A<{CRLF}/doc{CRLF}umen{CRLF}t>";
var expected = string_data.Replace(CRLF, "");
// to stream
var memory = new MemoryStream(Encoding.UTF8.GetBytes(string_data));
var data = new StreamReader(memory);
// act
var result = new CREliminator().Run(data);
// assert
var x = XDocument.Load(result);
Assert.Equal(expected, x.ToString());
}
}
But this code loads all stream in memory before to return the new stream.
My question is, how can do it in a Lazy mode? I mean, processing stream when some process is reading from new stream.
Thanks.
Only Stream.Read method needs to be implemented:
Read from source stream in chunks
Skip 2 bytes after every chunk
using System.Buffers;
using System.Text;
var sourceString = string.Concat(
Enumerable.Range(1, 10).Select(_ => "Foo \r\nBar \r\nBaz!\r\n"));
Console.WriteLine("Source: " + sourceString);
var encoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
var sourceBytes = encoding.GetBytes(sourceString);
using var stream = new MemoryStream(sourceBytes);
using var filter = new CrLfFilteringStream(stream, 4);
using var reader = new StreamReader(filter, encoding);
var res = reader.ReadToEnd();
Console.WriteLine("Result: " + res);
public class CrLfFilteringStream : Stream
{
private readonly Stream _stream;
private readonly int _chunkSize;
private readonly byte[] _chunk;
private int _chunkPosition;
private int _chunkLength;
public CrLfFilteringStream(Stream stream, int chunkSize)
{
_stream = stream ?? throw new ArgumentNullException(nameof(stream));
_chunkSize = chunkSize;
_chunkPosition = chunkSize;
_chunkLength = chunkSize;
_chunk = ArrayPool<byte>.Shared.Rent(chunkSize);
}
public override int Read(byte[] buffer, int offset, int count)
{
var toRead = count;
var bufferPosition = 0;
Span<byte> sink = stackalloc byte[2];
while (toRead > 0 && _chunkLength > 0)
{
if (_chunkPosition >= _chunkSize)
{
_chunkPosition = 0;
_chunkLength = _stream.Read(_chunk, 0, _chunkSize);
// Skip CR LF.
_stream.Read(sink);
}
var currentRead = Math.Min(_chunkLength, toRead);
Array.Copy(_chunk, _chunkPosition, buffer, bufferPosition, currentRead);
toRead -= currentRead;
bufferPosition += currentRead;
_chunkPosition += currentRead;
}
return count - toRead;
}
public override void Flush() => throw new NotSupportedException();
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => throw new NotSupportedException();
public override long Position
{
get => throw new NotSupportedException();
set => throw new NotSupportedException();
}
protected override void Dispose(bool disposing)
{
ArrayPool<byte>.Shared.Return(_chunk);
base.Dispose(disposing);
}
}
This code rents a single chunk-sized buffer from ArrayPool and does not allocate anything else (besides CrLfFilteringStream instance).
I have a class that contains as follow
public class foo
{
public long id= 0;
public char[] TestId = new char[25];// new char[25];
public char[] GUID = new char[64]; //new char[64];
public char[] sign = new char[61];//new char[61];
public int DemandDate = 0;
public int CompleteDate = 0;
public int DownloadDate = 0;
}
static public void WriteFile(string fileName, List<ClientInfo> list)
{
using (FileStream fs = new FileStream(fileName, FileMode.Create))
{
using (var writer = new BinaryWriter(fs, Encoding.UTF8, false))
{
foreach(ClientInfo ci in list)
{
WriteClientInfo(writer, offset, ociv);
offset += header.recordSize;
}
}
}
}
private static void WriteClientInfo(BinaryWriter writer, int offset, ClientInfo ci)
{
try
{
writer.Write(ci.ID);
writer.Write(ci.TestId, 0, 25);
writer.Write(ci.GUID, 0, 64 );
writer.Write(ci.Sign, 0, 61);
writer.Write(ci.DemandDate);
writer.Write(ci.CompleteDate);
writer.Write(ci.DownloadDate);
}
catch (Exception ex)
{
}
}
when I serialize this ci.TestId will crash, stating that the index is out of range.
I know that the value in ci.TestId = FD3394, but I need to be able to save the length of 25 char, not the actual length of the ci.TestId. Because I read this structure from a program that is written in C++, and it has to be the exact size, of the length of the char in the definition, not what is used. I modify it in C# but needs to be read back in C++ as well.
You can use the PadLeft() or PadRight() method like this
TestId = TestId.PadLeft(25);
The left side will be filled with spaces.
You have to convert it to a string and then re-convert it to char[]
TestId = _testId.PadLeft(25).ToCharArray();
I would check that serialized ClientInfo data isn't truncating/trimming the ci.TestId. Are you sure that the ci.TestId looks something like:
'FD3394 '. (i.e. with some sort of front for rear padding. Otherwise, my guess is that you will not be able to write 25 chars count if it doen't actually have 25 char. Maybe formatting ci.TestId with padding when writing.
Modify your writer code:
if(ci.TestId.Length >= 25){
writer.Write(ci.TestId, 0, 25);
} else {
writer.Write(ci.TestId);
writer.Write(new char[25-ci.TestId.Length]);
}
If you want the nils on the left, write them first:
else {
writer.Write(new char[25-ci.TestId.Length]);
writer.Write(ci.TestId);
}
I want to add echo to wave file, in order to do so I need to open it and then save it first. I've written some methods but output file is not correct.
Testing input file size: 731 014
Output file sieze: 730 898
But the "space on disc" is the same for both files, which is 733 184
Error code or whatever it is displayed when opened: 0xc00d36c4
What can be the issue here? This looks pretty simple and yet it's not working. Here's my header, read and write methods:
class WaveFile
{
struct WaveHeader
{
public byte[] RiffID;
public uint fileSize;
public byte[] format;
//Wave format chunk 1
public byte[] fmtID;
public uint fmtSize;
public ushort audioFormat;
public ushort channels;
public uint sampleRate;
public uint byteRate;
public ushort blockAlign;
public int bitsPerSample;
//Wave format chunk 2
public byte[] dataID;
public uint dataSize;
}
uint samples;
public List<short> L;
public List<short> R;
WaveHeader header = new WaveHeader();
//loading file, preparation for modyfying
public bool loadWaveFile(string filePath)
{
using (FileStream fs = File.Open(filePath, FileMode.Open))
using (BinaryReader reader = new BinaryReader(fs))
{
// chunk 0
header.RiffID = reader.ReadBytes(4);
header.fileSize = reader.ReadUInt32();
header.format = reader.ReadBytes(4);
// chunk 1
header.fmtID = reader.ReadBytes(4);
header.fmtSize = reader.ReadUInt32();
header.audioFormat = reader.ReadUInt16();
header.channels = reader.ReadUInt16();
header.sampleRate = reader.ReadUInt32();
header.byteRate = reader.ReadUInt32();
header.blockAlign = reader.ReadUInt16();
header.bitsPerSample = reader.ReadInt16();
// chunk 2
header.dataID = reader.ReadBytes(4);
header.dataSize = reader.ReadUInt32();
// DATA is stereo
L = new List<short>();
R = new List<short>();
samples = header.dataSize / header.blockAlign;
for (int i = 0; i < samples; i++)
{
L.Add((short)reader.ReadUInt16());
R.Add((short)reader.ReadUInt16());
}
reader.Close();
fs.Close();
}
return true;
}
public bool addEcho(int threadsNumber, int echoesNumber, int delay, int attenuation)
{
return true;
}
public bool saveWaveFile(string savingPath)
{
using (FileStream fs = new FileStream(#savingPath + "\\echo.wav", FileMode.Create))
using (BinaryWriter writer = new BinaryWriter(fs))
{
//chunk 0
writer.Write(header.RiffID);
writer.Write(header.fileSize);
writer.Write(header.format);
//chunk 1
writer.Write(header.fmtID);
writer.Write(header.fmtSize);
writer.Write(header.audioFormat);
writer.Write(header.channels);
writer.Write(header.sampleRate);
writer.Write(header.byteRate);
writer.Write(header.blockAlign);
writer.Write(header.bitsPerSample);
//chunk 2
writer.Write(header.dataID);
writer.Write(header.dataSize);
for (int i = 0; i < samples; i++)
{
writer.Write(L[i]);
writer.Write(R[i]);
}
writer.Close();
fs.Close();
return true;
}
}
}
I didn't find out what the issue was, but for echo purposes this class will work:
Class WaveFile
{
byte[] byteArray;
public void loadWaveFile(string filePath)
{
byteArray = File.ReadAllBytes(filePath);
}
public bool addEcho(int threadsNumber, int echoesNumber, int delay, int attenuation)
{
return true;
}
public bool saveWaveFile(string savingPath)
{
using (FileStream fs = new FileStream(#savingPath + "\\echo.wav", FileMode.Create))
using (BinaryWriter writer = new BinaryWriter(fs))
{
writer.Write(byteArray);
writer.Close();
fs.Close();
return true;
}
}
}
I need to communicate from a C# application to another application via encrypted messages in OFB mode. I know that RijndaelManaged does not have support for AES OFB mode. Is there anybody more experienced than me aware of any other way to encrypt/decrypt using OFB mode?
The following stream implements OFB by using a key stream generated by a zero-fed CBC cipher stream.
public class OFBStream : Stream
{
private const int BLOCKS = 16;
private const int EOS = 0; // the goddess of dawn is found at the end of the stream
private Stream parent;
private CryptoStream cbcStream;
private CryptoStreamMode mode;
private byte[] keyStreamBuffer;
private int keyStreamBufferOffset;
private byte[] readWriteBuffer;
public OFBStream (Stream parent, SymmetricAlgorithm algo, CryptoStreamMode mode)
{
if (algo.Mode != CipherMode.CBC)
algo.Mode = CipherMode.CBC;
if (algo.Padding != PaddingMode.None)
algo.Padding = PaddingMode.None;
this.parent = parent;
this.cbcStream = new CryptoStream (new ZeroStream (), algo.CreateEncryptor (), CryptoStreamMode.Read);
this.mode = mode;
keyStreamBuffer = new byte[algo.BlockSize * BLOCKS];
readWriteBuffer = new byte[keyStreamBuffer.Length];
}
public override int Read (byte[] buffer, int offset, int count)
{
if (!CanRead) {
throw new NotSupportedException ();
}
int toRead = Math.Min (count, readWriteBuffer.Length);
int read = parent.Read (readWriteBuffer, 0, toRead);
if (read == EOS)
return EOS;
for (int i = 0; i < read; i++) {
// NOTE could be optimized (branches for each byte)
if (keyStreamBufferOffset % keyStreamBuffer.Length == 0) {
FillKeyStreamBuffer ();
keyStreamBufferOffset = 0;
}
buffer [offset + i] = (byte)(readWriteBuffer [i]
^ keyStreamBuffer [keyStreamBufferOffset++]);
}
return read;
}
public override void Write (byte[] buffer, int offset, int count)
{
if (!CanWrite) {
throw new NotSupportedException ();
}
int readWriteBufferOffset = 0;
for (int i = 0; i < count; i++) {
if (keyStreamBufferOffset % keyStreamBuffer.Length == 0) {
FillKeyStreamBuffer ();
keyStreamBufferOffset = 0;
}
if (readWriteBufferOffset % readWriteBuffer.Length == 0) {
parent.Write (readWriteBuffer, 0, readWriteBufferOffset);
readWriteBufferOffset = 0;
}
readWriteBuffer [readWriteBufferOffset++] = (byte)(buffer [offset + i]
^ keyStreamBuffer [keyStreamBufferOffset++]);
}
parent.Write (readWriteBuffer, 0, readWriteBufferOffset);
}
private void FillKeyStreamBuffer ()
{
int read = cbcStream.Read (keyStreamBuffer, 0, keyStreamBuffer.Length);
// NOTE undocumented feature
// only works if keyStreamBuffer.Length % blockSize == 0
if (read != keyStreamBuffer.Length)
throw new InvalidOperationException ("Implementation error: could not read all bytes from CBC stream");
}
public override bool CanRead {
get { return mode == CryptoStreamMode.Read; }
}
public override bool CanWrite {
get { return mode == CryptoStreamMode.Write; }
}
public override void Flush ()
{
// should never have to be flushed, implementation empty
}
public override bool CanSeek {
get { return false; }
}
public override long Seek (long offset, System.IO.SeekOrigin origin)
{
throw new NotSupportedException ();
}
public override long Position {
get { throw new NotSupportedException (); }
set { throw new NotSupportedException (); }
}
public override long Length {
get { throw new NotSupportedException (); }
}
public override void SetLength (long value)
{
throw new NotSupportedException ();
}
}
Additional class ZeroStream required by OFBStream
class ZeroStream : System.IO.Stream
{
public override int Read (byte[] buffer, int offset, int count)
{
for (int i = 0; i < count; i++) {
buffer [offset + i] = 0;
}
return count;
}
public override bool CanRead {
get { return true; }
}
... the rest is not implemented
}
And you can use it as I do for a test vector:
// NIST CAVP test vector F.4.1: OFB-AES128.Encrypt from NIST SP 800-38A
RijndaelManaged aes = new RijndaelManaged ();
aes.Key = FromHex ("2b7e151628aed2a6abf7158809cf4f3c");
aes.IV = FromHex ("000102030405060708090A0B0C0D0E0F");
MemoryStream testVectorStream = new MemoryStream (FromHex (
"6bc1bee22e409f96e93d7e117393172aae2d8a571e03ac9c9eb76fac45af8e5130c81c46a35ce411e5fbc1191a0a52eff69f2445df4f9b17ad2b417be66c3710"));
OFBStream testOFBStream = new OFBStream (testVectorStream, aes, CryptoStreamMode.Read);
MemoryStream cipherTextStream = new MemoryStream ();
testOFBStream.CopyTo (cipherTextStream);
Console.WriteLine (ToHex (cipherTextStream.ToArray ()));
Note that the stream handling has not been fully tested (yet).
I have a long array. How to write this array to a binary file?
Problem is that if I convert it into byte array some values are changed.
The array is like:
long array = new long[160000];
Give some code snippet.
The BinaryFormatter will be the easiest.
Also valuetypes (I assume this is what you mean by long), serializes very efficiently.
var array = new[] { 1L, 2L, 3L };
using (var stream = new FileStream("test.bin", FileMode.Create, FileAccess.Write, FileShare.None))
using (var writer = new BinaryWriter(stream))
{
foreach (long item in array)
{
writer.Write(item);
}
}
How are the values changed? And an array of long can be copied into an array of byte very quickly, no need for serialization.
static void Main(string[] args) {
System.Random random = new Random();
long[] arrayOriginal = new long[160000];
long[] arrayRead = null;
for (int i =0 ; i < arrayOriginal.Length; i++) {
arrayOriginal[i] = random.Next(int.MaxValue) * random.Next(int.MaxValue);
}
byte[] bytesOriginal = new byte[arrayOriginal.Length * sizeof(long)];
System.Buffer.BlockCopy(arrayOriginal, 0, bytesOriginal, 0, bytesOriginal.Length);
using (System.IO.MemoryStream stream = new System.IO.MemoryStream()) {
// write
stream.Write(bytesOriginal, 0, bytesOriginal.Length);
// reset
stream.Flush();
stream.Position = 0;
int expectedLength = 0;
checked {
expectedLength = (int)stream.Length;
}
// read
byte[] bytesRead = new byte[expectedLength];
if (expectedLength == stream.Read(bytesRead, 0, expectedLength)) {
arrayRead = new long[expectedLength / sizeof(long)];
Buffer.BlockCopy(bytesRead, 0, arrayRead, 0, expectedLength);
}
else {
// exception
}
// check
for (int i = 0; i < arrayOriginal.Length; i++) {
if (arrayOriginal[i] != arrayRead[i]) {
throw new System.Exception();
}
}
}
System.Console.WriteLine("Done");
System.Console.ReadKey();
}