We Keep Coding

How do I add bits to a MemoryStream

So I've been trying to add bits of a value to a MemoryStream but the issue is I have no idea how. I've seen that it's used for performance when it comes to networking.
I know I want a function that takes the bit value and how many bits it takes to store that value. So for instance, to store the value 3 I would need to allocate 2 bits 0000 0000 0000 0011. I would essentially pack the bits into a byte array and then add that byte array to the MemoryStream
var ms = new MemoryStream();
ms.WriteByte(1);
ms.WriteByte(1);
ms.WriteByte(1);
ms.WriteByte(1);
ms.WriteByte(1);
WriteBits(2, 3);
WriteBits(1, 1);
void WriteBits(int numbBits, int value)
{
/* Convert the "value" to a byte or bytes and add it to the MemoryStream */
}
How do I properly implement this?
Java Example
public void writeBits(int numBits, int value) {
int bytePos = bitPosition >> 3;
int bitOffset = 8 - (bitPosition & 7);
bitPosition += numBits;
for (; numBits > bitOffset; bitOffset = 8) {
buffer[bytePos] &= ~bitMaskOut[bitOffset]; // mask out the desired area
buffer[bytePos++] |= (value >> (numBits - bitOffset))
& bitMaskOut[bitOffset];
numBits -= bitOffset;
}
if (numBits == bitOffset) {
buffer[bytePos] &= ~bitMaskOut[bitOffset];
buffer[bytePos] |= value & bitMaskOut[bitOffset];
} else {
buffer[bytePos] &= ~(bitMaskOut[numBits] << (bitOffset - numBits));
buffer[bytePos] |= (value & bitMaskOut[numBits]) << (bitOffset - numBits);
}
}

So I've been trying to add bits of a value to a MemoryStream
You don't, MemoryStream only handles bytes.
So for instance, to store the value 3 I would need to allocate 2 bits
This would only be true if the range of values you want to store is [0, 3]. If you want the possibility of storing any larger value you need more bits.
How do I properly implement this?
you would need to implement your own bit-stream. The java example looks like it has a byte[] buffer, and a bitPosition. You would need to implement this. The bif-fiddeling code looks like it should work just about the same in c#. Once you have a byte[] it is trivial to write this out to whatever stream you want, and usually possible to send directly over the network.
I've seen that it's used for performance when it comes to networking
I think there is a significant misunderstanding here. While you could manually manipulate individual bits, in most cases it would just be a waste of (development) time.
In general, a better way to get good performance is to use existing, well optimized and designed libraries. And there are a variety of serialization libraries that converts objects to byte-streams for you. An example would be protobuf (.net), this actually encodes numbers with a variable number of bytes.
If you still need smaller data it is usually more efficient to use some form of compression. The old classic deflate usually gives good compromise between size and performance, while algorithms like lz4 prioritizes speed over compression ratio.

I had exactly the same problem and wrote an entire BitStream library which can handle any reads and writes of an arbitrary number of bits to a MemoryStream (and any other stream, too). The library is open-source, MIT-licensed and fast (https://github.com/martinweihrauch/BitStream).
Writing bits to a MemoryStream.
These are the steps to write a value to a certain number of bits to a specific position in the MemoryStream:
Have a Stream available, e. g. a MemoryStream(), to which you want to write.
Connect this Stream to a new Bitstream
using SharpBitStream;
uint[] testDataUnsigned = { 5, 62, 17, 50, 33 };
var ms = new MemoryStream();
var bs = new BitStream(ms);
Now, you can start writing to the BitStream like this:
foreach(var bits in testDataUnsigned)
{
bs.WriteUnsigned(6, (ulong)bits);
}
Writing can be done as above by only providing the bitlength and the value, but you of course also have full controll of exactly where to write the bits like so:
bs.WriteUnsigned(3, 2, 4, 5);
// Overloaded signature of WriteUnsigned:
// public void WriteUnsigned(long offsetByteStream, int offsetBit, int bitLength, ulong value)
// For signed numbers (e. g. -17), use
// bs.WriteSigned(3, 2, 4, -5);
This means, you can control that you write to the 4th byte (3, because starting at 0) in the underlying byte Stream,
starting from the the 3rd (=2) position of the byte with a length of 6 bits and the value 5 (=0b0101);
Reading works similarly:
Just read the next 6 bits, wherever your byte and bit position is (e. g. for loops, etc):
ulong number = bs.ReadUnsigned(6);
// For Signed, use
// long number = bs.ReadSigned(6);
Read a specific position, in this example read 4 bits from 3rd byte in Stream (2= 3rd position), starting with bit #0:
ulong number = bs.ReadUnsigned(2, 0, 4);
// For signed, use
// long number = bs.ReadSigned(2, 0, 4);
Note: The bit offset is always counting from 0 from the left-most position.

Defining a bit[] array in C#

currently im working on a solution for a prime-number calculator/checker. The algorythm is already working and verry efficient (0,359 seconds for the first 9012330 primes). Here is a part of the upper region where everything is declared:
const uint anz = 50000000;
uint a = 3, b = 4, c = 3, d = 13, e = 12, f = 13, g = 28, h = 32;
bool[,] prim = new bool[8, anz / 10];
uint max = 3 * (uint)(anz / (Math.Log(anz) - 1.08366));
uint[] p = new uint[max];
Now I wanted to go to the next level and use ulong's instead of uint's to cover a larger area (you can see that already), where i tapped into my problem: the bool-array.
Like everybody should know, bool's have the length of a byte what takes a lot of memory when creating the array... So I'm searching for a more resource-friendly way to do that.
My first idea was a bit-array -> not byte! <- to save the bool's, but haven't figured out how to do that by now. So if someone ever did something like this, I would appreciate any kind of tips and solutions. Thanks in advance :)

You can use BitArray collection:
http://msdn.microsoft.com/en-us/library/system.collections.bitarray(v=vs.110).aspx
MSDN Description:
Manages a compact array of bit values, which are represented as Booleans, where true indicates that the bit is on (1) and false indicates the bit is off (0).

You can (and should) use well tested and well known libraries.
But if you're looking to learn something (as it seems to be the case) you can do it yourself.
Another reason you may want to use a custom bit array is to use the hard drive to store the array, which comes in handy when calculating primes. To do this you'd need to further split addr, for example lowest 3 bits for the mask, next 28 bits for 256MB of in-memory storage, and from there on - a file name for a buffer file.
Yet another reason for custom bit array is to compress the memory use when specifically searching for primes. After all more than half of your bits will be 'false' because the numbers corresponding to them would be even, so in fact you can both speed up your calculation AND reduce memory requirements if you don't even store the even bits. You can do that by changing the way addr is interpreted. Further more you can also exclude numbers divisible by 3 (only 2 out of every 6 numbers has a chance of being prime) thus reducing memory requirements by 60% compared to plain bit array.
Notice the use of shift and logical operators to make the code a bit more efficient.
byte mask = (byte)(1 << (int)(addr & 7)); for example can be written as
byte mask = (byte)(1 << (int)(addr % 8));
and addr >> 3 can be written as addr / 8
Testing shift/logical operators vs division shows 2.6s vs 4.8s in favor of shift/logical for 200000000 operations.
Here's the code:
void Main()
{
var barr = new BitArray(10);
barr[4] = true;
Console.WriteLine("Is it "+barr[4]);
Console.WriteLine("Is it Not "+barr[5]);
}
public class BitArray{
private readonly byte[] _buffer;
public bool this[long addr]{
get{
byte mask = (byte)(1 << (int)(addr & 7));
byte val = _buffer[(int)(addr >> 3)];
bool bit = (val & mask) == mask;
return bit;
}
set{
byte mask = (byte) ((value ? 1:0) << (int)(addr & 7));
int offs = (int)addr >> 3;
_buffer[offs] = (byte)(_buffer[offs] | mask);
}
}
public BitArray(long size){
_buffer = new byte[size/8 + 1]; // define a byte buffer sized to hold 8 bools per byte. The spare +1 is to avoid dealing with rounding.
}
}

Generate unique code from primary key

I have a table of orders and I want to give users a unique code for an order whilst hiding the incrementing identity integer primary key because I don't want to give away how many orders have been made.
One easy way of making sure the codes are unique is to use the primary key to determine the code.
So how can I transform an integer into a friendly, say, eight alpha numeric code such that every code is unique?

The easiest way (if you want an alpha numeric code) is to convert the integer primary key to HEX (like below). And, you can Use `PadLeft()' to make sure the string has 8 characters. But, when the number of orders grow, 8 characters will not be enough.
var uniqueCode = intPrimaryKey.ToString("X").PadLeft(8, '0');
Or, you can create an offset of your primary key, before converting it to HEX, like below:
var uniqueCode = (intPrimaryKey + 999).ToString("X").PadLeft(8, '0');

Assuming the total number of orders being created isn't going to get anywhere near the total number of identifiers in your pool, a reasonably effective technique is to simply generate a random identifier and see if it is used already; continue generating new identifiers until you find one not previously used.

A quick and easy way to do this is to have a guid column that has a default value of
left(newid(),8)
This solution will generally give you a unique value for each row. But if you have extremely large amounts of orders this will not be unique and you should use just the newid() value to generate the guid.

I would just use MD5 for this. MD5 offers enough "uniqueness" for a small subset of integers that represent your customer orders.
For an example see this answer. You will need to adjust input parameter from string to int (or alternatively just call ToString on your number and use the code as-is).

If you would like something that would be difficult to trace and you don;t mind it being 16 characters, you could use something like this that includes some random numbers and mixes the byte positions of the original input with them: (EDITED to make a bit more untraceable, by XOR-ing with the generated random numbers).
public static class OrderIdRandomizer
{
private static readonly Random _rnd = new Random();
public static string GenerateFor(int orderId)
{
var rndBytes = new byte[4];
_rnd.NextBytes(rndBytes);
var bytes = new byte[]
{
(byte)rndBytes[0],
(byte)(((byte)(orderId >> 8)) ^ rndBytes[0]),
(byte)(((byte)(orderId >> 24)) ^ rndBytes[1]),
(byte)rndBytes[1],
(byte)(((byte)(orderId >> 16)) ^ rndBytes[2]),
(byte)rndBytes[2],
(byte)(((byte)(orderId)) ^ rndBytes[3]),
(byte)rndBytes[3],
};
return string.Concat(bytes.Select(b => b.ToString("X2")));
}
public static int ReconstructFrom(string generatedId)
{
if (generatedId == null || generatedId.Length != 16)
throw new InvalidDataException("Invalid generated order id");
var bytes = new byte[8];
for (int i = 0; i < 8; i++)
bytes[i] = byte.Parse(generatedId.Substring(i * 2, 2), System.Globalization.NumberStyles.HexNumber);
return (int)(
((bytes[2] ^ bytes[3]) << 24) |
((bytes[4] ^ bytes[5]) << 16) |
((bytes[1] ^ bytes[0]) << 8) |
((bytes[6] ^ bytes[7])));
}
}
Usage:
var obfuscatedId = OrderIdRandomizer.GenerateFor(123456);
Console.WriteLine(obfuscatedId);
Console.WriteLine(OrderIdRandomizer.ReconstructFrom(obfuscatedId));
Disadvantage: If the algorithm is know, it is obviously easy to break.
Advantage: It is completely custom, i.e. not an established algorithm like MD5 that might be easier to guess/crack if you do not know what algorithm is being used.

Encrypt a number to another number of the same length

I need a way to take a 12 digit number and encrypt it to a different 12 digit number (no characters other than 0123456789). Then at a later point I need to be able to decrypt the encrypted number back to the original number.
It is important that it isn't obvious if 2 encrypted numbers are in order. So for instance if I encrypt 0000000000001 it should look totally different when encrypted than 000000000002. It doesn't have to be the most secure thing in the world, but the more secure the better.
I've been looking around a lot but haven't found anything that seems to be a perfect fit. From what I've seen some type of XOR might be the easiest way to go, but I'm not sure how to do this.
Thanks,
Jim

I ended up solving this thanks to you guys using "FPE from a prefix cipher" from the wikipedia page http://en.wikipedia.org/wiki/Format-preserving_encryption. I'll give the basic steps below to hopefully be helpful for someone in the future.
NOTE - I'm sure any expert will tell you this is a hack. The numbers seemed random and it was secure enough for what I needed, but if security is a big concern use something else. I'm sure experts can point to holes in what I did. My only goal for posting this is because I would have found it useful when doing my search for an answer to the problem. Also only use this in situations where it couldn't be decompiled.
I was going to post steps, but its too much to explain. I'll just post my code. This is my proof of concept code I still need to clean up, but you'll get the idea. Note my code is specific to a 12 digit number, but adjusting for others should be easy. Max is probably 16 with the way I did it.
public static string DoEncrypt(string unencryptedString)
{
string encryptedString = "";
unencryptedString = new string(unencryptedString.ToCharArray().Reverse().ToArray());
foreach (char character in unencryptedString.ToCharArray())
{
string randomizationSeed = (encryptedString.Length > 0) ? unencryptedString.Substring(0, encryptedString.Length) : "";
encryptedString += GetRandomSubstitutionArray(randomizationSeed)[int.Parse(character.ToString())];
}
return Shuffle(encryptedString);
}
public static string DoDecrypt(string encryptedString)
{
// Unshuffle the string first to make processing easier.
encryptedString = Unshuffle(encryptedString);
string unencryptedString = "";
foreach (char character in encryptedString.ToCharArray().ToArray())
unencryptedString += GetRandomSubstitutionArray(unencryptedString).IndexOf(int.Parse(character.ToString()));
// Reverse string since encrypted string was reversed while processing.
return new string(unencryptedString.ToCharArray().Reverse().ToArray());
}
private static string Shuffle(string unshuffled)
{
char[] unshuffledCharacters = unshuffled.ToCharArray();
char[] shuffledCharacters = new char[12];
shuffledCharacters[0] = unshuffledCharacters[2];
shuffledCharacters[1] = unshuffledCharacters[7];
shuffledCharacters[2] = unshuffledCharacters[10];
shuffledCharacters[3] = unshuffledCharacters[5];
shuffledCharacters[4] = unshuffledCharacters[3];
shuffledCharacters[5] = unshuffledCharacters[1];
shuffledCharacters[6] = unshuffledCharacters[0];
shuffledCharacters[7] = unshuffledCharacters[4];
shuffledCharacters[8] = unshuffledCharacters[8];
shuffledCharacters[9] = unshuffledCharacters[11];
shuffledCharacters[10] = unshuffledCharacters[6];
shuffledCharacters[11] = unshuffledCharacters[9];
return new string(shuffledCharacters);
}
private static string Unshuffle(string shuffled)
{
char[] shuffledCharacters = shuffled.ToCharArray();
char[] unshuffledCharacters = new char[12];
unshuffledCharacters[0] = shuffledCharacters[6];
unshuffledCharacters[1] = shuffledCharacters[5];
unshuffledCharacters[2] = shuffledCharacters[0];
unshuffledCharacters[3] = shuffledCharacters[4];
unshuffledCharacters[4] = shuffledCharacters[7];
unshuffledCharacters[5] = shuffledCharacters[3];
unshuffledCharacters[6] = shuffledCharacters[10];
unshuffledCharacters[7] = shuffledCharacters[1];
unshuffledCharacters[8] = shuffledCharacters[8];
unshuffledCharacters[9] = shuffledCharacters[11];
unshuffledCharacters[10] = shuffledCharacters[2];
unshuffledCharacters[11] = shuffledCharacters[9];
return new string(unshuffledCharacters);
}
public static string DoPrefixCipherEncrypt(string strIn, byte[] btKey)
{
if (strIn.Length < 1)
return strIn;
// Convert the input string to a byte array
byte[] btToEncrypt = System.Text.Encoding.Unicode.GetBytes(strIn);
RijndaelManaged cryptoRijndael = new RijndaelManaged();
cryptoRijndael.Mode =
CipherMode.ECB;//Doesn't require Initialization Vector
cryptoRijndael.Padding =
PaddingMode.PKCS7;
// Create a key (No IV needed because we are using ECB mode)
ASCIIEncoding textConverter = new ASCIIEncoding();
// Get an encryptor
ICryptoTransform ictEncryptor = cryptoRijndael.CreateEncryptor(btKey, null);
// Encrypt the data...
MemoryStream msEncrypt = new MemoryStream();
CryptoStream csEncrypt = new CryptoStream(msEncrypt, ictEncryptor, CryptoStreamMode.Write);
// Write all data to the crypto stream to encrypt it
csEncrypt.Write(btToEncrypt, 0, btToEncrypt.Length);
csEncrypt.Close();
//flush, close, dispose
// Get the encrypted array of bytes
byte[] btEncrypted = msEncrypt.ToArray();
// Convert the resulting encrypted byte array to string for return
return (Convert.ToBase64String(btEncrypted));
}
private static List<int> GetRandomSubstitutionArray(string number)
{
// Pad number as needed to achieve longer key length and seed more randomly.
// NOTE I didn't want to make the code here available and it would take too longer to clean, so I'll tell you what I did. I basically took every number seed that was passed in and prefixed it and postfixed it with some values to make it 16 characters long and to get a more unique result. For example:
// if (number.Length = 15)
// number = "Y" + number;
// if (number.Length = 14)
// number = "7" + number + "z";
// etc - hey I already said this is a hack ;)
// We pass in the current number as the password to an AES encryption of each of the
// digits 0 - 9. This returns us a set of values that we can then sort and get a
// random order for the digits based on the current state of the number.
Dictionary<string, int> prefixCipherResults = new Dictionary<string, int>();
for (int ndx = 0; ndx < 10; ndx++)
prefixCipherResults.Add(DoPrefixCipherEncrypt(ndx.ToString(), Encoding.UTF8.GetBytes(number)), ndx);
// Order the results and loop through to build your int array.
List<int> group = new List<int>();
foreach (string key in prefixCipherResults.Keys.OrderBy(k => k))
group.Add(prefixCipherResults[key]);
return group;
}

One more way for simple encryption, you can just substruct each number from 10.
For example
initial numbers: 123456
10-1 = 9
10-2 = 8
10-3 = 7
etc.
and you will get
987654
You can combine it with XOR for more secure encryption.

What you're talking about is kinda like a one-time pad. A key the same length as the plaintext and then doing some modulo math on each individual character.
A xor B = C
C xor B = A
or in other words
A xor B xor B = A
As long as you don't use the same key B on multiple different inputs (e.g. B has to be unique, every single time you encrypt), then in theory you can never recover the original A without knowing what B was. If you use the same B multiple times, then all bets are off.
comment followup:
You shouldn't end up with more bits aftewards than you started with. xor just flips bits, it doesn't have any carry functionality. Ending up with 6 digits is just odd... As for code:
$plaintext = array(digit1, digit2, digit3, digit4, digit5, digit6);
$key = array(key1, key2, key3, key4, key5, key6);
$ciphertext = array()
# encryption
foreach($plaintext as $idx => $char) {
$ciphertext[$idx] = $char xor $key[$idx];
}
# decryption
foreach($ciphertext as $idx => $char) {
$decrypted[$idx] = $char xor $key[$idx];
}
Just doing this as an array for simplicity. For actual data you'd work on a per-byte or per-word basis, and just xor each chunk in sequence. You can use a key string shorter than the input, but that makes it easier to reverse engineer the key. In theory, you could use a single byte to do the xor'ing, but then you've just basically achieved the bit-level equivalent of rot-13.

For example you can add digits of your number with digits some const (214354178963...whatever) and apply "~" operator (reverse all bits) this is not safely but ensure you can decrypt your number allways.

anyone with reflector or ildasm will be able to hack such an encryption algorithm.
I don't know what is your business requirement but you have to know that.

If there's enough wriggle-room in the requirements that you can accept 16 hexadecimal digits as the encrypted side, just interpret the 12 digit decimal number as a 64bit plaintext and use a 64 bit block cipher like Blowfish, Triple-DES or IDEA.

Simple Pseudo-Random Algorithm

I'm need a pseudo-random generator which takes a number as input and returns another number witch is reproducible and seems to be random.
Each input number should match to exactly one output number and vice versa
same input numbers always result in same output numbers
sequential input numbers that are close together (eg. 1 and 2) should produce completely different output numbers (eg. 1 => 9783526, 2 => 283)
It must not be perfect, it's just to create random but reproducible test data.
I use C#.
I wrote this funny piece of code some time ago which produced something random.
public static long Scramble(long number, long max)
{
// some random values
long[] scramblers = { 3, 5, 7, 31, 343, 2348, 89897 };
number += (max / 7) + 6;
number %= max;
// shuffle according to divisibility
foreach (long scrambler in scramblers)
{
if (scrambler >= max / 3) break;
number = ((number * scrambler) % max)
+ ((number * scrambler) / max);
}
return number % max;
}
I would like to have something better, more reliable, working with any size of number (no max argument).
Could this probably be solved using a CRC algorithm? Or some bit shuffling stuff.

I remove the microsoft code from this answer, the GNU code file is a lot longer but basically it contains this from http://cs.uccs.edu/~cs591/bufferOverflow/glibc-2.2.4/stdlib/random_r.c :
int32_t val = state[0];
val = ((state[0] * 1103515245) + 12345) & 0x7fffffff;
state[0] = val;
*result = val;
for your purpose, the seed is state[0] so it would look more like
int getRand(int val)
{
return ((val * 1103515245) + 12345) & 0x7fffffff;
}

You (maybe) can do this easily in C# using the Random class:
public int GetPseudoRandomNumber(int input)
{
Random random = new Random(input);
return random.Next();
}
Since you're explicitly seeding Random with the input, you will get the same output every time given the same input value.

A tausworthe generator is simple to implement and pretty fast. The following pseudocode implementation has full cycle (2**31 - 1, because zero is a fixed point):
def tausworthe(seed)
seed ^= seed >> 13
seed ^= seed << 18
return seed & 0x7fffffff
I don't know C#, but I'm assuming it has XOR (^) and bit shift (<<, >>) operators as in C.
Set an initial seed value, and invoke with seed = tausworthe(seed).

The first two rules suggest a fixed or input-seeded permutation of the input, but the third rule requires a further transform.
Is there any further restriction on what the outputs should be, to guide that transform? - e.g. is there an input set of output values to choose from?
If the only guide is "no max", I'd use the following...
Apply a hash algorithm to the whole input to get the first output item. A CRC might work, but for more "random" results, use a crypto hash algorithm such as MD5.
Use a next permutation algorithm (plenty of links on Google) on the input.
Repeat the hash-then-next-permutation until all required outputs are found.
The next permutation may be overkill though, you could probably just increment the first input (and maybe, on overflow, increment the second and so on) before redoing the hash.
For crypto-style hashing, you'll need a key - just derive something from the input before you start.