We Keep Coding

Using bitmask to look up used arrangements of ints in C#?

I have rarely used bitmasks and am trying to get more familiar with them. I understand various basic usages of them. As I understand them this should be able to work, but it seems to not work.
I have a use case where I have four different ints that may come in different arrangements, and I need to check if the current arrangement of ints has already come before as a different arrangement.
So one iteration they might come as:
2, 5, 10, 8
Next iteration:
1, 0, 2, 5
Now on the next iteration if this comes:
0, 1, 2, 5
It needs to discern that last set has already come in a different arrangement and skip it.
I am wondering, can I create a mask out of these ints, put them in a HashSet, so then I have easy lookup for whether or not that set of ints has come before?
Basically I am doing this:
int mask = int0 & int1 & int2 & int3;
if (checkHashSet.Contains(mask))
return; // int set already came, skip
//int set has not been processed, add mask and process
checkHashSet.Add(mask);
But that seems to be producing a mask that ends up equal to all following masks generated. So this doesn't work.
Can this work like this somehow? What would be the most performant way to check if a set of ints, no matter their arrangement, has already been processed?

Bit mask is generated by shift
int mask = (1 << int0) & (1 << int1) & (1 << int2) & (1 << int3);
HashSet.Add will check whether the item exists, Contains is redundant.
if(checkHashSet.Add(mask))
//int set has not been processed, add mask and process
else
// int set already came, skip
If the integer is greater than 31, you can use long or ulong, if it is greater than 64, then use 2 longs or BigInteger

To your question of "is there a better way". I think probably yes.
1) Sort your inputs and keep a map of entries you've already seen as a string. This is close to what you proposed but would be easier to read & implement.
2) An map having a key that is int[1000] would be easier than bit masks. As you process your input, for each number you find, increment the location in the array ++array[n]. You can then add that to a map with array[] as your key. You can search the map to work out if you've seen a particular combination before.
3) 1<< n only goes so far so you'd need array of ints to do it as a proper mask. array[n / 64] & (1 << (n % 64) ) or something like that. A good solution for embedded perhaps, but generally harder to understand. Also, it won't work reliably if any number is repeated as bits can only only mark one occurrence. Or use BigInteger as per Shingo's answer.
Personally I'd go the first one.

Bitwise Logic knowing what was enabled from the server

I am after some quick code to help with my logic on bitwise code (my first time doing it)
Basically I have 3 properties each with an ID of 1,2,4
int A = 1
int B = 2
int C = 4
Lets say A and C are checked, this will equal 5 and this value will be saved into the database. Good.
Now I need to work out on the way back what was checked with that value.
So in the example, the value is 5, so how will I know with bitwise logic what properties were checked?
So I am not asking for you to do my work, knowing what I do,
int i = A & C;
// if i == 5, then we know it was A and C checked
But do you need to do that for each possibility?
Thanks in Advance

You can simply check for these conditions:
i & A != 0 // true if A is set in i
i & B != 0 // true if B is set in i
Having said that, bitfields are inappropriate in lots of situations. Especially when you store the resulting value into a database it is usually better to just define some more columns storing each flag.

C# Performance and best practice to check the state of a bit Array

Here's my problem:
I have two classes: ClassObj and ClassProperty;
One of the properties of ClassObj (let's call it lstProperty) is a List of ClassProperty (representing the property that current instance of the object have);
At runtime I have stored somewhere an Array of all possible ClassProperty, we will call this Array arrPossibleProperty.
My issue is to find a REALLY fast way of checking if an instance of ClassObj match a particular set of ClassProperty (if he have in his lstProperty all the ClassProperty of a given set).
I was thinking about creating an array of Bit representing the sequence of ClassProperty that the ClassObj posesses. Using as reference the Array arrPossibleProperty, and the index of his Property.
So if for example we have 10 property and a InstanceA of ClassObj have the 1st the 4th and the 9th I would generate this array of bit:
1001000010
My question is, how can I check for example (Fastest and Most performing solution) that the array of bit have (for example) the 3rd and the 4th Property?
Of course if you need there's a more performing way of doing this let me know.

You'll want to use bitwise-operations, such as &, |, ^ or ~ (depending on your needs)
Taking your example of 1001000010, to find out if the 3rd bit is set, you'd want to do this: 1001000010 & 0000000100 != 0000000000, or rather
bool isSet = myProperty & (1 << (option - 1)) != 0; // to find out if `myProperty` has the property `option` set.
myProperty |= 1 << (option - 1); // to set property `option`
For more information about bitwise operations: http://en.wikipedia.org/wiki/Bitwise_operation
Alternatively and a lot more easy to implement, would be using enums.
[Flags]
enum ClassProperty
{
None = 0x00,
First = 0x01,
Second = 0x02,
Third = 0x04,
Fourth = 0x08
// add more if needed
}
Then, you'd use the enum like this
myProperty = ClassProperty.First | ClassProperty.Second; // sets both First and Second
bool thirdIsSet = myProperty.HasFlag(ClassProperty.Third);

You say your array of all possible ClassProperty-objects (arrPossibleProperty) are initalized at run-time. I then assume you don't know how large that array might be. That rules out Enums, which by definition are hard coded. If you instead go for some sort of bitmap as you indicate in your question, you need at some point to iterate arrPossibleProperty and your list of ClassProperty lstProperty in order to do the mapping and unmapping.
My point is you are making a lot of assumptions claiming that doing this bit-fiddling is faster, while you're actually working with high-level C# objects. Never assume. Test. Measure.
Here is a question quite similar to yours, in which the answer uses standard .NET library functions to do what you want:
HashSet<T>().IsSupertSetOf().
Is it fast enough? Then use it instead of creating far more unreadable code with bit-operations.

Testing for bitwise Enum values

I haven't really used bitwise enums before, and I just want to make sure my testing is correct. I am most interested in testing for the values None and All. We receive data from a webservice that utilises this enum to categorise certain pieces of the data. Given that, I am assuming that nether None nor All would ever be combined with any other value.
Given the following bitwise enum definition;
[System.FlagsAttribute()]
public enum TrainingComponentTypes : int
{
None = 0,
AccreditedCourse = 1,
Qualification = 2,
Unit = 4,
SkillSet = 8,
UnitContextualisation = 16,
TrainingPackage = 32,
AccreditedCourseModule = 64,
All = 127,
}
I read the following quote on this MSDN site about FlagAttributes;
Use None as the name of the flag
enumerated constant whose value is
zero. You cannot use the None
enumerated constant in a bitwise AND
operation to test for a flag because
the result is always zero. However,
you can perform a logical, not a
bitwise, comparison between the
numeric value and the None enumerated
constant to determine whether any bits
in the numeric value are set.
Does a logical comparison in this instance refer to a normal equality test for enums?
For example;
TrainingComponentTypes tct = TrainingComponentTypes.None;
if (tct == TrainingComponentTypes.None)
{ ... }
For a bitwise comparison, I am performing the following;
TrainingComponentTypes tct = TrainingComponentTypes.AccreditedCourse | TrainingComponentTypes.Qualification | TrainingComponentTypes.TrainingPackage;
Assert.IsTrue((tct & TrainingComponentTypes.AccreditedCourse) == TrainingComponentTypes.AccreditedCourse, "Expected AccreditedCourse as part the enum");
Assert.IsFalse((tct & TrainingComponentTypes.SkillSet) == TrainingComponentTypes.SkillSet, "Found unexpected SkillSet as part the enum");
Lastly, when testing for all, I have tried both a logical, and bitwise comparison, and they both return the same. Should I be using one over the other here? For example;
TrainingComponentTypes tct = TrainingComponentTypes.All;
Assert.IsTrue((tct & TrainingComponentTypes.All) == TrainingComponentTypes.All, "Expected All as part the enum");
Assert.IsTrue((tct) == TrainingComponentTypes.All, "Expected All as part the enum");
// The follow also pass the assertion for a value of All
Assert.IsTrue((tct & TrainingComponentTypes.Qualification) == TrainingComponentTypes.Qualification, "Expected Qualification as part the enum");
Assert.IsTrue((tct & TrainingComponentTypes.TrainingPackage) == TrainingComponentTypes.TrainingPackage, "Expected TrainingPackage as part the enum");
So in summary, I'd like to know the following about Bitwise enums;
Is my understanding of a logical
comparison correct given my example
above?
Is the way I am performing
a bitwise comparison correct?
What is the right way to handle the "All"
value (bitwise or logical). I am not sure if we'd ever receive a value where All was combined with other TrainingComponentTypes. I can't see why we would, but then, you never know?
Am I right in assuming that switch
statements basically shouldn't be
used for bitwise enums (given none
is appears to be a special case and
requires a logical comparison)?
Thanks,
Chris

Short answer: Yes :)
Longer:
1) All operations are performed on the integer value of the flags variable, so you can think about them in terms of this.
2) Yes.
3) Either works. However, it's worth noting that if someone shoves an invalid value into a variable then the == TrainingComponentTypes.All version will fail. For example:
var badValue = (TrainingComponentTypes)128 | TrainingComponentTypes.All;
// now badValue != TrainingComponentTypes.All
// but (badValue & TrainingComponentTypes.All) == TrainingComponentTypes.All
For this part:
I am not sure if we'd ever receive a value where All was combined with other TrainingComponentTypes.
I'm not sure you fully understand how the enum works under the covers.
The value of All is:
127 = 1111111 (binary)
The other values are:
AccreditedCourse = 0000001
Qualification = 0000010
Unit = 0000100
SkillSet = 0001000
UnitContextualisation = 0010000
TrainingPackage = 0100000
AccreditedCourseModule = 1000000
As you can see, All is simply the bitwise | of all these values together. You can't combine any other TraningComponentTypes with All, because All already includes them! Also, if you combine them all together with | yourself it's exactly the same as using All directly (so, All is simply a convenience when you define it inside an enum).
4) You could use it to check for None or All but not for other values.
It's worth noting that there is a convenience method on Enum that will do these checks for you: Enum.HasFlag.

Is my understanding of a logical comparison correct given my example above?
Yes, logical in this context means the equality and inequality operators.
Is the way I am performing a bitwise comparison correct?
Yes, but there is an easier way: Enum.HasFlag. For example:
tct.HasFlag(TrainingComponentTypes.Qualification)
instead of:
(tct & TrainingComponentTypes.Qualification) == TrainingComponentTypes.Qualification
What is the right way to handle the "All" value (bitwise or logical). I am not sure if we'd ever receive a value where All was combined with other TrainingComponentTypes. I can't see why we would, but then, you never know?
I think it is better to define All in the enum itself as the bitwise OR of all its parts. But you'll see people do it both ways.
Am I right in assuming that switch statements basically shouldn't be used for bitwise enums (given none is appears to be a special case and requires a logical comparison)?
No, not at all. Feel free to use them is switch statements. The case values must be constants but they can be expressions and are tested for equality. The compiler will tell you if you do something silly like try to use the same case value twice.

Yes.
Yes
Both logical and bitwise could be used. Usage depends on whether all is all bits set or just the bitwise OR of all the values you've defined.
Yes, but not because of None. A switch compares a single value, whereas a bit field can obviously have multiple values.
As others have noted Enum contains HasFlag().

1 and 2 - yes, however there is a way to make it a little easier to read:
TrainingComponentTypes tct = TrainingComponentTypes.AccreditedCourse | TrainingComponentTypes.Qualification;
Assert.IsTrue(tct.HasFlag(TrainingComponentTypes.AccreditedCourse), "Expected AccreditedCourse as part the enum");
3 - I am not sure if you need All value at all. I would remove it.
4 - Yes, switch statement usually doesn't make sense for Flags enumerations.

"3.What is the right way to handle the "All" value (bitwise or logical). I am not sure if we'd ever receive a value where All was combined with other TrainingComponentTypes. I can't see why we would, but "
It seems you misunderstand how the bitwise enum values work. 'All' is always combined with other values, in fact it is the combination of all the values. Looking at the binary values for your enum:
None = 0,
AccreditedCourse = 1,
Qualification = 10,
Unit = 100,
SkillSet = 1000,
UnitContextualisation = 10000,
TrainingPackage = 100000,
AccreditedCourseModule = 1000000,
All = 1111111
does that help your understanding?

1&2 look fine
3.
All as you have it defined can't be combined with anything without losing data. If "all" is a real value that you expect to receive from the server you should probably change it to 128.
otherwise it is a convenience value that you can use to test if any value is set... you shouldn't need this unless your flag values are sent as binary data and packed in a byte that may contain other data.
4.
Switch statements could be used but will not work well if/when you have values that have more than one flag, if there are small subsets of valid flag combinations they can still be useful.

In model
as_enum :shifts, { day_shift: 0,
evening_shift: 1,
night_shift: 2,
}, accessor: :bitwise
And in Views
Model.enum_collection(:shifts)

Using bitwise operators

I've been studying C# and ran accross some familiar ground from my old work in C++. I never understood the reason for bitwise operators in a real application. I've never used them and have never had in a reason to use them. I've been studying how they work; the example below shows the shift bitwise operator. What is the point of bitwise operators, their use and how they work?
Maybe I'm missing something in bitwise logic.
byte bitComp = 15; // bitComp = 15 = 00001111b
byte bresult = (byte) ~bitComp; // bresult = 240 = 11110000b
Here's an example for the ~complement bitwise operator:
byte bitComp = 15; // bitComp = 15 = 00001111b
byte bresult = (byte) ~bitComp; // bresult = 240 = 11110000b

A typical use is manipulating bits that represent mutually exclusive 'flags'.
Example from MSDN: Enumeration Types
[Flags]
enum Days2
{
None = 0x0,
Sunday = 0x1,
Monday = 0x2,
Tuesday = 0x4,
Wednesday = 0x8,
Thursday = 0x10,
Friday = 0x20,
Saturday = 0x40
}
class MyClass
{
Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
Days2 notWednesday = ~(Days2.Wednesday);
}
See also Stack Overflow question Most common C# bitwise operations.

Here's an everyday bitwise-op trick not many people have discovered:
When you have an enumerated type representing a bitfield, you need to define each enum entry as a distinct bit value, as in:
enum
{
Option1 = 1,
Option2 = 2,
Option3 = 4,
Option4 = 8,
Option5 = 16
};
but it's easy to forget that the next item in the sequence needs to be double the last number. Using bit shifting, it makes the sequence much easier to get right:
enum
{
Option1 = 1<<0,
Option2 = 1<<1,
Option3 = 1<<2,
Option4 = 1<<3,
Option5 = 1<<4
};

Another typical (but I think less common) usage is to compose several numbers into one big number. An example for this can be the windows RGB macro:
#define RGB(r, g ,b) ((DWORD) (((BYTE) (r) | ((WORD) (g) << 8)) | (((DWORD) (BYTE) (b)) << 16)))
Where you take 3 bytes and compose an integer from them the represent the RGB value.

Except for combining flags, bit logic isn't necessarily something you need in your UI code, but it is still tremendously important. For example, I maintain a binary serialization library, that needs to deal with all sorts of complex bit-packing strategies (variant length base-128 integer encoding, for example). This is one of the implementations (actually, this is a slower/safer version - there are other variants for dealing with buffered data, but they are harder to follow):
public static bool TryDecodeUInt32(Stream source, out uint value)
{
if (source == null) throw new ArgumentNullException("source");
int b = source.ReadByte();
if (b < 0)
{
value = 0;
return false;
}
if ((b & 0x80) == 0)
{
// single-byte
value = (uint) b;
return true;
}
int shift = 7;
value = (uint)(b & 0x7F);
bool keepGoing;
int i = 0;
do
{
b = source.ReadByte();
if (b < 0) throw new EndOfStreamException();
i++;
keepGoing = (b & 0x80) != 0;
value |= ((uint)(b & 0x7F)) << shift;
shift += 7;
} while (keepGoing && i < 4);
if (keepGoing && i == 4)
{
throw new OverflowException();
}
return true;
}
We have:
tests to see if the most-significant-bit is set (this changes the meaning of the data)
shifts-a-plenty
removal of the most-significant-bit
bitwise combination of values
This is real code, used in a real (and much used) protocol. In general, bit operations are used a lot in any kind of encoding layer.
It is also hugely important in graphics programming, for example. And lots of others.
There are also some micro-optimisations (maths intensive work etc) that can be done with bit operations.

An example from COM programming:
An HRESULT is an error code consisting of a 32 bit integer. The high bit is a flag indicating whether the code represents success (0) or failure (1). The next 15 bits are an integer representing what sort of error it is -- an ole automation error or a win32 error or whatever. The lower 16 bits are the actual error code.
Being able to shift bits around is quite useful when you want to get information into or out of an HRESULT.
Now, you almost always want to abstract away the bit twiddling. It's much better to have a method (or in C, a macro) that tells you whether the HRESULT is failure, rather than actually twiddling out the bit with (hr & 0x80000000) != 0 right there in your source code. Let the compiler inline it.
There are lots of examples of low-level data structures where information is crammed into words and needs to be extracted with bitwise operations.

Three major uses off of the top of my head:
1) In embedded applications, you often have to access memory-mapped registers, of which individual bits mean certain things (for instance the update bit in an ADC or serial register). This is more relevant to C++ than C#.
2) Calculations of checksums, like CRCs. These use shifts and masks very heavily. Before anybody says "use a standard library", I have come across non-standard checksums too many times, which have had to implemented from scratch.
3) When dealing with data which comes from another platform, which have a diffent bit or byte order (or both) from the one you are executing your code on. This is particularly true when doing software testing of embedded systems, receiving data across a network which has not been converted to network order, or processing bulk data from a data capture system. Check out the Wikipedia article on Endianness. If you are really interested, read the classic article On Holy Wars and a Call for Peace" by Danny Cohen.

A couple of examples:
Communication stacks: a header attached to data in a layer of a communication stack may contain bytes where individual bits within those bytes signify something, and so have to be masked before they can be processed. Similarly, when assembling the header in the response, individual bits will then need to be set or cleared.
Embedded software: embedded microcontrollers can have tens or hundreds of hardware registers, in which individual bits (or collections thereof) control different functions within the chip, or indicate the status of parts of the hardware.
Incidentally, in C and C++, bitfields are not recommended where portability is important, as the order of bits in a bitfield is compiler-dependent. Using masks instead guarantees which bit(s) will be set or cleared.

As to 'how they work': bitwise operations are one of the lowest level operations CPUs support, and in fact some bitwise operations, like NAND and NOR, are universal - you can build any operation at all out of a sufficiently large set of NAND gates. This may seem academic, but if you look at how things like adders are implemented in hardware, that's often basically what they boil down to.
As to the 'point': in a lot of higher level applications, of course, there is not much use for bit ops, but at the lowest levels of a system they are incredibly important. Just off the top of my head, things that would be very difficult to write without bit operations include device drivers, cryptographic software, error correction systems like RAID5 or erasure codes, checksums like CRC, video decoding software, memory allocators, or compression software.
They are also useful for maintaining large sets of integers efficiently, for instance in the fd_set used by the common select syscall, or when solving certain search/optimization problems.
Take a look at the source of an MPEG4 decoder, cryptography library, or operating system kernel sometime and you'll see many, many examples of bit operations being used.

Left and right shift operators (<< and >>) are often used in performance critical applications which do arithmetic operations and more specifically multiplications and divisions by powers of two.
For example suppose you had to calculate the mathematical expression 5*2^7. A naive implementation would be:
int result = 5 * (int)Math.Pow(2, 7);
Using left shift operator you could write:
int result = 5 << 7;
The second expression will be orders of magnitude faster than the first and yet yielding the same result.

Minimizing Memory Use
Naturally a very generalized reason is to cram a lot of data into a small amount of memory. If you consider an array of booleans like this:
bool data[64] = {...};
That can take 64 bytes (512 bits) of memory. Meanwhile the same idea can be represented with bits using 8 bytes (64-bits) of memory:
uint64_t data = ...;
And of course we have a boatload of DRAM these days so it might not seem like it'd matter to compact all this data into the minimum size, but we're still dealing with, say, 64-bit general purpose registers. We're still dealing with 64 byte cache lines, e.g., and kilobytes per physically-mapped page, and moving data down the memory hierarchy is expensive. So if you're processing a boatload of data sequentially, for example, and you can reduce that down to 1/8th its size, often you'll be able to process a whole lot more of it in a shorter amount of time.
So a common use of the analogy above to store a bunch of booleans in a small amount of space is when bit flags are involved, like this:
enum Flags
{
flag_selected = 1 << 0,
flag_hidden = 1 << 1,
flag_removed = 1 << 2,
flag_hovering = 1 << 3,
flag_minimized = 1 << 4,
...
};
uint8_t flags = flag_selected | flag_hovering;
Operating on Multiple Bits at Once
But on top of cramming all this data into a smaller amount of space, you can also do things like test for multiple bits simultaneously:
// Check if the element is hidden or removed.
if (flags & (flag_hidden | flag_removed))
{
...
}
And a smart optimizer will typically reduce that down to a single bitwise and if flag_hidden and flag_removed are literal constants known at compile-time.
As another example, let's go back to the example above:
bool data[64];
Let's say you wanted to test if all 64 booleans are set in which case you do something different. Given this type of representation, we might have to do this:
bool all_set = true;
for (int j=0; j < 64; ++j)
{
if (!data[j])
{
all_set = false;
break;
}
}
if (all_set)
{
// Do something different when all booleans are set.
...
}
And that's pretty expensive when the bitwise representation allows us to do this:
uint64_t data = ...;
if (data == 0xffffffffffffffff)
{
// Do something different when all bits are set.
...
}
This above version can perform the check for all 64 bits set in a single instruction on 64-bit machines. With SIMD registers, you can even test for more than 64 bits at a time with a single SIMD instruction.
As another example let's say you want to count how many of those booleans are set. In that case you might have to do this working with the boolean representation:
int count = 0;
for (int j=0; j < 64; ++j)
count += data[j];
// do something with count
Meanwhile if you used bitwise operations, you can do this:
uint64_t data = ...;
const int count = __popcnt64(data);
// do something with count
And some hardware can do that very efficiently as a native instruction. Others can still do it a whole, whole lot faster than looping through 64 booleans and counting the booleans set to true.
Efficient Arithmetic
Another common one is efficient arithmetic. If you have something like:
x = pow(2, n);
Where n is a runtime variable, then you can often get a much more efficient result doing:
x = 1 << n;
Of course an optimizing compiler using intrinsics for pow might be able to translate the former into the latter, but at least C and C++ compilers I've checked as of late cannot perform this optimization, at least when n is not known at compile-time.
Whenever you're working with power of two, you can often do a lot of things efficiently with bitshifts and other bitwise operations. For example, take this:
x = n % power_of_two;
... where power_of_two is a runtime variable that is always a power of two. In that case you can do:
x = n & (power_of_two - 1);
Which has the same effect as the modulo (only for power of two numbers). It's because a power of two value will always be a set bit followed by zeros. For example, 16 will be 0b10000. If you subtract one from that, it becomes: 0b1111, and using a bitwise and with that will effectively clear all upper bits in a way that gives you the analogical equivalent of n % 16.
Similar thing with left-shifts to multiply by a power of two, right-shifts to divide by a power of two, etc. One of the main reasons a lot of hardware favored power of two image sizes like 16x16, 32x32, 64x64, 256x256, etc. is due to the efficient arithmetic enabled by it using bitwise instructions.
Conclusion
So anyway, this is a brief introduction to what you can do with bitwise operations and instructions from fast arithmetic, reduced memory use, and being able to perform operations on potentially many bits at once without looping through them and operating on them one bit at a time.
And they're still very relevant today in performance-critical fields. For example, if you look at the Atomontage voxel rendering engine, it claims to be able to rep a voxel in just about a single bit, and that's important not just to fit huge voxel data in DRAM but also to render it really quickly from smaller, fast memory like registers. Naturally it can't do that if it's going to use 8 bits just to store a true/false kind of value which has to be checked individually.

I work in motion control (among other things) and the way you communicate with the drives is usually by using bit sequences. You set one bit pattern in a memory location x to set the motion profile, you set an enable bit at memory location y to start the motion, read a pattern from location z to get the status of the move, etc. The lower you go the more bit twiddling you have to perform.

There is a reason that, depending on the kind of person, can be important: cool-iness! ;)
Take this, an algorithm to calculate the absolute value of a number, without using any conditional branch instruction:
int abs(const int input) {
int temp = A >> 31;
return ( input ^ A ) - A;
}
A reason to avoid conditional branching is that it could stop your processor pre-fetching, waiting for the condition to be verified to know to which value the program counter should be set.
So, a part from the joke, there are very good technical reasons to do it.