Decode date/time hexadecimal values found embedded within binary files - c#

I am trying to write a C# program similar to the one on this website: http://www.digital-detective.co.uk/freetools/decode.asp
Can you please tell me how I can convert the hex numbers listed in the following bulletins to Date/Time values.
Windows 64 bit (little endian) hex value FF03D2315FE1C701 should
converts to = Sat, 18 August 2007 06:15:37 UTC
Windows 64 bit OLE hex value FBE8DF975D3FE340 should converts to =
Sun, 02 December 2007 22:11:42 UTC
Unix 32 bit (big endian) hex value 46C3B400 should converts to =
Thu, 16 August 2007 02:18:40 UTC
Apple Mac Absolute hex value 219216022 should converts to = Thu,
13 December 2007 05:20:22 UTC
HFS 32 bit (little endian) hex value CD4E55C3 should converts to =
Mon, 05 November 2007 22:50:53 Local
I was trying to use the following code to do that, but it doesn't return the correct result:
double decValue = int.Parse("A2C3B446", System.Globalization.NumberStyles.HexNumber);
System.DateTime dtDateTime = new DateTime(2013, 1, 1, 0, 0, 0, 0);
dtDateTime = dtDateTime.AddSeconds(decValue).ToLocalTime();
Console.WriteLine("Decimal Value: " + decValue);
Console.WriteLine(dtDateTime);

You'll first want a utility method that reads bytes from a stream and handles the endian-ness. That could look like:
public static byte[] ReadBytes(Stream s, int size, bool littleEndian) {
var bytes = new byte[size];
var len = s.Read(bytes, 0, size);
if (len != size) throw new InvalidOperationException("Unexpected end of file");
if (BitConverter.IsLittleEndian != littleEndian) Array.Reverse(bytes);
return bytes;
}
Windows dates are easy, supported by DateTime.FromFileTimeUtc() directly:
public static DateTime ConvertWindowsDate(byte[] bytes) {
if (bytes.Length != 8) throw new ArgumentException();
return DateTime.FromFileTimeUtc(BitConverter.ToInt64(bytes, 0));
}
Testing it with your value:
var date1 = DateReaders.ConvertWindowsDate(DateReaders.ReadBytes(
new MemoryStream(new byte[]{0xFF,0x03,0xD2,0x31,0x5F,0xE1,0xC7,0x01}), 8, true));
Produces {8/18/2007 6:15:37 AM} as expected.
OLE dates are easy, supported by DateTime.FromOADate() directly:
public static DateTime ConvertOLEDate(byte[] bytes) {
if (bytes.Length != 8) throw new ArgumentException();
return DateTime.FromOADate(BitConverter.ToDouble(bytes, 0));
}
Testing it with your value:
var date2 = DateReaders.ConvertOLEDate(DateReaders.ReadBytes(
new MemoryStream(new byte[] {0xFB,0xE8,0xDF,0x97,0x5D,0x3F,0xE3,0x40 }), 8, true));
Produces {12/2/2007 10:11:41 PM}
Unix date values are milliseconds from Jan 1st, 1970, 0:00 AM UTC:
public static DateTime ConvertUnixDate(byte[] bytes) {
if (bytes.Length != 4) throw new ArgumentException();
return new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddSeconds(
BitConverter.ToUInt32(bytes, 0));
}
Testing it with your value:
var date3 = DateReaders.ConvertUnixDate(DateReaders.ReadBytes(
new MemoryStream(new byte[] {0x46,0xC3,0xB4,0x00}), 4, false));
Produces {8/16/2007 2:18:40 AM}
Apple Mac absolute time is documented to be CPU dependent and requires conversion on the machine that generated it. The shown value "219216022" is quirky, it appears to decimal instead of hex like all the other ones. I'll follow Baldrick's lead with:
public static DateTime ConvertAppleDate(byte[] bytes) {
if (bytes.Length != 4) throw new ArgumentException();
return new DateTime(2001, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddSeconds(
BitConverter.ToUInt32(bytes, 0));
}
HFS dates are seconds since Jan 1st, 1904, 0:00 AM. Do note that HFS dates are local time but HFS Plus dates are UTC. I'll assume local since that's the result you documented:
public static DateTime ConvertHFSDate(byte[] bytes) {
if (bytes.Length != 4) throw new ArgumentException();
return new DateTime(1904, 1, 1, 0, 0, 0, DateTimeKind.Local).AddSeconds(
BitConverter.ToUInt32(bytes, 0));
}
Testing it with your value:
var date5 = DateReaders.ConvertHFSDate(DateReaders.ReadBytes(
new MemoryStream(new byte[] {0xCD,0x4E,0x55,0xC3 }), 4, true));
Produces {11/5/2007 10:50:53 PM}

Related

Converting HEX to (extended) ASCII 0-255 with NO UTF for DOS C#

Firstly, I don't want to be 'That Guy' when I ask this long question, even though I know it's been asked plenty of times in different way, but I'm having significant problems in getting a date format to store in a string correctly.
Some minor background.
I am using the DOS FileTime Date format that needs to be stored in an 8 character HEX format - as seen here: https://doubleblak.com/blogPosts.php?id=7
In short, the time and date are captured, then arranged in binary bits, and then converted to HEX.
What I need is to be able to do now, is store those HEX Values as a string, and be able to pass them to tagLib sharp to write a custom APE tag in an MP3 file. Easier said than done...
Writing the custom tags is easy, as it's basically just a matter of this:
TagLib.File file = TagLib.File.Create(filename);
TagLib.Ape.Tag ape_tag = (TagLib.Ape.Tag)file.GetTag(TagLib.TagTypes.Ape, true);
// Write - for my example
/* declarations:
public void SetValue(string key, string value);
public void SetValue(string key, uint number, uint count);
public void SetValue(string key, string[] value);
*/
ape_tag.SetValue("XLastPlayed", history );
So, on to the actual problem:
After the conversion of the date to the correct HEX Values, I get the following result:
928C9D51
However, to make this work and store it correctly, I need to convert it to ASCII values, so that it can then be stored by TagLibSharp.
If I convert this to ASCII, then I get the following: (which is wrong), as it should only be 4 ASCII characters long - even if they are unprintable, or sit in the > 127 character range.
"\u0092\u008c\u009dQ"
You can see in this image the extra HEX values that have been stored, which is incorrect.
This is a sample of the code of I've been trying to use, (in various forms) to get this to work.
string FirstHistory = "7D8C9D51";
String test1 = "";
for (int i = 0; i < FirstHistory.Length; i += 2)
{
string hs = FirstHistory.Substring(i, 2);
var enc = Encoding.GetEncoding("iso-8859-1"); //.ASCII;// .GetEncoding(437);
var bytes1 = enc.GetBytes(string.Format("{0:x1}", Convert.ToChar(Convert.ToUInt16(hs, 16))));
string unicodeString = enc.GetString(bytes1);
Console.WriteLine(unicodeString);
test1 = test1 + unicodeString;
}
// needs to be "00 00 00 21" for the standard date array for this file format.
byte[] bytesArray = { 0, 0, 0, 33 }; // A byte array containing non-printable characters
string s1 = "";
string history = "";
// Basically what the history will look like
// "???!???!???!???!???!???!???!???!???!???!???!???!???!???!???!???!???!"
for (int i =0; i < 18; i++)
{
if(i==0) {
history = test1; // Write the first value.
}
s1 = Encoding.UTF8.GetString(bytesArray); // encoding on this string won't effect the array date values
history = history + s1;
}
ape_tag.SetValue("XLastPlayed", history );
I am aware there are multiple encodings, and I've basically tried all that I can, and have read things, but I'm not getting anywhere.
Sometimes I think I've got it, but then when I look at the file I'm saving, it slips in a "C2" HEX value, when it shouldn't, and this is the unicode breaking everything. I've included an image of what it should be without these C2 Hex Values, and you can actually see the DOS Time and Date time appear correctly in the HxD Hex viewer.
I've tried various encodings such as 437, ios-8859-1, ASCII, and different methods such as using string builder, char, bytes, etc. and sometimes I get a date and time stamp where the values are correct, where the HEX values don't exceed in to the extended ASCII ranges, but then I run it again, and I'm back to square 1. It ALWAYS inserts those extended values as UTF8 entries and breaks regardless of what I do.
I'm sure there's not a bug in VS, but I'm running Microsoft Visual Studio Community 2019, Version 16.8.2 if that adds to the case.
I can not seem to find a way around this. Does anyone have any thoughts on this?
Thanks in advance.
*** UPDATE ***
This the update thanks to #xanatos
public static byte[] ConvertHexStringToByteArray(string str)
{
Dictionary<string, byte> hexindex = new Dictionary<string, byte>();
for (int i = 0; i <= 255; i++)
hexindex.Add(i.ToString("X2"), (byte)i);
List<byte> hexres = new List<byte>();
for (int i = 0; i < str.Length; i += 2)
hexres.Add(hexindex[str.Substring(i, 2)]);
return hexres.ToArray();
}
string FirstHistory = "7D8C9D51";
string s1 = "";
string history = "";
byte[] bytes = { 0, 0, 33, 0 }; // A byte array contains non-ASCII (or non-readable) characters
for (int i =0; i < 18; i++)
{
s1 = Encoding.UTF8.GetString(bytes); // ???
history = history + s1;
}
var theArray_SO = ConvertHexStringToByteArray(FirstHistory);
ape_tag.SetItem(new TagLib.Ape.Item("XLastPlayed", (new TagLib.ByteVector(theArray_SO)) + history));
*** UPDATE 2 - 30th Jan 2021 ***
After editing other values and resaving them, I ran into some trouble. It seems that there could be data corruption with TagLib and custom APE tags, specifically for this ByteVector data. If you just use the save method for editing other custom values then it's not a problem, but if you have custom values with these values with ByteVector values, you will most probably run in to trouble. This is what I still used for saving the files.
TagLib.File file = TagLib.File.Create(filename);
// changes
file.save();
However, to overcome this data corruption, I read (searched) the file first as a FileStream to locate the value I needed, and then put the values of 72 bytes after the found value to a new byte array, then save it back to the file.
I discovered that reading ByteVector data in through a string failed spectacularly and had results all over the place.
TagLib.Ape.Item item_Duration = ape_tag.GetItem("XLastScheduled");
While this can probably be rewritten a thousand ways, here's my code that works.
int foundlocation = 0;
int loop1 = 0;
byte[] sevenItems = new byte[80] { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };
string match = "XLastScheduled";
byte[] matchBytes = Encoding.ASCII.GetBytes(match);
{
using (var fs = new FileStream(filename, FileMode.Open))
{
int i = 0;
int readByte;
while ((readByte = fs.ReadByte()) != -1)
{
if (foundlocation == 0)
{
if (matchBytes[i] == readByte)
{
i++;
}
else
{
i = 0;
}
}
if (i == matchBytes.Length)
{
//Console.WriteLine("It found between {0} and {1}.", fs.Position - matchBytes.Length, fs.Position);
// set to true.
foundlocation = 1;
}
if (foundlocation==1)
{
//if (loop1 > 1)
{
// Start adding it at 2 bytes after it's found.
sevenItems[loop1] = (byte)readByte;
}
loop1++;
if(loop1 > 79)
{
fs.Close();
Console.WriteLine("Found the XLastScheduled data");
// 72/4 = 18 date/times
break;
}
}
// Then, I can save those values back as a vector byte array, instead of a string - hopefully...
}
fs.Close();
}
}
byte[] dst = new byte[sevenItems.Length - 8];
Array.Copy(sevenItems, 2, dst, 0, dst.Length);
TagLib.File file = TagLib.File.Create(filename);
// Get the APEv2 tag if it exists.
TagLib.Ape.Tag ape_tag = (TagLib.Ape.Tag)file.GetTag(TagLib.TagTypes.Ape, true);
// Save the new byteVector.
ape_tag.SetItem(new TagLib.Ape.Item("XLastScheduled", (new TagLib.ByteVector(dst))));
Console.WriteLine("XLastScheduled: set" );
There is another method for binary data:
var bytes = new byte[4] { 0xFF, 0, 0, 0xFF };
ape_tag.SetItem(new TagLib.Ape.Item("XLastPlayed", new ByteVector(bytes)));
Unclear if you need methods to convert from/to DOS FileTime:
public static uint ToDosFileTimeDate(DateTime dt)
{
ushort date = (ushort)(((dt.Year - 1980) << 9) | (dt.Month << 5) | (dt.Day));
ushort time = (ushort)((dt.Hour << 11) | (dt.Minute << 5) | (dt.Second >> 1));
uint dateTime = ((uint)date << 16) | time;
return dateTime;
}
public static DateTime FromDosFileTimeDate(uint ui)
{
ushort date = (ushort)(ui >> 16);
ushort time = (ushort)(ui & 0xFFFF);
var year = (date >> 9) + 1980;
var month = (date >> 5) & 0xF;
var day = date & 0x1F;
var hour = time >> 11;
var minute = (time >> 5) & 0x3F;
var second = (time & 0x1F) << 1;
return new DateTime(year, month, day, hour, minute, second, DateTimeKind.Local);
}
and to convert the uint to byte[4] arrays there are
uint ui = BitConverter.ToUInt32(bytes);
and
byte[] bytes = BitConverter.GetBytes(ui);

How to get timezone datetime correct if PC local time is wrong

How can I get datetime with timezone info (GMT/UTC +7)?
Let's say in real time it was 06:00 PM and then someone alters the local PC time to 01:00 PM. How do I get datetime 06:00 PM?
I have tried this:
System.Globalization.CultureInfo.CurrentCulture.ClearCachedData();
DateTime utcTime = DateTime.UtcNow;
TimeZoneInfo tzi = TimeZoneInfo.FindSystemTimeZoneById("SE Asia Standard Time");
DateTime localTime = TimeZoneInfo.ConvertTimeFromUtc(utcTime, tzi);
Console.WriteLine(localTime);
But I still got 01:00 PM.
Just contact NTP server and take it's time. Code found on stackoverflow. You need internet connection to do that of course.
public static DateTime GetNetworkTime() {
//default Windows time server
const string ntpServer = "time.windows.com";
// NTP message size - 16 bytes of the digest (RFC 2030)
var ntpData = new byte[48];
//Setting the Leap Indicator, Version Number and Mode values
ntpData[0] = 0x1B; //LI = 0 (no warning), VN = 3 (IPv4 only), Mode = 3 (Client Mode)
var addresses = Dns.GetHostEntry(ntpServer).AddressList;
//The UDP port number assigned to NTP is 123
var ipEndPoint = new IPEndPoint(addresses[0], 123);
//NTP uses UDP
using(var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp))
{
socket.Connect(ipEndPoint);
//Stops code hang if NTP is blocked
socket.ReceiveTimeout = 3000;
socket.Send(ntpData);
socket.Receive(ntpData);
socket.Close();
}
//Offset to get to the "Transmit Timestamp" field (time at which the reply
//departed the server for the client, in 64-bit timestamp format."
const byte serverReplyTime = 40;
//Get the seconds part
ulong intPart = BitConverter.ToUInt32(ntpData, serverReplyTime);
//Get the seconds fraction
ulong fractPart = BitConverter.ToUInt32(ntpData, serverReplyTime + 4);
//Convert From big-endian to little-endian
intPart = SwapEndianness(intPart);
fractPart = SwapEndianness(fractPart);
var milliseconds = (intPart * 1000) + ((fractPart * 1000) / 0x100000000L);
//**UTC** time
var networkDateTime = (new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc)).AddMilliseconds((long)milliseconds);
return networkDateTime.ToLocalTime(); }
// stackoverflow.com/a/3294698/162671 static uint SwapEndianness(ulong x) {
return (uint) (((x & 0x000000ff) << 24) +
((x & 0x0000ff00) << 8) +
((x & 0x00ff0000) >> 8) +
((x & 0xff000000) >> 24)); }
If you don't want to trust the information on the user's PC at all, then you will need:
An alternate source of time zone data
A network time source, such as via NTP
You can achieve the first with Noda Time, and the second with my NodaTime.NetworkClock add-on. Internally, it uses similar NTP code as that given by Martin's answer.
public static DateTime GetRealTimeInZone(string timeZoneId)
{
var clock = NetworkClock.Instance;
var now = clock.GetCurrentInstant();
var tz = DateTimeZoneProviders.Tzdb[timeZoneId];
return now.InZone(tz).ToDateTimeUnspecified();
}
Usage:
DateTime dt = GetRealTimeInZone("Asia/Bangkok");
Or, if you do trust the local time zone setting, just not the clock, then:
public static DateTime GetRealTimeInZone()
{
var clock = NetworkClock.Instance;
var now = clock.GetCurrentInstant();
var tz = DateTimeZoneProviders.Tzdb.GetSystemDefault();
return now.InZone(tz).ToDateTimeUnspecified();
}
Usage:
DateTime dt = GetRealTimeInZone();

6 bytes timestamp to DateTime

I use 3rd party API. According to its specification the following
byte[] timestamp = new byte[] {185, 253, 177, 161, 51, 1}
represents Number of milliseconds from Jan 1, 1970 when the message
was generated for transmission
The issue is that I don't know how it could be translated into DateTime.
I've tried
DateTime Epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
long milliseconds = BitConverter.ToUInt32(timestamp, 0);
var result = Epoch + TimeSpan.FromMilliseconds(milliseconds);
The result is {2/1/1970 12:00:00 AM}, but year 2012 is expected.
byte[] timestamp = new byte[] { 185, 253, 177, 161, 51, 1, 0, 0, };
DateTime Epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
ulong milliseconds = BitConverter.ToUInt64(timestamp, 0);
var result = Epoch + TimeSpan.FromMilliseconds(milliseconds);
Result is 11/14/2011
Adding padding code special for CodeInChaos:
byte[] oldStamp = new byte[] { 185, 253, 177, 161, 51, 1 };
byte[] newStamp = new byte[sizeof(UInt64)];
Array.Copy(oldStamp, newStamp, oldStamp.Length);
For running on big-endian machines :
if (!BitConverter.IsLittleEndian)
{
newStamp = newStamp.Reverse().ToArray();
}
I assume timestamp uses little endian format. I also left out parameter validation.
long GetLongLE(byte[] buffer,int startIndex,int count)
{
long result=0;
long multiplier=1;
for(int i=0;i<count;i++)
{
result += buffer[startIndex+i]*multiplier;
multiplier *= 256;
}
return result;
}
long milliseconds = GetLongLE(timestamp, 0, 6);

convert ticks to time_t or Filetime()

How do I convert ticks to time_t or filetime()?
time_t is defined as the number of seconds from the Unix epoch date January 1, 1970 # 00:00 UTC. So, you basically need to figure out how much time your ticks represent and subtract the epoch time. The following should do it.
double GetTimeTSecondsFrom(long ticks)
{
DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
return (new DateTime(ticks) - epoch).TotalSeconds;
}
For time_t:
long ticks = 633918528000000000;
DateTime target = new DateTime(ticks);
DateTime start = new DateTime(1970, 1, 1, 0, 0, 0);
double time_t = (target - start).TotalSeconds;

Sequential Guid Generator

Is there any way to get the functionality of the Sql Server 2005+ Sequential Guid generator without inserting records to read it back on round trip or invoking a native win dll call? I saw someone answer with a way of using rpcrt4.dll but I'm not sure if that would be able to work from my hosted environment for production.
Edit: Working with #John Boker's answer I attempted to turn it into more of a GuidComb generator instead of being dependent on the last generated Guid other than starting over. That for the seed instead of starting with Guid.Empty that I use
public SequentialGuid()
{
var tempGuid = Guid.NewGuid();
var bytes = tempGuid.ToByteArray();
var time = DateTime.Now;
bytes[3] = (byte) time.Year;
bytes[2] = (byte) time.Month;
bytes[1] = (byte) time.Day;
bytes[0] = (byte) time.Hour;
bytes[5] = (byte) time.Minute;
bytes[4] = (byte) time.Second;
CurrentGuid = new Guid(bytes);
}
I based that off the comments on
// 3 - the least significant byte in Guid ByteArray
[for SQL Server ORDER BY clause]
// 10 - the most significant byte in Guid ByteArray
[for SQL Server ORDERY BY clause]
SqlOrderMap = new[] {3, 2, 1, 0, 5, 4, 7, 6, 9, 8, 15, 14, 13, 12, 11, 10};
Does this look like the way I'd want to seed a guid with the DateTime or does it look like I should do it in reverse and work backwards from the end of the SqlOrderMap indexes? I'm not too concerned about their being a paging break anytime an initial guid would be created since it would only occur during application recycles.
Edit: 2020+ update
At this point I strongly prefer Snowflake identifiers using something like https://github.com/RobThree/IdGen
You could just use the same Win32 API function that SQL Server uses:
UuidCreateSequential
and apply some bit-shifting to put the values into big-endian order.
And since you want it in C#:
private class NativeMethods
{
[DllImport("rpcrt4.dll", SetLastError=true)]
public static extern int UuidCreateSequential(out Guid guid);
}
public static Guid NewSequentialID()
{
//Code is released into the public domain; no attribution required
const int RPC_S_OK = 0;
Guid guid;
int result = NativeMethods.UuidCreateSequential(out guid);
if (result != RPC_S_OK)
return Guid.NewGuid();
//Endian swap the UInt32, UInt16, and UInt16 into the big-endian order (RFC specified order) that SQL Server expects
//See https://stackoverflow.com/a/47682820/12597
//Short version: UuidCreateSequential writes out three numbers in litte, rather than big, endian order
var s = guid.ToByteArray();
var t = new byte[16];
//Endian swap UInt32
t[3] = s[0];
t[2] = s[1];
t[1] = s[2];
t[0] = s[3];
//Endian swap UInt16
t[5] = s[4];
t[4] = s[5];
//Endian swap UInt16
t[7] = s[6];
t[6] = s[7];
//The rest are already in the proper order
t[8] = s[8];
t[9] = s[9];
t[10] = s[10];
t[11] = s[11];
t[12] = s[12];
t[13] = s[13];
t[14] = s[14];
t[15] = s[15];
return new Guid(t);
}
See also
Is there a .NET equalent to SQL Servers newsequentialid()
Microsoft's UuidCreateSequential is just an implementation of a type 1 uuid from RFC 4122.
A uuid has three important parts:
node: (6 bytes) - the computer's MAC address
timestamp: (7 bytes) - number of 100 ns intervals since 00:00:00.00, 15 October 1582 (the date of Gregorian reform to the Christian calendar)
clockSequenceNumber (2 bytes) - counter in case you generate a guid faster than 100ns, or you change your mac address
The basic algorithm is:
obtain a system-wide lock
read the last node, timestamp and clockSequenceNumber from persistent storage (registry/file)
get the current node (i.e. MAC address)
get the current timestamp
a) if the saved state was not available or corrupted, or the mac address has changed, generate a random clockSequenceNumber
b) if the state was available, but the current timestamp is the same or older than the saved timestamp, increment the clockSequenceNumber
save node, timestamp and clockSequenceNumber back to persistent storage
release the global lock
format the guid structure according to the rfc
There is a 4-bit version number, and 2 bit variant that also need to be ANDed into the data:
guid = new Guid(
timestamp & 0xFFFFFFFF, //timestamp low
(timestamp >> 32) & 0xFFFF, //timestamp mid
((timestamp >> 40) & 0x0FFF), | (1 << 12) //timestamp high and version (version 1)
(clockSequenceNumber & 0x3F) | (0x80), //clock sequence number and reserved
node[0], node[1], node[2], node[3], node[4], node[5], node[6]);
Note: Completely untested; i just eyeballed it from the RFC.
the byte order might have to be changed (Here is byte order for sql server)
you might want to create your own version, e.g. Version 6 (version 1-5 are defined). That way you're guaranteed to be universally unique
this person came up with something to make sequential guids, here's a link
http://developmenttips.blogspot.com/2008/03/generate-sequential-guids-for-sql.html
relevant code:
public class SequentialGuid {
Guid _CurrentGuid;
public Guid CurrentGuid {
get {
return _CurrentGuid;
}
}
public SequentialGuid() {
_CurrentGuid = Guid.NewGuid();
}
public SequentialGuid(Guid previousGuid) {
_CurrentGuid = previousGuid;
}
public static SequentialGuid operator++(SequentialGuid sequentialGuid) {
byte[] bytes = sequentialGuid._CurrentGuid.ToByteArray();
for (int mapIndex = 0; mapIndex < 16; mapIndex++) {
int bytesIndex = SqlOrderMap[mapIndex];
bytes[bytesIndex]++;
if (bytes[bytesIndex] != 0) {
break; // No need to increment more significant bytes
}
}
sequentialGuid._CurrentGuid = new Guid(bytes);
return sequentialGuid;
}
private static int[] _SqlOrderMap = null;
private static int[] SqlOrderMap {
get {
if (_SqlOrderMap == null) {
_SqlOrderMap = new int[16] {
3, 2, 1, 0, 5, 4, 7, 6, 9, 8, 15, 14, 13, 12, 11, 10
};
// 3 - the least significant byte in Guid ByteArray [for SQL Server ORDER BY clause]
// 10 - the most significant byte in Guid ByteArray [for SQL Server ORDERY BY clause]
}
return _SqlOrderMap;
}
}
}
Here is how NHibernate implements the Guid.Comb algorithm:
private Guid GenerateComb()
{
byte[] guidArray = Guid.NewGuid().ToByteArray();
DateTime baseDate = new DateTime(1900, 1, 1);
DateTime now = DateTime.UtcNow;
// Get the days and milliseconds which will be used to build the byte string
TimeSpan days = new TimeSpan(now.Ticks - baseDate.Ticks);
TimeSpan msecs = now.TimeOfDay;
// Convert to a byte array
// Note that SQL Server is accurate to 1/300th of a millisecond so we divide by 3.333333
byte[] daysArray = BitConverter.GetBytes(days.Days);
byte[] msecsArray = BitConverter.GetBytes((long) (msecs.TotalMilliseconds / 3.333333));
// Reverse the bytes to match SQL Servers ordering
Array.Reverse(daysArray);
Array.Reverse(msecsArray);
// Copy the bytes into the guid
Array.Copy(daysArray, daysArray.Length - 2, guidArray, guidArray.Length - 6, 2);
Array.Copy(msecsArray, msecsArray.Length - 4, guidArray, guidArray.Length - 4, 4);
return new Guid(guidArray);
}
Maybe interesting to compare with the other suggestions:
EntityFramework Core also implements a sequentialGuidValueGenerator.
They generate randoms guids for each value and only change the most significant bytes based on a timestamp and thread-safe increments for sorting in SQL Server.
source link
This leads to values that are all very different but with a timestamp sortable.
C# Version
public static Guid ToSeqGuid()
{
Int64 lastTicks = -1;
long ticks = System.DateTime.UtcNow.Ticks;
if (ticks <= lastTicks)
{
ticks = lastTicks + 1;
}
lastTicks = ticks;
byte[] ticksBytes = BitConverter.GetBytes(ticks);
Array.Reverse(ticksBytes);
Guid myGuid = new Guid();
byte[] guidBytes = myGuid.ToByteArray();
Array.Copy(ticksBytes, 0, guidBytes, 10, 6);
Array.Copy(ticksBytes, 6, guidBytes, 8, 2);
Guid newGuid = new Guid(guidBytes);
string filepath = #"C:\temp\TheNewGuids.txt";
using (StreamWriter writer = new StreamWriter(filepath, true))
{
writer.WriteLine("GUID Created = " + newGuid.ToString());
}
return newGuid;
}
}
}
My solution (in VB but easy to convert). It changes the most significant (for SQL Server sorting) first 8 bytes of the GUID to DateTime.UtcNow.Ticks and also has extra code to help the issue of getting the same Ticks multiple times if you call for a new GUID faster than the system clock updates.
Private ReadOnly _toSeqGuidLock As New Object()
''' <summary>
''' Replaces the most significant eight bytes of the GUID (according to SQL Server ordering) with the current UTC-timestamp.
''' </summary>
''' <remarks>Thread-Safe</remarks>
<System.Runtime.CompilerServices.Extension()> _
Public Function ToSeqGuid(ByVal guid As Guid) As Guid
Static lastTicks As Int64 = -1
Dim ticks = DateTime.UtcNow.Ticks
SyncLock _toSeqGuidLock
If ticks <= lastTicks Then
ticks = lastTicks + 1
End If
lastTicks = ticks
End SyncLock
Dim ticksBytes = BitConverter.GetBytes(ticks)
Array.Reverse(ticksBytes)
Dim guidBytes = guid.ToByteArray()
Array.Copy(ticksBytes, 0, guidBytes, 10, 6)
Array.Copy(ticksBytes, 6, guidBytes, 8, 2)
Return New Guid(guidBytes)
End Function
Not specifically guid but I now normally use a Snowflake style sequential id generator. The same benefits of a guid while having even better clustered index compatibility than a sequential guid.
Flakey for .NET Core
IdGen for .NET Framework
I just took the NHibernate based answer by Moslem Ben Dhaou and made it an extension function:
using System;
namespace Atlas.Core.Kernel.Extensions
{
public static class Guids
{
public static Guid Comb(this Guid source)
{
byte[] guidArray = source.ToByteArray();
DateTime baseDate = new DateTime(1900, 1, 1);
DateTime now = DateTime.Now;
// Get the days and milliseconds which will be used to build the byte string
TimeSpan days = new TimeSpan(now.Ticks - baseDate.Ticks);
TimeSpan msecs = now.TimeOfDay;
// Convert to a byte array
// Note that SQL Server is accurate to 1/300th of a millisecond so we divide by 3.333333
byte[] daysArray = BitConverter.GetBytes(days.Days);
byte[] msecsArray = BitConverter.GetBytes((long)(msecs.TotalMilliseconds / 3.333333));
// Reverse the bytes to match SQL Servers ordering
Array.Reverse(daysArray);
Array.Reverse(msecsArray);
// Copy the bytes into the guid
Array.Copy(daysArray, daysArray.Length - 2, guidArray, guidArray.Length - 6, 2);
Array.Copy(msecsArray, msecsArray.Length - 4, guidArray, guidArray.Length - 4, 4);
return new Guid(guidArray);
}
}
}
As far I know NHibernate have special generator, called GuidCombGenerator. You can look on it.
I ended up writing this C# class to achieve the following that I needed with SQLite (in which GUIDs are stored as BLOBs and sort order is determined by memcmp). I suppose it is not a real GUID but it's tested and it does it job on SQLite.
Sortable (memcmp) sequential 16-byte array
Using counter to guarantee correct ordering
Thread safe
The time resolution seems to vary on OSes and on my Windows it's worse than 1ms. Therefore I chose to use a second bsed resolution where the first 34 bits represent the UnixTime (UTC), and then there is a 22 bit thread safe counter which increments for every request on the same second. If the counter reaches its max, the function sleeps for 500ms and tries again.
On my laptop I could generate and store ~3,2M 16 byte arrays per second.
The class returns a 16-byte array, not a GUID.
namespace SeqGuid
{
public static class SeqGuid
{
static private Object _lock = new Object();
static Random _rnd = new Random();
static UInt64 _lastSecond = 0;
static int _counter = 0;
public static UInt64 UnixTime()
{
return (UInt64)DateTime.UtcNow.Subtract(DateTime.UnixEpoch).TotalSeconds;
}
public static byte[] CreateGuid()
{
// One year is 3600*24*365.25 = 31557600 seconds
// With 34 bits we can hold ~544 years since 1970-01-01
//
UInt64 seconds = UnixTime();
lock (_lock)
{
if (seconds == _lastSecond)
{
// 22 bits counter, aka 11-1111-1111-1111-1111-1111 / 0x3F FFFF; 4.1M max / second, 1/4 ns
_counter++;
if (_counter >= 0x3F_FFFF)
{
Thread.Sleep(500);
// http://msdn.microsoft.com/en-us/library/c5kehkcz.aspx
// A lock knows which thread locked it. If the same thread comes again it just increments a counter and does not block.
return CreateGuid();
}
}
else
{
_lastSecond = seconds;
_counter = 0;
}
}
// Create 56 bits (7 bytes) {seconds (34bit), _counter(22bit)}
UInt64 secondsctr = (seconds << 22) | (UInt64)_counter;
byte[] byte16 = new byte[16] {
(byte) ((secondsctr >> 48) & 0xFF),
(byte) ((secondsctr >> 40) & 0xFF),
(byte) ((secondsctr >> 32) & 0xFF),
(byte) ((secondsctr >> 24) & 0xFF),
(byte) ((secondsctr >> 16) & 0xFF),
(byte) ((secondsctr >> 8) & 0xFF),
(byte) ((secondsctr >> 0) & 0xFF),
(byte) _rnd.Next(0,255),
(byte) _rnd.Next(0,255), (byte) _rnd.Next(0,255),
(byte) _rnd.Next(0,255), (byte) _rnd.Next(0,255),
(byte) _rnd.Next(0,255), (byte) _rnd.Next(0,255),
(byte) _rnd.Next(0,255), (byte) _rnd.Next(0,255)};
return byte16;
}
}
}

Categories