We Keep Coding

C# Datetime ISO 8601 format gives incorrect value

I am attempting to get a correctly formatted ISO 8601 date, and I keep hitting issues. My initial code seemed to be working, but I found that DST dates were not returning as expected. I made a .NET fiddle to ask about this issue here on stackoverflow, but it seems the way the "system" timezone works is going to cause me further problems when I deploy my code.
Here is a dotnet fiddle that displays something completely wrong:
using System;
public class Program
{
public static void Main()
{
var val3 = TimeZoneInfo.ConvertTimeFromUtc(new DateTime(2021, 10, 13, 18, 0, 0), TimeZoneInfo.FindSystemTimeZoneById("Central Standard Time"));
Console.WriteLine(val3.ToString("yyyy-MM-ddTHH:mm:sszzz"));
}
}
If I run this, I get the following:
2021-10-13T13:00:00+00:00
So the time is correct for CST, but the offset looks like it is reflecting the "system" timezone on the server. Taken altogether, the date is completely different than the input date.
If I run this code on my development system where the local timezone is CST, I get yet another different answer:
2021-10-13T08:00:00-06:00
Given that the date in question was in DST, I expect both of the above to return the following:
2021-10-13T13:00:00-05:00
What I am doing wrong?

Let me see if my understanding is correct (I'm still not entirely sure when the SOAP comes into play in the question).
You have a UTC DateTime:
var dt = new DateTime(2021, 10, 13, 18, 0, 0, DateTimeKind.Utc);
And you'd like to format that in CST -- which, on October 13, is actually CDT (-05:00)?
If that is correct, then you want to utilise DateTimeOffset, which has a better understanding of.. well, time offsets.
// Your original value. This will have an offset of "00:00"
// as the DateTime above was created as `DateTimeKind.Utc`
var timeAtUtc = new DateTimeOffset(dt);
// Alternatively:
// var timeAtUtc = new DateTimeOffset(2021, 10, 13, 18, 0, 0, TimeSpan.Zero);
// Find out the correct offset on that day (-5:00)
var cstOffsetAtTheTime = TimeZoneInfo
.FindSystemTimeZoneById("Central Standard Time")
.GetUtcOffset(timeAtUtc);
// And now apply the offset to your original value
var timeAtCst = timeAtUtc.ToOffset(cstOffsetAtTheTime);
// You will now get "2021-10-13T13:00:00-05:00"
// no matter where you run this from.
Console.WriteLine(timeAtCst.ToString("yyyy-MM-ddTHH:mm:sszzz"));
// Edit: If you pass it a date in Feb, you'll see that it will correctly be at -06:00.
Edit 2022-03-07 based on comment below. If you don't need to care about the offset value, you can simply do:
var timeAtUtc = ...;
var timeAtCst = TimeZoneInfo.ConvertTimeBySystemTimeZoneId(
timeAtUtc,
"Central Standard Time");

NPras's answer is good, but just to answer the question about what was wrong with the original code:
A DateTime object does not store time zone offset information. It only stores a Kind property, which can be one of three DateTimeKind values, either Utc, Local, or Unspecified.
Since your input was a DateTime, the output also has to be a DateTime, which has no time zone offset. Thus the offset related to your target time zone is discarded. Using a DateTimeOffset allows the target offset to persist.
In the final console output, you used zzz to show the offset. The .NET documentation explains why this used the server's time zone:
With DateTime values, the "z" custom format specifier represents the signed offset of the local operating system's time zone from Coordinated Universal Time (UTC), measured in hours. It doesn't reflect the value of an instance's DateTime.Kind property. For this reason, the "z" format specifier is not recommended for use with DateTime values.
With DateTimeOffset values, this format specifier represents the DateTimeOffset value's offset from UTC in hours.
As an aside, if you actually need to show an offset for a DateTime whose Kind is either Utc or Local, you should use the K specifier (the K is for Kind) instead of zzz. With the K specifier, Utc kinds will display a "Z", Local values will display the local offset such as "-05:00", and Unspecified values will display neither (an empty string).

DateTime conversion for HubSpot API is not working

I am pushing date/time values into the HubSpot CRM system via their API. For date/time values, the HS API requires UNIX format, which is milliseconds from Epoch (1/1/1970 12:00 AM). [HubSpot docs: https://developers.hubspot.com/docs/faq/how-should-timestamps-be-formatted-for-hubspots-apis]
But my dates are representing incorrectly. I am pulling dates from a SQL database that is in EST, and performing the following conversion:
string dbValue = "2019-02-03 00:00:00";
DateTime dt = Convert.ToDateTime(dbValue);
DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, 0, System.DateTimeKind.Utc);
long apiValue = Convert.ToInt64(dt.Subtract(epoch).TotalMilliseconds);
However, in HubSpot, the date field shows 2/2/2019. The time zone in HubSpot is set to UTC -4 Eastern Time.
It seems like there is some conversion issue, but I do not know what to do to correct it. I've tried using DateTime.SpecifiyKind to explicitly set dt to local before converting to long:
dt = DateTime.SpecifyKind(dt, System.DateTimeKind.Local);
But that hasn't worked either. I tried doing a basic test:
var dt1 = new DateTime(2019, 4, 1, 12, 0, 0, DateTimeKind.Local);
var dt2 = new DateTime(2019, 4, 1, 12, 0, 0, DateTimeKind.Utc);
Console.WriteLine(dt1.Subtract(dt2).TotalSeconds);
But the result was 0. I am in CST, and I was expecting like a 5 hour difference. I feel like I am missing some fundamental concept here on how DateTimes work in C#.

A few things:
Subtracting DateTime values does not take DateTimeKind into account.
.NET Framework 4.6 and higher have conversion functions to/from Unix time built in to the DateTimeOffset class, so you don't need to do any subtraction at all.
When you say EST or CST, I'll assume you meant US Eastern Time or US Central Time. Keep in mind that because of daylight saving time, that EDT or CDT might apply to some of your values.
You shouldn't be parsing from strings if the value is coming from your database. I'll assume you just gave that for the example here. But in your actual code, you should be doing something like:
DateTime dt = (DateTime) dataReader("field");
(If you're using Entity Framework or some other ORM, then this part would be handled for you.)
It doesn't matter what time zone the SQL Server is located in. It matters rather what time zone the dbValue you have is intended to represent. A best practice is to keep time in terms of UTC, in which case the server's time zone should be irrelevant.
If the datetime stored in your SQL server is actually in UTC, then you can simply do this:
long apiValue = new DateTimeOffset(dt, TimeSpan.Zero).ToUnixTimeMilliseconds();
If the datetime stored in your SQL server really is in US Eastern Time, then you will need to first convert from Eastern Time to UTC:
TimeZoneInfo tz = TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time");
DateTime utc = TimeZoneInfo.ConvertTimeToUtc(dt, tz);
long apiValue = new DateTimeOffset(utc).ToUnixTimeMilliseconds();
Note that the Windows time zone ID of "Eastern Standard Time" represents US Eastern Time including EDT when applicable, despite having the word "Standard" in the middle.
If you are running in .NET Core on a non-Windows platform, pass "America/New_York" instead. (And if you need to write for cross-platform resiliency, use my TimeZoneConverter library.)
Lastly, though it might be a bit dangerous to assume the time in the DB is in the same local time as the code accessing the DB, if you really wanted to make such a gamble, you could do it like this:
long apiValue = new DateTimeOffset(dt).ToUnixTimeMilliseconds();
That works only if dt.Kind is DateTimeKind.Unspecified or DateTimeKind.Local, as it would then apply the local time zone. But again, I would recommend against this.

Why doesn't C# detect that 1970/1/1 was under BST?

I'm working with a 3rd party API that returns Time of Day values as DateTime values filling in Jan 1, 1970 as the date part. So for 5AM, it will return something like 1969-12-31T21:03:00.000-08:00
The problem is that, when if the user was on London time, C# fails to apply BST adjustment for 1970-01-01.
For example, 1970-01-01 5AM in UTC should be 1970-01-01 6AM in London.
See conversion
But, C# doesn't seem to apply this conversion:
var utcTime = new DateTime(1970, 1, 1, 5, 0, 0, DateTimeKind.Utc);
var britishZone = TimeZoneInfo.FindSystemTimeZoneById("GMT Standard Time");
var ukTime = TimeZoneInfo.ConvertTime(utcTime, britishZone);
Console.WriteLine(ukTime);
The above code will still print 5AM.
However, if I set it to a more recent date that BST was effective such as Oct 1st, 2016, the same code works correctly printing 6AM.
Why does it behave this way?
How do I workaround this? Our app needs to work across any TimeZone (i.e. hard coding timezone isn't an option - we basically use the Windows local timezone).

When you use the TimeZoneInfo class on Windows to work with a system time zone ("GMT Standard Time" in this case), you are asking for time zone data from the Windows operating system, which is stored in the registry at:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones
Windows simply does not carry historical data back that far. For "GMT Standard Time", which is the time zone in London that alternates between GMT and BST, it only knows about one set of rules - the current ones that are in effect. It does not know about anything before that (it last changed in 1996).
Note that per Microsoft's DST/TZ Support Policy, only changes from 2010 forward are guaranteed to be recorded. There are several older changes that have historically been in Windows (such as the US 2007 DST change and others), which do indeed work, but from a global perspective you may not get the best results with dates before 2010.
For that, you'll need a full implementation of the IANA TZ Database, and in .NET, one of the best ways to do that is with the Noda Time library.
Your code, transliterated to Noda Time:
var utcTime = Instant.FromUtc(1970, 1, 1, 5, 0, 0);
var britishZone = DateTimeZoneProviders.Tzdb["Europe/London"];
var ukTime = utcTime.InZone(britishZone);
Console.WriteLine(ukTime.ToDateTimeUnspecified()); // just for equivalent output
// Prints: 1/1/1970 6:00:00 AM
Also note that if you are running .NET Core on Linux or OSX, then the time zones are indeed the IANA ones, thus your original code would work by just using the IANA identifer
var britishZone = TimeZoneInfo.FindSystemTimeZoneById("Europe/London");
See also the timezone tag wiki.

Have you looked at creating a custom time zone adjustment for the historical period(s) that are in question?
https://msdn.microsoft.com/en-us/library/bb397784(v=vs.110).aspx
which can be used in the case of:
The time zone does not have accurate information about time zone adjustments for a particular historic period.

I think the code is working correctly. BSt starts from last sunday of march until last sunday of october where the clock moves one hour forward . That is UTC+1. Since you are looking at a date in jan, london time will be same as UTC.that is utc+0
Check the wikipedia for timezone details https://en.m.wikipedia.org/wiki/British_Summer_Time

System.DateTime.Now.ToLocalTime not returning my local time

I'm getting a
"The supplied DateTime represents an invalid time. For example, when
the clock is adjusted forward, any time in the period that is skipped
is invalid"
message when I try to convert a time to UTC. The client that's experiencing the issue is in France, but the issue appears to have started when the US went to daylight savings time. I see there's a spot in the code that's using Eastern Time Zone. I'm thinking it must be somehow trying to use Eastern Standard time, somehow.
At any rate I tried mimicking their scenario by calling the same method after setting my locale to French/France and my timezone to Amsterdam, Berlin, etc. My test code is doing:
DateTime eventDate = System.DateTime.Now.ToLocalTime();
What doesn't make sense is that it's returning a date time as of Eastern Daylight Savings Time US, not the time that's being displayed by my system clock. Is the BIOS clock set to EDT or something? My intention was to pass eventDate to the method that seems to be working incorrectly for my French client.

What you've written is absolutely not possible.
You said you're getting the error:
"The supplied DateTime represents an invalid time. For example, when the clock is adjusted forward, any time in the period that is skipped is invalid"
That can only happen when converting a local time to UTC - not the other way around.
Also, it can only occur when using the conversion methods on TimeZoneInfo. For example, the following code will create an exception with that message:
TimeZoneInfo tz = TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time");
DateTime dt = new DateTime(2014, 3, 9, 2, 30, 0);
DateTime utc = TimeZoneInfo.ConvertTimeToUtc(dt, tz);
You might also think you'd get the same exception if your computer's time zone was set to "Eastern Standard Time" and you ran this code:
DateTime dt = new DateTime(2014, 3, 9, 2, 30, 0);
DateTime utc = dt.ToUniversalTime();
However, this method will silently allow invalid input without throwing an exception. It will just adjust forward an hour, as if you had passed 3:30 instead of 2:30.
Additionally, as others pointed out, DateTime.Now already is local and has Local kind. So calling ToLocalTime on it won't do anything. If you meant to call ToUniversalTime, then just use DateTime.UtcNow instead.
You may also be interested in reading the dst tag wiki, and the timezone tag wiki, which contain relevant details.

I added TimeZoneInfo.ClearCachedData(); and before my call to Now and it now returns the correct time.

Storing date/times as UTC in database

I am storing date/times in the database as UTC and computing them inside my application back to local time based on the specific timezone. Say for example I have the following date/time:
01/04/2010 00:00
Say it is for a country e.g. UK which observes DST (Daylight Savings Time) and at this particular time we are in daylight savings. When I convert this date to UTC and store it in the database it is actually stored as:
31/03/2010 23:00
As the date would be adjusted -1 hours for DST. This works fine when your observing DST at time of submission. However, what happens when the clock is adjusted back? When I pull that date from the database and convert it to local time that particular datetime would be seen as 31/03/2010 23:00 when in reality it was processed as 01/04/2010 00:00.
Correct me if I am wrong but isn't this a bit of a flaw when storing times as UTC?
Example of Timezone conversion
Basically what I am doing is storing the date/times of when information is being submitted to my system in order to allow users to do a range report. Here is how I am storing the date/times:
public DateTime LocalDateTime(string timeZoneId)
{
var tzi = TimeZoneInfo.FindSystemTimeZoneById(timeZoneId);
return TimeZoneInfo.ConvertTimeFromUtc(DateTime.UtcNow, tzi).ToUniversalTime().ToLocalTime();
}
Storing as UTC:
var localDateTime = LocalDateTime("AUS Eastern Standard Time");
WriteToDB(localDateTime.ToUniversalTime());

You don't adjust the date for DST changes based on whether you're currently observing them - you adjust it based on whether DST is observed at the instant you're describing. So in the case of January, you wouldn't apply the adjustment.
There is a problem, however - some local times are ambiguous. For example, 1:30am on October 31st 2010 in the UK can either represent UTC 01:30 or UTC 02:30, because the clocks go back from 2am to 1am. You can get from any instant represented in UTC to the local time which would be displayed at that instant, but the operation isn't reversible.
Likewise it's very possible for you to have a local time which never occurs - 1:30am on March 28th 2010 didn't happen in the UK, for example - because at 1am the clocks jumped forward to 2am.
The long and the short of it is that if you're trying to represent an instant in time, you can use UTC and get an unambiguous representation. If you're trying to represent a time in a particular time zone, you'll need the time zone itself (e.g. Europe/London) and either the UTC representation of the instant or the local date and time with the offset at that particular time (to disambiguate around DST transitions). Another alternative is to only store UTC and the offset from it; that allows you to tell the local time at that instant, but it means you can't predict what the local time would be a minute later, as you don't really know the time zone. (This is what DateTimeOffset stores, basically.)
We're hoping to make this reasonably easy to handle in Noda Time, but you'll still need to be aware of it as a possibility.
EDIT:
The code you've shown is incorrect. Here's why. I've changed the structure of the code to make it easier to see, but you'll see it's performing the same calls.
var tzi = TimeZoneInfo.FindSystemTimeZoneById("AUS Eastern Standard Time");
var aussieTime = TimeZoneInfo.ConvertTimeFromUtc(DateTime.UtcNow, tzi);
var serverLocalTime = aussieTime.ToLocalTime();
var utcTime = serverLocalTime.ToUniversalTime();
So, let's think about right now - which is 13:38 in my local time (UTC+1, in London), 12:38 UTC, 22:39 in Sydney.
Your code will give:
aussieTime = 22:39 (correct)
serverLocalTime = 23:39 (*not* correct)
utcTime = 22:39 (*not* correct)
You should not call ToLocalTime on the result of TimeZoneInfo.ConvertTimeFromUtc - it will assume that it's being called on a UTC DateTime (unless it's actually got a kind of DateTimeKind.Local, which it won't in this case).
So if you're accurately saving 22:39 in this case, you aren't accurately saving the current time in UTC.

It's good that you are attempting to store the dates and times as UTC. It is generally best and easiest to think of UTC as the actual date and time and local times are just pseudonyms for that. And UTC is absolutely critical if you need to do any math on the date/time values to get timespans. I generally manipulate dates internally as UTC, and only convert to local time when displaying the value to the user (if it's necessary).
The bug that you are experiencing is that you are incorrectly assigning the local time zone to the date/time values. In January in the UK it is incorrect to interpret a local time as being in a Summertime time zone. You should use the time zone that was in effect at the time and location that the time value represents.
Translating the time back for display depends entirely on the requirements of the system. You could either display the times as the user's local time or as the source time for the data. But either way, Daylight Saving/Summertime adjustments should be applied appropriately for the target time zone and time.

You could work around this by also storing the particular offset used when converting to UTC. In your example, you'd store the date as something like
31/12/2009 23:00 +0100
When displaying this to the user, you can use that same offset to convert, or their current local offset, as you choose.
This approach also comes with its own problems. Time is a messy thing.

The TimeZoneInfo.ConvertTimeFromUtc() method will solve your problem:
using System;
class Program {
static void Main(string[] args) {
DateTime dt1 = new DateTime(2009, 12, 31, 23, 0, 0, DateTimeKind.Utc);
TimeZoneInfo tz = TimeZoneInfo.FindSystemTimeZoneById("GMT Standard Time");
Console.WriteLine(TimeZoneInfo.ConvertTimeFromUtc(dt1, tz));
DateTime dt2 = new DateTime(2010, 4, 1, 23, 0, 0, DateTimeKind.Utc);
Console.WriteLine(TimeZoneInfo.ConvertTimeFromUtc(dt2, tz));
Console.ReadLine();
}
}
Output:
12/31/2009 11:00:00 PM
4/2/2010 12:00:00 AM
You'll need .NET 3.5 or better and run on an operating system that keeps historical daylight saving time changes (Vista, Win7 or Win2008).

Correct me if I am wrong but isn't
this a bit of a flaw when storing
times as UTC?
Yes it is. Also, days of the adjustment will have either 23 or 25 hours so the idiom of the prior day at the same time being local time - 24 hours is wrong 2 days a year.
The fix is picking one standard and sticking with it. Storing dates as UTC and displaying as local is pretty standard. Just don't use a shortcut of doing calculations local (+- somthing) = new time and you are OK.

This is a huge flaw but it isn't a flaw of storing times in UTC (because that is the only reasonable thing to do -- storing local times is always a disaster). This is a flaw is the concept of daylight savings time.
The real problem is that the time zone information changes. The DST rules are dynamic and historic. They time when DST starting in USA in 2010 is not the same when it started in 2000. Until recently Windows did not even contain this historic data, so it was essentially impossible to do things correctly. You had to use the tz database to get it right. Now I just googled it and it appears that .NET 3.5 and Vista (I assume Windows 2008 too) has done some improvement and the System.TimeZoneInfo actually handles historic data. Take a look at this.
But basically DST must go.