I'm writing a C# DKIM validator and have come across a problem that I cannot solve. Right now I am working on calculating the body hash, as described in Section 3.7 Computing the Message Hashes. I am working with emails that I have dumped using a modified version of EdgeTransportAsyncLogging sample in the Exchange 2010 Transport Agent SDK. Instead of converting the emails when saving, it just opens a file based on the MessageID and dumps the raw data to disk.
I am able to successfully compute the body hash of the sample email provided in Section A.2 using the following code:
SHA256Managed hasher = new SHA256Managed();
ASCIIEncoding asciiEncoding = new ASCIIEncoding();
string rawFullMessage = File.ReadAllText(#"C:\Repositories\Sample-A.2.txt");
string headerDelimiter = "\r\n\r\n";
int headerEnd = rawFullMessage.IndexOf(headerDelimiter);
string header = rawFullMessage.Substring(0, headerEnd);
string body = rawFullMessage.Substring(headerEnd + headerDelimiter.Length);
byte[] bodyBytes = asciiEncoding.GetBytes(body);
byte[] bodyHash = hasher.ComputeHash(bodyBytes);
string bodyBase64 = Convert.ToBase64String(bodyHash);
string expectedBase64 = "2jUSOH9NhtVGCQWNr9BrIAPreKQjO6Sn7XIkfJVOzv8=";
Console.WriteLine("Expected hash: {1}{0}Computed hash: {2}{0}Are equal: {3}",
Environment.NewLine, expectedBase64, bodyBase64, expectedBase64 == bodyBase64);
The output from the above code is:
Expected hash: 2jUSOH9NhtVGCQWNr9BrIAPreKQjO6Sn7XIkfJVOzv8=
Computed hash: 2jUSOH9NhtVGCQWNr9BrIAPreKQjO6Sn7XIkfJVOzv8=
Are equal: True
Now, most emails come across with the c=relaxed/relaxed setting, which requires you to do some work on the body and header before hashing and verifying. And while I was working on it (failing to get it to work) I finally came across a message with c=simple/simple which means that you process the whole body as is minus any empty CRLF at the end of the body. (Really, the rules for Body Canonicalization are quite ... simple.)
Here is the real DKIM email (right click and save it, the browsers eat the ending CRLF) with a signature using the simple algorithm (completely unmodified). Now, using the above code and updating the expectedBase64 hash I get the following results:
Expected hash: VnGg12/s7xH3BraeN5LiiN+I2Ul/db5/jZYYgt4wEIw=
Computed hash: ISNNtgnFZxmW6iuey/3Qql5u6nflKPTke4sMXWMxNUw=
Are equal: False
The expected hash is the value from the bh= field of the DKIM-Signature header. Now, the file used in the second test is a direct raw output from the Exchange 2010 Transport Agent. If so inclined, you can view the modified EdgeTransportLogging.txt.
At this point, no matter how I modify the second email, changing the start position or number of CRLF at the end of the file I cannot get the files to match. What worries me is that I have been unable to validate any body hash so far (simple or relaxed) and that it may not be feasible to process DKIM through Exchange 2010.
I tried this in python-dkim and I get a body hash mismatch too.
I think probably Exchange's GetMimeReadStream is not giving you the actual bytes as they were transmitted, therefore the hash doesn't match. Probably it's disassembling the message into its mime parts, and then GetMimeReadStream gives you a valid representation of the message, but not the one it was originally sent with.
Perhaps there's another API that will give you the real raw bytes?
Or perhaps by this point in the process the message has been torn apart and the original message thrown away, and you need to hook in earlier.
Probably you should try intercepting a DKIM-signed message by sending it to a non-Exchange server, and see if that works with your code. GetContentReadStream might possibly work?
Anyhow, what I would do next is try to find an API that gives you byte-for-byte what was sent.
Related
Completely stuck on a problem related to the inbound parse webhook functionality offered by SendGrid: https://sendgrid.com/docs/for-developers/parsing-email/setting-up-the-inbound-parse-webhook/
First off everything is working just fine with retrieving the mail sent to my application endpoint. Using Request.Form I'm able to retrieve the data and work with it.
The problem is that we started noticing question mark symbols instead of letters when recieving some mails (written in swedish using Å Ä and Ö). This occured both when sending plaintext mails, and mails with an HTML-body.
However, this only happens every now and then. After a lot of searching I found out that if the mail is sent from e.g. Postbox or Outlook (or the like), and the application has the charset set to iso-8859-1 that's when Å Ä Ö is replaced by question marks.
To replicate the error and be able to debug it I set up a HTML page with a form using the iso-8859-1 encoding, sending a similar payload as the one seen in the link above (the default one). And after that been through testing a multitude of things trying to get it to work.
As of now I'm trying to recode the input, without success. Code I'm testing:
Encoding wind1252 = Encoding.GetEncoding(1252);
Encoding utf8 = Encoding.UTF8;
byte[] wind1252Bytes = wind1252.GetBytes(Request.Form.["html"]);
byte[] utf8Bytes = Encoding.Convert(wind1252, utf8,wind1252Bytes);
string utf8String = Encoding.UTF8.GetString(utf8Bytes);
This only results in the utf8String producing the same result with "???" where Å Ä Ö should be. My guess here is that perhaps it's due to the Request.Form["html"] returning a UTF-16 string, of the content that is encoded already in the wrong encoding iso-8859-1.
The method for fetching the POST is as follows
public async Task<InboundParseModel> FetchMail(IFormCollection form)
{
InboundParseModel _em = new InboundParseModel
{
To = form["to"].SingleOrDefault(),
From = form["from"].SingleOrDefault(),
Subject = form["subject"].SingleOrDefault(),
Html = form["html"].SingleOrDefault(),
Text = System.Net.WebUtility.HtmlEncode(form["text"].SingleOrDefault()),
Envelope = form["envelope"].SingleOrDefault()
};
}
Called from another method that the POST is done to by FetchMail(Request.Form);
Project info: ASP.NET Core 2.2, C#
So as stated earlier, I am completely stuck and don't really have any ideas on how to solve this. Any help would be much appreciated!
I'll try to be brief, but I'll share the whole picture.
Problem Statement
I am using vector tile from tippecanoe from mapbox to create .pbtiles from my geojson data. The issue is, on a web client when I see the inspect element and download the .pbf and run it by this (mapbox-vector-tile-cs) library, I am able to successfully get the data from the tile. Which means that any one with some basic google search can also steal my data from the vector tiles.
What I was able to achieve
To avoid the security concern, with the short timeline I have, I came up with a quick and dirty way. After tippecanoe creates the .mbtiles sqlite db, I run a java utility I made to encrypt the data in the blob using AES 256 encryption and stored it in two different ways in two different sqlite db's:
Stored as bytes into a different .mbtiles sqlite db (which get's stored as Blob). Along with z, x, y and metadata
Encoded the encrypted data as base64 and then stored the base64encoded encrypted tile data into a string data type column. Along with z, x, y and metadata.
and stored the key (base64 encoded) and initialization vector (base64 encoded) into a file.
The API side (Question 1)
Now, when I get the non encrypted .pbf from the API, a header of type gzip and application/x-protobuf is set that helps to convert the unencrypted blob data to a protobuf and returns a .pbf file that gets downloaded.
Now when I try to get the encrypted data from the API with the same header as the non encrypted on, the download of the .pbf fails saying Failed - Network error. I realized that it's being caused as the header application/x-protobuf is trying to package the file into a .pbf while the contents of the blob might not be matching what's expected and hence the result.
I removed the header application/x-protobuf and since I can't gzip now, i removed the header of gzip too. Now the data gets displayed on the chrome browser instead of being downloaded, I figure as now it's just a random response.
The question is, How can I make it to send a .pbf that has encrypted data in it and this((mapbox-vector-tile-cs)) library can parse the data? I know the data will be need to be decrypted first before I pass it for parsing assuming that it's decrypted and I have the data that was stored into the blob of the .mbtiles.
This Library with a UWP project (Question 2)
So now currently as mentioned above (since i don't have a solution to the headers part) I removed the headers and let the API return me a direct response.
The Issue now I am facing is that when I pass in the decryted (I checked the decryption was successful and the decrypted data is an exact match to the what was stored in the Blob) Blob data to the
var layerInfos = VectorTileParser.Parse(stream);
code line returns me an IEnumerable<Tile> that is not null but has 0 layers in it. while the actual tile contains 5 layers in it.
My Question is, how do I get this((mapbox-vector-tile-cs)) library to return me the layers.
The code to fetch the tile from the server and decrypt before I send it for parsing is as below:
//this code downloads the tile, layerInfos is returned as an empty collection
private async Task<bool> ProcessTile(TileData t, int xOffset, int yOffset)
{
var stream = await GetTileFromWeb(EncryptedTileURL,true);
if (stream == null)
return false;
var layerInfos = VectorTileParser.Parse(stream);
if (layerInfos.Count == 0)
return false;
return true;
}
The tiles are fetched from the server using a GetTileFromWeb() method:
private async Task<Stream> GetTileFromWeb(Uri uri, bool GetEnc = false)
{
var handler = new HttpClientHandler();
if (!GetEnc)
handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
var gzipWebClient = new HttpClient(handler);
var bytes = gzipWebClient.GetByteArrayAsync(uri).Result;
if (GetEnc)
{
var decBytes = await DecryptData(bytes);
return decBytes;
}
var stream = new MemoryStream(bytes);
return stream;
}
PS: Sorry for such a long question, I am not used to such elaborate detail, but seemed I need to share more as Encryption is my forte while map data vector tiles isn't.
I'm currently working on a project where pdf's can be decrypted after a successful api call that returns the password.
I've browsed through SO and pdftron SDK but can't find a definitive solution on how to insert this password into the security handler.
Things I have tried:
None of the GetSecurityHandler() methods seem to handle password insertion:
SecurityHandler handler = m_PdfDocument.GetSecurityHandler();
Takes a password string but throws error:
m_PdfDocument.InitStdSecurityHandler(pwd);
error: Message: Not a standard security handler. The custom filter needs to be registered.
Judging from the message I assumed I needed m_PdfDocument.InitSecurityHandler() instead, but that method doesn't take a string, only int.
Anyone can bump me onto the right track ?
Thank you for sending the file. This file is encrypted using custom encryption. Your DRM. No PDF reader can open the file, but your own custom PDF reader.
To open the PDF with PDFNet, you need to find out how the file was encrypted in the first place, and essentially do the opposite. I assume the other team that did the encryption was also decrypting, for at least testing purposes?
It might as simple as following example 3 in our Encryption sample. In which case you just need to register under the filter name that the other team used. I think I know what that is, but won't post here, and will email you instead.
But for others, if the PDF was encrypted with a filter called "Frodo", then it would be
CreateDelegate frodo_del = new CreateDelegate(FrodoSecurityHandler.Create);
SecurityManagerSingleton.Instance().RegisterSecurityHandler("Frodo", new SecurityDescriptor("Frodo Security", frodo_del));
Well according to this page, GetSecurityHandler() is used after you initialize another handler, so since InitSecurityHandler() takes an int you could do this
string password = "9quali52ty3";
// Convert the string into a byte[].
byte[] asciiBytes = Encoding.ASCII.GetBytes(password);
string compiledBytes = System.Text.Encoding.ASCII.GetString(asciiBytes);
int convertedBytes = int.Parse(compiledBytes);
m_PdfDocument.InitSecurityHandler(convertedBytes);
m_PdfDocument.GetSecurityHandler();
A good rule of thumb for programming: There is always a way to get from one datatype to another.
Credit to: #Brig Lamoreaux, #Zanoni and #Brandon on the following pages.
Brig Zanoni Brandon
I am trying to get Avatar from google talk.
I received packet from goole talk server like:
<presence from="xxxxxxxxxxxxx#gmail.com/MessagingA3e8c9465" to="xxxxxxxxxx#gmail.com/Jabber.NetF5D1AB65">
<show>away</show>
<caps:c ver="1.1" node="http://www.android.com/gtalk/client/caps" xmlns:caps="http://jabber.org/protocol/caps" />
<x xmlns="vcard-temp:x:update">
<photo>6373f2ccdf12ef06292ca2257dc0bdc9aa1040c2</photo>
</x>
I thought the hex vale of '<photo>' tag is the avatar (display image) of the contact. (Please correct me if I am wrong.)
I converted that value to byte[]
and used following code to display the image.
pictureBox1.Image = Image.FromStream(new MemoryStream(byte_array));
// byte_array is byte[] converted from hex value.
It raises exception saying:
Parameter is not valid.
I am using the following function to covert from hex to byte[]:
private static byte[] HexString2Bytes(string hexString)
{
int bytesCount = (hexString.Length) / 2;
byte[] bytes = new byte[bytesCount];
for (int x = 0; x < bytesCount; ++x)
{
bytes[x] = Convert.ToByte(hexString.Substring(x * 2, 2), 16);
}
return bytes;
}
I tries many ways but same result.
I also tried to convert the hex value to uppercase, but no luck, same result.
I am using .net 3.5 on windows 8.1 machine.
Thanks
Updated:
Thanks to every one for their comments and answer.
I was wrong the hex value was not avatar (display image).
I sent 'iq' request to server and it gives the avatar.
Thanks a lot.
Happy Coding.
http://www.xmpp.org/extensions/xep-0153.html says following:
Next, the user's client computes the SHA1 hash of the avatar image data itself (not the base64-encoded version) in accordance with RFC 3174 [4]. This hash is then included in the user's presence information as the XML character data of the child of an element qualified by the 'vcard-temp:x:update' namespace, as shown in the following example:
Example 3. User's Client Includes Avatar Hash in Presence Broadcast
So, basically hex value of '' tag is not the avatar, but SHA1 hash of the avatar image.
The hex value that you see is not the display image of the contact. It is a hash of the display image. The logic to get the display image is as follows.
After login on the XMPP client, you start receiving presence messages from the XMPP server.
In the presence message, you receive the hash of the avatar.
Check your local storage, if you have a binary image against the received hash.
If you have a binary image against the hash, then display the avatar on your client from the local storage.
If you do not have a binary image against the hash, send a request for v-card to the XMPP server, for the user against which you received the presence.
On receiving the v-card response, you will find the hash and the display image binary. Store this in some local storage.
For details on the XMPP packets Read section 3.2 on http://www.xmpp.org/extensions/xep-0153.html#retrieve
According to this, the photo is Base64-encoded. So you simple need to call Convert.FromBase64String to get the byte array from the photo element InnerText.
I'm struggling finding a feasible solution to this. I've tried looking around but can't find any documentation regarding this issue. If a customer sends out a message with quote(s), it break the payload syntax and android spits me back a 400 Bad Request error.
The only solution I can think of is by doing my own translations and validations. Allow only the basics, and for the restricted do my own "parsing" Ie take a quote, replace them with "/q" and then replace "/q" on the App when received. I don't like this solution because it involves logic on the App that if, I forget something. I want to be able to change it immediately rather then update everyones phone, app, etc.
I'm looking for an existing encoding I could apply that is processed correctly by the GCM servers. Allowing them to be accepted then broadcasted. Received by the phone with the characters intact.
Base64 encoding should get rid of the special characters. Just encode it before sending and decode it again on receiving:
Edit: sorry, just got a java/android sample here, I don't know how exactly xamarin works and what functions it provides:
// before sending
byte[] data = message.getBytes("UTF-8");
String base64Message = Base64.encodeToString(data, Base64.DEFAULT);
// on receiving
byte[] data = Base64.decode(base64Message , Base64.DEFAULT);
String message= new String(data, "UTF-8");
.Net translation of #tknell solution
Decode:
Byte[] data = System.Convert.FromBase64String(encodedString);
String decoded = System.Text.Encoding.UTF8.GetString(data);
Encode:
Byte[] data = System.Text.Encoding.UTF8.GetBytes(decodedString);
String encoded = System.Convert.ToBase64String(data);