We Keep Coding

c++ Azure Storage Queue is sending the wrong type of string

I have inherited a C++ app (of which Im no expert, Im a .NET guy) which sends messages to an azure queue in a JSON form. This works fine, its when I try and pick the message off the queue in my .NET console app that gives me the following message:
“The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or a non-white space character among the padding characters”.
The C++ code looks like this (notice the commented out dummy message which gives an example of what it looks like)
void send(utility::string_t msg) {
// Define the connection-string with your values.
const utility::string_t storage_connection_string(U("DefaultEndpointsProtocol=https;AccountName=bogus;AccountKey=YcG8FP9HdaB+r5jDTruTzZy8dXku+fLr4hvPcq+C6Uzhh7UOB6C7MemYluQMz28JlzwZIcn6Vw=="));
// Retrieve storage account from connection string.
azure::storage::cloud_storage_account storage_account = azure::storage::cloud_storage_account::parse(storage_connection_string);
// Create a queue client.
azure::storage::cloud_queue_client queue_client = storage_account.create_cloud_queue_client();
// Retrieve a reference to a queue.
azure::storage::cloud_queue queue = queue_client.get_queue_reference(U("beam-queue"));
// Create the queue if it doesn't already exist.
queue.create_if_not_exists();
// Create a message and add it to the queue.
//Dummy message
//azure::storage::cloud_queue_message message(U("[{\"url\":\"https://www.google.com.au\",\"app\":null,\"email\":\"jv#jv.ie\",\"dbId\":\"323e3098-cc87-4b37-8eb5-85a6d6ddba1c\",\"seconds\":147.0490574,\"date\":\"2016-11-17T00:00:00+11:00\"}]"));
azure::storage::cloud_queue_message message(msg);
queue.add_message(message);
lastsendtime = GetTickCount();
}
I can even see the message in the storage explorer:
But it seems to be in the wrong format as when i pick the message off the queue:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(GetConnectionString());
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueueClient clnt = storageAccount.CreateCloudQueueClient();
CloudQueue queue = clnt.GetQueueReference("bogus");
queue.EncodeMessage = true;
List<Service> userServices = null;
CloudQueueMessage retrievedMessage = queue.GetMessage();
List<CloudAppItem> items = JsonConvert.DeserializeObject<List<CloudAppItem>>(queue.GetMessage().AsString);
It fails in the last line. And its not because of the serializer. The queue.GetMessage().AsString returns the error.
UPDATE (Still not working)
I took out the encodedmessage statement, as I had previous tried to add it in to make it work. It still doesnt work. I also show the raw string, not accessible due to the fact it is private method on the class:

I took out the encodedmessage statement, as I had previous tried to add it in to make it work. It still doesnt work.
In your screenshot, we could find that you just remove queue.EncodeMessage = true;, but the default value of CloudQueue.EncodeMessage property is true, please explicitly set CloudQueue.EncodeMessage to false.
CloudQueueClient clnt = storageAccount.CreateCloudQueueClient();
CloudQueue queue = clnt.GetQueueReference("bogus");
queue.EncodeMessage = false; //explicitly set CloudQueue.EncodeMessage to false

The reason you're getting this error is because you're instructing the SDK to decode the message from a base64 encoded string however the message contents is not base64 encoded (you're saving the message as plain text).
Please change the following line of code:
queue.EncodeMessage = true;
to
queue.EncodeMessage = false;
And that should take care of the problem. From CloudQueue.EncodeMessage documentation:
Gets or sets a value indicating whether to apply base64 encoding when
adding or retrieving messages.

GetMessage() take excessive long time MailKit

When I call
var result = MailKit.GetMessage(uid)
Then the call takes around 4 seconds to return. Is there any better/faster way to get the mime message from the server??
Currently doing the following:
1. Fetch(0, -1, MessageSummaryItems.UniqueId | MessageSummaryItems.Envelope | MessageSummaryItems.Flags)
2. Loop the fetch result to find a messageId match
3. Send GetMessage() with the UID found previously.
I can see that FETCH in most cases takes around 2 secs, all in all my process will take something like 6 seconds, which is kinda "long" before I can present the email in the client.
Is there a faster way to get a mime message from a mail server using MailKit, when all I know about the email is the MessageID??

You could cache all of the message summaries so that you don't have to ask the server for them each time...
You could also try:
var uids = folder.Search (SearchQuery.HeaderContains ("Message-Id", messageId));
if (uids.Count > 0)
message = folder.GetMessage (uids[0]);

Is it possible to detect message type in protobuf-csharp-port?

A java client constructs a Message according to this skeleton :
package tutorial;
option java_package = "com.example.sff.piki2";
option java_outer_classname = "MsgProtos";
message MSG {
required string guid = 1;
required int32 MsgCode = 2;
required int32 From = 3; //sender
...
This message is sent to a C# program ( server side).
The server knows how to read the bytes ( first byte is the number of bytes to read which represents the size of the following message ).
This is how MSG is being constructed by a byte array.
MSG Msg = MSG.CreateBuilder().MergeFrom(buffer).Build();
Where buffer is the byte array which read from the socket.
But now I'm in a situation where a client needs to send "Heartbeat" message( another message) in order to check if the server is alive or not. ( the server should respond : "yes i'm alive")
Sure , I can add another field to the MSG class. but I don't want to because the MSG class has a lot of unnecessary fields - for a Heartbeat operation.
Question :
The server read n bytes. Is there anyway I can know if this is a MSG message or a "Heartbeat" message ?

Is there anyway I can know if this is a MSG message or a "Heartbeat" message?
No. Protocol buffer messages don't contain any such type information. Typically the way round this is to have a "wrapper" type with a field for each message you might want to send. You'd ideally want to express this as a oneof, but my port doesn't support that (yet).
The overhead is minimal - but it will be a protocol change, of course, so you'll need to think about any existing servers etc.

Unable to transfer files from Windows to Linux via Bluetooth using 32feet.Net C# library

First linux service listening process is started using the following command:
obexpushd –B[00:15:83:3D:0A:57]:9 –d –o /home/myfolder
On windows the following code is used to perform the obex transfer:
InTheHand.Net.BluetoothAddress address = peerDevice.DeviceAddress;
System.Uri uri = new Uri("obex://" + address.ToString() + "/" + srcfile.Name);
request = new ObexWebRequest(uri);
startcopy = DateTime.Now;
request.ReadFile(file); // this performs the file read from the hard drive
try
{
response = (ObexWebResponse)request.GetResponse(); // here file should be pushed to the listening service
}
catch (System.InvalidOperationException ex)
{
if (response != null) {
response.Close();
}
return;
}
Devices see each other and their obex services are visible as well.
Transfer seems to be successful, but no data is actually transferred.
The code works between windows and windows without a problem.
Obexpushd process ouput shows:
obexpushd 0.10.2 Copyright (C) 2006-2010 Hendrik Sattler
This software comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions.
Listening on bluetooth/[00:15:83:3D:0A:57]:9
OBEX_EV_ACCEPTHINT, OBEX_CMD_CONNECT
0: Connection from "bluetooth/[00:09:DD:50:94:0B]:9"
0: OBEX_EV_REQHINT, OBEX_CMD_CONNECT
0: OBEX_EV_REQ, OBEX_CMD_CONNECT
0: Sending response code 0
0: OBEX_EV_REQDONE, OBEX_CMD_CONNECT
0: OBEX_EV_REQHINT, OBEX_CMD_PUT
0.1: OBEX_EV_REQCHECK, OBEX_CMD_PUT
0.1: OBEX_EV_REQDONE, OBEX_CMD_PUT
0.1: OBEX_EV_REQHINT, OBEX_CMD_DISCONNECT
0.1: Sending response code 100
0.1: OBEX_EV_REQ, OBEX_CMD_DISCONNECT
0.1: OBEX_EV_REQDONE, OBEX_CMD_DISCONNECT
I have also tried to disable Authentication in C# code but that did not help.
Does any one have idea how to nail this problem down or where to even start looking?

Ok so it seems that no one is much interested in this topic. :) Fortunately I have found the solution myself.
However, it involved a lot of analysis (of both obex transfer protocol and 32feet library) and a bit of luck.
The difference between Linux obexpushd implementation lies in its interpretation of OBEX transfer packets.
I found the OBEX specification on page: OBEX specification.
After debugging the internals of the 23feet obex transfer I found where the code sends the OBEX PUT command used to send file to the receiver. Obex specification gives the following example for PUT inititialization packet:
PUT Command | length of packet | Name header | Length of Name header | name of object | Length header | Length of object | Object Body chunk header | Length of Body header | bytes of body
The 32feet library sends the first packet without the Body Header which causes error in obexpushd Linux command.
Not shure if it is error in 32feet library, obexpushd or if the OBEX specification is not precise enough, but adding the Body header to the first packet solved the problem. From my experiments it turns out that at least 2 first bytes of the objectt must be sent in the first packet. Moreover, adding the header does not crash anything else and Windows<->Windows transfer still works very well.

protobuf-csharp-port

I'm using Jon Skeet's (excellent) port of Google's Protocol Buffers to C#/.Net.
For practice, I have written a dummy Instant Messenger app that sends some messages down a socket. I have a message definition as follows:-
message InstantMessage {<br/>
required string Message = 1;<br/>
required int64 TimeStampTicks = 2; <br/>
}
When the sender serialises the message, it sends it really elegantly:-
...
InstantMessage.Builder imBuild = new InstantMessage.Builder();
imBuild.Message = txtEnterText.Text;
imBuild.TimeStampTicks = DateTime.Now.Ticks;
InstantMessage im = imBuild.BuildPartial();
im.WriteTo(networkStream);
...
This works great. But at the other end, I'm having trouble getting the ParseFrom to work.
I want to use:-
InstantMessage im = InstantMessage.ParseFrom(networkStream);
But instead I have had to read it to bytes and then parse it from here. This is obviously not ideal for a number of reasons. Current code is:-
while (true)
{
Byte[] byteArray = new Byte[10000000];
int intMsgLength;
int runningMsgLength = 0;
DateTime start = DateTime.Now;
while (true)
{
runningMsgLength += networkStream.Read(byteArray, runningMsgLength, 10000000 - runningMsgLength);
if (!networkStream.DataAvailable)
break;
}
InstantMessage im = InstantMessage.ParseFrom(byteArray.Take(runningMsgLength).ToArray());
When I try to use ParseFrom, control does not return to the calling method even when I know a valid GB message is on the wire.
Any advice would be gratefully received,
PW

Sorry for taking a while to answer this. As Marc says, protocol buffers don't have a terminator, and they aren't length prefixed unless they're nested. However, you can put on the length prefix yourself. If you look at MessageStreamIterator and MessageStreamWriter, you'll see how I do this - basically I pretend that I'm in the middle of a message, writing a nested message as field 1. Unfortunately when reading the message, I have to use internal details (BuildImpl).
There's now another API to do this: IMessage.WriteDelimitedTo and IBuilder.MergeDelimitedFrom. This is probably what you want at the moment, but I seem to remember there's a slight issue with it in terms of detecting the end of the stream (i.e. when there isn't another message to read). I can't remember whether there's a fix for it at the moment - I have a feeling it's changed in the Java version and I may not have ported the change yet. Anyway, that's definitely the area to look at.

Protobuf has no terminator - so either close the stream, or use your own length prefix etc. Protobuf-net exposes this easily via SerializeWithLenghtPrefix / DeserializeWithLengthPrefix.
Simply: without this, it can't know where each message ends, so keeps trying to read to the end of the stream.