I'm pretty new to Microsoft Bot Framework. I have a bot that receives messages from one user and can be viewed by another. I want to add a feature where the viewer can then reply to the sender. I figured the best way to do this is to send a proactive message to the original sender. However, I'm having trouble understanding the documentation Microsoft provides and other sources are pretty dated.
Right now this is what I have.
MessageDetails.RelatesTo contains the ConversationReference:
private async Task<DialogTurnResult> FinalStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
var messageDetails = (MessageDetails)stepContext.Options;
var replyText = (string)stepContext.Result;
var messageactivity = messageDetails.RelatesTo.GetContinuationActivity();
await stepContext.Context.SendActivityAsync($"{messageDetails.RelatesTo}");
var client = new ConnectorClient(new Uri(messageactivity.ServiceUrl));
if (messageDetails.IsTrustedServiceUrl)
{
MicrosoftAppCredentials.TrustServiceUrl(messageactivity.ServiceUrl);
}
var triggerReply = messageactivity.CreateReply();
triggerReply.Text = $"NoReply from {stepContext.Context.Activity.Name}: {replyText}";
await client.Conversations.ReplyToActivityAsync(triggerReply);
await stepContext.Context.SendActivityAsync($"Your reply has been sent to {messageDetails.RelatesTo.User.Name}.");
return await stepContext.EndDialogAsync(messageDetails, cancellationToken);
}
This code doesn't work and I'm not entirely sure why. (I would also appreciate any advice on how I might troubleshoot the problem). I realize that this doesn't make use of the controller shown in the sample code provided by Microsoft. Honestly, I don't understand how the notify controller works. So if a solution involves that, it would be great to get an explanation on some of the details.
Fixing Your Problem
My guess is that you're getting this error:
System.ArgumentNullException: 'Value cannot be null. Parameter name: clientId'
This is because you're not specifying the appId in new ConnectorClient. You can disregard that, as you're better off doing something more like:
await stepContext.Context.Adapter.ContinueConversationAsync("<yourAppId>", messageDetails.RelatesTo, async (ITurnContext turnContext, CancellationToken cancel) =>
{
await turnContext.SendActivityAsync(triggerReply);
}, cancellationToken);
The key is making sure you have a ConversationReference for the user that you need to send the proactive message to. If you don't have one, you should be able to CreateConversation to establish one
Note that you also have:
if (messageDetails.IsTrustedServiceUrl)
{
MicrosoftAppCredentials.TrustServiceUrl(messageactivity.ServiceUrl);
}
This is basically saying, "if we already trust the serviceUrl, trust it again". Instead, you need: if (!messageDetails.IsTrustedServiceUrl)
How Proactive Bot Works
Here's a brief explanation of how the proactive sample works. I recommend downloading it and playing around with it to get a better understanding.
The bot establishes listeners on api/messages (all bots do this) and api/notify (only this bot does this).
When a user messages the bot, it goes through api/messages and is processed through the ActivityHandler.
On any message, the bot saves the conversation reference
When somebody visits <theBotUrl>/api/notify, it loops through each saved conversation reference, calls ContinueConversation, and sends the message to all of the users in the saved conversation references
Related
I'm creating a bot and every time the conversation end, I'll to clear the conversation state. For That I'm sending an activity with type endOfConversation after the last message.
At the method OnMessageActivityAsync of my bot I've added this code.
turnContext.OnSendActivities(OnSendActivitiesHandlerAsync);
Witch must execute this code:
private async Task<ResourceResponse[]> OnSendActivitiesHandlerAsync(ITurnContext turnContext, List<Activity> activities, Func<Task<ResourceResponse[]>> next)
{
foreach (Activity activity in activities)
{
if (activity.Type == ActivityTypes.EndOfConversation)
{
await _conversationState.ClearStateAsync(turnContext); // <-- `_conversationState` is a global variable type of `BotState`.
}
}
return await next();
}
It works very good in the Bot Framework Emulator, but when it goes live on Microsoft Teams I had errors. When looking for that message I've found this:
That particular type is not supported in Teams, sorry.
https://github.com/microsoft/botframework-sdk/issues/3300
In the logs I see this when I want to send an endOfConversation type.
{
"code": "BadArgument",
"message": "Unknown activity type",
"innerHttpError": null
}
I would to send an end of conversation type because based on that type I could clear the conversation state on the server side of the bot. In that state I store some objects that aren't not relevant anymore when the conversation is ended.
So my question is now how could I clear the conversation state at the end of an conversation without using the endOfConversation type?
"RichMoe" in the link you've provided states that Teams does not support this operation (2017) and I think the same is still true - I don't think Teams has a concept of the "end" of a conversation, in the way that, for example, a support bot on a website would - like any conversation, Teams will try to keep the entire chat history forever.
Perhaps you can explain why you need to "end" the conversation, if there's something related I can try help with.
I have a Bot created with v4 of the Microsoft Bot Framework. I can successfully use this bot in the "Test in Web Chat" portion in the Azure Portal. I can also successfully use this bot in an app that I've created in Microsoft Teams. I now want to send a notification from the "Test in Web Chat" piece to a specific user in Teams. For example, in the "Test in Web Chat" piece, I'd like to enter
Hello someuser#mytenant.com
When this is sent via the "Test in Web Chat" piece, I'd like to show "Hello" in Microsoft Teams to only someuser#mytenant.com. I have successfully tokenized the string from the "Test in Web Chat". Thus, I know what I want to send, and who I want to send it to. However, I do not know how to actually send it.
Currently, I have the following in my bot:
public class EchoBot : ActivityHandler
{
private ConcurrentDictionary<string, ConversationReference> _conversationReferences;
public EchoBot(ConcurrentDictionary<string, ConversationReference> conversationReferences)
{
_conversationReferences = conversationReferencs;
}
private void AddConversationReference(Activity activity)
{
var reference = activity.GetConversationReference();
_conversationReferences.AddOrUpdate(reference.User.Id, reference, (key, newValue) => reference);
}
protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> context, CancellationToken cancellationToken)
{
AddConversationReference(context.Activity as Activity);
var parameters = GetParameters(); // Parses context.Activity.Text;
// Send a message to the target (i.e. someuser#mytenant.com)
var connection = new Microsoft.Bot.Connector.ConnectorClient(new Uri(context.Activity.ServiceUrl));
var tenant = context.Activity.GetChannelData<TeamsChannelData>().Tenant;
// how do I send the message to parameters.Target?
// Confirm message was sent to the sender
var confirmation = $"Message was sent to {parameters.Target}.";
await context.SendActivityAsync(MessageFactory.Text(confirmation));
}
}
I've reviewed how to send proactive notifications to users. However, I've been unsuccessful in a) getting the user specified in parameters.Target and b) sending a notification to that user. What am I missing?
First, you'll need to map user#email.com to their Teams userId (maybe with a static dictionary), which is in the format of:
29:1I9Is_Sx0O-Iy2rQ7Xz1lcaPKlO9eqmBRTBuW6XzXXXXXXXXMij8BVMdBcL9L_RwWNJyAHFQb0TXXXXXX
You can get the Teams UserId by either:
Querying the roster, or
Having the user message the bot, and setting a breakpoint on an incoming message, looking at the Activity.ChannelData for the Teams userId, or
Dynamically build a static dictionary of all incoming messages that stores the user's email mapped to their Teams userId (I believe both are found in Activity.ChannelData).
Note: #1 and #2 both require a user to message the bot, first, which sort of defeats the purpose of proactive messages
After you have the appropriate Teams IDs, you just send a proactive message to a Teams user. The end of this link also mentions trustServiceUrl, which you may find handy if you run into permissions/auth issues when trying to send a proactive message.
I have a web app deployed in azure that opens in Messenger Webview. I made a empty text prompt for user to click for the user to be able to continue when they close the Webview. However, users sometimes forget to click the button. I read this doc but i can't manage to do it as i learn best seeing actual examples and codes. How can the dialog automatically continue when the user closes or press a button in the Webview? Thank you.
The web view is a set of questions and the answer of the users are save in Cosmos DB and when they close the web view the bot access their scores in Cosmos DB and calculate their scores.
This is my current code.
private static async Task<DialogTurnResult> ThirdStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken = default(CancellationToken))
{
Activity reply = stepContext.Context.Activity.CreateReply();
reply.ChannelData = JObject.FromObject(
new
{
attachment = new
{
type = "template",
payload = new
{
template_type = "generic",
elements = new[]
{
new
{
title = "<title>",
buttons = new object[]
{
new
{
type = "web_url",
title = "<title>",
url = webAppUrl,
messenger_extensions="true",
webview_height_ratio = "tall",
},
new
{
type = "postback",
title = "Done ✔️",
payload = "Done ✔️",
},
},
},
},
},
},
});
await stepContext.Context.SendActivityAsync(reply);
return await stepContext.PromptAsync(
nameof(TextPrompt),
new PromptOptions
{
Prompt = MessageFactory.Text(string.Empty),
RetryPrompt = MessageFactory.Text("Click Done to proceed."),
});
}
private static async Task<DialogTurnResult> FourthStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
var result = stepContext.Result.ToString().ToLower();
if (result == "done ✔️" || result == "done")
{
return await stepContext.NextAsync();
}
await stepContext.Context.SendActivityAsync(
MessageFactory.Text(
$"Please press done to proceed, thank you!"));
return await stepContext.ReplaceDialogAsync(nameof(CalendarIncomeExpensesDialogV2));
}
update rafa:
Since you don't even know what kind of web app you're trying to create or what language you're writing the web app in or how you plan on hosting the web app, your first step is to figure all that out. If you need help with any part of that process then you'll need to ask a new very specific question about it and include what you've tried so far along with all the relevant code.
When you open the web app in your Facebook webview, you'll need to make sure the web app has all the information it needs to send an activity to the bot (I've been calling it a proactive message but those normally refer to bot-to-user messages). The needed credentials should already be built into the web app, so the only thing you'll need to send to the web app's endpoint is the user ID so that the web app can pass it along in the activity and the bot can identify which conversation the activity pertains to. The conventional way to send an activity to the bot is using Direct Line, though you may figure out a way to do it by just sending an HTTP request to the bot's endpoint or even using your Facebook app's callback URL.
There is absolutely no need to involve LUIS in this. LUIS should only be used to interpret messages from the user when you don't know what the user will say. Any time you're in control of the message that's getting sent to the bot, it doesn't make any sense to use LUIS. Keep in mind that there are 15+ activity types and your activity doesn't need to be a "message" activity. There are many ways you can identify the activity and respond accordingly in your bot. I recommend using an event activity.
Based on my understanding of your proficiency level, you will likely need to do a lot of research to accomplish what you're trying to accomplish. The documentation is a great place to start: https://learn.microsoft.com/en-us/azure/bot-service/
I am in the process of building a chatbot which is more conversational using the Microsoft Bot Framework v4.
What form of structure can I implement in the architecture to achieve a bot that can handle complex scenarios as described below
User: I'd like to add a Mac to my renters insurance
{detects intent}
Bot: Sure, which laptop do you want to get.
User: A used Pro from 2018
{extracts "pro" and "2018" from user utterance}
Bot:Alright, It will cost $5 per month to add your MacBook Pro 2018. Does that work for you?
User: Oh Sorry, It is actually a 2017
{Bot detect intent and extracts entities}
Bot:No worries, it would still be $5 per month to add your MacBook Pro 2017, Works for you?
User: By the way what is the weather in New York?
{Bot detect interruption}
Bot: The weather in New York is 28 Degrees Celsius
Bot:Do you still want to add your MacBook to your rentals?
User:Sounds great
Bot: Perfect, Your MacBook is not insured.
Bot: By the way, Happy Birthday!!.
I understand that bot framework can be integrated with LUIS to detect user intents from utterance however how can I place Luis in the chatbot project for it to handle all user utterances as the dialog flows.
So in scenarios as described above, LUIS can help continuing extracting entities from user utterances.
The core bot here shows both how to handle LUIS and interruptions.
Basically, instead of adding LUIS in as middleware, you're going to add it in as a "helper" class:
The LUIS helper class will keep track of the intents:
public static class LuisHelper
{
public static async Task<BookingDetails> ExecuteLuisQuery(IConfiguration configuration, ILogger logger, ITurnContext turnContext, CancellationToken cancellationToken)
{
var bookingDetails = new BookingDetails();
try
{
// Create the LUIS settings from configuration.
var luisApplication = new LuisApplication(
configuration["LuisAppId"],
configuration["LuisAPIKey"],
"https://" + configuration["LuisAPIHostName"]
);
var recognizer = new LuisRecognizer(luisApplication);
// The actual call to LUIS
var recognizerResult = await recognizer.RecognizeAsync(turnContext, cancellationToken);
var (intent, score) = recognizerResult.GetTopScoringIntent();
if (intent == "Book_flight")
{
// We need to get the result from the LUIS JSON which at every level returns an array.
bookingDetails.Destination = recognizerResult.Entities["To"]?.FirstOrDefault()?["Airport"]?.FirstOrDefault()?.FirstOrDefault()?.ToString();
bookingDetails.Origin = recognizerResult.Entities["From"]?.FirstOrDefault()?["Airport"]?.FirstOrDefault()?.FirstOrDefault()?.ToString();
// This value will be a TIMEX. And we are only interested in a Date so grab the first result and drop the Time part.
// TIMEX is a format that represents DateTime expressions that include some ambiguity. e.g. missing a Year.
bookingDetails.TravelDate = recognizerResult.Entities["datetime"]?.FirstOrDefault()?["timex"]?.FirstOrDefault()?.ToString().Split('T')[0];
}
}
catch (Exception e)
{
logger.LogWarning($"LUIS Exception: {e.Message} Check your LUIS configuration.");
}
return bookingDetails;
}
}
In your main dialog, you're going to call that LUIS helper like this:
private async Task<DialogTurnResult> ActStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
// Call LUIS and gather any potential booking details. (Note the TurnContext has the response to the prompt.)
var bookingDetails = stepContext.Result != null
?
await LuisHelper.ExecuteLuisQuery(Configuration, Logger, stepContext.Context, cancellationToken)
:
new BookingDetails();
// In this sample we only have a single Intent we are concerned with. However, typically a scenario
// will have multiple different Intents each corresponding to starting a different child Dialog.
// Run the BookingDialog giving it whatever details we have from the LUIS call, it will fill out the remainder.
return await stepContext.BeginDialogAsync(nameof(BookingDialog), bookingDetails, cancellationToken);
}
As for interruptions, the simple way to handle those are by creating them as a component dialog, and then extending that dialog on all the other dialogs:
I recommend giving the Core Bot a look over, as it gives a basic outline of what you're looking for. For a more complicated example, the Bot Framework also has the Virtual Assistant, here.
The Scenario
I have a bot built using the Bot Framework with a series of dialogs. One of these dialogs gives the user the option of inputting some complex data via a web page by presenting a button to them. Clicking the button they are then taken to the site, fill out the data, save and are then directed back to the bot.
I want my bot to pause the dialog until it receives an event from my web page telling me the user has saved the data and then continue asking the user questions.
Before
I had a version implemented whereby I would store a ConversationReference before the user clicked the button and then when the external event happened I would send the cards and next messages I wanted to show (not in a dialog) from a webhook, that was fine but it got quite complicated/messy - I'd rather keep the whole app in one continuous dialog.
Idea 1: Use DirectLine API
I did some research and many people were suggesting using the DirectLine API. So I implemented this:
public async Task SendEventAsync(InternalEventMessage message, ConversationReference reference) {
var client = new DirectLineClient(!String.IsNullOrEmpty(_settings.DirectLineSecret) ? _settings.DirectLineSecret : null);
if (_settings.SiteUrl.Contains("localhost")) {
client.BaseUri = new Uri(_settings.DirectLineServiceUrl);
}
var eventMessage = Activity.CreateEventActivity();
//Wrong way round?!?
eventMessage.From = reference.Bot;
eventMessage.Type = ActivityTypes.Event;
eventMessage.Value = message;
var conversation = await client.Conversations.PostActivityAsync(reference.Conversation.Id, eventMessage as Activity);
}
This uses the DirectLine client to send an event message to the serviceUrl using a stored ConversationReference, basically imitating a user (bot and user seem to be the wrong way round in the SDK). Checking for localhost was so that the DirectLine library pointed at the emulator server rather than https://directline.botframework.com.
In my dialog I call:
//method above shows input button and links to web page
context.Wait(WaitForAddressInput);
}
private async Task WaitForAddressInput(IDialogContext context, IAwaitable<IActivity> result) {
var message = await result;
switch (message.Type) {
case ActivityTypes.Message:
//TODO: Add response
break;
case ActivityTypes.Event:
var eventMessage = message as IEventActivity;
if (((JObject)eventMessage.Value).ToObject<InternalEventMessage>().Type == EventType.AddressInputComplete) {
_addressResult = (await _tableService.ReadOrderById(Order.OrderId)).Address;
await context.PostAsync($"Great}");
context.Done(_addressResult);
}
break;
}
}
This waits for any message from the user after the button has been shown and if our event matches then we proceed with the dialog.
This works locally using the emulator but, frustratingly, doesn't live. It fails to recognise channels created via webchat or Messenger. That is explained here: Microsoft Bot Framework DirectLine Can't Access Conversations
For security reasons, you can't use DirectLine to spy on messages from
another conversation.
So I can't access a channel that I haven't created using DirectLine.
Idea 2: BotConnector
So I thought I'd try the BotConnector using similar code:
public async Task SendEventAsync(InternalEventMessage message, Microsoft.Bot.Connector.DirectLine.ConversationReference reference) {
var botAccount = new ChannelAccount(reference.User.Id, reference.User.Name);
var userAccount = new ChannelAccount(reference.Bot.Id, reference.Bot.Name);
MicrosoftAppCredentials.TrustServiceUrl(reference.ServiceUrl);
var connector = new ConnectorClient(new Uri(reference.ServiceUrl), new MicrosoftAppCredentials("xxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxx"));
connector.Credentials.InitializeServiceClient();
var eventMessage = Activity.CreateMessageActivity();
eventMessage.Recipient = botAccount;
eventMessage.From = userAccount;
eventMessage.Type = ActivityTypes.Event;
eventMessage.Conversation = new ConversationAccount(id: reference.Conversation.Id);
eventMessage.ServiceUrl = reference.ServiceUrl;
eventMessage.Timestamp = DateTimeOffset.UtcNow;
eventMessage.LocalTimestamp = DateTime.Now;
eventMessage.ChannelId = reference.ChannelId;
var result = await connector.Conversations.SendToConversationAsync(eventMessage as Microsoft.Bot.Connector.Activity);
}
This doesn't crash and I can see the event appear in the emulator request console but nothing happens, it seems to be ignored!
Idea 3: Try to imitate the bot service calling my bot
I haven't tried this yet because I think it might be the most time consuming but I was reading here about the service authentication and wondered if it would be possible to imitate the hosted bot service sending a message and send my event that way with the required data?
This seems like a fairly common scenario so I'm surprised I haven't come across a way to do this yet. If anyone has any other ideas on how I can send an event message to my bot from an external service then I'd love to hear it.
Update:
See my answer below Eric's to see what I did.
Idea 1:
DirectLine is a channel, not a library to use in order to connect to channels. (For instance: you would not use Facebook Messenger to connect to Skype) DirectLineClient is useful for creating a client application that connects to the DirectLine channel through the Direct Line connector service.
Idea 2:
This method should work. In fact, the BotAuth library uses this method for the MagicNumber login flow within the CallbackController: https://github.com/MicrosoftDX/botauth/blob/9a0a9f1b665f4aa95b6d60d09346dda90d8b314e/CSharp/BotAuth/Controllers/CallbackController.cs
For your scenario, you should be able to construct a CardAction of type ActionTypes.OpenUrl that contains a value with the ConversationReference encoded in the url. Clicking the button will call an mvc controller that displays a page (saving the ConversationReference in a cookie or something) and when the user finishes adding the address on the page, use the ConversationReference to send an event to the bot (similar to how BotAuth resumes the conversation in the CallbackController).
Idea 3:
This would bypass the connector services, and is not a supported scenario. The link you shared explains the details of how authentication works in the Bot Framework, not how to bypass the connector services.
Eric's answer led me to solve the issue using the BotAuth example but, for completeness, here is what I did using Idea 2.
I created a CallbackController on my Bot Framework endpoint and then used the following code to send an event back to the awaiting dialog:
MicrosoftAppCredentials.TrustServiceUrl(reference.ServiceUrl);
var message = reference.GetPostToBotMessage();
message.Value = new InternalEventMessage(type);
message.Type = ActivityTypes.Event;
await Conversation.ResumeAsync(reference, message);
The dialog awaits with this code and continues:
context.Wait(WaitForAddressInput);
}
private async Task WaitForAddressInput(IDialogContext context,
IAwaitable<IActivity> result)
{
var message = await result;
switch (message.Type)
{
case ActivityTypes.Message:
//TODO: Add response
break;
case ActivityTypes.Event:
//Process event and continue!
break;
}
}
This is the most complicated issue I've had with the Bot Framework and I found the docs a little lacking. Hope this helps someone!