I am trying to move the events from event hub to the Blob storage. I have created trigger on the eventhub to trigger whenever a message comes to event hub. Also, i have configured outputs as Blob Storage. Right now, what i am doing within the function is:
public static void Run(string myEventHubMessage, out string outputBlob, TraceWriter log)
{
outputBlob = myEventHubMessage;
}
This will create a new blob in the container mentioned in the Ouputs configuration. But, I want is to create a blob with a specified name based on the data present in the eventhub message and need to set the content type and other metadata while saving the data to Azure Blob. Can someone help how to do this?
Regards,
John
You might no longer need to do that.
Event hub now supports piping to blob storage out of the box.
There are several possibilities to configure your output binding.
If you need to set the Blob path based on event hub message properties, you can declare your strongly typed message
public class MyEvent
{
public string SomeName { get; set; }
// more properties
}
then bind it declaratively to blob path, e.g.
{
"type": "blob",
"name": "outputBlob",
"path": "mycontainer/{SomeName}.json",
"connection": "...",
"direction": "out"
},
and modify the function accordingly
public static void Run(MyEvent myEventHubMessage, out MyEvent outputBlob)
{
outputBlob = myEventHubMessage;
}
If you need more advanced calculation to determine the output path, you can remove declarative output binding from function.json and use imperative binding:
public static async Task Run(string myEventHubMessage, Binder binder)
{
var path = ...;
using (var writer = binder.Bind<TextWriter>(new BlobAttribute(path)))
{
writer.Write(myEventHubMessage);
}
}
If you need to set more properties of Blob, bind to ICollector<CloudBlockBlob>
var collector = binder.Bind<ICollector<CloudBlockBlob>>(new BlobAttribute(path)));
collector.Add(new CloudBlockBlob { ... });
You should play with all those options to identify which scenario works for you.
Related
I'm having difficulty setting up an end-point to receive Google Play Developer Notifications via Pub/Sub in a c# controller. I've set everything up against the app to publish to a topic and I have successfully setup a subscription in Google Pub/Sub...and the test notification is coming through on Google Pub/Sub...the problem is I am having trouble receiving the push notification on my server side c# controller end-point...I'm not sure how to set that up and if I have the correct json signature. I can't find any examples or documentation around this. Any help would be appreciated!
This is my first "test" of Pub/Sub and this sample worked for me.
See all samples here: https://github.com/GoogleCloudPlatform/dotnet-docs-samples/tree/main/pubsub/api/Pubsub.Samples
These steps needs to be done:
Create a topic here: https://console.cloud.google.com/cloudpubsub/topic/ , in the example we call it "iap"
Under permission for "iap", add google-play-developer-notifications#system.gserviceaccount.com as Pub/Sub publisher. This will allow Google Play to publish on this topic.
Under subscriptions https://console.cloud.google.com/cloudpubsub/subscription add your service account/personal gmail or what ever that is linked to your c# server later on. I tested firebase-adminsdk#yourproject.iam.gserviceaccount.com and it worked fine. Check your environment variable "GOOGLE_APPLICATION_CREDENTIALS" and extract this user as Pub/Sub subscriber in permissions for "iap-sub".
Play store needs to be configured under "Monetization setup". String are for instance: projects/yourproject/topics/iap
Press a test message (you can also see it in the Cloud console)
Test message could look something like this:
20:16:07: Received message 6108129433484522 20:16:07:
{"version":"1.0","packageName":"com.yourproject","eventTimeMillis":"1666642564858","testNotification":{"version":"1.0"}}
Class below runs the client in the background without waiting.
If you just wanna try in a console, use the Console.ReadLine()
public class GCloudPubSub : IDisposable
{
public String projectId { get; set; }
public String subscriptionId { get; set; }
private SubscriberClient _client;
public FirebasePubSub() {
projectId = "yourproject";
subscriptionId = "iap-sub";
}
public async void Start()
{
SubscriptionName subscriptionName = SubscriptionName.FromProjectSubscription(projectId, subscriptionId);
_client = await SubscriberClient.CreateAsync(subscriptionName);
await _client.StartAsync(HandleMessage);
}
public async void Stop()
{
await _client.StopAsync(CancellationToken.None);
}
public void Dispose()
{
Stop();
}
static Task<SubscriberClient.Reply> HandleMessage(PubsubMessage message, CancellationToken token)
{
Log($"Received message {message.MessageId}");
string text = System.Text.Encoding.UTF8.GetString(message.Data.ToArray());
Log(text);
return Task.FromResult(SubscriberClient.Reply.Ack);
}
static void Log(string text) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss}: {text}");
}
Hopefully this will lead you on the right way :)
I have an Azure Function based on a QueueTrigger. This gets triggered when something appears on the queue, but after I processed this item, I then want to put new items back on the queue.
Is there a way to do this directly from within the Azure Function?
[Function("Batch")]
public async Task Run([QueueTrigger("batch", Connection = "DataQueue")] string data,
FunctionContext context)
{
var model = JsonConvert.DeserializeObject<MyObject>(data);
// 1. process model
// 2. Put items back on queue?
}
You can use output bindings as the following:
[StorageAccount("MyStorageConnectionAppSetting")]
public static class QueueFunctions
{
[FunctionName("QueueOutput")]
[return: Queue("myqueue-items")]
public static string QueueOutput([HttpTrigger] dynamic input, ILogger log)
{
log.LogInformation($"C# function processed: {input.Text}");
return input.Text;
}
}
Is it possible to get the queue name in a service bus triggered Azure Function?
I know it is possible if the QueueName or TopicName+SubscriberName are hard coded. In that case I can make a const string and use it inside the function. However my service bus trigger gets the name from the settings using "%ServiceBusSettings:QueueName%", "%ServiceBusSettings:TopicName%" and "%ServiceBusSettings:SubscriberName%".
How to get the Queue or Topic+Subscriber name in this configurable case?
[FunctionName("topic-and-subscriber-function")]
public async Task Run(
[ServiceBusTrigger("%ServiceBusSettings:TopicName%", "%ServiceBusSettings:SubscriptionName%", Connection = "ServiceBusSettings:ServiceBusConnection")] Message message,
ILogger log, MessageReceiver messageReceiver)
{
// Get TopicName and SubscriberName
}
[FunctionName("queue-function")]
public async Task Run(
[ServiceBusTrigger("%ServiceBusSettings:QueueName%", Connection = "ServiceBusSettings:ServiceBusConnection")] Message message,
ILogger log, MessageReceiver messageReceiver)
{
// Get QueueName
}
What you need to get is the name of the queue, which is a property of QueueClient, but Microsoft.Azure.ServiceBus.QueueClient itself cannot be serialized. The Microsoft.Azure.ServiceBus.Message can be serialized but does not have a queuename attribute. (And Microsoft.Azure.ServiceBus.Message has no parent class. This means that it is generally impossible to obtain queuename through this class.)
You can pass in queuename as part of the message, such as input json format message:
{
"queuename":"testname",
...
}
then you can get the queuename in code.(Also You can get this value in bindings by this way. All key values passed in in json format can be obtained in the binding.)
Assume the following typical Queue Trigger function:
public void Run([QueueTrigger("queue1")]object data, ILogger log)
{
// Do something with data
}
My problem is that "queue1" has to be a constant field, so it has to be defined at compile time.
Also, I'd want to have a base class for Queue Triggers, that could work like this:
public abstract class QueueBase<TModel>
{
public void Run([QueueTrigger("queueName")]TModel data, ILogger log)
{
// Do something with data, log something etc.
OnRunExecuted(data);
// Do something with data, log something etc.
}
public abstract void OnRunExecuted(TModel data);
}
with this, I could write own classes which inherit from QueueBase but can even live inside a library which doesn't have Microsoft.Azure.WebJobs dependency:
public class MyQueueHandler : QueueBase<MyModel>
{
public void OnRunExecuted(MyModel data) => ...;
}
But it's impossible to pass in a Queue name... is it?
See binding expressions:
In short, you can pass in a variable queue name as "%queue-name-variable%"
[FunctionName("QueueTrigger")]
public static void Run(
[QueueTrigger("%queue-name-variable%")]string myQueueItem,
ILogger log)
{
log.LogInformation($"C# Queue trigger function processed: {myQueueItem}");
}
Where input-queue-name is defined in your configuration like
{"queue-name-variable": "queue-name-in-current-env"}
As i remember attribute QueueTrigger accept only const string, so you can try make some tricks using environment variables like in post how to pass dynamic queue name
I have an Azure Function which is triggered whenever an image is uploaded to a container in a Azure storage account. I read the stream and do some actions.
I would also like to get the uri of the blob that is triggering the Function but how would I do this? Do I have to work with additional inputs / outputs?
public static void Run(Stream myBlob, TraceWriter log)
{
//get byte array of the stream
byte[] image = ReadStream(myBlob);
// ...
}
If you really want the full URI, as opposed to just the blob relative path (which string blobTrigger would give you), you'll need to do something like this:
public static void Run(CloudBlockBlob myBlob, TraceWriter log)
{
// ...
}
At that point, you can use the CloudBlockBlob object model both to get the URI (e.g. StorageUri or some other related props) and to get the stream (e.g. BeginDownloadToStream). Note that when you do this, you can no longer directly receive the Stream as an input param.
Based on the webjobs' documentation for blobs, you can simply add a string blobTrigger parameter:
public static void Run(Stream myBlob, string blobTrigger, TraceWriter log)
In your binding, you can define a variable name for blob path:
"path": "foo/{name}.bar",
Then add name as another function parameter:
public static void Run(Stream myBlob, string name, TraceWriter log)