I have implemented dequeuing of messages from Oracle AQ and exposing it to the system as IObservable. The workflow is as follows:-
Application receives a call back event from Oracle about a new message.
Application de-queues the message and add it to IObservable (Message is dequeued as part of a transaction which is committed as soon as the message is dequeued).
I realized a potential issue and that is when the message is dequeued, the transaction is committed straight away rather than waiting for it to be consumed successfully by the application. Below is the code which I am using but need a suggestion where/how to commit the transaction after it has been consumed successfully by the application. Currently it starts and commits/rollbacks the transaction in the private Dequeue method.
public sealed class Queue<T> : IQueue<T> where T : IQueueDataType
{
private readonly OracleConnection _connection;
private readonly string _consumerName;
private readonly IQueueSetting _queueSetting;
private readonly IDbConnectionFactory _dbConnectionFactory;
private OracleAQQueue _queue;
private IObservable<T> _messages;
private bool _isDisposed;
public Queue(IDbConnectionFactory dbConnectionFactory, IDalSettings dalSettings, IQueueSetting queueSetting)
{
_dbConnectionFactory = dbConnectionFactory;
_connection = dbConnectionFactory.Create() as OracleConnection;
_consumerName = dalSettings.Consumer;
_queueSetting = queueSetting;
}
public void Connect()
{
_connection.Open();
_queue = new OracleAQQueue(_queueSetting.QueueName, _connection)
{
DequeueOptions = { Wait = 10, Visibility = OracleAQVisibilityMode.Immediate , ConsumerName = _consumerName, NavigationMode = OracleAQNavigationMode.FirstMessage, DequeueMode = OracleAQDequeueMode.Remove},
UdtTypeName = _queueSetting.QueueDataTypeName,
MessageType = OracleAQMessageType.Udt
};
_queue.NotificationConsumers = new[] { _consumerName };
_messages = Observable
.FromEventPattern<OracleAQMessageAvailableEventHandler, OracleAQMessageAvailableEventArgs>(
h => _queue.MessageAvailable += h, h => _queue.MessageAvailable -= h)
.Where(x => x.EventArgs.AvailableMessages > 0)
.Select(x =>
{
try
{
Log.Info("Msg received", "Queue", _queueSetting.QueueName);
OracleAQMessage msg = Dequeue();
Log.Info("Msg received id " + msg.MessageId, "Queue", _queueSetting.QueueName);
return (T)msg.Payload;
}
catch (Exception e)
{
}
}).Publish().RefCount();
}
private OracleAQMessage Dequeue()
{
using (var connection = _dbConnectionFactory.Create() as OracleConnection)
{
try
{
connection.Open();
using (OracleTransaction transaction = connection.BeginTransaction())
{
try
{
OracleAQMessage msg = _queue.Dequeue();
**transaction.Commit();**
return msg;
}
catch (Exception e)
{
**transaction.Rollback();**
throw;
}
}
}
catch (Exception e)
{
Log.Error(string.Format("Error occurred while connecting to database to dequeue new message. Error : {0}", e),
"Dequeue", GetType().FullName);
throw;
}
finally
{
connection.Close();
}
}
}
public IObservable<T> GetMessages()
{
return _messages;
}
public void Dispose()
{
if (!_isDisposed)
{
if (_queue != null)
{
_queue.Dispose();
}
_connection.Dispose();
_isDisposed = true;
}
}
}
If rather than using IObservable, I just expose an event committing and rolling back transaction would be very easy but I like what I can do with IObservable i.e I can run a Linq but don't know how to commit the transaction.
I don't think there's an easy fix here. If I understand this correctly:
An event is pushed from Oracle,
You want to expose the event via an IObservable stream,
The application 'handles' it,
You want to commit if handled successfully, roll-back if not.
The problem is that IObservable is a one-way mechanism. Once you publish a message (in our case that you got something from this Oracle queue), the intention isn't to track it, and decide later whether to commit/rollback. So your options are pretty much to stuff your app-logic into a handler of some form:
Func<OracleMessage, bool> isMessageCommitable; //...application handling logic here
var appHandledMessages = oracleSourceMessages
.Select(m => Tuple.Create(m, isMessageCommitable(m)))
.Publish()
.RefCount();
appHandledMessages
.Where(t => t.Item2)
.Subscribe(t => Commit(t.Item1));
appHandledMessages
.Where(t => !t.Item2)
.Subscribe(t => Rollback(t.Item1));
...or to set up IObservables pointing the other way which would push from the application back to the queue which messages should be committed/rolled back. You would probably want two, one for committing, one for rolling back, and those should probably be passed into the constructor of Queue<T>.
Good luck.
Related
I've recently discovered the System.IO.Pipelines namespace and it looks interesting. I've been trying to implement the IDuplexPipe interface in the context of a simple TCP server which accepts connections and then communicates back and forth with the connected client.
However, I'm struggling to make it stable. It feels like I've misunderstood something fundamental. I've also been googling for reference implementations of the interface to guide me in the right direction.
This is to my knowledge the most complete document on System.IO.Pipelines out there. The exmple I've provided below is heavily borrowing from it.
https://github.com/davidfowl/DocsStaging/blob/master/Pipelines.md
https://devblogs.microsoft.com/dotnet/system-io-pipelines-high-performance-io-in-net/
My question: what would a typical implementation of the IDuplexPipe interface look like in the context of a TCP server?
Btw, this is what I have currently. The idea is to setup a new "duplex communication" by providing an established SslStream:
public class DuplexCommunication : IDuplexPipe
{
public PipeReader Input => _receivePipe.Reader;
public PipeWriter Output => _transmitPipe.Writer;
private readonly SslStream _stream;
// Data received from the SslStream will end up on this pipe
private readonly Pipe _receivePipe = new Pipe();
// Data that is to be transmitted over the SslStream ends up on this pipe
private readonly Pipe _transmitPipe = new Pipe();
private readonly CancellationToken _cts;
private Task _receive;
private Task _transmit;
public DuplexCommunication(SslStream stream, CancellationToken cts)
{
_stream = stream;
_cts = cts;
_receive = Receive();
_transmit = Transmit();
}
private async Task Receive()
{
Exception error = null;
try
{
while (!_cts.IsCancellationRequested)
{
var buffer = _receivePipe.Writer.GetMemory(1);
var bytes = await _stream.ReadAsync(buffer, _cts);
_receivePipe.Writer.Advance(bytes);
if (bytes == 0) {
break;
}
var flush = await _receivePipe.Writer.FlushAsync(_cts);
if (flush.IsCompleted || flush.IsCanceled)
{
break;
}
}
}
catch (Exception ex)
{
// This might be "stream is closed" or similar, from when trying to read from the stream
Console.WriteLine($"DuplexPipe ReceiveTask caugth an exception: {ex.Message}");
error = ex;
}
finally
{
await _receivePipe.Writer.CompleteAsync(error);
}
}
private async Task Transmit()
{
Exception error = null;
try
{
while (!_cts.IsCancellationRequested)
{
var read = await _transmitPipe.Reader.ReadAsync(_cts);
var buffer = read.Buffer;
if (buffer.IsEmpty && read.IsCompleted)
{
break;
}
foreach (var segment in buffer)
{
await _stream.WriteAsync(segment, _cts);
}
_transmitPipe.Reader.AdvanceTo(buffer.End);
await _stream.FlushAsync(_cts);
}
}
catch (Exception e)
{
Console.WriteLine($"DuplexPipe Transmit caught an exception: {e.Message}");
error = e;
}
finally
{
await _transmitPipe.Reader.CompleteAsync(error);
}
}
}
So, I spent some more time searching around the internet and found something that sort of solves my problem. I found, among some other things, this question: How to handle incoming TCP messages with a Kestrel ConnectionHandler?
Here it seems like the ConnectionHandler class provides exactly what I need, including some very handy plumbing for handling SSL certificates, port listening, etc that comes for free when building an ASP.NET application.
I'm working on a Chat server which receives connections from multiple clients and sends/receives messages.
This is how it gets connections from the clients:
public void StartServer()
{
tcpListener = new TcpListener(IPAddress.Any, 60000);
tcpListener.Start();
listenTask = Task.Factory.StartNew(() => ListenLoop());
}
private async void ListenLoop()
{
int i = 0;
for (; ; )
{
var socket = await _tcpListener.AcceptSocketAsync();
if (socket == null)
break;
var c = new Client(socket, i);
i++;
}
}////Got this code from somewhere here, not really what I want to use (discussed down)
This is the Client class:
public class Client
{
//irrelevant stuff here//
public Client(Socket socket, int number)
{
//irrelevant stuff here//
Thread ct = new Thread(this.run);
ct.Start();
}
public void run()
{
writer.Write("connected"); //Testing connection
while (true)
{
try
{
string read = reader.ReadString();
// Dispatcher.Invoke(new DisplayDelegate(DisplayMessage), new object[] { "[Client] : " + read });
}////////Not working, because Client needs to inherit from MainWindow.
catch (Exception z)
{
MessageBox.Show(z.Message);
}
}
}
}
Ok so problem is, to update the UI Client class must inherit from MainWindow, but when it does, I get "the calling thread must be sta because many UI components require this" error. When it doesn't inherit it works just fine.
Another problem is, I want to use a Client[] clients array and then when a user connects, it adds him to the array so that i can individually write/read to/from specific clients.
while (true)
{
try
{
clients[counter] = new Client(listener.AcceptSocket(), counter);
counter ++;
MessageBox.Show("client " + counter.ToString());
}
catch (Exception e) { MessageBox.Show(e.Message); }
}
Problem here is, i get "Object refrence not set to an instance of an object" when a client connects.
Any ideas how to fix both/any of these problems?
Sorry code might be a bit messed up but I tried lots of stuff to get it working so I ended up with lots of junk in the code.
Thanks in advance.
i have a console applicaiton . in this application i make contact with a queue. first i check to see if for example queue with name 'exampleQueue' exists or not. and if it does not exist i create it.
after creating and returing the path or just returning the path . i want to attach the ReceiveCompleted event to this queue.
i have two approach i can use 'using' keyword to make the queue dispoed after my work or i can just use normal way to create queue object .
in below u can see my code :
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
CreateQueue(".\\exampleQueue", false);
using (var queue = new MessageQueue(".\\exampleQueue"))
{
queue.ReceiveCompleted += MyReceiveCompleted;
queue.BeginReceive();
}
}
public static void CreateQueue(string queuePath, bool transactional)
{
if (!MessageQueue.Exists(queuePath))
{
MessageQueue.Create(queuePath, transactional);
}
else
{
Console.WriteLine(queuePath + " already exists.");
}
}
private static void MyReceiveCompleted(Object source,
ReceiveCompletedEventArgs asyncResult)
{
var queue = (MessageQueue)source;
try
{
var msg = queue.EndReceive(asyncResult.AsyncResult);
Console.WriteLine("Message body: {0}", (string)msg.Body);
queue.BeginReceive();
}
catch (Exception ex)
{
var s = ex;
}
finally
{
queue.BeginReceive();
}
return;
}
}
}
the problem that i face is that whenever i use this code for create queue object
using (var queue = new MessageQueue(".\\exampleQueue"))
{
queue.ReceiveCompleted += MyReceiveCompleted;
queue.BeginReceive();
}
the MyReceiveCompleted event does not work properly .
but when i use this
var queue = new MessageQueue(".\\exampleQueue");
queue.ReceiveCompleted += MyReceiveCompleted;
queue.BeginReceive();
every thing just work in proper way.
my question is that which approcah is the best one ?
and if i choose to use the first approcah , how can i make it work ?
accept my apologize for bad typing .
You can use your original approach, but you have to make sure, that the Main method is blocking inside the using statement:
static void Main(string[] args)
{
CreateQueue(".\\exampleQueue", false);
using (var queue = new MessageQueue(".\\exampleQueue"))
{
queue.ReceiveCompleted += MyReceiveCompleted;
queue.BeginReceive();
// here you have to insert code to block the
// execution of the Main method.
// Something like:
while(Console.ReadLine() != "exit")
{
// do nothing
}
}
}
.Net application which will subscribe to streaming notification. This application executes well, this subscription disconnected after 30 mints so I added the code to reconnect the connection .I tested the application by adding break point at the reconnect line and waited for some time before it established the connection again. During that time I created a few new emails and watched if console displays those events. It did not as connection was disconnected then I run the code again and I was able to see the events which were created during the time connection disconnected and connected. I need to know if this is the right way to make this process run continuously and track the events which occurred during the disconnect and reconnect of the application. also need to know why the below code is not notifying about delete mail event. kindly looking for help.
namespace NewMailNotification
{
class Program
{
static void Main(string[] args)
{
ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2010_SP2);
//***********New**********************
ExchangeService mailbox = new ExchangeService(ExchangeVersion.Exchange2010_SP2);
string mailboxEmail = ConfigurationSettings.AppSettings["user-id"];
WebCredentials wbcred = new WebCredentials(ConfigurationSettings.AppSettings["user"], ConfigurationSettings.AppSettings["PWD"]);
mailbox.Credentials = wbcred;
// mailbox.ImpersonatedUserId = new ImpersonatedUserId(ConnectingIdType.SmtpAddress, mailboxEmail);
mailbox.AutodiscoverUrl(mailboxEmail, RedirectionUrlValidationCallback);
mailbox.HttpHeaders.Add("X-AnchorMailBox", mailboxEmail);
FolderId mb1Inbox = new FolderId(WellKnownFolderName.Inbox, mailboxEmail);
SetStreamingNotification(mailbox, mb1Inbox);
bool run = true;
bool reconnect = false;
while (run)
{
System.Threading.Thread.Sleep(100);
}
}
internal static bool RedirectionUrlValidationCallback(string redirectionUrl)
{
//The default for the validation callback is to reject the URL
bool result=false;
Uri redirectionUri=new Uri(redirectionUrl);
if(redirectionUri.Scheme=="https")
{
result=true;
}
return result;
}
static void SetStreamingNotification(ExchangeService service,FolderId fldId)
{
StreamingSubscription streamingssubscription=service.SubscribeToStreamingNotifications(new FolderId[]{fldId},
EventType.NewMail,
EventType.Created,
EventType.Deleted);
StreamingSubscriptionConnection connection=new StreamingSubscriptionConnection(service,30);
connection.AddSubscription(streamingssubscription);
//Delagate event handlers
connection.OnNotificationEvent+=new StreamingSubscriptionConnection.NotificationEventDelegate(OnEvent);
connection.OnDisconnect += new StreamingSubscriptionConnection.SubscriptionErrorDelegate(Connection_OnDisconnect);
connection.OnSubscriptionError+=new StreamingSubscriptionConnection.SubscriptionErrorDelegate(OnError);
connection.Open();
}
static private void Connection_OnDisconnect(object sender, SubscriptionErrorEventArgs args)
{
StreamingSubscriptionConnection connection = (StreamingSubscriptionConnection)sender;
if (!connection.IsOpen)
{
// Console.WriteLine("no connection");
connection.Open();
}
}
static void OnEvent(object sender,NotificationEventArgs args)
{
StreamingSubscription subscription=args.Subscription;
if(subscription.Service.HttpHeaders.ContainsKey("X-AnchorMailBox"))
{
Console.WriteLine("event for nailbox"+subscription.Service.HttpHeaders["X-AnchorMailBox"]);
}
//loop through all the item-related events.
foreach(NotificationEvent notification in args.Events)
{
switch(notification.EventType)
{
case EventType.NewMail:
Console.WriteLine("\n----------------Mail Received-----");
break;
case EventType.Created:
Console.WriteLine("\n-------------Item or Folder deleted-------");
break;
case EventType.Deleted:
Console.WriteLine("\n------------Item or folder deleted---------");
break;
}
//Display notification identifier
if(notification is ItemEvent)
{
//The NotificationEvent for an email message is an ItemEvent
ItemEvent itemEvent=(ItemEvent)notification;
Console.WriteLine("\nItemId:"+ itemEvent.ItemId.UniqueId);
Item NewItem=Item.Bind(subscription.Service,itemEvent.ItemId);
if(NewItem is EmailMessage)
{
Console.WriteLine(NewItem.Subject);
}
}
else
{
//the Notification for a Folder is an FolderEvent
FolderEvent folderEvent=(FolderEvent)notification;
Console.WriteLine("\nFolderId:"+folderEvent.FolderId.UniqueId);
}
}
}
static void OnError(object sender,SubscriptionErrorEventArgs args)
{
//Handle error conditions.
Exception e=args.Exception;
Console.WriteLine("\n-----Error-----"+e.Message+"--------");
}
}
}
Here is an example :
_BackroundSyncThread = new Thread(streamNotification.SynchronizeChangesPeriodically); _BackroundSyncThread.Start();
private void SynchronizeChangesPeriodically()
{
while (true)
{
try
{
// Get all changes from the server and process them according to the business
// rules.
SynchronizeChanges(new FolderId(WellKnownFolderName.Calendar));
}
catch (Exception ex)
{
Console.WriteLine("Failed to synchronize items. Error: {0}", ex);
}
// Since the SyncFolderItems operation is a
// rather expensive operation, only do this every 10 minutes
Thread.Sleep(TimeSpan.FromMinutes(10));
}
}
public void SynchronizeChanges(FolderId folderId)
{
bool moreChangesAvailable;
do
{
Debug.WriteLine("Synchronizing changes...");
// Get all changes since the last call. The synchronization cookie is stored in the _SynchronizationState field.
// Only the the ids are requested. Additional properties should be fetched via GetItem calls.
var changes = _ExchangeService.SyncFolderItems(folderId, PropertySet.FirstClassProperties, null, 512,
SyncFolderItemsScope.NormalItems, _SynchronizationState);
// Update the synchronization cookie
_SynchronizationState = changes.SyncState;
// Process all changes
foreach (var itemChange in changes)
{
// This example just prints the ChangeType and ItemId to the console
// LOB application would apply business rules to each item.
Console.WriteLine("ChangeType = {0}", itemChange.ChangeType);
Console.WriteLine("Subject = {0}", itemChange.Item.Subject);
}
// If more changes are available, issue additional SyncFolderItems requests.
moreChangesAvailable = changes.MoreChangesAvailable;
} while (moreChangesAvailable);
}
The _SynchronizationState field is like a Cookie that contains some informations about your last sync process . So next time the thread will synchronize all the items since the last sync .
Hope it helps
I implemented my own code of Publish/subscribe pattern implementation with WCF with WSDualHttpBinding, but i've a little problem with timeouts that i explain later, for now let me show what i'm doing:
public interface IEventSubscriber
{
[OperationContract]
void NotifyEvent(EventNotification notification);
[OperationContract]
void NotifyServiceDisconnecting();
}
[ServiceContract(SessionMode = SessionMode.Required, CallbackContract = typeof(IEventSubscriber))]
public interface IEventPublisherService
{
[OperationContract(IsOneWay = false, IsInitiating = true)]
void Subscribe(string culture);
[OperationContract(IsOneWay = false, IsInitiating = false, IsTerminating = true)]
void Unsubscribe();
}
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]
internal class EventPublisherServiceImpl : IEventPublisherService
{
ServiceHost host;
public bool StartService()
{
bool ret = false;
try
{
Uri baseAddress = new Uri(ConfigurationManager.AppSettings[GlobalConstants.CfgKeyConfigEventPublishserServiceBaseAddress].ToString());
EventHelper.AddEvent(string.Format("Event Publisher Service on: {0}", baseAddress.ToString()));
host = new ServiceHost(this, baseAddress);
// duplex session enable http binding
WSDualHttpBinding httpBinding = new WSDualHttpBinding(WSDualHttpSecurityMode.None);
httpBinding.ReceiveTimeout = TimeSpan.FromMinutes(10);
httpBinding.ReliableSession = new ReliableSession();
httpBinding.ReliableSession.Ordered = true;
httpBinding.ReliableSession.InactivityTimeout = TimeSpan.FromMinutes(10);
host.AddServiceEndpoint(typeof(IEventPublisherService), httpBinding, baseAddress);
// Enable metadata publishing.
ServiceMetadataBehavior smb = new ServiceMetadataBehavior();
smb.HttpGetEnabled = true;
smb.MetadataExporter.PolicyVersion = PolicyVersion.Policy15;
host.Description.Behaviors.Add(smb);
// Open the ServiceHost to start listening for messages.
host.Open();
ret = true;
}
catch (Exception e)
{
EventHelper.AddException(e.Message);
}
return ret;
}
...
}
now in my implementation class i have a list of subscribers that are stored in memory, when a new notification arrived the following code is performed for each subscriber:
...
/// <summary>
/// List of active subscribers
/// </summary>
private static Dictionary<IEventSubscriber, string> subscribers = new Dictionary<IEventSubscriber, string>();
...
that i use it like this:
internal void Subscribe(string culture)
{
lock (subscribers)
{
// Get callback contract as specified on the service definition
IEventSubscriber callback = OperationContext.Current.GetCallbackChannel<IEventSubscriber>();
// Add subscriber to the list of active subscribers
if (!subscribers.ContainsKey(callback))
{
subscribers.Add(callback, culture);
}
}
}
...
private void OnNotificationEvent(NormalEvent notification)
{
lock (subscribers)
{
List<IEventSubscriber> listToRemove = new List<IEventSubscriber>();
// Method signature: Parallel.ForEach(IEnumerable<TSource> source, Action<TSource> body)
Parallel.ForEach(subscribers, kvp =>
{
try
{
kvp.Key.NotifyEvent(new EventNotification(notification, kvp.Value));
}
catch (Exception ex)
{
EventHelper.AddException(string.Format("Error notifying event notification to client: {0} - removing this one", ex.Message));
listToRemove.Add(kvp.Key);
}
} //close lambda expression
); //close method invocation
Parallel.ForEach(listToRemove, subs =>
{
try
{
subs.NotifyServiceDisconnecting();
}
catch (Exception ex) {
EventHelper.AddException(string.Format("Failed to notify client that is to be removed: {0}",
ex.Message));
}
subscribers.Remove(subs);
}
);
}
}
What is the problem with this, when timeouts are achieved (note that i set 10 minutes for ReceiveTimeout and inactive timeout) the subscribers that are in the list go to fault state, and the following exception is catched in the OnNotificationEvent
*The operation 'NotifyEvent' could not be completed because the sessionful channel timed out waiting to receive a message. To increase the timeout, either set the receiveTimeout property on the binding in your configuration file, or set the ReceiveTimeout property on the Binding directly. *
Ok i can increase the timeout value, but if i do this it will happen some time, eventually.
My question are: i'm doing something wrong when trying to implement this pattern? or exists any other way of implementing this pattern a better way that avoid this problem? or exist any way of reconnecting the faulted callback channel (for what i'm reading it's not possible, but due to that i can't notify the client that is connection was lost, letting the client blind not knowing that the communication ended!? or is a way to give knowledge that he lost communication with the publisher!?)
Of course solution like ping messages are out of date :) but ok, if nothing better appears look like i've to implement something like that...
Thanks
For now the solution was to change the timeouts to have a infinite value:
// duplex session enable http binding
WSDualHttpBinding httpBinding = new WSDualHttpBinding(WSDualHttpSecurityMode.None);
httpBinding.ReceiveTimeout = TimeSpan.MaxValue;
httpBinding.ReliableSession = new ReliableSession();
httpBinding.ReliableSession.Ordered = true;
httpBinding.ReliableSession.InactivityTimeout = TimeSpan.MaxValue;
You are using Parallel.ForEach but I'm not sure that this is enough. AFAIR Parallel.ForEach does not execute each iteration in the separate thread.
I would like to suggest to start separate threads for each subscriber in the OnNotificationEvent and use lock to make sure that foreach will not be breaked by Collection modified exception:
lock (_subscribersSync)
foreach (var chatter in subscribers)
{
Logger.Log.DebugFormat("putting all users to {0}", subscribers.Key.Name);
Thread th = new Thread(PublishAllUserMessage);
th.Start(new MessageData() { Message = "", Subscriber = chatter.Key};
}
void PublishAllUserMessage(object messageData)
{
MessageData md = (MessageData)messageData;
try
{
md.Subscriber.NotifyEvent(...event parameters here ...);
}
catch (Exception ex)
{
Logger.Log.Error(string.Format("failed to publish message to '{0}'", md.Subscriber.Name), ex);
KickOff(md.Subscriber);
}
}
object _subscribersSync = new object();
void KickOff(IEventSubscriber p)
{
lock (_subscribersSync)
{
subscribers.Remove(p);
Logger.Log.WarnFormat("'{0}' kicked off", p.Name);
}
}
public class MessageData
{
public string Message;
public IEventSubscriber Subscriber;
}