We Keep Coding

Using "Queue" in C# for sending SMS

I am trying to send SMS to clients from transactional data. It works fine but sometimes the staging table gets tanked due to huge data load. I was thinking if there is any way to queue data in a First-In-First-Out (FIFO) order. I am still learning C#. Please help me to implement the following code in a smart way.
I am sharing what I did.
private static void Main(string[] args)
{
var datetime = DateTime.Now;
var hr = datetime.Hour;
var mm = datetime.Minute + 1;
var ss = double.Parse(ConfigurationManager.AppSettings["SMSTIME"]);
// For Interval in Seconds
// This Scheduler will start at current time + 1 minute and call after every 3 Seconds
// IntervalInSeconds(start_hour, start_minute, seconds)
SchedulerTasks.IntervalInSeconds(hr, mm, ss,
() =>
{
CreateNewSms.Instance.PostNewSmsText();
});
Console.ReadLine();
}
public async void PostNewSmsText()
{
var smsDt = await Task.Run(() => DataAccessLayer.Instance.InsertSmsDataToLocalDb());
if (smsDt.Rows.Count != 0)
{
var smsData = GetSmsData(smsDt);
var xmlRespDoc = new XmlDocument();
await Task.Run(() =>
{
foreach (var sms in smsData)
{
var smsText = GetSmsText(sms.Channel, sms.AccountNumber, sms.Amount, sms.DrCr, sms.CurrentBalance, sms.BranchName,
sms.CurrencyName, sms.TrnDescription, sms.UbsTrnDateTime);
var smsSubmitTime = DateTime.Now.ToString(CultureInfo.InvariantCulture);
var response = ExecuteSmsService.Instance.PostSmsText(sms.MobileNo, smsText, sms.TrnRefNo);
var smsDeliveryTime = DateTime.Now.ToString(CultureInfo.InvariantCulture);
var operatorInfo = ConfigurationManager.AppSettings["Operator"];
xmlRespDoc.LoadXml(response);
var smsCsmsId = xmlRespDoc.SelectSingleNode("//SMSINFO/CSMSID")?.InnerText;
var smsRefNo = xmlRespDoc.SelectSingleNode("//SMSINFO/REFERENCEID")?.InnerText;
var smsDeliveryStatus = xmlRespDoc.SelectSingleNode("//SMSINFO/MSISDNSTATUS")?.InnerText;
if (smsRefNo != null) //Save response data if sms delivery successful
{
DataAccessLayer.Instance.InsertSmsResponseDetails(
sms.EntrySerialNo,
smsCsmsId,
smsRefNo,
smsText,
smsSubmitTime,
smsDeliveryTime,
true,
"Success",
operatorInfo);
DataAccessLayer.Instance.UpdateSmsStatus(sms.EntrySerialNo);
DataAccessLayer.Instance.DeleteSmsData(sms.EntrySerialNo, true);
Console.WriteLine("SMS successfully sent to: " + sms.MobileNo);
}
else
{
//If any sms delivery failed here we set a counter to retry again in next date
DataAccessLayer.Instance.SetSmsCounter(sms.EntrySerialNo);
//Save response data if sms delivery unsuccessful with failed status
DataAccessLayer.Instance.InsertSmsResponseDetails(
sms.EntrySerialNo,
smsCsmsId,
null,
smsText,
smsSubmitTime,
smsDeliveryTime,
false,
smsDeliveryStatus,
operatorInfo);
DataAccessLayer.Instance.DeleteSmsData(sms.EntrySerialNo, false);
Console.WriteLine("Invalid mobile no: " + sms.MobileNo);
}
}
});
}
else
{
Console.WriteLine("Waiting for transactions.....");
}
}
I was searching on Queue Implementations and found this but confused how to use it here in my code.
public class SendSmsQueue
{
private ConcurrentQueue<object> _jobs = new ConcurrentQueue<object>();
public SendSmsQueue()
{
var thread = new Thread(new ThreadStart(OnStart));
thread.IsBackground = true;
thread.Start();
}
public void Enqueue(object job)
{
_jobs.Enqueue(job);
}
private void OnStart()
{
while (true)
{
if (_jobs.TryDequeue(out object result))
{
Console.WriteLine(result);
}
}
}
}

ZeroMQ C# Ironhouse example

I am fairly new to ZeroMQ and have been comparing security of messages using the ZeroMQ NuGet package and the NetMQ & NetMQ Security NuGet packages.
So far, I have not been able to find a C# version of the Ironhouse example using Curve Security. There is a "todo" item on the ZGuides repo but so far nothing implemented. (https://github.com/metadings/zguide/issues/1)
I am also trying to determine whether the NetMQ.Security approach to security is better than the curve security approach that is built into ZeroMQ 4. It seems like most information about Curve is at least from 2014 or earlier.
Any information would be greatly appreciated!

Both publisher and subscriber need to use its own set of public\private keys. In your sample code for subscriber you set CurvePublicKey (to that of server, which is wrong but still) but do not set CurveSecretKey - that's why you get "cannot open client INITIATE vouch". Here is your sample from another question fixed:
public class Program
{
static void Main(string[] args) {
using (var context = new ZContext()) {
Console.WriteLine($"Curve Supported: {ZeroMQ.ZContext.Has("curve")}");
byte[] serverPublicKey;
byte[] serverSecretKey;
Z85.CurveKeypair(out serverPublicKey, out serverSecretKey);
var publisher = new ZSocket(context, ZSocketType.PUB);
publisher.CurvePublicKey = serverPublicKey;
publisher.CurveSecretKey = serverSecretKey;
publisher.CurveServer = true;
publisher.Bind("tcp://*:5050");
var subscriber = new ZSocket(context, ZSocketType.SUB);
byte[] subPublicKey;
byte[] subSecretKey;
Z85.CurveKeypair(out subPublicKey, out subSecretKey);
subscriber.CurvePublicKey = subPublicKey;
subscriber.CurveSecretKey = subSecretKey;
subscriber.CurveServerKey = serverPublicKey;
ZError connectError;
subscriber.Connect("tcp://mybox:5050", out connectError);
if (connectError != null) {
Console.WriteLine($"Connection error: {connectError.Name} - {connectError.Number} - {connectError.Text}");
}
subscriber.SubscribeAll();
// Publish some messages
Task.Run(() => {
for (var i = 1; i <= 5; i++) {
var msg = $"Pub msg: {Guid.NewGuid().ToString()}";
using (var frame = new ZFrame(msg)) {
publisher.Send(frame);
}
}
});
Task.Run(() => {
// Receive some messages
while (true) {
using (var frame = subscriber.ReceiveFrame()) {
var msg = frame.ReadString();
Console.WriteLine($"Received: {msg}");
}
}
});
Console.WriteLine("Press ENTER to exit");
Console.ReadLine();
ZError subError;
subscriber.Disconnect("tcp://mybox:5050", out subError);
subscriber.Dispose();
ZError pubError;
publisher.Disconnect("tcp://*:5050", out pubError);
publisher.Dispose();
}
}
}

Indeed, there are not much C# examples with NetMQ. I found this that works "CurveTests.cs":
public void CurveTest()
{
var serverPair = new NetMQCertificate();
using var server = new DealerSocket();
server.Options.CurveServer = true;
server.Options.CurveCertificate = serverPair;
server.Bind($"tcp://127.0.0.1:55367");
var clientPair = new NetMQCertificate();
using var client = new DealerSocket();
client.Options.CurveServerKey = serverPair.PublicKey;
client.Options.CurveCertificate = clientPair;
client.Connect("tcp://127.0.0.1:55367");
for (int i = 0; i < 100; i++)
{
client.SendFrame("Hello");
var hello = server.ReceiveFrameString();
Assert.Equal("Hello", hello);
server.SendFrame("World");
var world = client.ReceiveFrameString();
Assert.Equal("World", world);
}
}
Important note - if you want to share server public key between different applications, don't use string representation (serverPair.PublicKeyZ85), because encryption won't work. I assume it related to encoding. Better save byte array representation to some file and share it instead:
File.WriteAllBytes("serverPublicKey.txt", serverPair.PublicKey);

SqlWorkflowInstanceStore WaitForEvents returns HasRunnableWorkflowEvent but LoadRunnableInstance fails

Dears
Please help me with restoring delayed (and persisted) workflows.
I'm trying to check on self-hosted workflow store is there any instance that was delayed and can be resumed. For testing purposes I've created dummy activity that is delayed and it persists on delay.
generally resume process looks like:
get WF definition
configure sql instance store
call WaitForEvents
is there event with HasRunnableWorkflowEvent.Value name and if it is
create WorkflowApplication object and execute LoadRunnableInstance method
it works fine if store is created|initialized, WaitForEvents is called, store is closed. In such case store reads all available workflows from persisted DB and throws timeout exception if there is no workflows available to resume.
The problem happens if store is created and loop is started only for WaitForEvents (the same thing happens with BeginWaitForEvents). In such case it reads all available workflows from DB (with proper IDs) but then instead of timeout exception it is going to read one more instance (I know exactly how many workflows is there ready to be resumed because using separate test database). But fails to read and throws InstanceNotReadyException. In catch I'm checking workflowApplication.Id, but it was not saved with my test before.
I've tried to run on new (empty) persistent database and result is the same :(
This code fails:
using (var storeWrapper = new StoreWrapper(wf, connStr))
for (int q = 0; q < 5; q++)
{
var id = Resume(storeWrapper); // InstanceNotReadyException here when all activities is resumed
But this one works as expected:
for (int q = 0; q < 5; q++)
using (var storeWrapper = new StoreWrapper(wf, connStr))
{
var id = Resume(storeWrapper); // timeout exception here or beginWaitForEvents continues to wait
What is a best solution in such case? Add empty catch for InstanceNotReadyException and ignore it?
Here are my tests
const int delayTime = 15;
string connStr = "Server=db;Database=AppFabricDb_Test;Integrated Security=True;";
[TestMethod]
public void PersistOneOnIdleAndResume()
{
var wf = GetDelayActivity();
using (var storeWrapper = new StoreWrapper(wf, connStr))
{
var id = CreateAndRun(storeWrapper);
Trace.WriteLine(string.Format("done {0}", id));
}
using (var storeWrapper = new StoreWrapper(wf, connStr))
for (int q = 0; q < 5; q++)
{
var id = Resume(storeWrapper);
Trace.WriteLine(string.Format("resumed {0}", id));
}
}
Activity GetDelayActivity(string addName = "")
{
var name = new Variable<string>(string.Format("incr{0}", addName));
Activity wf = new Sequence
{
DisplayName = "testDelayActivity",
Variables = { name, new Variable<string>("CustomDataContext") },
Activities =
{
new WriteLine
{
Text = string.Format("before delay {0}", delayTime)
},
new Delay
{
Duration = new InArgument<TimeSpan>(new TimeSpan(0, 0, delayTime))
},
new WriteLine
{
Text = "after delay"
}
}
};
return wf;
}
Guid CreateAndRun(StoreWrapper sw)
{
var idleEvent = new AutoResetEvent(false);
var wfApp = sw.GetApplication();
wfApp.Idle = e => idleEvent.Set();
wfApp.Aborted = e => idleEvent.Set();
wfApp.Completed = e => idleEvent.Set();
wfApp.Run();
idleEvent.WaitOne(40 * 1000);
var res = wfApp.Id;
wfApp.Unload();
return res;
}
Guid Resume(StoreWrapper sw)
{
var res = Guid.Empty;
var events = sw.GetStore().WaitForEvents(sw.Handle, new TimeSpan(0, 0, delayTime));
if (events.Any(e => e.Equals(HasRunnableWorkflowEvent.Value)))
{
var idleEvent = new AutoResetEvent(false);
var obj = sw.GetApplication();
try
{
obj.LoadRunnableInstance(); //instancenotready here if the same store has read all instances from DB and no delayed left
obj.Idle = e => idleEvent.Set();
obj.Completed = e => idleEvent.Set();
obj.Run();
idleEvent.WaitOne(40 * 1000);
res = obj.Id;
obj.Unload();
}
catch (InstanceNotReadyException)
{
Trace.TraceError("failed to resume {0} {1} {2}", obj.Id
, obj.DefinitionIdentity == null ? null : obj.DefinitionIdentity.Name
, obj.DefinitionIdentity == null ? null : obj.DefinitionIdentity.Version);
foreach (var e in events)
{
Trace.TraceWarning("event {0}", e.Name);
}
throw;
}
}
return res;
}
Here is store wrapper definition I'm using for test:
public class StoreWrapper : IDisposable
{
Activity WfDefinition { get; set; }
public static readonly XName WorkflowHostTypePropertyName = XNamespace.Get("urn:schemas-microsoft-com:System.Activities/4.0/properties").GetName("WorkflowHostType");
public StoreWrapper(Activity wfDefinition, string connectionStr)
{
_store = new SqlWorkflowInstanceStore(connectionStr);
HostTypeName = XName.Get(wfDefinition.DisplayName, "ttt.workflow");
WfDefinition = wfDefinition;
}
SqlWorkflowInstanceStore _store;
public SqlWorkflowInstanceStore GetStore()
{
if (Handle == null)
{
InitStore(_store, WfDefinition);
Handle = _store.CreateInstanceHandle();
var view = _store.Execute(Handle, new CreateWorkflowOwnerCommand
{
InstanceOwnerMetadata = { { WorkflowHostTypePropertyName, new InstanceValue(HostTypeName) } }
}, TimeSpan.FromSeconds(30));
_store.DefaultInstanceOwner = view.InstanceOwner;
//Trace.WriteLine(string.Format("{0} owns {1}", view.InstanceOwner.InstanceOwnerId, HostTypeName));
}
return _store;
}
protected virtual void InitStore(SqlWorkflowInstanceStore store, Activity wfDefinition)
{
}
public InstanceHandle Handle { get; protected set; }
XName HostTypeName { get; set; }
public void Dispose()
{
if (Handle != null)
{
var deleteOwner = new DeleteWorkflowOwnerCommand();
//Trace.WriteLine(string.Format("{0} frees {1}", Store.DefaultInstanceOwner.InstanceOwnerId, HostTypeName));
_store.Execute(Handle, deleteOwner, TimeSpan.FromSeconds(30));
Handle.Free();
Handle = null;
_store = null;
}
}
public WorkflowApplication GetApplication()
{
var wfApp = new WorkflowApplication(WfDefinition);
wfApp.InstanceStore = GetStore();
wfApp.PersistableIdle = e => PersistableIdleAction.Persist;
Dictionary<XName, object> wfScope = new Dictionary<XName, object> { { WorkflowHostTypePropertyName, HostTypeName } };
wfApp.AddInitialInstanceValues(wfScope);
return wfApp;
}
}

I'm not workflow foundation expert so my answer is based on the official examples from Microsoft. The first one is WF4 host resumes delayed workflow (CSWF4LongRunningHost) and the second is Microsoft.Samples.AbsoluteDelay. In both samples you will find a code similar to yours i.e.:
try
{
wfApp.LoadRunnableInstance();
...
}
catch (InstanceNotReadyException)
{
//Some logging
}
Taking this into account the answer is that you are right and the empty catch for InstanceNotReadyException is a good solution.

Configure kafka-net to stop sending latest messages

I'm using kafka 0.8.1.1 on a Red Hat VM with kafka-net plugin. How can I configure my consumer to stop receiving earlier messages from kafka?
My consumer code:
var options = new KafkaOptions(new Uri("tcp://199.53.249.150:9092"), new Uri("tcp://199.53.249.151:9092"));
Stopwatch sp = new Stopwatch();
var router = new BrokerRouter(options);
var consumer = new Consumer(new ConsumerOptions("Test", router));
ThreadStart start2 = () =>
{
while (true)
{
sp.Start();
foreach (var message in consumer.Consume())
{
if (MessageDecoderReceiver.MessageBase(message.Value) != null)
{
PrintMessage(MessageDecoderReceiver.MessageBase(message.Value).ToString());
}
else
{
Console.WriteLine(message.Value);
}
}
sp.Stop();
}
};
var thread2 = new Thread(start2);
thread2.Start();

The Consumer in Kafka-net does not currently auto track the offsets being consumed. You will have to implement the offset tracking manually.
To Store the offset in kafka version 0.8.1:
var commit = new OffsetCommitRequest
{
ConsumerGroup = consumerGroup,
OffsetCommits = new List<OffsetCommit>
{
new OffsetCommit
{
PartitionId = partitionId,
Topic = IntegrationConfig.IntegrationTopic,
Offset = offset,
Metadata = metadata
}
}
};
var commitResponse = conn.Connection.SendAsync(commit).Result.FirstOrDefault();
To set the consumer to start importing at a specific offset point:
var offsets = consumer.GetTopicOffsetAsync(IntegrationConfig.IntegrationTopic).Result
.Select(x => new OffsetPosition(x.PartitionId, x.Offsets.Max())).ToArray();
var consumer = new Consumer(new ConsumerOptions(IntegrationConfig.IntegrationTopic, router), offsets);
Note the above code will set the consumer to start consuming at the very end of the log, effectively only receiving new messages.

How do I implement a C# Thrift service and consume it with a Silverlight client?

I'm current looking at Thrift to use as a RPC framework for our apps (mostly written in C# and Silverlight). I've come as far as implementing a service and consuming it from a C# console app (using a socket as transport).
For the C# server side code my code looked like: (basically copying the tutorials included with the source code)
MyServiceHandler handler = new MyServiceHandler();
MyService.Processor processor = new MyService.Processor(handler);
TServerTransport serverTransport = new TServerSocket(9090);
TServer server = new TSimpleServer(processor, serverTransport);
server.Serve();
For the client side code it looked like:
TTransport transport = new TSocket("localhost", 9090);
TProtocol protocol = new TBinaryProtocol(transport);
MyService.Client client = new MyService.Client(protocol);
transport.Open();
client.SomeServiceCall();
However, we will be consuming the service from a Silverlight client, and unfortunately there is no support for sockets in Silverlight for Thrift. I assume I'm forced to use HTTP communication between the client and service, using Thrift's C# THttpClient and THttpHandler classes? I could not find any examples of how to do this out there, can anyone point me in the right direction? Some example server and client side code would be appreciated.

It seems that this issue was already addressed by this guy. According to this JIRA, the fix is available in Thrift 0.9. You can either try this snapshot (note that, as it's not a final release, it might not be stable) or you can apply this patch to the 0.8 release.

I believe by now you would have understood, there is no direct way of communicating from Silverlight to the Cassandra database either using Thrift or any other clients.
I have one simple option related to this. Write a Silverlight enabled web service and consume it from the client.
For example, on the server side you can have a web service which does insert/update/read etc., like this. I just managed to pull out some code which we use for our project. Hope this helps.
using Apache.Cassandra;
using Thrift.Protocol;
using Thrift.Transport;
namespace CassandraWebLibrary
{
public class MyDb
{
String _host;
int _port;
String _keyspace;
bool _isConnected;
TTransport _transport = null;
Apache.Cassandra.Cassandra.Client _client = null;
String columnFamily = "ColumnFamilyName";
public VazhikaattiDB(String host, int port, String keyspace)
{
_host = host;
_port = port;
_keyspace = keyspace;
_isConnected = false;
}
public bool Connect()
{
try
{
_transport = new TFramedTransport(new TSocket(_host, _port));
TProtocol protocol = new TBinaryProtocol(_transport);
_client = new Apache.Cassandra.Cassandra.Client(protocol);
_transport.Open();
_client.set_keyspace(_keyspace);
_isConnected = true;
}
catch (Exception ex)
{
log.Error(ex.ToString());
}
return _isConnected;
}
public bool Close()
{
if (_transport.IsOpen)
_transport.Close();
_isConnected = false;
return true;
}
public bool InsertData(Send your data as parameters here)
{
try
{
List<Column> list = new List<Column>();
string strKey = keyvalue;
#region Inserting into Coulmn family
List<Byte> valbytes = new List<byte>(BitConverter.GetBytes(value)); //You might have to pad this with more bytes to make it length of 8 bytes
Column doublecolumn1 = new Column()
{
Name = Encoding.UTF8.GetBytes("column1"),
Timestamp = timestampvalue,
Value = valbytes.ToArray()
};
list.Add(doublecolumn1);
Column stringcolumn2 = new Column()
{
Name = Encoding.UTF8.GetBytes("column2"),
Timestamp = timestampvalue,
Value = Encoding.UTF8.GetBytes("StringValue")
};
list.Add(stringcolumn2);
Column timecolumn3 = new Column()
{
Name = Encoding.UTF8.GetBytes("column3"),
Timestamp = timestampvalue,
Value = BitConverter.GetBytes(DateTime.Now.Ticks)
};
list.Add(timecolumn3);
#endregion
ColumnParent columnParent = new ColumnParent();
columnParent.Column_family = columnFamily;
Byte[] key = Encoding.UTF8.GetBytes(strKey);
foreach (Column column in list)
{
try
{
_client.insert(key, columnParent, column, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
log.Error(e.ToString());
}
}
return true;
}
catch (Exception ex)
{
log.Error(ex.ToString());
return false;
}
}
public List<YourReturnObject> GetData(parameters)
{
try
{
ColumnParent columnParent = new ColumnParent();
columnParent.Column_family = columnFamily;
DateTime curdate = startdate;
IndexExpression indExprsecondkey = new IndexExpression();
indExprsecondkey.Column_name = Encoding.UTF8.GetBytes("column");
indExprsecondkey.Op = IndexOperator.EQ;
List<Byte> valbytes = PadLeftBytes((int)yourid, 8);
indExprsecondkey.Value = valbytes.ToArray();
indExprList.Add(indExprsecondkey);
IndexClause indClause = new IndexClause()
{
Expressions = indExprList,
Count = 1000,
Start_key = Encoding.UTF8.GetBytes("")
};
SlicePredicate slice = new SlicePredicate()
{
Slice_range = new SliceRange()
{
//Start and Finish cannot be null
Start = new byte[0],
Finish = new byte[0],
Count = 1000,
Reversed = false
}
};
List<KeySlice> keyslices = _client.get_indexed_slices(columnParent, indClause, slice, ConsistencyLevel.ONE);
foreach (KeySlice ks in keyslices)
{
String stringcolumnvalue = Encoding.UTF8.GetString(cl.Column.Value);
double doublevalue= (Double)BitConverter.ToDouble(cl.Column.Value);
long timeticks = BitConverter.ToInt64(cl.Column.Value, 0);
DateTime dtcolumntime = new DateTime(timeticks);
}
}
catch (Exception ex)
{
log.Error(ex.ToString());
}
return yourdatalist;
}
}
}
Now the above class can be used by your webservice, which in turn will be used by Silverlight. Btw, you'll have to take care of other silverlight issues like size of data to be downloaded from server/webservice etc.,
FYI, our client service of Cassandra runs on port 9160..