I try to code a MongoDB change stream. But if I do some changes, my code doesn't detect the changes. If the code detects the changes I have to get the operation type - according to my understanding. Moreover, how can I insert the changes in the other database? Is it so "easy", that I code a new database connection and update/insert/delete the collection/database?
What did I Code?
`
lang-cs
string mongo_db_client = "mongodb://localhost:27017";
string m_database = "myDataBase";
string m_collection ="myCollection";
MongoClient dbClient = new MongoClient(mongo_db_client);
var database = dbClient.GetDatabase(m_database);
var collection = database.GetCollection<BsonDocument>(m_collection);
var cursor = collection.Watch();
var the_operation_type = null;
if(cursor != null)
{
using(cursor)
{
var curr = cursor.Current;
if (curr != null)
{
foreach (var change in curr)
{
try
{
the_operation_type= change.OperationType;
}
catch(NullReferenceException e)
{
Log(e.ToString());
}
}
}
}
}
`
Related
I am trying to detect which message is edited or deleted on a subscribed channel on telegram with TLSharp library in c#.
1- in a while(true) loop I am getting latest updates.
2- when I delete or edit a message for test, I receive TLUpdateChannelTooLong only.
3- then I use client.GetHistoryAsync function to get channel messages, and check their EditDate.
But I don't know how much should I go deep in history and I can not find deleted message with this code easily.
Is there any solution to find deleted/edited messages easy and safe?
Part of my code:
state = await client.SendRequestAsync<TLState>(new TLRequestGetState());
while (true)
{
await Task.Delay(1000);
var req = new TLRequestGetDifference() { Date = state.Date, Pts = state.Pts, Qts = state.Qts };
TLDifference diff = null;
try
{
diff = await client.SendRequestAsync<TLAbsDifference>(req) as TLDifference;
}
catch (Exception ex)
{
HandleThisException(ex);
}
//--
if (diff != null)
{
state = await client.SendRequestAsync<TLState>(new TLRequestGetState());
foreach (var upd in diff.OtherUpdates.OfType<TLUpdateNewChannelMessage>())
{
var tm = (upd.Message as TLMessage);
if (tm == null) { continue; } // ?
var textMessage = tm.Message;
if (tm.Media != null)
{
if (tm.Media.GetType().ToString() == "TeleSharp.TL.TLMessageMediaPhoto")
{
var tLMessageMediaPhoto = (tm.Media as TLMessageMediaPhoto);
textMessage = tLMessageMediaPhoto.Caption;
}
}
try
{
var from = (tm.ToId as TLPeerChannel).ChannelId;
long replyTo = tm.ReplyToMsgId == null ? 0 : (long)tm.ReplyToMsgId;
await AnalyzeNewMessage( ... );
}
catch (Exception exParsing)
{
HandleThisException(exParsing);
}
}
// Checking Edited/Deleted Messages
foreach(var upLong in diff.OtherUpdates.OfType<TLUpdateChannelTooLong>())
{
TLChannel theChat = null;
foreach(var chat in diff.Chats.OfType<TLChannel>())
{
if(chat.Id == upLong.ChannelId) { theChat = chat; break; }
}
if (theChat != null)
{
var x = await client.GetHistoryAsync(
new TLInputPeerChannel { ChannelId = theChat.Id, AccessHash = (long)theChat.AccessHash },
0,-1,2
); // checking only 2 last messages!
var ChMsgs = x as TLChannelMessages;
foreach (var msg in ChMsgs.Messages.OfType<TLMessage>())
{
if(msg.EditDate != null)
{
var txt = msg.Message;
if (msg.Media != null)
{
if (msg.Media.GetType().ToString() == "TeleSharp.TL.TLMessageMediaPhoto")
{
txt = (msg.Media as TLMessageMediaPhoto).Caption;
}
}
await AnalyzeEditedMessage( ... );
}
}
}
}
}
}
I'm struggling to find an answer to a problem I've been experiencing with C#'s popular Postgres library, Npgsql. I'm not sure if it's wholly a problem with Npgsql or not, though I suspect it is because my code is very straight forward. The issue I'm seeing is: when I call an async method on Npgsql sometimes, not all the time, the thread becomes locked. This issue occurs randomly from what I can tell. The result, being that I'm running in a Microsoft Orleans environment (which may be relevant to finding the solution), is that the thread locks indefinitely, thus making one of Orleans' worker threads unable to process work. As I make more Npgsql calls, these locked threads stack up and eventually the Orleans system is choked by thread exhaustion.
So I'm really at a loss for what the problem could be, but because the locking always happens in the same method, and because it appears to be happening in some subroutine of Npgsql, I think it's fair to investigate Npgsql further.
Here's my code which is used in an Orleans storage provider (special class which handles the system's persistence layer.)
var sql = $"SELECT * FROM objects WHERE id = #id";
using (var connection = new NpgsqlConnection(connectionString))
using (var cmd = new NpgsqlCommand(sql, connection))
{
try
{
await connection.OpenAsync();
cmd.Parameters.AddWithValue("id", id);
using(var reader = await connection.ExecuteReaderAsync(cmd))
{
if (reader.HasRows)
{
var objects = await ProtobufSQL.DataReaderToType(modelType, reader);
var data = objects[0];
state.Data = data;
}
}
catch (Exception e)
{
Log.Error(1, e.Message, e);
}
}
This is the source for the ProtobufSQL class:
public class ProtobufSQL
{
public static List<Tuple<string, object>> FlattenToSQLColumns(IMessage message, MessageDescriptor descriptor, string prefix = null)
{
var fields = descriptor.Fields.InDeclarationOrder();
var columns = new List<Tuple<string, object>>();
for (var i = 0; i < fields.Count; i++)
{
var field = fields[i];
var columnName = field.Name.ToLower();
if (field.Name == "id")
{
ByteString bytes = (ByteString)field.Accessor.GetValue(message);
var uuid = new Guid(bytes.ToByteArray());
columns.Add(new Tuple<string, object>("id", uuid));
}
else if (field.FieldType == FieldType.Message)
{
var embeddedDescriptor = field.MessageType;
var embeddedMessage = field.Accessor.GetValue(message);
if (field.IsRepeated)
{
throw new Exception("Repeated complex types are not supported, create a foreign key reference in a new object instead.");
}
else
{
columns.AddRange(FlattenToSQLColumns((IMessage)embeddedMessage, embeddedDescriptor, $"{columnName}."));
}
}
else if (field.FieldType == FieldType.Group)
{
throw new Exception("Groups are not supported by ProtobufSQL.");
}
else
{
var columnValue = field.Accessor.GetValue(message);
var key = prefix + columnName;
if (field.IsRepeated)
{
var enumerableColumnValue = columnValue as IEnumerable;
Type listTypeOf = enumerableColumnValue.GetType().GetGenericArguments()[0];
Type listType = typeof(List<>).MakeGenericType(listTypeOf);
dynamic valueList = Activator.CreateInstance(listType);
foreach (var item in enumerableColumnValue)
{
valueList.Add((dynamic)item);
}
columns.Add(new Tuple<string, object>(key, valueList.ToArray()));
}
else
{
columns.Add(new Tuple<string, object>(key, columnValue));
}
}
}
return columns;
}
public static async Task<IMessage[]> DataReaderToType(Type type, DbDataReader reader)
{
var descriptor = (MessageDescriptor)type.GetProperty("Descriptor").GetValue(null);
IList<IMessage> objects = new List<IMessage>();
while (await reader.ReadAsync())
{
var obj = Activator.CreateInstance(type);
TraverseDbRow(reader, descriptor, obj);
objects.Add((IMessage)obj);
}
return objects.ToArray();
}
private static void TraverseDbRow(DbDataReader reader, MessageDescriptor descriptor, object obj, string prefix = null)
{
var fields = descriptor.Fields.InFieldNumberOrder();
for (var i = 0; i < fields.Count; i++)
{
var field = fields[i];
if (field.FieldType == FieldType.Message)
{
if (field.IsRepeated)
{
// Repeated embedded types will be broken out into a separate table,
// so there's no need to handle them here.
}
else if (field.IsMap)
{
throw new Exception("Maps are not yet supported by ProtobufSQL.");
}
else
{
var embeddedDescriptor = field.MessageType;
var embeddedObj = Activator.CreateInstance(embeddedDescriptor.ClrType);
TraverseDbRow(reader, embeddedDescriptor, embeddedObj, $"{prefix}{field.Name}.");
}
}
else if (field.FieldType == FieldType.Group)
{
throw new Exception("Groups are not supported by ProtobufSQL.");
}
else
{
var columnName = prefix + field.Name;
try
{
var columnValue = reader[columnName];
var propertyInfo = obj.GetType().GetProperty(field.Name, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
if (field.Name == "id")
{
var guid = (Guid)columnValue;
ByteString bytes = ByteString.CopyFrom(guid.ToByteArray());
propertyInfo.SetValue(obj, bytes);
}
else if (field.IsRepeated)
{
var repeated = propertyInfo.GetValue(obj);
var addRange = repeated.GetType().GetMethod("AddRange");
addRange.Invoke(repeated, new object[] { columnValue });
}
else if (field.IsMap)
{
throw new Exception("Maps are not yet supported by ProtobufSQL.");
}
else
{
propertyInfo.SetValue(obj, Convert.ChangeType(columnValue, propertyInfo.PropertyType));
}
}
catch (IndexOutOfRangeException e)
{
// columnName was not present in the response
}
}
}
}
}
I also have this screenshot of the thread's stack when it got locked:
I'm really unsure what to make of all this. Hopefully someone is out there with a bit of knowledge that can help me proceed! Thanks.
Here is my code
public static string UpdateEmptyCaseRevierSet() {
string response = string.Empty;
using (System.Transactions.TransactionScope tran = new System.Transactions.TransactionScope()) {
using (var db = new Entities.WaveEntities()) {
var maxCaseReviewersSetID = db.CaseReviewerSets.Select(crs => crs.CaseReviewersSetId).Max();
var emptyCHList = db.CaseHistories.Where(ch => ch.CaseReviewersSetID == null && ch.IsLatest == true && ch.StatusID != 100).ToList();
for(int i=0; i < emptyCHList.Count; i++) {
var emptyCH = emptyCHList[i];
var newCaseReviewerSET = new Entities.CaseReviewerSet();
newCaseReviewerSET.CreationCHID = emptyCH.CHID;
db.CaseReviewerSets.Add(newCaseReviewerSET);
emptyCH.CaseReviewerSet = newCaseReviewerSET;
}
db.SaveChanges();
}
tran.Complete();
}
return response;
}
The exception occures on "db.SaveChanges()"
I saw in another post with the same error message something about "it seems I cannot have two connections opened to the same database with the TransactionScope block." but I dont think that this has anything to do with my case.
Additionally the number of records to insert and update in total are 2700, witch is not that many really. But it does take quite a lot of time to complete the for statement (10 minutes or so). Since everything happening within the for statement is actually happening in the memory can someone please explane why is this taking so long ?
You can try as shown below using latest db.Database.BeginTransaction API.
Note : use foreach instead of for
using (var db = new Entities.WaveEntities())
{
using (var dbContextTransaction = db.Database.BeginTransaction())
{
try
{
var maxCaseReviewersSetID = db.CaseReviewerSets.Select(crs => crs.CaseReviewersSetId).Max();
var emptyCHList = db.CaseHistories.Where(ch => ch.CaseReviewersSetID == null && ch.IsLatest == true && ch.StatusID != 100).ToList();
foreach(var ch in emptyCHList) {
var newCaseReviewerSET = new Entities.CaseReviewerSet();
newCaseReviewerSET.CreationCHID = ch.CHID;
db.CaseReviewerSets.Add(newCaseReviewerSET);
}
db.SaveChanges();
dbContextTransaction.Commit();
}
catch (Exception)
{
dbContextTransaction.Rollback();
}
}
}
Help is very welcome and extremely appreciated, thank you. What this program is, is a ProxyChecker, because i've bought a bunch and will continue to do so of proxies with different user/passes etc, however some have expired. I added a break point and what it's doing is actually skipping the ProxyClient code and going straight to for each var item in 1, if item accepts connection etc, it then just returns false and finishes.
private static void CheckProxy(object state)
{
var u = user[0];
var p = pass[0];
var l = new List<MyIP>();
Parallel.ForEach(l.ToArray(), (ip_item) =>
{
try
{
string ip = ip_item.IP;
using (var client = new ProxyClient(ip, u, p))
{
Console.WriteLine(ip, user, pass);
client.Connect();
ip_item.AcceptsConnection = client.IsConnected;
}
}
catch
{
l.Remove(ip_item);
}
});
foreach (var item in l)
{
if (item.AcceptsConnection == true)
{
WriteToFile(user[0], pass[0]);
}
Console.WriteLine(item.IP + " is " + (item.AcceptsConnection) + " accepts connections" + " doesn not accept connections");
}
}
Load ips function:#
private static void loadips()
{
using (TextReader tr = new StreamReader("ips.txt"))
{
var l = new List<MyIP>();
string line = null;
while ((line = tr.ReadLine()) != null)
{
l.Add(new MyIP { IP = line });
}
}
}
I have added this in response to the answer. I believe this is a variable issue as the variable is locally declared not publicly any ideas how to fix? i'm unable to find a way to get this working seems like i'm being dumb. thanks.
Your code "skips" the Parallel.ForEach because the local variable l is an empty list (you just created it with new List<MyIP>().
Your loadips() method only fills a list referenced only by (another) local variable l, which is out of scope when the using block is left.
There are several ways to solve that. If both methods are in the same class, you could declare a member variable in that class of type List<MyIP> and fill that list in your loadips().
My suggestion would be to change loadips() so that it returns the list of read ip addresses and call this from CheckProxy():
private static List<MyIP> loadips() // changed return type to List<MyIP>
{
using (TextReader tr = new StreamReader("ips.txt"))
{
var l = new List<MyIP>();
string line = null;
while ((line = tr.ReadLine()) != null)
{
l.Add(new MyIP { IP = line });
}
return l; // return the list!
}
}
and the CheckProxy:
private static void CheckProxy(object state)
{
var u = user[0];
var p = pass[0];
var l = loadips(); // load IPs here
Parallel.ForEach(l.ToArray(), (ip_item) =>
{
...
In my c# code I have the following method that creates a document in the database, adds metadata regarding the document to the database and then updates some information regarding the date the repository was last updated. This method is often called numerous times in quick succession as multiple file uploads are common. However I am having problems with the code failing due to deadlock in sql server.
private IEnumerable<DocumentMetadata> CreateDoc(int? jobId, int?repositoryId, int? folderId, string documentTypeString, IEnumerable<DocumentModel> files)
{
if ((jobId == null && repositoryId == null) || (jobId != null && repositoryId != null))
{
throw new InvalidOperationException("Either job id or repository id must be specified");
}
using (var tran = new TransactionScope())
{
List<DocumentMetadata> newDocuments = new List<DocumentMetadata>();
var documentType = GetDocumentTypeByPrefix(documentTypeString);
if (folderId == null)
{
// Find the root folder
var job = getJob(jobId);
var rootFolder = getRootFolder(job);
// If we can't find a root folder, create one
if (rootFolder == null)
{
rootFolder = CreateRootDirectory(job);
}
folderId = rootFolder.FolderId;
}
User currentUser = _userService.GetCurrentUser();
foreach (var file in files)
{
var document = new Document() { Document1 = file.Data };
var documentMetadata = new DocumentMetadata
{
Document = document,
CreatedDate = file.CreatedDate,
FileName = file.Filename,
FileSize = file.Data.Length,
FolderId = folderId,
DocumentType = documentType,
JobId = jobId,
RepositoryId = repositoryId,
User = currentUser
};
_unitOfWork.DocumentMetadata.Add(documentMetadata);
newDocuments.Add(documentMetadata);
}
// set repository updated date
if (repositoryId != null)
{
DocumentRepository repo = GetDocumentRepository(repositoryId);
if (repo != null)
{
repo.UpdatedDate = new DateTimeOffset(DateTime.Now);
}
}
_unitOfWork.SaveChanges();
tran.Complete();
return newDocuments;
}
}
After some debugging it would appear that the updating of the repository id is causing the deadlock problem. If I remove this code block outside of the transaction all files are saved with no errors.
Why would this code block
if (repositoryId != null)
{
DocumentRepository repo = GetDocumentRepository(repositoryId);
if (repo != null)
{
repo.UpdatedDate = new DateTimeOffset(DateTime.Now);
}
}
cause the deadlock? No other access is being made to the DocumentRepository table apart from in this method - as the locks are obtained in the same order surely there should be no deadlock?
What is it about this code that is leading to deadlock?
Updated: The code for GetDocumentRepository is:
public DocumentRepository GetDocumentRepository(int repositoryId)
{
var result = DocumentRepositories.SingleOrDefault(x => x.RepositoryId == repositoryId); return result;
}
Have you checked the code without defining a transaction explicitly? Based on your code I would say that you are trying to read something that has been modified but not commited.
Another test you could do is to try to add a breakpoint in your code and try to get the DocumentRepository using READ UNCOMMITTED.