We Keep Coding

MongoDB Change Stream

I try to code a MongoDB change stream. But if I do some changes, my code doesn't detect the changes. If the code detects the changes I have to get the operation type - according to my understanding. Moreover, how can I insert the changes in the other database? Is it so "easy", that I code a new database connection and update/insert/delete the collection/database?
What did I Code?
`
lang-cs
string mongo_db_client = "mongodb://localhost:27017";
string m_database = "myDataBase";
string m_collection ="myCollection";
MongoClient dbClient = new MongoClient(mongo_db_client);
var database = dbClient.GetDatabase(m_database);
var collection = database.GetCollection<BsonDocument>(m_collection);
var cursor = collection.Watch();
var the_operation_type = null;
if(cursor != null)
{
using(cursor)
{
var curr = cursor.Current;
if (curr != null)
{
foreach (var change in curr)
{
try
{
the_operation_type= change.OperationType;
}
catch(NullReferenceException e)
{
Log(e.ToString());
}
}
}
}
}
`

Mongo Tailable cursor not working when trying to run in separate thread

I am trying to use Mongo Tailable cursor in my application I am using below code to create capped collection and tailable cursor
using MongoDB.Bson;
using MongoDB.Driver;
public static class TailableCursor
{
private const string DbUrl = ""//TODO use actual connectionstring
private const string DbName = "Test_App";
private const string CollectionName = "ExchangeMessage";
public static bool CreateCollection()
{
var name = "ExchangeMessage";
var count = 100;
var client = new MongoClient(new MongoUrl(DbUrl));
#pragma warning disable CS0618 // Type or member is obsolete
var server = client.GetServer();
#pragma warning restore CS0618 // Type or member is obsolete
var db = server.GetDatabase(DbName);
if (db == null)
{
return false;
}
if (!db.CollectionExists(name))
{
var options = new CollectionOptionsDocument(new Dictionary<string, object> {
{ "capped", true },
{ "size", count * 1024 },
{ "max", count }
});
db.CreateCollection(name, options);
var collection = db.GetCollection(name);
collection.Insert(new ExchangeMessage());
}
else
{
var collection = db.GetCollection(name);
if (collection.Count() == 0)
{
collection.Insert(new ExchangeMessage());
}
}
server.Disconnect();
return true;
}
public static void SubscribeTailAbleCursor()
{
var client = new MongoClient(new MongoUrl(DbUrl));
var database = client.GetDatabase(DbName);
var collection = database.GetCollection<BsonDocument>(CollectionName);
// Set lastInsertDate to the smallest value possible
BsonValue lastInsertDate = BsonMinKey.Value;
var options = new FindOptions<BsonDocument>
{
// Our cursor is a tailable cursor and informs the server to await
CursorType = CursorType.TailableAwait,
NoCursorTimeout = true
};
// Initially, we don't have a filter. An empty BsonDocument matches everything.
BsonDocument filter = new BsonDocument();
// NOTE: This loops forever. It would be prudent to provide some form of
// an escape condition based on your needs; e.g. the user presses a key.
int i = 0;
while (true)
{
//// Start the cursor and wait for the initial response
//using (var cursor = collection.FindSync(filter, options))
var cursor = collection.FindSync(filter, options);
{
foreach (var document in cursor.ToEnumerable())
{
i++;
// Set the last value we saw
lastInsertDate = document["InsertDate"];
Console.WriteLine(i.ToString());
// Write the document to the console.
Console.WriteLine(document.ToString());
}
}
// The tailable cursor died so loop through and restart it
// Now, we want documents that are strictly greater than the last value we saw
filter = new BsonDocument("$gt", new BsonDocument("InsertDate", lastInsertDate));
}
}
}
//The above code work when called as below from startup or main method but it blocks main //thread
TailableCursor.CreateCollection();
TailableCursor.SubscribeTailAbleCursor();
//When I try to run code in separate thread tailable cursor does not work.
Task.Run(async () =>{
TailableCursor.CreateCollection();
TailableCursor.SubscribeTailAbleCursor();
});

IS there a way to add CommandFlags to a helper function?

Currently trying to build a command which iterates through a BlockTable recursively, creating new drawings for each component BlockTableRecord. To do so, I need to lock each new document to properly edit it, which I'm trying to do via the Document.LockDocument() function. Since this command uses a recursive helper function, however, it throws a "DocumentLock is a type which is not valid in the given context" error which I believe is caused by the function lacking the "CommandFlags.Session" flag. Is there any way to attach this flag to a helper function? I've included my function's code below, thanks.
public List<string> BTRRecursor(BlockTableRecord bTR, string filepath)
{
Document doc = Application.DocumentManager.MdiActiveDocument;
Database db = doc.Database;
Editor ed = doc.Editor;
var filepaths = new List<string>();
foreach (ObjectId internalObj in bTR)
{
Transaction trans = db.TransactionManager.StartTransaction();
try
{
BlockTableRecord internalBTR = trans.GetObject(internalObj, OpenMode.ForRead) as BlockTableRecord;
if (internalBTR.Name.Contains("_MR_"))
{
string strTemplatePath = "acad.dwt";
Document newDoc = Application.DocumentManager.Add(strTemplatePath);
using (DocumentLock lock = newDoc.LockDocument()) {
BlockTable nDBT = trans.GetObject(newDoc.Database.BlockTableId, OpenMode.ForWrite) as BlockTable;
nDBT.Add(internalBTR);
//toDWG
//Create new file
//Open its BTR for write
//Set its BTR to internalBTR
filepaths.Append("DWG Filepath");
}
}
else if (internalBTR.Name.Contains("_NMR_"))
{
BTRRecursor(internalBTR, Directory.CreateDirectory(filepath + #"\" + internalBTR.Name).FullName);
}
}
finally
{
trans.Dispose();
}
}
return filepaths;
}

I posted an example in response to your post in the Autodesk forum. Here's another one in a more functional style.
static IEnumerable<BlockTableRecord> GetNestedBlockTableRecords(BlockTableRecord source, Transaction tr)
{
foreach (var btr in source.Cast<ObjectId>()
.Where(id => id.ObjectClass.DxfName == "INSERT")
.Select(id => (BlockReference)tr.GetObject(id, OpenMode.ForRead))
.Select(br => (BlockTableRecord)tr.GetObject(br.BlockTableRecord, OpenMode.ForRead)))
{
yield return btr;
foreach (var item in GetNestedBlockTableRecords(btr, tr))
{
yield return item;
}
}
}

Programmatically Import Block Into AutoCAD (C#)

I'm writing a plugin for AutoCAD and want to import all the blocks it will use at the beginning to make sure that they are available when needed. To do that, I use this method
public static void ImportBlocks(string[] filesToTryToImport, string filter = "")
{
foreach (string blockToImport in filesToTryToImport)
{
if (blockToImport.Contains(filter))
{
Database sourceDb = new Database(false, true); //Temporary database to hold data for block we want to import
try
{
sourceDb.ReadDwgFile(blockToImport, System.IO.FileShare.Read, true, ""); //Read the DWG into a side database
ObjectIdCollection blockIds = new ObjectIdCollection(); // Create a variable to store the list of block identifiers
Autodesk.AutoCAD.DatabaseServices.TransactionManager tm = sourceDb.TransactionManager;
using (Transaction myT = tm.StartTransaction())
{
// Open the block table
BlockTable bt = (BlockTable)tm.GetObject(sourceDb.BlockTableId, OpenMode.ForRead, false);
// Check each block in the block table
foreach (ObjectId btrId in bt)
{
BlockTableRecord btr = (BlockTableRecord)tm.GetObject(btrId, OpenMode.ForRead, false);
// Only add named & non-layout blocks to the copy list
if (!btr.IsAnonymous && !btr.IsLayout)
{
blockIds.Add(btrId);
}
btr.Dispose();
}
}
// Copy blocks from source to destination database
IdMapping mapping = new IdMapping();
sourceDb.WblockCloneObjects(blockIds, _database.BlockTableId, mapping, DuplicateRecordCloning.Replace, false);
_editor.WriteMessage("\nCopied " + blockIds.Count.ToString() + " block definitions from " + blockToImport + " to the current drawing.");
}
catch (Autodesk.AutoCAD.Runtime.Exception ex)
{
_editor.WriteMessage("\nError during copy: " + ex.Message);
}
finally
{
sourceDb.Dispose();
}
}
}
}
That method appears to work because it successfully executes. However when I go to insert a block in the drawing via AutoCAD's interface it doesn't show up as an option and when I try to insert it programmatically it throws a FileNotFound exception meaning it didn't work. What's wrong with this method? Thanks in advance!
EDIT: Here is a less complicated method with a test method
public static void ImportSingleBlock(string fileToTryToImport)
{
using (Transaction tr = _database.TransactionManager.StartTransaction())
{
Database sourceDb = new Database(false, true); //Temporary database to hold data for block we want to import
try
{
sourceDb.ReadDwgFile(fileToTryToImport, System.IO.FileShare.Read, true, ""); //Read the DWG into a side database
_database.Insert(fileToTryToImport, sourceDb, false);
_editor.WriteMessage("\nSUCCESS: " + fileToTryToImport);
}
catch (Autodesk.AutoCAD.Runtime.Exception ex)
{
_editor.WriteMessage("\nERROR: " + fileToTryToImport);
}
finally
{
sourceDb.Dispose();
}
tr.Commit();
}
}
[CommandMethod("TESTSINGLEBLOCKIMPORTING")]
public void TestSingleBlockImporting()
{
OpenFileDialog ofd = new OpenFileDialog();
DialogResult result = ofd.ShowDialog();
if (result == DialogResult.Cancel) //Ending method on cancel
{
return;
}
string fileToTryToImport = ofd.FileName;
using (Transaction tr = _database.TransactionManager.StartTransaction())
{
EntityMethods.ImportSingleBlock(fileToTryToImport);
tr.Commit();
}
}
This file is the block I'm trying to import. Hope this inspires someone cause I am desperately lost right now.

Your code is correct and should work. In fact I did tried it and works fine. You're probably missing to Commit() a outer transaction (where you call this ImportBlocs() method). Check:
using (Transaction trans = _database.TransactionManager.StartTransaction())
{
ImportBlocks(... parameters here ...);
trans.Commit(); // remember to call this commit, if omitted, Abort() is assumed
}

I had the same issue, very similar code. Issue was that
_database.Insert(fileToTryToImport, sourceDb, false);
should be
_database.Insert(blockName, sourceDb, false);
You can see that the first parameter has to be "blockName", and not the file path.

First time creating a method that uses parallel linq and getting out of memory exception

I wrote a method to download data from the internet and save it to my database. I wrote this using PLINQ to take advantage of my multi-core processor and because it is downloading thousands of different files in a very short period of time. I have added comments below in my code to show where it stops but the program just sits there and after awhile, I get an out of memory exception. This being my first time using TPL and PLINQ, I'm extremely confused so I could really use some advice on what to do to fix this.
UPDATE: I found out that I was getting a webexception constantly because the webclient was timing out. I fixed this by increasing the max amount of connections according to this answer here. I was then getting exceptions for the connection not opening and I fixed it by using this answer here. I'm now getting connection timeout errors for the database even though it is a local sql server. I still haven't been able to get any of my code to run so I could totally use some advice
static void Main(string[] args)
{
try
{
while (true)
{
// start the download process for market info
startDownload();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
}
public static void startDownload()
{
DateTime currentDay = DateTime.Now;
List<Task> taskList = new List<Task>();
if (Helper.holidays.Contains(currentDay) == false)
{
List<string> markets = new List<string>() { "amex", "nasdaq", "nyse", "global" };
Parallel.ForEach(markets, market =>
{
Downloads.startInitialMarketSymbolsDownload(market);
}
);
Console.WriteLine("All downloads finished!");
}
// wait 24 hours before you do this again
Task.Delay(TimeSpan.FromHours(24)).Wait();
}
public static void startInitialMarketSymbolsDownload(string market)
{
try
{
List<string> symbolList = new List<string>();
symbolList = Helper.getStockSymbols(market);
var historicalGroups = symbolList.AsParallel().Select((x, i) => new { x, i })
.GroupBy(x => x.i / 100)
.Select(g => g.Select(x => x.x).ToArray());
historicalGroups.AsParallel().ForAll(g => getHistoricalStockData(g, market));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
}
public static void getHistoricalStockData(string[] symbols, string market)
{
// download data for list of symbols and then upload to db tables
Uri uri;
string url, line;
decimal open = 0, high = 0, low = 0, close = 0, adjClose = 0;
DateTime date;
Int64 volume = 0;
string[] lineArray;
List<string> symbolError = new List<string>();
Dictionary<string, string> badNameError = new Dictionary<string, string>();
Parallel.ForEach(symbols, symbol =>
{
url = "http://ichart.finance.yahoo.com/table.csv?s=" + symbol + "&a=00&b=1&c=1900&d=" + (DateTime.Now.Month - 1) + "&e=" + DateTime.Now.Day + "&f=" + DateTime.Now.Year + "&g=d&ignore=.csv";
uri = new Uri(url);
using (dbEntities entity = new dbEntities())
using (WebClient client = new WebClient())
using (Stream stream = client.OpenRead(uri))
using (StreamReader reader = new StreamReader(stream))
{
while (reader.EndOfStream == false)
{
line = reader.ReadLine();
lineArray = line.Split(',');
// if it isn't the very first line
if (lineArray[0] != "Date")
{
// set the data for each array here
date = Helper.parseDateTime(lineArray[0]);
open = Helper.parseDecimal(lineArray[1]);
high = Helper.parseDecimal(lineArray[2]);
low = Helper.parseDecimal(lineArray[3]);
close = Helper.parseDecimal(lineArray[4]);
volume = Helper.parseInt(lineArray[5]);
adjClose = Helper.parseDecimal(lineArray[6]);
switch (market)
{
case "nasdaq":
DailyNasdaqData nasdaqData = new DailyNasdaqData();
var nasdaqQuery = from r in entity.DailyNasdaqDatas.AsParallel().AsEnumerable()
where r.Date == date
select new StockData { Close = r.AdjustedClose };
List<StockData> nasdaqResult = nasdaqQuery.AsParallel().ToList(); // hits this line
break;
default:
break;
}
}
}
// now save everything
entity.SaveChanges();
}
}
);
}

Async lambdas work like async methods in one regard: They do not complete synchronously but they return a Task. In your parallel loop you are simply generating tasks as fast as you can. Those tasks hold onto memory and other resources such as DB connections.
The simplest fix is probably to just use synchronous database commits. This will not result in a loss of throughput because the database cannot deal with high amounts of concurrent DML anyway.