We Keep Coding

Doc2Vec (Or Word2Vec) In Catalyst C#: How Do I get it to give results? (FastText)

I'm trying to replicate results from Gensim in C# to compare results and see if we need to bother trying to get Python to work within our broader C# context. I have been programming in C# for about a week, am usually a Python coder. I managed to get LDA to function and assign topics with C#, but there is no Catalyst model (that I could find) that does Doc2Vec explicitly, but rather I need to do something with FastText as they have in their sample code:
// Training a new FastText word2vec embedding model is as simple as this:
var nlp = await Pipeline.ForAsync(Language.English);
var ft = new FastText(Language.English, 0, "wiki-word2vec");
ft.Data.Type = FastText.ModelType.CBow;
ft.Data.Loss = FastText.LossType.NegativeSampling;
ft.Train(nlp.Process(GetDocs()));
ft.StoreAsync();
The claim is that it is simple, and fair enough... but what do I do with this? I am using my own data, a list of IDocuments, each with a label attached:
using (var csv = CsvDataReader.Create("Jira_Export_Combined.csv", new CsvDataReaderOptions
{
BufferSize = 0x20000
}))
{
while (await csv.ReadAsync())
{
var a = csv.GetString(1); // issue key
var b = csv.GetString(14); // the actual bug
// if (jira_base.Keys.ToList().Contains(a) == false)
if (jira.Keys.ToList().Contains(a) == false)
{ // not already in our dictionary... too many repeats
if (b.Contains("{panel"))
{
// get just the details/desc/etc
b = b.Substring(b.IndexOf("}") + 1, b.Length - b.IndexOf("}") - 1);
try { b = b.Substring(0, b.IndexOf("{panel}")); }
catch { }
}
b = b.Replace("\r\n", "");
jira.Add(a, nlp.ProcessSingle(new Document(b,Language.English)));
} // end if
} // end while loop
From a set of Jira Tasks and then I add labels:
foreach (KeyValuePair<string, IDocument> item in jira) { jira[item.Key].Labels.Add(item.Key); }
Then I add to a list (based on a breakdown from a topic model where I assign all docs that are at or above a threshold in that topic to the topic, jira_topics[n] where n is the topic numner, as such:
var training_lst = new List<IDocument>();
foreach (var doc in jira_topics[topic_num]) { training_lst.Add(jira[doc]); }
When I run the following code:
// FastText....
var ft = new FastText(Language.English, 0, $"vector-model-topic_{topic_num}");
ft.Data.Type = FastText.ModelType.Skipgram;
ft.Data.Loss = FastText.LossType.NegativeSampling;
ft.Train(training_lst);
var wtf = ft.PredictMax(training_lst[0]);
wtf is (null,NaN). [hence the name]
What am I missing? What else do I need to do to get Catalyst to vectorize my data? I want to grab the cosine similarities between the jira tasks and some other data I have, but I can't even get the Jira data into anything resembling a vectorization I can apply to something. Help!
Update:
So, Predict methods apparently only work for supervised learning in FastText (see comments below). And the following:
var wtf = ft.CompareDocuments(training_lst[0], training_lst[0]);
Throws an Implementation error (and only doesn't work with PVDM). How do I use PVDM, PVDCbow in Catalyst?

C# MongoDB update definition from filled model

How i can combine type C# filled model and mongo language for create update request?
i try this but get types error:
var update = new ObjectUpdateDefinition<User>(user) & Builders<User>.Update.Inc(f => f.FindCount, 1);
FullCode:
var user = new Model
{
Email = email,
Language = language,
Password = password,
// +17 fields
};
// how i can convert all fields to set? and join with query Inc (like AllFieldToSet)?
var update = user.AllFieldToSet() & Builders<User>.Update.Inc(f => f.FindCount, 1);
Models.FindOneAndUpdate<Model>(
f => f.Email == email,
update,
new FindOneAndUpdateOptions<Model> {IsUpsert = true});

Firstly I would begin to ask why do you update so many fields at once, this can have a significant effect on performance and scalability, especially when your collection has indexes, replication and/or sharding involved. Whenever you have to update an indexed field it needs to update the index as well.
There are multiple solutions:
ReplaceOne:
Inc only increments the value, this can be done manually in the model: FindCount++
Manually build the update operator: Just manually build using the builder they provided
Write your own extension using reflection Personally I like type safety but you can use this extension that I wrote (you will just have to extend it and build in null checks, etc.)
public static class UpdateBuilderExtensions
{
public static UpdateDefinition<TDocument> SetAll<TDocument, TModel>(this UpdateDefinition<TDocument> updateBuilder, TModel value, params string[] excludeProperties)
{
foreach (var property in value.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance).Where(x => excludeProperties?.Contains(x.Name) == false))
updateBuilder = Builders<TDocument>.Update.Combine(updateBuilder.Set(property.Name, property.GetValue(value)));
return updateBuilder;
}
}
Usage of it:
You have to exclude properties that have been set already and you have to exclude the BsondId field (if you do not set it manually)
var update = Builders<User>.Update.Inc(x => x.FindCount, 1).SetAll(model, nameof(model.Id), nameof(model.FindCount));
Models.UpdateOne(x => x.Email== "some-email", update, new UpdateOptions
{
IsUpsert = true
});

unable to get data from database by passing string value in entity frame work asp.net MVC

i am trying to get data from database passing a string value. but get null value instead of the data.
i have tried the following code
order getCustomerOrder(string or_n)
{
using (foodorderingEntities db = new foodorderingEntities ())
{
var result = db.orders.Where(or => or.order_no == or_n).FirstOrDefault();
return result;
}
}
please some one guide me to solve this problem.

Please paste your order object, so we know if its reference object/primitive etc, you can use this to understand the two ways to retrieve orders.
I would recommend you read this answer & this MSDN string compare, if your default culture is causing an issue in the comparison, it will help you understand whats going on
Options 1:
// Query syntax
IEnumerable<CustomerOrder> queryResultsCustomerOrder =
from order in orders
where order.number == myOrderNumber
select order;
Options 2:
// Method-based syntax
IEnumerable<CustomerOrder> queryResultsCustomerOrder2 = orders.Where(order => order.Number == myOrderNumber);
using the above, now you can get however many orders the customer has. I am assuming your order is a number, but you can change it to whatever like a string.
Int based order comparison sample
IEnumerable<CustomerOrder> getCustomerOrder(int myOrderNumber)
{
if(myOrderNumber <1) return null;
using (foodorderingEntities dbContextOrderSet = new foodorderingEntities())
{
IEnumerable<CustomerOrder> resultsOneOrManyOrders = orders.Where(order => order.Number == myOrderNumber);
return resultsOneOrManyOrders ;
}
}
string based order comparison
IEnumerable<CustomerOrder> getCustomerOrder(string myOrderNumber)
{
//no orders
if(String.IsNullOrEmpty(myOrderNumber)) return null;
using (foodorderingEntities dbContextOrderSet = new foodorderingEntities())
{
IEnumerable<CustomerOrder> resultsOneOrManyOrders = orders.Where(order => order.Number == myOrderNumber);
// you can replace the *** comparison with .string.Compare and try inside the block
return resultsOneOrManyOrders ;
}
}

MongoDB C# Driver how to update a collection of updated documents

I have a Mongo database with lots of documents. All of the documents have a field called "source" which contains the origin name for the document. But a lot of old documents have this field containing "null" (because I hadn't have source by that time) string value. I want to select all those documents and fix this problem by replacing their "source = "null"" values by new values parsed from another fields of the aforementioned documents.
Here's what I'm doing to fix the this:
public void FixAllSources() {
Task.Run(async ()=> {
var filter = Builders<BsonDocument>.Filter.And(new List<FilterDefinition<BsonDocument>>() {
Builders<BsonDocument>.Filter.Exists("url"),
Builders<BsonDocument>.Filter.Ne("url", BsonNull.Value),
Builders<BsonDocument>.Filter.Eq("source", "null")
});
var result = await m_NewsCollection.FindAsync(filter);
var list = result.ToList();
list.ForEach(bson => {
bson["source"] = Utils.ConvertUrlToSource(bson["url"].AsString);
});
});
}
as you can see, I'm fetching all those documents and replacing their source field by a new value. So now I've got a List of correct BsonDocuments which I need to put back into the database.
Unfortunately, I'm stuck on this. The problem is that all of the "Update" methods require filters and I have no idea what filters should I pass there. I just need to put back all those updated documents, and that's it.
I'd appreciate any help :)
p.s.
I've came up with an ugly solution like this:
list.ForEach(bson => {
bson["source"] = Utils.ConvertUrlToSource(bson["url"].AsString);
try {
m_NewsCollection.UpdateOne( new BsonDocument("unixtime", bson["unixtime"]), new BsonDocument {{"$set", bson}},
new UpdateOptions {IsUpsert = true});
}
catch (Exception e) {
WriteLine(e.StackTrace);
}
});
It works for now. But I'd still like to know the better one in case I need to do something like this again. So I'm not putting my own solution as an answer

List 'areas' used in TFS project

I've been playing about with the WorkItem objects from the Microsoft.TeamFoundation schemas in C#, but was wondering if anyone knows how I would refer to an object of type 'Area' or, for that matter, 'Iteration'.
It seems that these are treated as objects in TFS, but I haven't come across any information on how to refer to these in C#.
You can filter WorkItems by [Area] or [Iteration] using WIQL, but what if I wanted to populate a ComboBox with all Areas or Iterations?
Also, how can I view the database structure of my workplace's TFS project?
Thanks guys,
Andy

Have a look at this Blog Post. There's sample code and a demo.
Here's a quick LINQPad Query that should do the job (download VS2010 / VS2012):
void Main()
{
const String CollectionAddress = "http://tfsserver:8080/tfs/MyCollection";
const String ProjectName = "MyProject";
using (var tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(
new Uri(CollectionAddress)))
{
tfs.EnsureAuthenticated();
var server = tfs.GetService<ICommonStructureService>();
var projectInfo = server.GetProjectFromName(ProjectName);
var nodes = server.ListStructures(projectInfo.Uri).Dump();
// You should be able to re-factor this with "Iteration"
// for getting those too.
var nodesXml = server.GetNodesXml(
nodes
.Where(node => node.Name == "Area")
.Select(node => node.Uri).ToArray(),
true);
var areaPathAndId =
XElement.Parse(nodesXml.OuterXml)
.Descendants("Node")
.Select(xe => new
{
Path = xe.Attribute("Path").Value,
ID = xe.Attribute("NodeID").Value,
})
.Dump();
}
}