We Keep Coding

Faster insert sqlite3 processing

I have this function where it opens a .TXT file with some products and insert line by line on the sqlitedb. The process is working fine, the problem is. This file contains 2000+ lines, because of that, the process is taking several hours to finish. I wonder if there is a way to make the process a little bit faster.
here is the function:
private void carrega_produtos()
{
var assembly = typeof(sincroniza_page).GetTypeInfo().Assembly;
foreach (var res in assembly.GetManifestResourceNames())
{
if (res.Contains("produtos.txt"))
{
Stream stream = assembly.GetManifestResourceStream(res);
var st = res.Count();
using (var reader = new StreamReader(stream))
{
string linha;
acesso_banco_produtos banco = new acesso_banco_produtos();
while ((linha = reader.ReadLine()) != null)
{
List<string> lista = linha.Split(new char[] { '§' }).ToList();
var cod = int.Parse(lista.ElementAt(0));
var nome_prod = lista.ElementAt(1);
var cod_grupo = lista.ElementAt(2);
var nm_grupo = lista.ElementAt(3);
var ind_ativo = lista.ElementAt(4);
var val_custo_unit = lista.ElementAt(5);
var val_custo = lista.ElementAt(6);
var perc_imposto = lista.ElementAt(7);
var unidade_med = lista.ElementAt(8);
var qtd_mes_1 = lista.ElementAt(9);
var qtd_mes_2 = lista.ElementAt(10);
var qtd_mes_3 = lista.ElementAt(11);
var qtd_mes_6 = lista.ElementAt(12);
var qtd_mes_12 = lista.ElementAt(13);
var data = lista.ElementAt(14);
var bd = new banco_produtos()
{
cod_produto = cod,
nm_produto = nome_prod,
cod_grupo = cod_grupo,
nm_grupo = nm_grupo,
ind_ativo = ind_ativo,
val_custo_unitario = Double.Parse(val_custo_unit),
val_lista_preco = val_custo,
perc_impostos = perc_imposto,
unidade_medida = unidade_med,
qtde_vendida_mes_1 = qtd_mes_1,
qtde_vendida_mes_2 = qtd_mes_2,
qtde_vendida_mes_3 = qtd_mes_3,
qtde_vendida_mes_6 = qtd_mes_6,
qtde_vendida_mes_12 = qtd_mes_12
};
//here i look in the DB if already exists the new product
var procura = banco.get_produto(cod);
if (procura == null)
{
// here is inserted to the db
banco.inserir_produto(bd);
}
}
valor += 1;
}
}
}
}

I'm not sure what is inside your method which inserts data into db but the most common issue with SQLite and massive inserts is the fact that SQLite by default wraps every insert with transaction which creates significant overhead. A good practice for such cases is to make signle transaction for all insterts which should singificantly improve the perfomance, see the example.

I did what #Dmytro said, I used the method "insertORIgnore". It improved a lot using that method.thank you for the help.

Trouble Fetching Value in Variable

Basically here's my code which I'm having trouble with. Insanely new to mongoDB and would love to understand how to get values out of a JSON string that is returns in the variable 'line'.
public string get_data()
{
var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<BsonDocument>("metacorp");
var cursor = collection.Find("{'movie_name' : 'Hemin'}").ToCursor();
var line = "";
foreach (var document in cursor.ToEnumerable())
{
using (var stringWriter = new StringWriter())
using (var jsonWriter = new JsonWriter(stringWriter))
{
var context = BsonSerializationContext.CreateRoot(jsonWriter);
collection.DocumentSerializer.Serialize(context, document);
line = stringWriter.ToString();
}
}
var js = new JavaScriptSerializer();
var d = js.Deserialize<dynamic>(line);
var a = d["movie_name"];
return line;
}
This is the output I get if I return line:
{ "_id" : ObjectId("58746dcafead398e4d7233f5"), "movie_name" : "Hemin"
}
I want to be able to fetch the value 'Hemin' into 'a'.

I know this is not what you're asking for but since you're using the c# driver then I would recommend the following. Assumes you have a c# class corresponding to metacorp collection or at least a serializer that handles it. Hope it helps.
var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<MetaCorp>("metacorp");
var m = collection.SingleOrDefault(x => x.Movie_Name == "Hemin"); // Assuming 0 or 1 with that name. Use Where otherwise
var movieName = "Not found";
if(m!= null)
movieName = m.Movie_Name;

You could have your dto class for movie ans just fetch the data from collection:
public class Movie
{
public ObjectId Id { get; set; }
public string movie_name { get; set;}
}
...
var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<BsonDocument>("metacorp");
var movies = collection.Find(x=>x.movie_name == "Hemin").ToEnumerable();

How to add/update milestones in a feature using rally rest API and C#?

I am not able to add or update milestones field for the Features in the Rally. If anyone having the code available using C# to update the same, please share with me. I am searching and doing from last one week with no luck.
When I am trying to add/Update milestones in the Features. I am getting the error as "Could not read: Could not read referenced object null". My code is as follows:-
public DynamicJsonObject UpdateFeaturesbyName(string fea, string bFun)
{
//getting list of Feature.
Request feat = new Request("PortfolioItem/Feature");
feat.Query = new Query("Name", Query.Operator.Equals, fea);
QueryResult TCSResults = restApi.Query(feat);
foreach (var res in TCSResults.Results)
{
var steps = res["Milestones"];
Request tsteps = new Request(steps);
QueryResult tstepsResults = restApi.Query(tsteps);
foreach (var item in tstepsResults.Results)
{
}
if (res.Name == fea)
{
var targetFeature = TCSResults.Results.FirstOrDefault();
DynamicJsonObject toUpdate = new DynamicJsonObject();
//toUpdate["Milestones"] = "";
// CreateResult createResult = restApi.Create(steps._ref, toUpdate);
// String contentRef = steps._ref;
//String contentRef = createResult._ref;
string[] value = null;
string AccCri = string.Empty;
if (!string.IsNullOrWhiteSpace(bFun))
{
value = bFun.Split(new string[] { "\r\n", "\n" }, StringSplitOptions.None);
foreach (string item in value)
{
//if (string.IsNullOrWhiteSpace(AccCri))
// AccCri = item;
//else
// AccCri = AccCri + "<br/>" + item;
if (!string.IsNullOrWhiteSpace(item))
{
//Query for Milestone.
Request ms = new Request("Milestone");
ms.Fetch = new List<string>() { "Name", "ObjectID" };
ms.Query = new Query("Name", Query.Operator.Equals, item);
QueryResult msResults = restApi.Query(ms);
var targetMLResult = msResults.Results.FirstOrDefault();
long MLOID = targetMLResult["ObjectID"];
DynamicJsonObject tarML = restApi.GetByReference("Milestone", MLOID, "Name", "_ref", "DisplayColor");
DynamicJsonObject targetML = new DynamicJsonObject();
targetML["Name"] = tarML["Name"];
//targetML["_ref"] = tarML["_ref"];
targetML["_ref"] = "/milestone/" + Convert.ToString(MLOID);
targetML["DisplayColor"] = tarML["DisplayColor"];
// Grab collection of existing Milestones.
var existingMilestones = targetFeature["Milestones"];
long targetOID = targetFeature["ObjectID"];
// Milestones collection on object is expected to be a System.Collections.ArrayList.
var targetMLArray = existingMilestones;
var tagList2 = targetMLArray["_tagsNameArray"];
tagList2.Add(targetML);//
//targetMLArray.Add(targetML);
targetMLArray["_tagsNameArray"] = tagList2;
toUpdate["Milestones"] = targetMLArray;
OperationResult updateResult = restApi.Update(res._ref, toUpdate);
bool resp = updateResult.Success;
}
}
}
//toUpdate["c_AcceptanceCriteria"] = AccCri;
//OperationResult updateResult = restApi.Update(res._ref, toUpdate);
}
}
var features = TCSResults.Results.Where(p => p.Name == fea).FirstOrDefault();
var featuresref = features._ref;
return features;
}

Now that v3.1.1 of the toolkit has been released you can use the AddToCollection method to do this.
Otherwise, you can still always just update the full collection. The value should be an arraylist of objects with _ref properties.
Check out this example (which adds tasks to defects, but should be very similar to what you're doing): https://github.com/RallyCommunity/rally-dot-net-rest-apps/blob/master/UpdateTaskCollectionOnDefect/addTaskOnDefect.cs

Nested XML from list with needed data on multiple lines

I need to format an XML to a given hierarchy from a list (List<>). The list contains, for the banking information, data spread across multiple rows as shown in the image.
The XML output needs to be formatted like this:
<ROOT>
<DocumentElement>
<Supplier>
<Company>STV</Company>
<Code>000199</Code>
<Name>TrafTrans</Name>
<BankAccounts>
<SupplierBankAccount>
<Bban>220-012510-63</Bban>
<Name>B1</Name>
</SupplierBankAccount>
<SupplierBankAccount>
<Bban>RUIL</Bban>
<Name>RUIL</Name>
</SupplierBankAccount>
</BankAccounts>
<SupplierAddresses>
<SupplierAddress>
<Type>PostalAddress</Type>
<Name>Loc TrafTrans</Name>
<IsDefault>true</IsDefault>
<AddressParts>
<SupplierAddressPart>
<AddressPartKey>STREET_NAME</AddressPartKey>
<AddressPartText>Somewhere</AddressPartText>
</SupplierAddressPart>
<SupplierAddressPart>
<AddressPartKey>COUNTRY</AddressPartKey>
<AddressPartText>SPAIN</AddressPartText>
</SupplierAddressPart>
</AddressParts>
</SupplierAddress>
</SupplierAddresses>
</Supplier>
</DocumentElement>
</ROOT>
I already have a method that converts a list to an XML and returns a string. But the problem is that this only formats one item from the list and there could be additional info in the following items.
public static string SuppliersToXML(List<SupplierItem> supplier)
{
CultureInfo ci = new CultureInfo("en-US");
XmlDocument doc = new XmlDocument();
var root = doc.CreateElement("ROOT");
var rootNode = doc.AppendChild(root);
var docElem = doc.CreateElement("DocumentElement");
var docElemNode = rootNode.AppendChild(docElem);
foreach (var item in supplier)
{
var supplierElem = doc.CreateElement("Supplier");
var companyElem = (XmlNode)doc.CreateElement("Company");
companyElem.InnerText = item.Company.ToString();
//more code...
supplierElem.AppendChild(companyElem);
//more code...
}
return doc.OuterXml;
}

I have found the answer myself, it may not be the prettiest code ever written. But it does the job.
public static string SuppliersToXML(List<SupplierItem> supplier)
{
//A distinct select is needed because bank info can be on multiple lines. So first a list is generated with correct info except for bank information
List<SupplierItem> distinctsupplier = supplier
.GroupBy(p => p.Code)
.Select(g => g.First())
.ToList();
CultureInfo ci = new CultureInfo("en-US");
XmlDocument doc = new XmlDocument();
var root = doc.CreateElement("ROOT");
var rootNode = doc.AppendChild(root);
var docElem = doc.CreateElement("DocumentElement");
var docElemNode = rootNode.AppendChild(docElem);
foreach (var item in distinctsupplier)
{
var supplierElem = doc.CreateElement("Supplier");
var companyElem = (XmlNode)doc.CreateElement("Company");
companyElem.InnerText = item.Company.ToString();
var codeElem = (XmlNode)doc.CreateElement("Code");
codeElem.InnerText = item.Code.ToString();
var codeName = (XmlNode)doc.CreateElement("Name");
codeName.InnerText = item.Name.ToString();
//supplieridentifier part
var supplierIdsElem = doc.CreateElement("SupplierIdentifiers");
var supplierIdElem = doc.CreateElement("SupplierIdentifier");
var supplierIdValueElem = (XmlNode)doc.CreateElement("SupplierIDValue");
supplierIdValueElem.InnerText = item.SupplierIDValue;
//supplieraddress part
var supplierAddressesElem = doc.CreateElement("SupplierAddresses");
var supplierAddressElem = doc.CreateElement("SupplierAddress");
var supplierTypeValueElem = (XmlNode)doc.CreateElement("Type");
supplierTypeValueElem.InnerText = item.Type;
var supplierNameValueElem = (XmlNode)doc.CreateElement("Name");
supplierNameValueElem.InnerText = item.AddressName;
var supplierDefaultValueElem = (XmlNode)doc.CreateElement("IsDefault");
supplierDefaultValueElem.InnerText = item.AddressIsDefault;
//address part
var AddressPartElem = doc.CreateElement("AddressParts");
var supplierAddressPartsElem = doc.CreateElement("SupplierAddressPart");
//Street
var AddressPartElemStreetKeyElem = (XmlNode)doc.CreateElement("AddressPartKey");
AddressPartElemStreetKeyElem.InnerText = item.AddressStreetKey;
var AddressPartElemStreetValueElem = (XmlNode)doc.CreateElement("AddressPartText");
AddressPartElemStreetValueElem.InnerText = item.AddressStreetText;
//Country
var AddressPartElemCountryKeyElem = (XmlNode)doc.CreateElement("AddressPartKey");
AddressPartElemCountryKeyElem.InnerText = item.AddressCountryKey;
var AddressPartElemCountryValueElem = (XmlNode)doc.CreateElement("AddressPartText");
AddressPartElemCountryValueElem.InnerText = item.AddressCountryText;
//add elements to supplierelem
supplierElem.AppendChild(companyElem);
supplierElem.AppendChild(codeElem);
supplierElem.AppendChild(codeName);
//bankaccounts part
var bankAccountElem = doc.CreateElement("BankAccounts");
//select all rows that contain multiple lines, so the bank info can be extracted, for a certain supplier
var bankInformation = supplier.Where(s => s.Code == item.Code).ToList();
foreach (var bankitem in bankInformation)
{
if (item.Code == bankitem.Code)
{
var bankAccountSupplElem = doc.CreateElement("SupplierBankAccount");
var bankAccountBbanElem = doc.CreateElement("Bban");
var bankAccountBbanValueElem = (XmlNode)doc.CreateElement("Bban");
bankAccountBbanValueElem.InnerText = bankitem.Bban;
var bankAccountNameElem = doc.CreateElement("Name");
var bankAccountNameValueElem = (XmlNode)doc.CreateElement("Name");
bankAccountNameValueElem.InnerText = bankitem.BankName;
var bankAccountElemNode = supplierElem.AppendChild(bankAccountElem);
var bankAccountSupplElemNode = bankAccountElemNode.AppendChild(bankAccountSupplElem);
bankAccountSupplElemNode.AppendChild(bankAccountBbanValueElem);
bankAccountSupplElemNode.AppendChild(bankAccountNameValueElem);
}
}
//add address elements to supplierelem
var supplierAddressesElemNode = supplierElem.AppendChild(supplierAddressesElem);
var supplierAddressElemNode = supplierAddressesElemNode.AppendChild(supplierAddressElem);
supplierAddressElemNode.AppendChild(supplierTypeValueElem);
supplierAddressElemNode.AppendChild(supplierNameValueElem);
supplierAddressElemNode.AppendChild(supplierDefaultValueElem);
//add addressparts to supplieraddressesnode
//Street
var AddressPartElemNode = supplierAddressElemNode.AppendChild(AddressPartElem);
var supplierAddressPartsElemNode = AddressPartElemNode.AppendChild(supplierAddressPartsElem);
supplierAddressPartsElemNode.AppendChild(AddressPartElemStreetKeyElem);
supplierAddressPartsElemNode.AppendChild(AddressPartElemStreetValueElem);
//Country (first reinitialize supplieraddresspart)
supplierAddressPartsElem = doc.CreateElement("SupplierAddressPart");
var supplierAddressPartsCountryElemNode = AddressPartElemNode.AppendChild(supplierAddressPartsElem);
supplierAddressPartsCountryElemNode.AppendChild(AddressPartElemCountryKeyElem);
supplierAddressPartsCountryElemNode.AppendChild(AddressPartElemCountryValueElem);
//add all supplierelements to docelemnode
docElemNode.AppendChild(supplierElem);
}
return doc.OuterXml;
}

Import data into Quickbooks General Journal Entries

I try to import data from my POS System to Quickbooks to create General Journal entries. When i add some static(hard coded) data all saved successfully. But when I try to add data from my app QB add data to dataService but when i sync QB data didn't come. Please could you help with my problem. Here is my code.
//main code to add General Journal Entry to QB
var header = GenerateReportHeader(model);
var creditInfo = GenerateJournalEntryLines(model.CreditInformation, PostingTypeEnum.Credit);
var debitInfo = GenerateJournalEntryLines(model.DebitInformation, PostingTypeEnum.Debit);
var allLines = creditInfo.Concat(debitInfo).ToArray();
var result = new JournalEntry();
result.Header = header;
result.Line = allLines;
dataService.Add(result);
//Add Header
private JournalEntryHeader GenerateReportHeader(GeneralJournalModel model)
{
var result = new JournalEntryHeader
{
TxnDate = new DateTime(2013,7,1),
Status = "Payable",
Adjustment = false,
TxnDateSpecified = true
};
if (!String.IsNullOrEmpty(model.EntryNo))
{
result.DocNumber = model.EntryNo;
}
return result;
}
//Add Line
private JournalEntryLine[] GenerateJournalEntryLines(List<GeneralJournalEntryModel> model, PostingTypeEnum postType)
{
var result = new JournalEntryLine[model.Count];
for (int i = 0; i < model.Count; i++)
{
var journalEntryLine = model[i];
var account = GetAccountByNumber(journalEntryLine.AccountNumber);
var toAdd = new JournalEntryLine
{
AccountId = account.Id,
AccountType = account.Type,
AccountName = account.Name,
Amount = Convert.ToDecimal(journalEntryLine.Amount),
AmountSpecified = true,
PostingType = postType,
AccountTypeSpecified = true,
Id = new IdType(),
PostingTypeSpecified = true
};
if (journalEntryLine.EntityId != null)
{
toAdd.EntityId = GetEntityId(journalEntryLine.EntityType, journalEntryLine.EntityId);
toAdd.EntityType = GetEntityType(journalEntryLine.EntityType);
toAdd.EntityName = GetEntityName(journalEntryLine.EntityType, journalEntryLine.EntityId);
toAdd.EntityTypeSpecified = true;
}
result[i] = toAdd;
}
return result;
}

Did you check the SyncStatus, and find out why not?
This should be where you start:
https://developer.intuit.com/docs/0025_quickbooksapi/0050_data_services/v2/0500_quickbooks_windows/0600_object_reference/syncstatus
It will give you more detail about specifically why the data failed to sync over.