We Keep Coding

Faster insert sqlite3 processing

I have this function where it opens a .TXT file with some products and insert line by line on the sqlitedb. The process is working fine, the problem is. This file contains 2000+ lines, because of that, the process is taking several hours to finish. I wonder if there is a way to make the process a little bit faster.
here is the function:
private void carrega_produtos()
{
var assembly = typeof(sincroniza_page).GetTypeInfo().Assembly;
foreach (var res in assembly.GetManifestResourceNames())
{
if (res.Contains("produtos.txt"))
{
Stream stream = assembly.GetManifestResourceStream(res);
var st = res.Count();
using (var reader = new StreamReader(stream))
{
string linha;
acesso_banco_produtos banco = new acesso_banco_produtos();
while ((linha = reader.ReadLine()) != null)
{
List<string> lista = linha.Split(new char[] { '§' }).ToList();
var cod = int.Parse(lista.ElementAt(0));
var nome_prod = lista.ElementAt(1);
var cod_grupo = lista.ElementAt(2);
var nm_grupo = lista.ElementAt(3);
var ind_ativo = lista.ElementAt(4);
var val_custo_unit = lista.ElementAt(5);
var val_custo = lista.ElementAt(6);
var perc_imposto = lista.ElementAt(7);
var unidade_med = lista.ElementAt(8);
var qtd_mes_1 = lista.ElementAt(9);
var qtd_mes_2 = lista.ElementAt(10);
var qtd_mes_3 = lista.ElementAt(11);
var qtd_mes_6 = lista.ElementAt(12);
var qtd_mes_12 = lista.ElementAt(13);
var data = lista.ElementAt(14);
var bd = new banco_produtos()
{
cod_produto = cod,
nm_produto = nome_prod,
cod_grupo = cod_grupo,
nm_grupo = nm_grupo,
ind_ativo = ind_ativo,
val_custo_unitario = Double.Parse(val_custo_unit),
val_lista_preco = val_custo,
perc_impostos = perc_imposto,
unidade_medida = unidade_med,
qtde_vendida_mes_1 = qtd_mes_1,
qtde_vendida_mes_2 = qtd_mes_2,
qtde_vendida_mes_3 = qtd_mes_3,
qtde_vendida_mes_6 = qtd_mes_6,
qtde_vendida_mes_12 = qtd_mes_12
};
//here i look in the DB if already exists the new product
var procura = banco.get_produto(cod);
if (procura == null)
{
// here is inserted to the db
banco.inserir_produto(bd);
}
}
valor += 1;
}
}
}
}

I'm not sure what is inside your method which inserts data into db but the most common issue with SQLite and massive inserts is the fact that SQLite by default wraps every insert with transaction which creates significant overhead. A good practice for such cases is to make signle transaction for all insterts which should singificantly improve the perfomance, see the example.

I did what #Dmytro said, I used the method "insertORIgnore". It improved a lot using that method.thank you for the help.

Perform Multiple Posts WIth Parallel.ForEach

Here is my syntax, but I keep have a compile error of on my line Parallel.ForEach()
System.Data.DataRow is a type but is used like a variable
which I am sure is something simple that I am just overlooking. Below is my full syntax, if someone could assist me with what exactly I am missing, I will greatly appreciate it!
private void TryParallel()
{
Dictionary<string, string> dic = new Dictionary<string, string>();
string strEndpointURL = string.Format("http://sitetosenddatato.com/post");
SqlDataReader reader;
string strPostData = "";
string strMessage = "";
DataSet grds = new DataSet();
grds = GetSQLResults();
if (grds.Tables[0].Rows.Count >= 1)
{
Parallel.ForEach(DataRow, grds.Tables[0].Rows =>
{
dic.Add("userID", reader.GetValue(0).ToString());
dic.Add("name", reader.GetValue(1).ToString());
dic.Add("address", reader.GetValue(2).ToString());
dic.Add("city", reader.GetValue(3).ToString());
dic.Add("state", reader.GetValue(4).ToString());
dic.Add("zip", reader.GetValue(5).ToString());
dic.Add("Phone", reader.GetValue(6).ToString());
});
}
System.Web.Script.Serialization.JavaScriptSerializer json = new System.Web.Script.Serialization.JavaScriptSerializer();
foreach (var d in dic) { strPostData += d.Key + "=" + Server.UrlEncode(d.Value) + "&"; }
strPostData += "hs_context=";
S ystem.Net.HttpWebRequest r = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(strEndpointURL);
r.Method = "POST";
r.Accept = "application/json";
r.ContentType = "application/x-www-form-urlencoded";
r.ContentLength = strPostData.Length;
r.KeepAlive = false;
using (System.IO.StreamWriter sw = new System.IO.StreamWriter(r.GetRequestStream()))
{
try { sw.Write(strPostData); }
catch (Exception ex) { strMessage = ex.Message; }
}
var response = r.GetResponse();
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
var result = readStream.ReadToEnd();
var xml = System.Xml.Linq.XElement.Parse(result);
if (xml.Elements("success").FirstOrDefault().Value == "1") { strMessage = "Success"; }
else
{
var errors = xml.Elements("errors");
foreach (var error in errors.Elements("error")) { strMessage = error.Value; }
}
}
EDIT
Following the example outlined below by #Glen Thomas - I altered my code to
if (grds.Tables[0].Rows.Count == 1)
{
Parallel.ForEach(rows, row =>
{
dic.Add("userID", reader.GetValue(0).ToString());
//More Here
}
}
which presents a compile error of:
Use of unassigned local variable 'reader'
But I have reader declared at the top of my method?

You are specifying a type name as the first parameter. This should be the collection you are iterating. The second parameter is a function to perform, with a parameter for each element in the collection.
The correct usage of Parallel.ForEach is like this:
var rows = new DataRow[0]
Parallel.ForEach(rows, row =>
{
// Do something with row here
});
For your code:
Parallel.ForEach(grds.Tables[0].Rows.OfType<DataRow>(), row =>
{
dic.Add("userID", reader.GetValue(0).ToString());
dic.Add("name", reader.GetValue(1).ToString());
dic.Add("address", reader.GetValue(2).ToString());
dic.Add("city", reader.GetValue(3).ToString());
dic.Add("state", reader.GetValue(4).ToString());
dic.Add("zip", reader.GetValue(5).ToString());
dic.Add("Phone", reader.GetValue(6).ToString());
});

I believe you want to do this instead:
Parallel.ForEach(grds.Tables[0].Rows.OfType<DataRow>(), (row) =>
{
dic.Add("userID", reader.GetValue(0).ToString());
dic.Add("name", reader.GetValue(1).ToString());
dic.Add("address", reader.GetValue(2).ToString());
dic.Add("city", reader.GetValue(3).ToString());
dic.Add("state", reader.GetValue(4).ToString());
dic.Add("zip", reader.GetValue(5).ToString());
dic.Add("Phone", reader.GetValue(6).ToString());
//though realistically you should be doing something with your specific row
});
The answer is in the error message you recieved - DataRow is not defined as an object in the code you provided.
However, this isn't even actually solving your actual problem which I believe is performing multiple HTTP posts in parallel - so you'd need to put your post logic within the anonymous function of your Parallel.ForEach()

Read text file from specific position and store in two arrays

I have text file which contains line like this:
#relation SMILEfeatures
#attribute pcm_LOGenergy_sma_range numeric
#attribute pcm_LOGenergy_sma_maxPos numeric
#attribute pcm_LOGenergy_sma_minPos numeric...
Where are about 6000 lines of these attributes, after attributes where are lines like this:
#data
1.283827e+01,3.800000e+01,2.000000e+00,5.331364e+00
1.850000e+02,4.054457e+01,4.500000e+01,3.200000e+01...
I need to seperate these strings in two different arrays. So far I only managed to store everything in one array.
Here is my code for storing in array:
using (var stream = new FileStream(filePath, FileMode.OpenOrCreate))
{
using (var sr = new StreamReader(stream))
{
String line;
while ((line = sr.ReadLine()) != null)
{
sb.AppendLine(line);
}
}
string allines = sb.ToString();
Console.WriteLine(sb);
}

All strings after #relation SMILEfeatures and contains #attribute are stored in first array. All the strings after #data should are stored in the second array. Hope this is what you wanted.
var relationLineNumbers = new List<int>();
var dataLineNumbers = new List<int>();
var relation = new StringBuilder();
var data = new List<string>();
using (var stream = new FileStream(filepath, FileMode.OpenOrCreate))
{
using (var sr = new StreamReader(stream))
{
string line;
bool isRelation = false;
bool isData = false;
int lineNumber = 0;
while ((line = sr.ReadLine()) != null)
{
lineNumber++;
if (line.StartsWith("#relation SMILEfeatures"))
{
isRelation = true;
isData = false;
continue;
}
if (line.StartsWith("#data"))
{
isData = true;
isRelation = false;
continue;
}
if (isRelation)
{
if (line.StartsWith("#attribute"))
{
relation.Append(line);
relationLineNumbers.Add(lineNumber);
}
}
if (isData)
{
data.AddRange(line.Split(','));
dataLineNumbers.Add(lineNumber);
}
}
}
Console.WriteLine("Relation");
Console.WriteLine(relation.ToString());
Console.WriteLine("Data");
data.ForEach(Console.WriteLine);

All strings which starts with #relation SMILEfeatures and contains #attribute should be stored in first array. Numbers which starts with #data should be stored in second array.
Use string.Contains() and string.StatsWith() for checking.
Read every line and decide in wich array / list you want to put this line
void ReadAndSortInArrays(string fileLocation)
{
List<string> noData = new List<string>();
List<string> Data = new List<string>();
using(StreamReader sr = new StreamReader(fileLocation))
{
string line;
while(!sr.EndOfStream)
{
line = sr.ReadLine();
if(line.StartsWith("#relation") && line.Contains("#attribute"))
{
noData.Add(line);
}
else if(line.StartsWith("#data")
{
Data.Add(line);
}
else
{
// This is stange
}
}
}
var noDataArray = noData.ToArray();
var DataArray = Data.ToArray();
}
But i think that not every line is beginning with "#data"
So you may want to Read all lines and do somethink like this:
string allLines;
using(StreamReader sr = new StreamReader(yourfile))
{
allLines = = sr.ReadToEnd();
}
var arrays = allLines.Split("#data");
// arrays[0] is the part before #data
// arrays[1] is the part after #data (the numbers)
// But array[1] does not contain #data

The question is not really very clear. But my take is, collect all lines that start with #relation or #attribute in one bucket, then collect all number lines in another bucket. I have chosen to ignore the #data lines, as they do not seem to contain any extra information.
Error checking may be performed by making sure that the data lines (i.e. number lines) contain comma separated lists of parsable numerical values.
var dataLines = new List<string>();
var relAttLines = new List<string>();
foreach (var line in File.ReadAllLines())
{
if (line.StartsWith("#relation") || line.StartsWith("#attribute"))
relAttLines.Add(line);
else if (line.StartsWith("#data"))
//ignore these
continue;
else
dataLines.Add(line);
}

change string into var datatype in c#

I am reading data from file like this
using (StreamReader r = new StreamReader("newfile.txt"))
{
string lines1;
lines1 = r.ReadLine();
var lines = lines1;
foreach (string line in lines)
{
var dictionary = new Dictionary<String, Int32>();
var records = line.Split(',');
How can i convert string data type to var data type?
if i used this line
var lines = lines1;
then the error accur at
var records = line.Split(',');

There is no such thing as a var datatype. "var", in C#, just means "let the compiler figure out the data type for me based on the context". in this case, using:
var lines = lines1;
or:
string lines = lines1;
Will produce the exact same code. The error message is not related to the usage of var instead of string.
The problem here is this:
lines1 = r.ReadLine();
var lines = lines1;
// ...
foreach (string line in lines)
You're reading a single line as a single string, then trying to do: "foreach string in my single string". I suspect you want something more like:
using (StreamReader r = new StreamReader("newfile.txt"))
{
while (r.Peek() >= 0)
{
string line = r.ReadLine();
var dictionary = new Dictionary<String, Int32>();
var records = line.Split(',');
// use records.... Note that above is the same as:
// string[] records = line.Split(',');
}
}

The "split" function returns an array.

C#: What's an efficient way of parsing a string with one delimiter through ReadLine() of TextReader?

C#: What's an efficient way to parse a string with one delimiter for each ReadLine() of TextReader?
My objective is to load a list of proxies to ListView into two columns (Proxy|Port) reading from a .txt file. How would I go upon splitting each readline() into the proxy and port variables with the delimiter ":"?
This is what I've got so far,
public void loadProxies(string FilePath)
{
string Proxy; // example/temporary place holders
int Port; // updated at each readline() loop.
using (TextReader textReader = new StreamReader(FilePath))
{
string Line;
while ((Line = textReader.ReadLine()) != null)
{
// How would I go about directing which string to return whether
// what's to the left of the delimiter : or to the right?
//Proxy = Line.Split(':');
//Port = Line.Split(':');
// listview stuff done here (this part I'm familiar with already)
}
}
}
If not, is there a more efficient way to do this?

string [] parts = line.Split(':');
string proxy = parts[0];
string port = parts[1];

You could split them this way:
string line;
string[] tokens;
while ((Line = textReader.ReadLine()) != null)
{
tokens = line.Split(':');
proxy = tokens[0];
port = tokens[1];
// listview stuff done here (this part I'm familiar with already)
}
it's best practise to use small letter names for variables in C#, as the other ones are reserved for class / namespace names etc.

How about running a Regex on the whole file?
var parts=
Regex.Matches(input, #"(?<left>[^:]*):(?<right>.*)",RegexOptions.Multiline)
.Cast<Match>()
.Where(m=>m.Success)
.Select(m => new
{
left = m.Groups["left"],
right = m.Groups["right"]
});
foreach(var part in parts)
{
//part.left
//part.right
}
Or, if it's too big, why not Linqify the ReadLine operation with yielding method?
static IEnumerable<string> Lines(string filename)
{
using (var sr = new StreamReader(filename))
{
while (!sr.EndOfStream)
{
yield return sr.ReadLine();
}
}
}
And run it like so:
var parts=Lines(filename)
.Select(
line=>Regex.Match(input, #"(?<left>[^:]*):(?<right>.*)")
)
.Where(m=>m.Success)
.Select(m => new
{
left = m.Groups["left"],
right = m.Groups["right"]
});
foreach(var part in parts)
{
//part.left
//part.right
}

In terms of efficiency I expect you'd be hard-pressed to beat:
int index = line.IndexOf(':');
if (index < 0) throw new InvalidOperationException();
Proxy = line.Substring(0, index);
Port = int.Parse(line.Substring(index + 1));
This avoids the array construction / allocation associated with Split, and only looks as far as the first delimited. But I should stress that this is unlikely to be a genuine performance bottleneck unless the data volume is huge, so pretty-much any approach should be fine. In fact, perhaps the most important thing (I've been reminded by the comment below) is to suspend the UI while adding:
myListView.BeginUpdate();
try {
// TODO: add all the items here
} finally {
myListView.EndUpdate();
}

You might want to try something like this.
var items = File.ReadAllText(FilePath)
.Split(new[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries)
.Select(line => line.Split(':'))
.Select(pieces => new {
Proxy = pieces[0],
Port = int.Parse(pieces[1])
});
If you know that you won't have a stray newline at the end of the file you can do this.
var items = File.ReadAllLines(FilePath)
.Select(line => line.Split(':'))
.Select(pieces => new {
Proxy = pieces[0],
Port = Convert.ToInt32(pieces[1])
});