Rewriting foreach using IObservable and Reactive Framework - c#

I'm in VS2008 with Entity Framework. I'm accessing objects from the database using esql for WHERE IN functionality. I'm passing a ton of IDs to the select statement so I chunk it up into sets of 800. Then I merge the results together from each chunk. My goal is to obtain results for each chunk in parallel, rather than waiting synchronously. I installed Reactive Framework and am pretty sure I need to make use of ForkJoin. However, I can't figure out how to convert this function to use it. Here's my existing code:
public static IList<TElement> SelectWhereIn<TElement, TValue>(this ObjectContext context, string fieldName, IList<TValue> idList)
{
var chunkedIds = idList.Split(CHUNK_SIZE);
string entitySetName = typeof(TElement).Name + "Set";
var retList = new List<TElement>();
foreach (var idChunk in chunkedIds)
{
string delimChunk = string.Join(",", idChunk.Select(x => x.ToString()).ToArray());
ObjectQuery<TElement> query = context.CreateQuery<TElement>("SELECT VALUE x FROM " + entitySetName + " AS x");
query = query.Where("it." + fieldName + " IN {" + delimChunk + "}");
retList.AddRange(query);
}
return retList;
}
Thanks!
EDIT >>>
I modified the code to use Poor Man's as below:
public static IList<TElement> SelectWhereIn<TElement, TValue>(this ObjectContext context, string fieldName, IList<TValue> idList)
{
var chunkedIds = idList.Split(CHUNK_SIZE);
string entitySetName = typeof(TElement).Name + "Set";
var chunkLists = new List<IEnumerable<TElement>>();
Parallel.ForEach(chunkedIds, idChunk =>
{
string delimChunk = string.Join(",", idChunk.Select(x => x.ToString()).ToArray());
ObjectQuery<TElement> query = context.CreateQuery<TElement>("SELECT VALUE x FROM " + entitySetName + " AS x");
query = query.Where("it." + fieldName + " IN {" + delimChunk + "}");
chunkLists.Add(query.ToList());
});
var retList = new List<TElement>();
foreach (var chunkList in chunkLists)
{
retList.AddRange(chunkList);
}
return retList;
}
It worked great the first time. But the second time I ran it, I got this error:
The connection was not closed. The connection's current state is connecting.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.InvalidOperationException: The connection was not closed. The connection's current state is connecting.
Source Error:
Line 49: foreach (var iAsyncResult in resultList)
Line 50: {
Line 51: del.EndInvoke(iAsyncResult);
Line 52: iAsyncResult.AsyncWaitHandle.Close();
Line 53: }
It's interesting, b/c Emre (the author of the library) has an edit to his original post talking about how he added those lines of code for added safety. Am i using it right? Or was his v1 safer after all?

VS2010 does have that with PLINQ. Using the extensions AsParallel().WithDegreeOfParallelism(nbProcessors) would do what you need.
With VS2008, I've used Poor Man's Parallel.ForEach Iterator by Emre Aydinceren in the past when I was trying to work around a performance bottleneck, try to give it a shot.
EDIT: In reaction to the error you added, it might be a random shot in the dark, but seperate contexts for each thread ? Like so:
Parallel.ForEach(chunkedIds, idChunk =>
{
ObjectContext context = new MyContext(connStr);//depending what's your config
// like, with or w/o conn string
string delimChunk = string.Join(",", idChunk.Select(x => x.ToString()).ToArray());
ObjectQuery<TElement> query = context.CreateQuery<TElement>("SELECT VALUE x FROM " + entitySetName + " AS x");
query = query.Where("it." + fieldName + " IN {" + delimChunk + "}");
chunkLists.Add(query.ToList());
});
You might have to tweak around some things (like take the connextion string from the Context extended to instantiate new Contexts).

Related

address1_latitude and address1_longitude is not getting from RetrieveMultiple

Hi i have tried to get address1_latitude and address1_longitude from CRM using CRM SDK here is mu code
var querybyattribute11 = new QueryByAttribute("account");
querybyattribute11.ColumnSet = new ColumnSet("name", "address1_city", "statuscode", "address1_postalcode", "address1_latitude", "address1_longitude");
querybyattribute11.Attributes.AddRange("name");
querybyattribute11.Values.AddRange("ASSOCIATED COMBUSTION INC");
EntityCollection entities = service.RetrieveMultiple(querybyattribute11);
foreach (Entity item in entities.Entities)
{
// Console.WriteLine("Name: {0}. Id: {1}", role.Name, role.Id);
list += item.Attributes["name"].ToString() + " " + item.Attributes["address1_longitude"] .ToString() + "\n";
}
But I am not geting it item.Attributes["address1_longitude"]
error message is
'The given key was not present in the dictionary.'
It might be because it's null.
Try either one of these 2 options:
item["address1_longitude"] (shouldn't raise exception, it would return null if blank, otherwise the address longitude)
To check if the column exists:
item.Attributes.ContainsKey("address1_longitude")

Converting a single string to Array results in {"Non-static method requires a target."}

I am converting a single string into a string[] but I can getting the {"Non-static method requires a target."} error message. I did a search on the web and there is many cases that people have received that error but non seem to be in line with the situation I have.
Here is the code that I receive the error at:
string[] techEmail = { db.Users.Single(u => u.NTUserName.Equals(ticket.Technician.NTUserName)).EmailAddress.ToString() };
I even tried without the .ToString() and also tried to use this method:
string[] result = " [null,null,'3',null,null,null]".Replace("[","").Replace("]","").Split(',');
Here is the second attempt using the code above:
string[] techEmail = " [" + db.Users.Single(u => u.NTUserName.Equals(ticket.Technician.NTUserName)).EmailAddress.ToString() + "]".Replace("[", "").Replace("]", "").Split(',');
but nothing worked. This seems like something that should be so simple.
You can use List and add your string to it later.
Something like that:
string email = db.Users.Single(u => u.NTUserName.Equals(ticket.Technician.NTUserName)).EmailAddress.ToString();
List<string> techEmail = new List<string>();
techEmail.Add(email);

C# return linq result as a list from a wcf service method then use it in aspx web forms page

I'm a newbie in C#
I'm trying to use a wcf service application which references a data entity model to select a row from a table.Searching s.o. I found a way to return linq query results as a list though I haven't found a way yet to use the list in my aspx web forms page.I don't know how to load the list in the aspx page and so far my research in msdn hasn't helped me.
I tried to express my problem the best way I could so you would understand, here is my code:
the wcf service application code:
public List<string> getAccountInfo(int uid)
{
List<string> result = new List<string>();
try
{
using (paragon_db_Models.user_accounts_Model context = new paragon_db_Models.user_accounts_Model())
{
var query = from uacc in context.user_accounts
where uacc.user_account_id == uid
select uacc;
foreach (var c in query)
{
string row = c.user_account_id + ";" + c.order_id + ";" + c.order_state + ";" + c.estimated_cost + ";" + c.instance_form + ";" + c.time_scedule + ";" + c.invoice + ";" + c.notification + ";" + c.user_account_type + ";" + c.username + ";" + c.password;
result.Add(row);
}
}
return result;
}
catch (Exception)
{
return result;
}
}
the aspx.cs code
protected void Page_Load(object sender, EventArgs e)
{
accountInfo_Ref.IaccountInfoSrvcClient accInfoClient = new accountInfo_Ref.IaccountInfoSrvcClient();
int id = (int)Session["UserId"];
List<string> columns = new List<string>(accInfoClient.getAccountInfo(id));
id_lbl.Text = columns[0];
order_id_lbl.Text = columns[1];
}
The service works fine. I'm also open to suggestions to better ways to do this.
Or you could not return strings from your service, you could return a list of objects.
If you MUST return strings you must split the serialized data using the String.Split method but this is really a poor method.
If you MUST return strings you could at least use a better serialization strategy such as JSON or XML.
But really consider changing your service interface.
Now let's get back to using your results:
you are using a service to return a List of records with that ID
i assume that the ID is unique and you will allways get 0 or 1 results in your list
each "row" added to the list contains the information about an account
so each row must be splitted to get the information
Code:
protected void Page_Load(object sender, EventArgs e)
{
accountInfo_Ref.IaccountInfoSrvcClient accInfoClient = new accountInfo_Ref.IaccountInfoSrvcClient();
int id = (int)Session["UserId"];
List<string> rows = new List<string>(accInfoClient.getAccountInfo(id));
// display the first row
string row = rows.FirstOrDefault();
if (String.IsNullOrEmpty(row))
{
// record cannot be found
}
else
{
string[] details = row.Split(';');
id_lbl.Text = details[0];
order_id_lbl.Text = details[1];
}
}
the method is returning a List<string>.You just have to store it in an instance of List<string>.
You have to do like this:
List<string> columns = accInfoClient.getAccountInfo(id);
UPDATE:
it is returning an array as you said in comments:
string[] columns = accInfoClient.getAccountInfo(id);
or use implict variable:
var columns = accInfoClient.getAccountInfo(id);
If i understood your question properly Simply write like this
List<string> columns = accInfoClient.getAccountInfo(id).ToList<string>();
You will get list filled with data.

LINQ results change at end of for loop

When performing a set of LINQ queries against a data-source (I'm using LINQ-to-SQL, but it happens here too using just a List<string> object), I end up getting a different result at the end of my checks.
Specifically, the code below is trying to find if a Fully Qualified Domain Name (FQDN) either exists in a list of host names (not all of which will be FQDNs or in the same domain, but the host identifier is what matters to me). The search is trying to find whether "host-6.domain.local" or any of its sub-components exist (i.e, "host-6.domain" or "host-6") in the list, which they do not. While inside the for-loop, we get the results we expect, but as soon as the for loop is finished, I get a result that has all of the contents of the list, which to me sounds like it is trying to find elements that match the empty string.
void MyMethod()
{
string fqdn = "host-6.domain.local";
string[] splitFqdn = fqdn.Split('.');
List<string> values = new List<string>();
values.add("host-1");
values.add("host-2.domain.local");
values.add("host-3.domain.local");
values.add("host-4");
values.add("host-5.other.local");
IEnumerable<string> queryResult = null;
for (int i = splitFqdn.Length; i > 0; i--)
{
result =
from value in values
where value.StartsWith(
string.Join(".", splitFqdn.Take(i)))
select value;
Console.WriteLine(
"Inside for loop, take " + i + ": " + result.Count());
}
Console.WriteLine();
Console.WriteLine(
"Outside for loop: " + result.Count());
}
Why is this happening and how can I get accurate results that I can still access after the for loop is finished?
You are getting bitten by LINQ's lazy execution and closure.
When you create an enumerable like you are doing here...
result =
from value in values
where value.StartsWith(
string.Join(".", splitFqdn.Take(i)))
select value;
It doesn't get evaluated until you do something that forces it to get evaluated... for instance when you do result.count()
Then later outside of your loop when you evaluate it again result.count() is evaluated with the last value of i that existed in your for loop which is not giving you what you want.
Try forcing evaluation by doing .ToList() on your enumerable like so... This code shows both values so you can compare.
void MyMethod()
{
string fqdn = "host-6.domain.local";
string[] splitFqdn = fqdn.Split('.');
List<string> values = new List<string>();
values.add("host-1");
values.add("host-2.domain.local");
values.add("host-3.domain.local");
values.add("host-4");
values.add("host-5.other.local");
IEnumerable<string> queryResult = null;
List<string> correctResult = null;
for (int i = splitFqdn.Length; i > 0; i--)
{
queryResult =
from value in values
where value.StartsWith(
string.Join(".", splitFqdn.Take(i)))
select value;
correctResult = queryResult.ToList();
Console.WriteLine(
"Inside for loop, take " + i + ": " + queryResult.Count());
}
Console.WriteLine();
Console.WriteLine(
"Outside for loop queryResult: " + queryResult.Count());
Console.WriteLine(
"Outside for loop correctResult: " + correctResult.Count());
}
EDIT: Thanks nlips for pointing out that I hadn't fully answered the question... and apologies for converting to method syntax but it would have taken longer to convert to query syntax.
void MyMethod()
{
string fqdn = "host-6.domain.local";
string[] splitFqdn = fqdn.Split('.');
List<string> values = new List<string>();
values.Add("host-1");
values.Add("host-2.domain.local");
values.Add("host-3.domain.local");
values.Add("host-4");
values.Add("host-5.other.local");
values.Add("host-5.other.local");
IEnumerable<string> queryResult = null;
List<string> correctResult = new List<string>();
for (int i = splitFqdn.Length; i > 0; i--)
{
correctResult = correctResult
.Union(values.Where(
value => value.StartsWith(string.Join(".", splitFqdn.Take(i)))))
.ToList();
}
}
I really like Kevin's answer to my question, but I wasn't a huge fan of calling .ToList() on the result since this would cause all of the objects that matched to be pulled from the database (eating up more memory) rather than executing a query that simply got the count of matching objects (which is a little faster and doesn't take the memory to store the objects), so using the information from his post, I have this additional solution that doesn't require pulling all objects from a database, and only runs a COUNT query (in the SQL sense).
To avoid the issue caused by capturing i which then becomes 0 at the end of the for-loop, I simply set up a temporary variable to hold the value I'm searching for.
void MyMethod()
{
string fqdn = "host-6.domain.local";
string[] splitFqdn = fqdn.Split('.');
List<string> values = new List<string>();
values.add("host-1");
values.add("host-2.domain.local");
values.add("host-3.domain.local");
values.add("host-4");
values.add("host-5.other.local");
IEnumerable<string> queryResult = null;
for (int i = splitFqdn.Length; i > 0; i--)
{
//taking the line referencing i out of the
//query expression prevents referencing i
//after it is set to 0 outside the for loop
string temp = string.Join(".", splitFqdn.Take(i));
//since temp isn't changed anywhere else, it won't
//get set to an invalid value after the loop exits
result =
from value in values
where value.StartsWith(temp)
select value;
Console.WriteLine(
"Inside for loop, take " + i + ": " + result.Count());
}
Console.WriteLine();
Console.WriteLine(
"Outside for loop: " + result.Count());
}
I think you need to call ToList when assigning to the result variable like this:
result =
(from value in values
where value.StartsWith(
string.Join(".", splitFqdn.Take(i)))
select value).ToList();

Unreliable parallel loop fails 4 out of 400 times

I have a Parallel foreach function that creates a new instance of a class, that manipulates a picture, and saves it to the disk...
However approximately 4 times out of 400, the picture gets saved to the disk, but without being manipulated, my theory is that when it happens, some of the propperties existing in my class is null, when they are not suppost to...
The 4 (sometimes 3) errors mostly occurs in the first 10 images of the parallel loop.
There is no error message, it justs skip some of my code, for some reason... My breakpoint doesn't work when it is parralel, so it is hard to debug.
Any advice on how to proceed / debug / fix ?
The code as requested
private static void GenerateIcons(Effects effect)
{
DirectoryInfo dir = new DirectoryInfo(HttpContext.Current.Server.MapPath(#"~\Icons\Original\"));
FileInfo[] ff = dir.GetFiles();
string mappath = HttpContext.Current.Server.MapPath(#"~\Icons\");
List<string> paths = new List<string>();
string ids = GetAllEffectIds(effect.TinyUrlCode);
Parallel.ForEach(ff, item =>
{
if (!File.Exists(mappath + #"Generated\" + ids + "-" + item.Name))
{
paths.Add(mappath + #"Generated\" + ids + "-" + item.Name);
ApplyEffects f = new ApplyEffects(effect, item.Name, mappath);
f.SaveIcon();
}
});
//Zip icons!
ZipFiles(paths, effect.TinyUrlCode, ids, effect.General.Prefix);
}
You can re-write it in a more functional style, to hopefully remove the threading issues:
private static void GenerateIcons(Effects effect)
{
var dir = new DirectoryInfo(HttpContext.Current.Server.MapPath(#"~\Icons\Original\"));
var mappath = HttpContext.Current.Server.MapPath(#"~\Icons\");
var ids = GetAllEffectIds(effect.TinyUrlCode);
var filesToProcess = dir
.EnumerateFiles()
.AsParallel()
.Select(f => new { info = f, generated = File.Exists(mappath + #"Generated\" + ids + "-" + f.Name) })
.ToList();
Parallel.ForEach(filesToProcess.Where(f => !f.generated), file =>
{
new ApplyEffects(effect, file.info.Name, mappath).SaveIcon();
});
//Zip icons!
ZipFiles(filesToProcess.Select(f => f.info), effect.TinyUrlCode, ids, effect.General.Prefix);
}
My theory is that your list of paths is not being updated properly due to List<T> not being thread safe. Essentially if two threads try and add an item to the list at the same time any number of weird things could happen, like 4 items missing from the resulting list. Try using the lock statement.
Parallel.ForEach(ff, item =>
{
if (!File.Exists(mappath + #"Generated\" + ids + "-" + item.Name))
{
lock(paths)
{
paths.Add(mappath + #"Generated\" + ids + "-" + item.Name);
}
ApplyEffects f = new ApplyEffects(effect, item.Name, mappath);
f.SaveIcon();
}
});
Have you checked with the none parallel version?
Are you using any
API functions that is not marked as thread safe?
To answer 1), mutex lock the entire function and test for errors.
To answer 2) reduce the amount of code in the mutex until you find the offending function. You can do this by bisection.
ConcurrentBag<T> Is a thread safe container.

Categories