We Keep Coding

Getting Missing Rows inside a table and copy them to another table

var FileProducts = from ProductsRow in ProductRangesDt.AsEnumerable()
join Filee in FileTb.AsEnumerable() on ProductsRow["GEN_CODE"].ToString() equals Filee["GEN_CODE"].ToString()
select new
{
PRODUCT_ID = ProductsRow["PRODUCT_ID"],
PRODUCT_NAME = ProductsRow["PRODUCT_NAME"],
PROVIDER_ID = ProductsRow["PROVIDER_ID"],
PROVIDER_NAME = ProductsRow["PROVIDER_NAME"],
GEN_CODE = ProductsRow["GEN_CODE"],
MIN_QUANTITY = Filee["MIN_QUANTITY"],
MAX_QUANTITY = Filee["MAX_QUANTITY"],
DISCOUNT_VALUE = Filee["DISCOUNT_VALUE"]
};
var s = (from b in FileProducts
select b.PRODUCT_ID).Distinct(); // count=285
var Products = (from ProductsRow in ProductRangesDt.AsEnumerable()
select ProductsRow["PRODUCT_ID"]).Distinct(); // count=7159
var result = Products.Except(s); // it's count should be 7159-285
I want to get all the products ID that are in Products and don't exist in FileProducts how can i do this ? result always return 0 as count

From the MSDN documentation about Except extension method:
This method is implemented by using deferred execution. The immediate
return value is an object that stores all the information that is
required to perform the action. The query represented by this method
is not executed until the object is enumerated either by calling its
GetEnumerator method directly or by using foreach in Visual C# or For
Each in Visual Basic.
So in order to get the real value form your Set differentiation, you need to enumerate your result either by a call to the Count()-Method (result.Count()) on using foreach (foreach (var r in result) { ... }).
I can't test with your data, but with test data at my disposition, the Except-extension did delivered the correct results.

LINQ and creating NON anonymous return values

I think I understand returning records of an anonymous type from But in this I want to create NEW CatalogEntries, and set them from the values selected. (context is a Devart LinqConnect database context, which lets me grab a view).
My solution works, but it seems clumsy. I want to do this in one from statement.
var query = from it in context.Viewbostons
select it;
foreach (GPLContext.Viewboston item in query)
{
CatalogEntry card = new CatalogEntry();
card.idx = item.Idx;
card.product = item.Product;
card.size = (long)item.SizeBytes;
card.date = item.Date.ToString();
card.type = item.Type;
card.classification = item.Classification;
card.distributor = item.Distributor;
card.egplDate = item.EgplDate.ToString();
card.classificationVal = (int)item.ClassificationInt;
card.handling = item.Handling;
card.creator = item.Creator;
card.datum = item.Datum;
card.elevation = (int)item.ElevationFt;
card.description = item.Description;
card.dirLocation = item.DoLocation;
card.bbox = item.Bbox;
card.uniqID = item.UniqId;
values.Add(card);
}
CatalogResults response = new CatalogResults();
I just tried this:
var query2 = from item in context.Viewbostons
select new CatalogResults
{ item.Idx,
item.Product,
(long)item.SizeBytes,
item.Date.ToString(),
item.Type,
item.Classification,
item.Distributor,
item.EgplDate.ToString(),
(int)item.ClassificationInt,
item.Handling,
item.Creator,
item.Datum,
(int)item.ElevationFt,
item.Description,
item.DoLocation,
item.Bbox,
item.UniqId
};
But I get the following error:
Error 79 Cannot initialize type 'CatalogService.CatalogResults' with a
collection initializer because it does not implement
'System.Collections.IEnumerable' C:\Users\ysg4206\Documents\Visual
Studio
2010\Projects\CatalogService\CatalogService\CatalogService.svc.cs 91 25 CatalogService
I should tell you what the definition of the CatalogResults is that I want to return:
[DataContract]
public class CatalogResults
{
CatalogEntry[] _results;
[DataMember]
public CatalogEntry[] results
{
get { return _results; }
set { _results = value; }
}
}
My mind is dull today, apologies to all. You are being helpful. The end result is going to be serialized by WCF to a JSON structure, I need the array wrapped in a object with some information about size, etc.

Since .NET 3.0 you can use object initializer like shown below:
var catalogResults = new CatalogResults
{
results = context.Viewbostons
.Select(it => new CatalogEntry
{
idx = it.Idx,
product = it.Product,
...
})
.ToArray()
};
So if this is only one place where you are using CatalogEntry property setters - make all properties read-only so CatalogEntry will be immutable.
MSDN, Object initializer:
Object initializers let you assign values to any accessible fields or properties of an
object at creation time without having to explicitly invoke a constructor.

The trick here is to create a IQueryable, and then take the FirstOrDefault() value as your response (if you want a single response) or ToArray() (if you want an array). The error you are getting (Error 79 Cannot initialize type 'CatalogService.CatalogResults' with a collection initializer because it does not implement 'System.Collections.IEnumerable') is because you're trying to create an IEnumerable within the CatalogEntry object (by referencing the item variable).
var response = (from item in context.Viewbostons
select new CatalogEntry()
{
idx = item.Idx,
product = item.Product,
size = (long)item.SizeBytes,
...
}).ToArray();

You don't have to create anonymous types in a Linq select. You can specify your real type.
var query = context.Viewbostons.Select( it =>
new CatalogEntry
{
idx = it.idx,
... etc
});

This should work:
var query = from it in context.Viewbostons
select new CatalogEntry()
{
// ...
};

Linq To Data set error System.InvalidCastException: Specified cast is not valid

I get the following exception using Linq to data set "System.InvalidCastException: Specified cast is not valid."
The problem is as follows I have a model with two value of type int?. The values in the database are not required so some fields are blank. I have read the table into a data set and now I need to query the data set using the following code.
//model
public class Model
{
// Public Properties
...
...
...
public int? YearBegin { get; set; }
public int? YearEnd { get; set; }
}
//query
var list = from m in data.Tables["Models"].AsEnumerable()
select new Model
{
// rest of members omitted to simplify
YearBegin = m.Field<int>("YearBegin"),
YearEnd = m.Field<int>("YearEnd")
};
I have tried the following none have worked:
m.Field<int?>("YearBegin")
YearEnd = m.IsNull("YearEnd") ? null, m.Field<int>("YearEnd")
Is there another way to check if the field has a value similar to String.IsNullOrEmpty().
Using string as the type is not a possibility...
Thanks

You aren't using a typed DataSet, so my first question would be is the does the DataTable know that those fields are supposed to be 'int?' in the first place, or are they listed as strings? If the DataTable is treating those fields as strings, you will experience that error. The following code assumes a TestData DataSet with a Models DataRow, with two nullable string columns as YearBegin and YearEnd:
using (TestData ds = new TestData())
{
// Typed Rows
ds.Models.AddModelsRow("1", "2");
ds.Models.AddModelsRow(ds.Models.NewModelsRow()); // NULL INFO TEST
// Untyped rows
DataRow r = ds.Models.NewRow();
r[0] = "4";
r[1] = "5";
ds.Models.Rows.Add(r);
//query
var list = from m in ds.Tables["Models"].AsEnumerable()
select new Model
{
// rest of members omitted to simplify
YearBegin = m.Field<int?>("YearBegin"),
YearEnd = m.Field<int?>("YearEnd"),
};
}
That code will encounter the InvalidCastException. However, when I flip the types on the DataTable to nullable Int32, then the nearly identical code works properly:
using (TestData ds = new TestData())
{
// Typed Rows
ds.Models.AddModelsRow(1, 2);
ds.Models.AddModelsRow(ds.Models.NewModelsRow()); // NULL INFO TEST
// Untyped rows
DataRow r = ds.Models.NewRow();
r[0] = 4;
r[1] = 5;
ds.Models.Rows.Add(r);
//query
var list = from m in ds.Tables["Models"].AsEnumerable()
select new Model
{
// rest of members omitted to simplify
YearBegin = m.Field<int?>("YearBegin"),
YearEnd = m.Field<int?>("YearEnd"),
};
}
Take a look at your DataTable. You can correct your issue there. The Field cast to int? will not work unless your DataTable field matches the int? type.

Problem solved, I am working against a legacy access database and the data type was stored as Integer instead on Long Integer meaning it is represented as an Int16 in the data set hence the Invalid cast exception...

Db4o query: find all objects with ID = {anything in array}

I've stored 30,000 SimpleObjects in my database:
class SimpleObject
{
public int Id { get; set; }
}
I want to run a query on DB4O that finds all SimpleObjects with any of the specified IDs:
public IEnumerable<SimpleObject> GetMatches(int[] matchingIds)
{
// OH NOOOOOOES! This activates all 30,000 SimpleObjects. TOO SLOW!
var query = from SimpleObject simple in db
join id in matchingIds on simple.Id equals id
select simple;
return query.ToArray();
}
How do I write this query so that DB4O doesn't activate all 30,000 objects?

I am not an expert on this, and it might be good to post on the DB4O forums about it, but I think I have a solution. It involves not using LINQ and using SODA.
This is what I did. I created a quick project that populates the database with 30000 SimpleObject based on your post's definition. I then wrote a query to grab all the SimpleObjects from the database:
var simpleObjects = db.Query<SimpleObject>(typeof(SimpleObject));
When I wrapped a StopWatch around it, that run takes about 740 milliseconds. I then used your code to search for a 100 random numbers between 0 and 2999. The response was 772 ms, so based on that number I am assuming that it is pulling all the objects out of the database. I am not sure how to verify that, but later I think I proved it with performance.
I then went lower. From my understanding the LINQ provider from the DB4O team is just doing a translation into SODA. Therefore I figured that I would write a SODA query to test, and what I found was that using SODA against a property is bad for performance because it took 19902 ms to execute. Here is the code:
private SimpleObject[] GetSimpleObjectUsingSodaAgainstAProperty(int[] matchingIds, IObjectContainer db)
{
SimpleObject[] returnValue = new SimpleObject[matchingIds.Length];
for (int counter = 0; counter < matchingIds.Length; counter++)
{
var query = db.Query();
query.Constrain(typeof(SimpleObject));
query.Descend("Id").Constrain(matchingIds[counter]);
IObjectSet queryResult = query.Execute();
if (queryResult.Count == 1)
returnValue[counter] = (SimpleObject)queryResult[0];
}
return returnValue;
}
So thinking about why this would be so bad, I decided to not use an auto-implemented property and define it my self because Properties are actually methods and not values:
public class SimpleObject
{
private int _id;
public int Id {
get
{ return _id; }
set
{ _id = value; }
}
}
I then rewrote the query to use the _id private field instead of the property. The performance was much better at about 91 ms. Here is that code:
private SimpleObject[] GetSimpleObjectUsingSodaAgainstAField(int[] matchingIds, IObjectContainer db)
{
SimpleObject[] returnValue = new SimpleObject[matchingIds.Length];
for (int counter = 0; counter < matchingIds.Length; counter++)
{
var query = db.Query();
query.Constrain(typeof(SimpleObject));
query.Descend("_id").Constrain(matchingIds[counter]);
IObjectSet queryResult = query.Execute();
if (queryResult.Count == 1)
returnValue[counter] = (SimpleObject)queryResult[0];
}
return returnValue;
}
Just to make sure that it is was not a fluke, I ran the test run several times and recieved similar results. I then added another 60,000 records for a total of 90,000, and this was the performance differences:
GetAll: 2450 ms
GetWithOriginalCode: 2694 ms
GetWithSODAandProperty: 75373 ms
GetWithSODAandField: 77 ms
Hope that helps. I know that it does not really explain why, but this might help with the how. Also the code for the SODA field query would not be hard to wrap to be more generic.

If you try to run this query using LINQ it'll run unoptimized (that means that db4o are going to retrieve all objects of type SimpleObject and delegate the rest to LINQ to objects)
The best approach would be to run n queries (since the id field is indexed, each query should run fast) and aggregate the results as suggested by "Mark Hall".
You can even use LINQ for this (something like)
IList<SimpleObject> objs = new List<SimpleObject>();
foreach(var tbf in ids)
{
var result = from SimpleObject o in db()
where o.Id = tbf
select o;
if (result.Count == 1)
{
objs.Add(result[0]);
}
}
Best

I haven't done much with db4o LINQ. But you can use the DiagnosticToConsole (or ToTrace) and add it to the IConfiguration.Diagnostic().AddListener. This will show you if the query is optimized.
You don't give a lot of details, but is the Id property on SimpleObject indexed?
Once diagnostics are turned on, you might try the query like so...
from SimpleObject simple in db
where matchingIds.Contains(simple.Id)
select simple
See if that gives you a different query plan.

You could 'build' a dynamic linq query. For example the API could look like this:
First parameter: a expression which tells on which property you search
The other parameters: the id's or whatever you're searching for.
var result1 = db.ObjectByID((SimpleObject t) => t.Id, 42, 77);
var result2 = db.ObjectByID((SimpleObject t) => t.Id, myIDList);
var result3 = db.ObjectByID((OtherObject t) => t.Name, "gamlerhart","db4o");
The implementation builds a dynamic query like this:
var result = from SimpleObject t
where t.Id = 42 || t.Id==77 ... t.Id == N
select t
Since everything is combined with OR the can be evaluated directly on the indexes. It doesn't need activation. Example-Implementations:
public static class ContainerExtensions{
public static IDb4oLinqQuery<TObjectType> ObjectByID<TObjectType, TIdType>(this IObjectContainer db,
Expression<Func<TObjectType, TIdType>> idPath,
params TIdType[] ids)
{
if(0==ids.Length)
{
return db.Cast<TObjectType>().Where(o=>false);
}
var orCondition = BuildOrChain(ids, idPath);
var whereClause = Expression.Lambda(orCondition, idPath.Parameters.ToArray());
return db.Cast<TObjectType>().Where((Expression<Func<TObjectType, bool>>) whereClause);
}
private static BinaryExpression BuildOrChain<TIdType, TObjectType>(TIdType[] ids, Expression<Func<TObjectType, TIdType>> idPath)
{
var body = idPath.Body;
var currentExpression = Expression.Equal(body, Expression.Constant(ids.First()));
foreach (var id in ids.Skip(1))
{
currentExpression = Expression.OrElse(currentExpression, Expression.Equal(body, Expression.Constant(id)));
}
return currentExpression;
}
}

Regex Replace to assist Orderby in LINQ

I'm using LINQ to SQL to pull records from a database, sort them by a string field, then perform some other work on them. Unfortunately the Name field that I'm sorting by comes out of the database like this
Name
ADAPT1
ADAPT10
ADAPT11
...
ADAPT2
ADAPT3
I'd like to sort the Name field in numerical order. Right now I'm using the Regex object to replace "ADAPT1" with "ADAPT01", etc. I then sort the records again using another LINQ query. The code I have for this looks like
var adaptationsUnsorted = from aun in dbContext.Adaptations
where aun.EventID == iep.EventID
select new Adaptation
{
StudentID = aun.StudentID,
EventID = aun.EventID,
Name = Regex.Replace(aun.Name,
#"ADAPT([0-9])$", #"ADAPT0$1"),
Value = aun.Value
};
var adaptationsSorted = from ast in adaptationsUnsorted
orderby ast.Name
select ast;
foreach(Adaptation adaptation in adaptationsSorted)
{
// do real work
}
The problem I have is that the foreach loop throws the exception
System.NotSupportedException was unhandled
Message="Method 'System.String Replace(System.String, System.String,
System.String)' has no supported translation to SQL."
Source="System.Data.Linq"
I'm also wondering if there's a cleaner way to do this with just one LINQ query. Any suggestions would be appreciated.

Force the hydration of the elements by enumerating the query (call ToList). From that point on, your operations will be against in-memory objects and those operations will not be translated into SQL.
List<Adaptation> result =
dbContext.Adaptation
.Where(aun => aun.EventID = iep.EventID)
.ToList();
result.ForEach(aun =>
aun.Name = Regex.Replace(aun.Name,
#"ADAPT([0-9])$", #"ADAPT0$1")
);
result = result.OrderBy(aun => aun.Name).ToList();

Implement a IComparer<string> with your logic:
var adaptationsUnsorted = from aun in dbContext.Adaptations
where aun.EventID == iep.EventID
select new Adaptation
{
StudentID = aun.StudentID,
EventID = aun.EventID,
Name = aun.Name,
Value = aun.Value
};
var adaptationsSorted = adaptationsUnsorted.ToList<Adaptation>().OrderBy(a => a.Name, new AdaptationComparer ());
foreach (Adaptation adaptation in adaptationsSorted)
{
// do real work
}
public class AdaptationComparer : IComparer<string>
{
public int Compare(string x, string y)
{
string x1 = Regex.Replace(x, #"ADAPT([0-9])$", #"ADAPT0$1");
string y1 = Regex.Replace(y, #"ADAPT([0-9])$", #"ADAPT0$1");
return Comparer<string>.Default.Compare(x1, y1);
}
}
I didn't test this code but it should do the job.

I wonder if you can add a calculated+persisted+indexed field to the database, that does this for you. It would be fairly trivial to write a UDF that gets the value as an integer (just using string values), but then you can sort on this column at the database. This would allow you to use Skip and Take effectively, rather than constantly fetching all the data to the .NET code (which simply doesn't scale).