We Keep Coding

Way to save an entity framework query for use later - report creation

In our company we create custom reports for clients who use our services. The data we collect is contact data (first name, last name, company, email etc) of attendees at events but there is also dynamic data captured. Each event where we collect their data, there are surveys which are custom to the event and company.
For example, at event 123 there might be a survey with 2 questions. Q1 might be an open question/free text, and Q2 will be multiple choice (select n from a list of options that we define).
To code a report like that, it is currently a manual process as we need to
1. Filter on all contact data for this event and the specific client
2. Create a temporary table with standard columns (first name, last name, etc)
3. But also with custom columns for however many survey questions we have (2)
4. Populate the temp table with the data from step 1
5. Use a set based update to get survey responses for each question
6. Format the report into the required format (order of columns)
This is currently done in a stored procedure, which is time consuming, error-prone and tedious.
Is there a way to
Somehow save a EF query (like from events in db.Events join ....) somewhere, and able to retrieve that and execute?
Similarly to the above, save the order of the output columns and have that reflected when the report is run
Currently we do this in sprocs but ideally we want to do this using EF in a report generation tool.
Edit: Here is a stripped down version of the SQL required for a report.
-- Step 1. Filter lead data into temp table. Let's say there are 10 leads, and 8 have survey responses for the 2 questions.
SELECT leadID,FirstName,LastName,eMail
INTO #leads
FROM dbo.leads
WHERE eventid = 123 and companyid = 456
-- Step 2 and 3. Create a temporary table with standard contact fields plus survey questions.
CREATE TABLE #Data (
LeadId NVARCHAR(50) PRIMARY KEY,
FirstName NVARCHAR(100),
LastName NVARCHAR(100),
EmailAddress NVARCHAR(100),
[Question1] NVARCHAR(1000),
[Question2] NVARCHAR(1000)
)
-- Step 4. Populate temp table.
INSERT INTO #Data (LeadId,
FirstName,
LastName,
EmailAddress)
SELECT leadid,
FirstName,
LastName,
Email
FROM #leads L
-- Step 5. Use a set based update to get survey responses for each question. 8 leads in each update statement would be updated as 2 leads did not answer the survey.
UPDATE TTARGET SET
Question1 = OA.Title
FROM #Data TTARGET
JOIN answerset answers on TTARGET.LeadId = answers.leadid
JOIN que.answer A on answers.answersetid = A.answersetid
JOIN que.optionalanswer OA ON A.possibleAnswerId = OA.optionalAnswerId
WHERE A.questionid IN ('3A2FFD77-BF37-4580-938A-FF234395B602') AND A.Active = 1
UPDATE TTARGET SET
Question2 = OA.Title
FROM #Data TTARGET
JOIN answerset answers on TTARGET.LeadId = answers.leadid
JOIN que.answer A on answers.answersetid = A.answersetid
JOIN que.optionalanswer OA ON A.possibleAnswerId = OA.optionalAnswerId
WHERE A.questionid IN ('75C5FA00-439F-4DA9-9662-9C683A0E6719') AND A.Active = 1
-- Step 6. Format the report.
SELECT
FirstName,
LastName
EmailAddress,
LeadId,
Question1 AS 'the title of the first question',
Question2 AS 'the title of the second question'
FROM #Data

Comparing and reporting differing data between two databases

The database for my application contains tables (not editable by the user) that are necessary for my application to run. For instance, there is a Report table containing a list of my SSRS reports.
Except for the Auto-Increment and GUID fields, the data in my Report Table should match across all databases.
To keep existing client databases in synch with the ones created from scratch, there is a database updater app that runs scripts to update the existing client base.
There are Unit Tests to ensure Reports run correctly on both types of databases. However, other than developer eye, there is no system check to ensure the rows and values in those rows match among the tables. This is prone to human error.
To fix, I plan to add a small report to Unit Test report that will inform development of the following:
Records missing from the "Made From Scratch" database that exist in the "Updated" Database
Records missing from the "Updated" database that exist in the "Made From Scratch" Database
Fields that do not match between the tables
So far, I have a query to report the above information for all tables involved.
A sample query would look something like this:
--Take the fields I want to compare from TableToCompare in MadeFromScratch and put them in #First_Table_Var
--NOTE: MyFirstField should match in both tables in order to compare the values between rows
DECLARE #First_Table_Var table(
MyFirstField Varchar(255),
MySecondField VarChar(255),
MyThirdField Varchar(255),
);
INSERT INTO #First_Table_Var
SELECT
r.MyFirstField,
r.MySecondField,
l.MyThirdField
FROM
MadeFromScratch.dbo.TableToCompare r
INNER JOIN MadeFromScratch.dbo.LookUpTable l ON r.ForeignKeyID = l.PrimaryKeyID
--Take the fields I want to compare from TableToCompare in UpdatdDatabase and put them in #Second_Table_Var
DECLARE #Second_Table_Var table(
MyFirstField Varchar(255),
MySecondField VarChar(255),
MyThirdField Varchar(255),
);
INSERT INTO #Second_Table_Var
SELECT
r.MyFirstField,
r.MySecondField,
l.MyThirdField
FROM
UpdatdDatabase.dbo.TableToCompare r
INNER JOIN UpdatdDatabase.dbo.LookUpTable l ON r.ForeignKeyID = l.PrimaryKeyID
--**********************
-- CREATE OUTPUT
--**********************
--List Rows that exist in #Second_Table but not #First_Table
--(e.g. these rows need to be added to the table in MadeFromScratch)
SELECT
Problem = '1 MISSING ROW IN A MADE-FROM-SCRATCH DATABASE',
hur.MyFirstField,
hur.MySecondField,
hur.MyThirdField
FROM
#Second_Table_Var hur
WHERE
NOT EXISTS
(SELECT
*
FROM
#First_Table_Var hu
WHERE
hu.MyFirstField = hur.MyFirstField
)
UNION
--List Rows that exist in #First_Table but not #Second_Table
--(e.g. these rows need to be added to the table in UpdatdDatabase)
SELECT
Problem = '2 MISSING IN UPDATE DATABASE',
hur.MyFirstField,
hur.MySecondField,
hur.MyThirdField
FROM
#First_Table_Var hur
WHERE
NOT EXISTS
(SELECT
*
FROM
#Second_Table_Var hu
WHERE
hu.MySecondField = hur.MySecondField
)
UNION
--Compare fields among the tables where MyFirstField matches, but
SELECT
Problem = '3 MISMATCHED FIELD',
h.MyFirstField,
MySecondField = CASE WHEN h.MySecondField = hu.MySecondField THEN '' ELSE 'Created Value: ' + h.MySecondField + ' Updated Value: ' + hu.MySecondField END,
MyThirdField = CASE WHEN h.MyThirdField = hu.MyThirdField THEN '' ELSE 'Created Value: ' + CAST(h.MyThirdField AS VARCHAR(4)) + ' Updated Value: ' + CAST(hu.MyThirdField AS VARCHAR(4)) END,
FROM
#First_Table_Var h
INNER JOIN #Second_Table_Var hu on h.MyFirstField = hu.MyFirstField
WHERE
NOT EXISTS
(SELECT
*
FROM
#Second_Table_Var hu
WHERE
hu.MyFirstField = h.MyFirstField and
hu.MySecondField = h.MySecondField and
hu.MyThirdField = h.MyThirdField and
)
ORDER BY Problem
I won't have any problem writing code to parse through the results, but this methodology feels antiquated for the following reasons:
Several queries (which essentially do the same thing) will need to be written
Maintenance for this process can get cumbersome
I would like to be able to write something where the list of tables and fields to compare is maintained by some kind of file (XML?). So, whether fields are added or changes all the user has to do is update this file.
Is there a way to use LINQ and/or Reflection (or any feature in .NET 4.0 for that matter) where I could compare tables between two databases and maintain them like I've listed above?
Ideas are welcome. Ideas with an example would be great! :D

you said "Except for the Auto-Increment and GUID fields, the data in my Report Table should match across all databases."
I assume that these fields are ID fields, ideally, replication of the database should replicate the id fields too ensuring this will allow you to check for new inserts by ID, in case of updates, you can set a timestamp field for comparison.

Sql Group by for rows with all columns the same except one

I have a table with a number of columns that will be used to save searches stored by a user.
Every day an email needs to be sent out containing the latest search results.
There could be thousands of saved searches.
My columns in my saved search table are similar to as follows:
Id
userEmail
SearchParam1
SearchParam2
SearchParam3
As there will be so many searches I anticipate there will be many that are the same so I would like to get every row where param1,2 and 3 are the same but still have access to the users email so I can send the same results to each without having to run the same search multiple times.
Ideally I'd like something like this back:
SearchParam1, SearchParam2, SearchParam3
|
|____Email1
|
|____Email2
|
|____Email3
Kind of grouped by the search params but linked to the emails some how.
I don't even know where to start, hence why I have not added any code.
Can anyone give me some suggestions?
EDIT
Just to clarify. One email would not exist more than once in the table. So technically they are only allowed to save 1 search each.
There are no users as such, just email addresses.
Bex

What you really should consider is data normalization. This looks like a many-to-many relationship, so a structure like this would be idea:
create table yourUsersTable
(
id int identity(1, 1) primary key clustered not null,
name varchar(1000) not null,
email varchar(1000) not null
)
go
create table searchParams
(
id int identity(1, 1) primary key clustered not null,
searchText varchar(1000) not null
)
go
create table userSearchParams
(
fkYourUsersTable int not null references yourUsersTable(id),
fkSearchParams int not null references searchParams(id)
)
go
This way, if you wanted to get all the email addresses for users subscribed to a search parameters, it'd be a simple query:
select u.email
from yourUsersTable u
inner join userSearchParams up
on u.id = up.fkYourUsersTable
inner join searchParams p
on p.id = up.fkSearchParams
where p.searchText = 'your search parameter here'
This is a basic example of implementing data normalization with a many-to-many relationship. It utilizes a join table to create the relationship between users and search parameters. Data retrieval has become much simpler with the above design, and you aren't constricted by your original denormalized data (with columns like searchParam1, searchParam2, etc.).
Please let me know if that made sense.

In SQL, a starting point would be:
select SearchParam1, SearchParam2, SearchParam3, userEmail, count(*) query_count
from MyTable
group by SearchParam1, SearchParam2, SearchParam3, userEmail
order by 1,2,3,4
(If you don't want to see how many times the same user ran the same query, you can omit the group by clause and the count(*) item, and just include select distinct instead.)
This will repeat the search parameters next to each user's e-mail.
If you want to only include the search parameters as a heading each time one of them changes, I suggest using the above query in a reporting tool (such as SQLServer reporting services) - group on the search parameters, include the parameter fields in the group header only and include the user eMail in the detail line only.
EDIT: if userEmail uniquely identifies a record, the query can be simplified to:
select SearchParam1, SearchParam2, SearchParam3, userEmail
from MyTable
order by 1,2,3,4

Turning a table "on it's side" in asp.net - how?

How do I turn this table:
+------------+-----------------+
| Category + Subcategory |
+------------+-----------------+
|Cat..........+ Persian.........|
|Cat..........+ Siamese........|
|Cat..........+ Tabby...........|
|Dog.........+ Poodle..........|
|Dog.........+ Boxer............|
+------------+----------------+
on it's side to get the following:
+------------+-----------------+
| Cat......... + Dog............. |
+------------+-----------------+
+ Persian..+ Poodle.........+
+ Siamese + Boxer...........+
+ Burmese + ...................+
+------------+-----------------+
The initial table is from the following MySQL query:
select c.CATEGORYNAME, sc.NAME from subcategorydefinition sc
join categorydefinition c on sc.CATEGORYID = c.CATEGORYID
where c.ISDELETED = 0
order by CATEGORYNAME, NAME ASC
And I want to display it in (probably) a Gridview.
Cheers!

Pivot is static in SQL. You need to know in advance the columns you want in output, so if the list of categories is not fixed, you can't use pivot directly.
If you were using Microsoft SQL Server (which I know you're not, but it's for the sake of example), you could use a dynamic query in a stored procedure, as described here:
http://www.simple-talk.com/community/blogs/andras/archive/2007/09/14/37265.aspx
Now, in MySql, there is no way to execute dynamic SQL on the sql side (no equivalent of EXECUTE or sp_executeqsl), so your best choice would be to generate a similar SQL query server-side (aspnet server-side).
Another simpler idea IMHO would be to forget about doing it in SQL, but to do the aggregation in your C# code.

You should use pivot

To do this in SQL, you'd need to dynamically generate your query based on the available set of values in the "Category" column. This is usually fairly painful and error prone, regardless of whether you do it in pure SQL (in a sproc) or in code (dynamic SQL).
I'd recommend reading your values from the database in the way that they are stored, then dynamically creating a DataTable or similar structure to use as the datasource for your UI.

I don't have a working version of MySql handy but this will work as long as there is always more cats than dogs because of the left join at the end of the script. I forgot that there isn't a full outer join in MySql but you could use this logic to try it out.
But the point of this is that if you have two tables with arbitrary keys you can join on the keys to get the results lined up like you want.
-- drop tables
DROP TABLE dbo.cat
DROP TABLE dbo.dog
--create dog table
create table dog (
dog_id int IDENTITY(1,1) NOT NULL
,dog varchar(50)
)
--add dogs only
insert into dog (dog)
select subcategory
FROM play.dbo.test
where category = 'Dog'
--create cat table
create table cat (
cat_id int IDENTITY(1,1) NOT NULL
,cat varchar(50)
)
--add cats only
insert into cat (cat)
select subcategory
FROM play.dbo.test
where category = 'cat'
-- disply everything
SELECT cat
, dog
from dog d
--full outer join cat c
left join dog d
on d.dog_id = c.cat_id

Adding a Column Programmatically to a SQL Server database

I've taken over an ASP.NET application that needs to be re-written. The core functionality of this application that I need to replicate modifies a SQL Server database that is accessed via ODBC from third party software.
The third-party application creates files that represent printer labels, generated by a user. These label files directly reference an ODBC source's fields. Each row of the table represents a product that populates the label's fields. (So, within these files are direct references to the column names of the table.)
The ASP.NET application allows the user to create/update the data for these fields that are referenced by the labels, by adding or editing a particular row representing a product.
It also allows the occasional addition of new fields... where it actually creates a new column in the core table that is referenced by the labels.
My concern: I've never programmatically altered an existing table's columns before. The existing application seems to handle this functionality fine, but before I blindly do the same thing in my new application, I'd like to know what sort of pitfalls exist in doing this, if any... and if there are any obvious alternatives.

It can become problem when too many columns are added to tables, and you have to be careful if performance is a consideration (covering indexes are not applicable, so expensive bookmark lookups might be performed).
The other alternative is a Key-Value Pair structure: Key Value Pairs in Database design, but that too has it's pitfalls and you are better off creating new columns, as you are suggesting. (KVPs are good for settings)

One option I think is to use a KVP table for storing dynamic "columns" (as first mentioned by Mitch), join the products table with the KVP table based on the product id then pivot the results in order to have all the dynamic columns in the resultset.
EDIT: something along these lines:
Prepare:
create table Product(ProductID nvarchar(50))
insert Product values('Product1')
insert Product values('Product2')
insert Product values('Product3')
create table ProductKVP(ProductID nvarchar(50), [Key] nvarchar(50), [Value] nvarchar(255))
insert ProductKVP values('Product1', 'Key2', 'Value12')
insert ProductKVP values('Product2', 'Key1', 'Value21')
insert ProductKVP values('Product2', 'Key2', 'Value22')
insert ProductKVP values('Product2', 'Key3', 'Value23')
insert ProductKVP values('Product3', 'Key4', 'Value34')
Retrieve:
declare #forClause nvarchar(max),
#sql nvarchar(max)
select #forClause = isnull(#forClause + ',', '') + '[' + [Key] + ']' from (
select distinct [Key] from ProductKVP /* WHERE CLAUSE */
) t
set #forClause = 'for [Key] in (' + #forClause + ')'
set #sql = '
select * from (
select
ProductID, [Key], [Value]
from (
select k.* from
Product p
inner join ProductKVP k on (p.ProductID = k.ProductID)
/* WHERE CLAUSE */
) sq
) t pivot (
max([Value])' +
#forClause + '
) pvt'
exec(#sql)
Results:
ProductID Key1 Key2 Key3 Key4
----------- --------- --------- --------- -------
Product1 NULL Value12 NULL NULL
Product2 Value21 Value22 Value23 NULL
Product3 NULL NULL NULL Value34

It very much depends on the queries you want to run against those tables. The main disadvantage of KVP is that more complex queries can become very inefficient.
A "hybrid" approach of both might be interesting.
Store the values you want to query in dedicated columns and leave the rest in an XML blob (MS SQL has nice features to even query inside the XML) or alternatively in a KVP bag. Personally I really don't like KVPs in DBs because you cannot build application logic specific indixes anymore.
Just another approach would be not to model the specific columns at all. You create generic "custom attribute" tables like: Attribute1, Attribute2, Attribute3, Attribute4 (for the required data type etc...) You then add meta data to your database that describes what AttrX means for a specific type of printer label.
Again, it really depends on how you want to use that data in the end.

One risk is the table getting too wide. I used to maintain a horrible app that added 3 columns "automagically" when new values were added to some XML (for some reason it thought everything would be a string a date or a number- hence the creation of 3 columns).
There are other techniques like serializing a BLOB or designing the tables differently that may help.