We Keep Coding

C# - SQL - Linq Query including 4 tables

Am a front-end dev. Let's get that out of the way... My issue is I need to get some data out of a few tables within the DB. The backend is written in C#. (I will try to answer questions on it if need be) which is not my area.
My SQL is just about average, I can understand enough to write the query that I need see below:
(I've sanitized the table names)
SELECT *
FROM (SELECT * FROM table_1 UNION SELECT * FROM table_2) name
JOIN (SELECT OrderNo, ID FROM table_3 UNION SELECT OrderNo, ID from table_4) name_2
ON name.OrderNo = name_2.OrderNo
WHERE name_2 = <queryParam>
The difficult part is that I need to be able to run this query by varying search terms, by order number, by telephone number and by branch ID using linq.
Can anyone help and also point me in the direction of (preferably easy) resources to learn?

Best way to save a static list using SQL Server, EF6

I'm starting a rewrite of an existing application. One of the current performance bottle necks is saving user generated lists of Employees into a static list, so that they can come back and view the same list of employees at a later date.
Below is a simple example of the funcionality I am looking for. The list generated would be by a more complex query than in the example.
Scenario: A user searches for all Employees on night shift and wants to save this list to load it later. They want the list to always return the result as it was the first time they ran the search. I.e. if a new employee is added to the night shift they should not appear on the list when they pull it up.
What I have tried:
Currently, there is a very poor solution of storing all of the ID's in the resulting list as a string array and then rebuilding a query using those ID's. This is very inefficient and causes problems with large lists having too many parameters.
I've also played around with constructing a table from a saved array of ID's and then joining, however, this is extremely slow on large lists (20,000+ employees) and often results in timeouts.
My current thought is to create a new table for each list and then call a JOIN on that table and the Employee table. However, if 100 users save 10 large lists (20,000+ employees) each, it quickly become a storage and table management nightmare.
I assume that this is a fairly common problem with a solution. But, I haven't been able to find any sort of examples or best practices on how to store static lists (I'm probably searching for the wrong thing). Does anyone have any general concepts on how to best handle this type of use case scenario?
Update:
I think I tried the following setup before, but it wasn't working for some reason or another, this was years ago; but looking back it seems to make the most sense. I think the issue I had was with NHibernate having issues with subselects in Linq. But, that is no longer a limitation.
I'm thinking I have a table of StaticSavedLists and index table linking the Person (Employee in previous example) linking the List to Employee in a many-to-many mapping.
The classes in c# would look like this:
public class StaticSavedList : BaseModel
{
public string Name { get; set; }
public IList<StaticSavedListPersonIdx> PersonsIdx { get; set; } //Has many persons
}
public class StaticSavedListPersonIdx : BaseModel
{
public StaticSavedList StaticSavedList { get; set; }
public Person Person { get; set; }
}

You probably need to have 1 table containing the "header" detail for the searches, which will include an ID for the search and then a second table with the entries for every search. Then you just need to get the ID somehow (maybe by userid and date of shift) and use that to join the results and employees table.
Here's how a schema could look
ShiftSearches
SearchID int (PK)
ShiftDate datetime
SearchResults
SearchID int (PK)
EmployeeID int (PK)
Employees
EmployeeID int (PK)
FirstName varchar
etc ...
Possible LINQ Query
DateTime shiftDate = new DateTime(2014,11,26);
int searchId = db.ShiftSearches.Single(s => s.ShiftDate == shiftDate).SearchID;
var results = from r in db.SearchResults where r.SearchID == searchId
join e in Employees on r.EmployeeID equals e.EmployeeID
select e;
This way you only need one table, and it is very thin, so should not take up much room - as you are just storing the required search and employee IDs, you can't really get the data smaller anyway.
The class structure you posted pretty much matches this concept.

Entity Framework may not be the appropriate choice of technology in all cases, and dealing with batches of 20k rows at a time might be one of these cases.
However, I believe your design of the data model is a good one. Below, on Sql Only, it can be shown that 60k rows containing (ListId, EmployeeId) pairs can be inserted into a sensibly clustered ListSearchEmployee table in under one second, and subsequently, one of the lists of 20k rows can be joined back to the full Employee row within 1.8 seconds from a cold start.
The performance bottleneck is more likely to be the original user search - presumably this can be a nearly arbitrary query executed against your Employees + related tables, which will be difficult to index for all permutations.
Some performance suggestions (for the List save + refetch):
Use SqlBulkCopy to bulk dump EmployeeId's into the List tables
Use something low level, like a basic SqlReader to fetch the list data back for the UI (although I guess it will be paged? - If so, a DbSet.SqlQuery with AsNoTracking() turned off might suffice.
Don't bother with Foreign Keys on the List table - this will slow down inserts.
Don't try and synchronously cleanup unwanted old List searches - queue them for deletion and have a background process to do the deletion.
As a result, the ListSearchEmployee table will frequently need reindexing due to the large amount of churn on it.
-- Sample data setup - not considered in the timing
CREATE TABLE Employee
(
EmployeeID INT identity(1,1),
Name NVARCHAR(100),
SomeOtherFieldToLessenTheDensityOfEmployee CHAR(500),
PRIMARY KEY(EmployeeID)
);
CREATE TABLE ListSearch
(
ListSearchID INT IDENTITY(1,1) PRIMARY KEY
-- Other fields you may want to identify the search, e.g. date, which user, which filters etc
)
CREATE TABLE ListSearchEmployee
(
ListSearchID INT,
EmployeeID INT, -- Don't bother Foreign Keying for performance
PRIMARY KEY CLUSTERED (ListSearchID, EmployeeID)
);
-- Insert 1M Employees
WITH cteData AS
(
SELECT top 1000000 sc1.name, ROW_NUMBER() OVER (ORDER BY sc1.object_id) AS rn
FROM sys.columns sc1 CROSS JOIN sys.columns sc2 CROSS JOIN sys.columns sc3
)
INSERT INTO Employee(Name)
SELECT name + CAST(rn AS VARCHAR)
FROM cteData;
-- Timing : 0.972 seconds on SQLExpress 2012 on an i3
-- Inserting 3 x 20 k lists of pseudo random employees (but not contigious on the EmployeeId Cluster)
WITH cteData AS
(
SELECT top 20000 1 as listid, ROW_NUMBER() OVER (ORDER BY sc1.object_id) * 50 AS empid
FROM sys.columns sc1 CROSS JOIN sys.columns sc2
UNION ALL
SELECT top 20000 2 as listid, ROW_NUMBER() OVER (ORDER BY sc1.object_id) * 30 AS empid
FROM sys.columns sc1 CROSS JOIN sys.columns sc2
UNION ALL
SELECT top 20000 3 as listid, ROW_NUMBER() OVER (ORDER BY sc1.object_id) * 41 AS empid
FROM sys.columns sc1 CROSS JOIN sys.columns sc2
)
INSERT INTO ListSearchEmployee(ListSearchID, EmployeeID)
SELECT listid, empid
FROM cteData;
DBCC DROPCLEANBUFFERS;
-- Timing : 1.751 seconds on SQLExpress 2012 on an i3
-- Joining 20k rows
SELECT *
FROM ListSearchEmployee el INNER JOIN Employee e on el.EmployeeID = e.EmployeeID
WHERE el.ListSearchID = 2

SQL Select All Without Values in Another Table

I'm building an application to manage tools that are on loan to staff members and the way that the table is set up is there has a Tools table which has a unique tool code and serial number and a Loans table that contains the unique tool code and a return date. Each time there is a loan, it creates a new entry, the return date when the loan is created is null.
The query I have at the moment searches for all tools based on a text input. I've tried a lot of code with inner joins with no success and as I'm not extremely experienced with SQL, I couldn't figure out how to make it work.
SELECT [Tools].[ToolID], [Tools].[Type], [Tools].[Brand], [Tools].[Serial], [Tools].[Year], [Tools].[Code] FROM [Tools]
WHERE ([Tools].Serial LIKE '%234%' OR [Tools].Code LIKE '%234%')
My issue occurs when I'm searching for a tool in the loan database. I want it to show tools from the Tools table that:
A) Have no entries in the loan database (i.e. a new tool)
B) All entries in the database have a return date (tool has been returned)
Thanks.
EDIT: Combined the two queries from the answer
SELECT [Tools].[ToolID], [Tools].[Type], [Tools].[Brand], [Tools].[Serial], [Tools].[Year], [Tools].[Code]
FROM [Tools] left outer join
Loan ON [Tools].code = Loan.toolcode
WHERE Loan.toolcode is null AND ([Tools].Code LIKE '%234%' OR [Tools].Serial LIKE '%234%')
UNION
SELECT [Tools].[ToolID], [Tools].[Type], [Tools].[Brand], [Tools].[Serial], [Tools].[Year], [Tools].[Code]
FROM Loan
INNER JOIN Tools ON Tools.Code = Loan.ToolCode
GROUP BY [Tools].[ToolID], [Tools].[Type], [Tools].[Brand], [Tools].[Serial], [Tools].[Year], [Tools].[Code]
HAVING count(returndate) = count(*)

To show tools that have no entry in the loan database:
SELECT t.*
from tools t left outer join
loans l
on t.code = l.toolcode WHERE l.toolcode is null;
I assume that for the second question, you want all tools where all entries have a return date -- that is, no entry is NULL:
select l.toolcode
from loans l
group by l.toolcode
having sum(case when returndate is null then 1 else 0 end) = 0
This formulation counts up the number of records where the returndate is null and only returns tools where there are no such records. An alternative way of writing the same logic is:
having count(returndate) = count(*)

Ranking Database Entries

I have a database that contains:
user_id | category_id | liked_id | disliked_id
(thanks to stack overflow users for helping me get my database setup properly in the first place!!)
Last time I used food as an example but this time I'm going to use people.
The user is given 2 images (male vs male or female vs female) and he/she simply chooses which one he/she thinks is more attractive. The user repeats this process as long as he/she wishes. Each selection is entered into the database showing which person they liked and which they disliked (also a button would be available if you think the two are similar).
Now that I have my table full of entries, I'm trying to develop an algorithm that will take all of those "votes" and translate it into a ranked list of who the user finds most attractive (based on hundreds or maybe even thousands of ranking entries).
I've been at the drawing board for hours and can't seem to think of an effective way of doing this.
Any help would be appreciated.
P.S.: The idea is also to have this be a multi-user thing, where other users can see your "like" tables and also have globally averaged tables showing how all users in general rank things.

So you posted your question in the c# group. I want to give you, however, a solution that is implemented in the database, making it more independent of your program.
What you probably want to do first is to get the number of times an image has been liked and disliked. This SQL statement should do that for you (if you are using a database supporting grouping sets it would probably be easier to write):
SELECT t1.liked_id as id, t1.c_liked, t2.c_disliked
FROM
(SELECT liked_id, COUNT(*) as c_liked FROM table GROUP BY liked_id) t1
LEFT JOIN
(SELECT disliked_id, COUNT(*) c_disliked FROM table GROUP BY disliked_id) t2
ON
t1.liked_id = t2.disliked_id
Then it's up to you what you do with the numbers. In the outermost SELECT-statement, you could put a very complicated function, e.g. you could choose to weigh the dislikes less than the likes. To give you an idea of a possible very simple function:
SELECT t1.liked_id as id,
(t1.c_liked/(t1.c_liked + t2.c_disliked) - t2.c_disliked/(t1.c_liked + t2.c_disliked)) as score
This returns you values [-1, 1] (which you could normalize to [0, 1] if you like, but don't have to), which you then can sort as in this example:
SELECT t1.liked_id as id,
(t1.c_liked/(t1.c_liked + t2.c_disliked) - t2.c_disliked/(t1.c_liked + t2.c_disliked)) as score
FROM
(SELECT liked_id, COUNT(*) as c_liked FROM table GROUP BY liked_id) t1
LEFT JOIN
(SELECT disliked_id, COUNT(*) c_disliked FROM table GROUP BY disliked_id) t2
ON
t1.liked_id = t2.disliked_id
ORDER BY score

limiting the records inserted in a sql table

I have the requirement to build a asp.net sign up form which will allow students to register a training. So far I built a database in sql server and 3 tables: student, training & studenttraining
My question is, how can I limit the form from displaying the dates available once a particular training gets full, or meabe how can I prevent by checking the tables that the user can register?

Select count(*) as SeatsFilled, t.TrainingKey, t.TrainingDate
From Training t
Inner Join StudentTraining st on t.TrainingKey = st.TrainingKey
Group By t.TrainingKey, t.TrainingDate
Having count(*) < t.TotalSeats
TotalSeats is a column in the Training table that specifies how many seats the training provides. I assumed StudentTraining is a many-to-many bridge table between Students and Training.

You'll need to establish what "full" is first. Then, you can do a simple
SELECT COUNT(id) FROM table to determine if the full amount is already reached.

I guess you could have a MaxTraining column in the training table and when you get the data for your form, you can count the training entries in studenttraining, and if it equals MaxTraining, then don't bring that training entry, cause it means it's already full.