We Keep Coding

Fast Way to Replace Names with Ids in Datatable?

I have a very large CSV file I have to load on a regular basis that contains time series data. Examples of the headers are below:
| SiteName | Company | Date | ResponseTime | Clicks |
This data comes from a service external to the uploader. SiteName and Company are both string fields. In the database these are normalized. There is a Site table and a Company table:
CREATE TABLE [dbo].[Site] (
[Id] INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
[Name] NVARCHAR(MAX) NOT NULL
)
CREATE TABLE [dbo].[Company] (
[Id] INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
[Name] NVARCHAR(MAX) NOT NULL
)
As well as the data table.
CREATE TABLE [dbo].[SiteStatistics] (
[Id] INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
[CompanyId] INT NOT NULL,
[SiteId] INT NOT NULL,
[DataTime] DATETIME NOT NULL,
CONSTRAINT [SiteStatisticsToSite_FK] FOREIGN KEY ([SiteId]) REFERENCES [Site]([Id]),
CONSTRAINT [SiteStatisticsToCompany_FK] FOREIGN KEY ([CompanyId]) REFERENCES [Company]([Id])
)
At around 2 million rows in the CSV file any sort of IO-bound iteration isn't going to work. I need this done in minutes, not days.
My initial thought is that I could pre-load Site and Company into DataTables. I already have the CSV loaded into a datatable in the format that matches the CSV columns. I need to now replace every SiteName with the Id field of Site and every Company with the Id field of Company. What is the quickest, most efficient way to handle this?

If you go with Pre-Loading the Sites and Company's you can get the distinct values using code:
DataView view = new DataView(table);
DataTable distinctCompanyValues = view.ToTable(true, "Company")
DataView view = new DataView(table);
DataTable distinctSiteValues = view.ToTable(true, "Site")
Then load those two DataTables into their SQL Tables using Sql-Bulk-Copy.
Next dump all the data in:
CREATE TABLE [dbo].[SiteStatistics] (
[Id] INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
[CompanyId] INT DEFAULT 0,
[SiteId] INT DEFAULT 0,
[Company] NVARCHAR(MAX) NOT NULL,
[Site] NVARCHAR(MAX) NOT NULL,
[DataTime] DATETIME NOT NULL
)
Then do an UPDATE to set the Referential Integrity fields:
UPDATE [SiteStatistics] ss SET
[CompanyId] = (SELECT Id FROM [Company] c Where ss.[Company] = c.Name),
[SiteId] = (SELECT Id FROM [Site] s Where ss.[Site] = s.Name)
Add the Foreign Key constraints:
ALTER TABLE [SiteStatistics] ADD CONSTRAINT [SiteStatisticsToSite_FK] FOREIGN KEY ([SiteId]) REFERENCES [Site]([Id])
ALTER TABLE [SiteStatistics] ADD CONSTRAINT [SiteStatisticsToCompany_FK] FOREIGN KEY ([CompanyId]) REFERENCES [Company]([Id])
Finally delete the Site & Company name fields from SiteStatistics:
ALTER TABLE [SiteStatistics] DROP COLUMN [Company];
ALTER TABLE [SiteStatistics] DROP COLUMN [Site];

ASP.NET MVC 5 - LINQ Query to Select Data from Database

I'm working on a school project in ASP.NET MVC 5. The project is about creating a social network. After the user logs in, he will see all public posts on his newsfeed.
I am having issues, though, in showing the public posts' data from the database.
This is the script of the database :
create table Utilizador(
id_utilizador integer not null identity(1,1),
nome varchar(50) not null,
apelido varchar(50) not null,
username varchar(15) not null unique,
pass varchar(50) not null,
email varchar(50) not null unique,
sexo char(1) not null CHECK (sexo IN('M', 'F')),
país varchar(50) not null,
imagem_perfil varchar(50) not null,
data_nascimento date not null,
estado int not null default 2, --0->Bloqueado 1-Activo, 2-por activar
primary key (id_utilizador),
check (email LIKE '%#%.%')
)
create table Post(
id_post integer not null identity(1,1),
texto varchar(400) not null,
primary key(id_post)
)
create table Publish_Post(
id_post integer not null,
id_utilizador integer not null,
data timestamp not null,
primary key(id_post),
foreign key(id_post) references Post(id_post),
foreign key(id_utilizador) references Utilizador(id_utilizador)
)
create table Privacy(
id_privacidade integer not null identity(1,1), --> 1 public, 2 private
nome varchar(50) not null,
primary key(id_privacidade)
)
create table Have_Privacy(
id_post integer not null,
id_privacidade integer not null,
primary key(id_post),
foreign key(id_post) references Post(id_post),
foreign key(id_privacidade) references Privacidade(id_privacidade)
)
Let me explain why I create the database the way I do:
The user creates and publishes some posts that have will have a privacy value (1 or 2). After the user logs in, all public posts(1) should appear on his newsfeed.
So far I have this LINQ query in C#:
var id_posts = from p in db.Posts
select p.texto;
ViewBag.Posts = id_posts;
Can someone help me?
Thanks in advance :)

Just do this
var id_posts = from p in db.Posts
join hp in db.Have_Privacy on p.id_post equals hp.id_post
join prv in db.Privacy on hp.id_privacidade equals prv.id_privacidade
where prv.nome = 'Private'
select p.texto;
Tell how it goes

Why not just add a field in Post called isprivate with boolean type of BIT that determines if it's private or not and then use query for provided data with where clause:
var id_posts = from p in db.Posts
where isprivate == false
select p.texto;
If you want to have more than 2 types of privacy and just stick with DB schema you provided, you can go with a JOIN:
If id decides it is private:
var id_posts = from p in db.Posts
join hp in db.Have_Privacy on p.id_post equals hp.id_post
where hp.id_privacidade = 1
select p.texto;
If name decides it is private:
var id_posts = from p in db.Posts
join hp in db.Have_Privacy on p.id_post equals hp.id_post
join prv in db.Privacy on hp.id_privacidade equals prv.id_privacidade
where prv.nome = 'Private'
select p.texto;
Also please note that naming tables in one language and columns in other is considered as bad design. It's hard for others (in this example me) to read it, even if I know what it should mean.
Two last queries use your schema with no changes implemented.

Optimizing query with many-to-many relationship on big data set

I have a database (SQLite) constructed with similar DDL:
CREATE TABLE [Player] (
[PlayerID] INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
[Name] TEXT UNIQUE NULL
);
CREATE TABLE [Position] (
[PlayerID] INTEGER NOT NULL,
[SingleHandID] INTEGER NOT NULL,
[Position] INTEGER NULL,
PRIMARY KEY ([PlayerID],[SingleHandID])
);
CREATE TABLE [SingleHand] (
[SingleHandID] INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
[Stake] FLOAT NULL,
[Date] DATE NULL,
DataSetID INTEGER NULL
[IsPreflopAllIn] BOOLEAN NULL
);
CREATE UNIQUE INDEX [NameIndex] ON [Player](
[Name] ASC
CREATE INDEX [DataSetIndex] ON [SingleHand](
[DataSetID] ASC
);
It is mapped to Entity Framework model. I am working on large data sets up to 10 million records each.
My problem is, that I need to find all Hands where a specific Player is sting on any given Postion (plus some other filters, like date range).
While I can scan the database very quickly, to find data from a single table, for example:
//[playerIDs and selectedPos are cashed in memory]
context.Positions.Where(p => playerIDs.Contains(p.PlayerID) && selectedPos.Contains(p.Position)).Select(p => p.SingleHandID).Take(maxHands ?? 1);
When I need to do any join between tables, it starts to run very slowly, for example:
//accesing both Position and SingleHand table
context.Positions.Where(p => playerIDs.Contains(p.PlayerID) && selectedPos.Contains(p.Position) && p.SingleHand.DataSetID == dataSetNumber).Select(p => p.SingleHandID).Take(maxHands ?? 1);
What clever trick, comining queries, and code (ex, with local caching) can I pull, to make this run most efficent? I am using System.Data.SQLite provider.
Maybe I should add redundant DataSetID to Position table, then I can do my main query only on the Position table? Later, when I will have the IDs of all the matching hands, it should be quicker to add additional conditions (like date checking)

Create a new index:
CREATE INDEX [DataSetIndex2] ON [SingleHand](
[SingleHandID] ASC,
[DataSetID] ASC
);
That should help a lot.
You can also try something like this:
context.Positions
.Where(p => playerIDs.Contains(p.PlayerID) && SelectedPos.Contains(p.Position))
.Select(p => p.SingleHandID)
.Intersect(context.SingleHand
.Where(s=>s.DataSetId==dataSetNumber)
.Select(s=>s.SingleHandID))
.Take(maxHands ?? 1);

Order by child records using dynamic field with entity framework

I am interested in sorting a query using entity framework and am having trouble sorting using a dynamic order by expression,
I have a parent record (DatasetRecord) with a one to many relationship consisting of a number of associated values (DatasetValues), I would like to be able to sort based on the data stored within the value if it matches a certain field,
So I need to sort by the Value of DatasetValues where it matches a supplied FieldID
public IEnumerable<DatasetRecordDto> FetchData(ModuleSettingsDto settings)
{
var query = _context.DatasetRecords.AsQueryable().Where(r => r.DatasetId == settings.DatasetId && r.IsDeleted == false);
foreach (var filter in settings.Filters)
{
// this works fine, check we have current field and value matches
query = query.Where(i => i.DatasetValues.Any(x => x.DatasetFieldId == filter.DatasetFieldId && x.Value == filter.Value));
}
foreach (var sort in settings.Sort)
{
switch (sort.SortDirection)
{
case "asc":
// THIS IS WHERE I NEED THE DYNAMIC ORDER BY EXPRESSION,
// THIS SHOULD SORT BY DATETIME WHERE FIELD ID MATCHES
query = query.OrderBy(i => i.DatasetValues.Any(x => x.ValueDateTime && x.DatasetFieldId == sort.DatasetFieldId));
break;
}
}
return ShapeResults(query);
}
And here is my database schema:
CREATE TABLE [dbo].[DatasetRecords](
[Id] [int] IDENTITY(1,1) NOT NULL,
[DatasetId] [int] NOT NULL
)
CREATE TABLE [dbo].[DatasetFields](
[Id] [int] IDENTITY(1,1) NOT NULL,
[Label] [varchar](50) NOT NULL,
[DatasetId] [int] NULL,
[DataType] [int] NOT NULL
)
CREATE TABLE [dbo].[DatasetValues](
[Id] [int] IDENTITY(1,1) NOT NULL,
[DatasetRecordId] [int] NOT NULL,
[DatasetFieldId] [int] NOT NULL,
[ValueString] [varchar](8000) NULL,
[ValueInt] [int] NULL,
[ValueDateTime] [datetime] NULL,
[ValueDecimal] [decimal](18, 2) NULL,
[ValueBit] [bit] NULL
)
Please let me know if you need any further info, any help is greatly appreciated,
UPDATE: I am not having an issue with creating a dynamic order by expression, I am trying to apply a conditional sort order for parent records based on the values in a child table

Have you tried running the Linq code on something like LinqPad, it lets you run Linq code against a dataset and shows you the generated SQL. You can take that SQL and run it manually to make sure it isn't some invalid SQL that is causing the error.

Database Hotel Reservation

So i have a bit of a problem, on my ASP.Net website i want it to show all the rooms which are currently available and not in use. When i say not in use i mean at present there is no one reserved to that room, so the systemdate is not between any values of the checkin and checkoutdate.
Part of my schema:
CREATE TABLE "Rooms"(
RoomNo int NOT NULL,
RoomType nvarchar(20) NULL,
PricePerNight money NULL,
MaximumOccupancy int NULL,
NoOfBeds int NULL,
NoOfBathrooms int NULL,
Entertainment bit NULL,
RoomService bit NULL,
Gym bit NULL,
CONSTRAINT PK_Rooms PRIMARY KEY(RoomNo)
)
CREATE TABLE "Reservation"(
ReservationID int IDENTITY (1,1) NOT NULL,
CustomerID int NOT NULL,
RoomNo int NOT NULL,
CheckInDate date NOT NULL,
CheckOutDate date NOT NULL,
NoOfDays int NOT NULL,
CONSTRAINT PK_Reservation PRIMARY KEY(ReservationID),
CONSTRAINT FK_Reservation_Customers_CustID FOREIGN KEY(CustomerID)
REFERENCES dbo.Customers(CustomerID),
CONSTRAINT FK_Reservation_Rooms_RoomNo FOREIGN KEY(RoomNo)
REFERENCES dbo.Rooms(RoomNo)
)
So heres my idea, im assuming im going to need an sql query which goes something like this:
Select All FROM Rooms Where &CheckInDate != "Select CheckIndate From
Reservation" AND &CheckOutDate != "Select CheckOutDate From
Reservation".
But this has some flaws in it.
Can someone please tell me how i can do this, how i can make my Sue-do-code (lets call it) more viable, as i'm not sure if something like that will work, and if necessarily suggest improvements i could make to my schema.
TLDR; i need a query where someone enters a checkindate and checkoutdate and it returns all the rooms which are available.

Try this..
SELECT *
FROM Rooms
WHERE NOT EXISTS
(SELECT *
FROM Room r, Reservation rs
WHERE r.RoomNo = rs.RoomNo AND GETDATE() BETWEEN rs.CheckInDate AND rs.CheckOutDate)

This is what i wanted.
SELECT dbo.Rooms.RoomNo
FROM dbo.Rooms JOIN dbo.Reservation
ON (dbo.Rooms.RoomNo = dbo.Reservation.RoomNo)
WHERE '2012-01-01' NOT BETWEEN dbo.Reservation.CheckInDate AND dbo.Reservation.CheckOutDate
AND '2012-01-05' NOT BETWEEN dbo.Reservation.CheckInDate AND dbo.Reservation.CheckOutDate;
Took time but i got it in the end. the where statement initial date can be replaced with &Date.