I want to create and compile a SQL query using SqlKata and then in some other function execute this query. It's not obvious based on the documentation how this is done, especially for complex queries.
I have tried converting the query to a string and then running it, but this does not seem to be the right thing to do and the string does not compile well for complex queries.
public class TradeLoader : ITradeLoader
{
public SqlResult CreateTradeQuery(Target targets, Group groups, DateTime fromTime)
{
var compiler = new SqlServerCompiler();
var query = new Query().FromRaw(#"
[trades] AS t WITH (NOLOCK)
LEFT JOIN [info] AS i WITH (NOLOCK)
ON t.Id = i.Id
")
.Select("i.Name", "t.Price", "t.Volume")
.WhereTime("t.CreatedDate", ">", fromTime);
var subQuery = new Query(#"
[trades] AS t WITH (NOLOCK)
LEFT JOIN [info] AS i WITH (NOLOCK)
ON t.Id = i.Id
");
foreach (var target in targets)
{
subQuery.OrWhere(q => q.WhereIn("i.Name", groups.Keys).WhereIn("i.GroupName", target.Value));
}
query.Where(subQuery);
SqlResult result = compiler.Compile(query);
return result; // How to execute this somewhere else?
}
}
It's not clear to me how I can execute this query somewhere else, either using the SqlKata.Execution or System.Data.SqlClient. Preferably I should be able to run it in both.
I don't understand what you want to do.
You can use it this way.
public SqlResult CreateTradeQuery(Target targets, Group groups, DateTime fromTime)
{
var query = GetDefaultQuery();
query.Select("i.Name", "t.Price", "t.Volume");
query.Where("t.CreatedDate", ">", fromTime);
query.Where(GetSubQuery(targets,groups),"subQuery");
SqlResult result = compiler.Compile(query);
return result;
}
private Query GetSubQuery(Target targets, Group groups)
{
var subQuery = GetDefaultQuery();
foreach (var target in targets)
{
subQuery.OrWhere(q => q.WhereIn("i.Name", groups.Keys).WhereIn("i.GroupName", target.Value));
}
return subQuery;
}
private Query GetDefaultQuery()
{
var query = new Query("trades as t");
query.LeftJoin("info as i","i.Id","t.Id");
return query;
}
This is probably basic knowledge for those who are proficient in C#/.NET, but what I wanted to do was to inject params into the SQL query created with SqlKata and then run it without using SqlKata. In the end, I did it in the following way (and there are probably other ways to do this too):
public class TradeLoader : ITradeLoader
{
...
public async Task<SomeDataType> RunQuery()
{
...
var sqlCommand = CreateTradeQuery(someTargets, someGroups, someDateTime);
using (SqlConnection sql = new SqlConnection(_connectionString))
{
sql.Open();
var command = new SqlCommand(sqlCommand.Sql, sql) {CommandTimeout = _timeout};
foreach (var (name, value) in sqlCommand.NamedBindings)
{
var parameter = command.CreateParameter();
parameter.ParameterName = name;
parameter.Value = value;
command.Parameters.Add(parameter);
}
using (SqlDataReader dataReader = await command.ExecuteReaderAsync())
{
while (await dataReader.ReadAsync())
{
// read data here
}
}
}
return data;
}
}
This allows you to build your queries with SqlKata, but run the query using other libraries like in this example where I use System.Data.SqlClient.
Related
In Dapper it is possible to query a list with where in using this code:
var sql = "SELECT * FROM Invoice WHERE Kind IN #Kind;";
using (var connection = My.ConnectionFactory())
{
connection.Open();
var invoices = connection.Query<Invoice>(sql, new {Kind = new[] {InvoiceKind.StoreInvoice, InvoiceKind.WebInvoice}}).ToList();
}
But I'd like to query a list of products by multiple suppliers and their product code. So to do this, I tried creating an array of arrays. This is my method in my repository:
public Dictionary<int, Dictionary<string, Product>> GetGroupedListByRelationIdAndProductCode(Dictionary<int, List<string>> productKeysByRelationId)
{
Dictionary<int, Dictionary<string, Product>> list = new Dictionary<int, Dictionary<string, Product>>();
string sql = "SELECT * FROM Products WHERE 1=1 ";
int i = 0;
foreach (KeyValuePair<int, List<string>> productKeys in productKeysByRelationId)
{
sql = sql + " AND (ManufacturerRelationId = " + productKeys.Key + " AND ManufacturerProductCode in #ProductCodeList[" + i + "] )";
++i;
}
using (var conn = _connectionFactory.CreateConnection())
{
conn.Open();
var param = new { ProductCodeList = productKeysByRelationId.Select(x => x.Value.ToArray()).ToArray() };
var productsList = conn.Query<Product>(sql, param).ToList();
if (productsList.Count > 0)
{
foreach (var product in productsList)
{
list[product.ManufacturerRelationId][product.ManufacturerProductCode] = product;
}
}
}
return list;
}
This gives me this error though: System.ArgumentException: 'No mapping exists from object type System.String[] to a known managed provider native type.'
Any suggestions on how to do this?
You have several problems with your code.
If you have more than one item in your productKeysByRelationId you will end up getting SQL like:
WHERE 1=1 AND ManufacturerRelationId = 1 ... AND ManufacturerRelationId = 2
That's not likely to return any results, you need to sprinkle some OR's in there.
The error you are getting is because you have something like:
AND ManufacturerProductCode in #ProductCodeList[0]
Dapper cannot handle that. It mostly expects the query parameters to be an object with aptly named members of a few simple types or array.
Thankfully Dapper has a solution to that, DynamicParameters to the rescue!
You can construct your query like this:
var queryParams = new DynamicParameters();
foreach (var productKeys in productKeysByRelationId)
{
sql = sql + $" ... AND ManufacturerProductCode in #ProductCodeList{i} )";
queryParams.Add($"ProductCodeList{i}", productKeys.Value);
i++;
}
Now you have your query parameters in the right format, so you can just do this:
var productsList = conn.Query<Product>(sql, queryParams).ToList();
That ought to fix it, but you really should try to parameterize the ManufacturerRelationId also. It's not just about SQL injection, there might be some SQL cache performance related things.
You might also gain some clarity in code by using the SqlBuilder from Dapper.Contrib.
I try to combine a projection and a distinct with the MongoDB driver but don't get anywhere...
I have:
var coll = db.GetCollection<Vat>(CommonConstants.VatCodeCollection);
// I like to combine in one statement:
var result = coll.Distinct<DateTime>("ValidSince", filter).ToList();
var projection = Builders<Vat>.Projection.Expression(x => new VatPeriod { ValidSince = x.ValidSince });
So at the end I like to get a List<VatPeriod> as a result of one statement. Of course I could do something like
var coll = db.GetCollection<Vat>(CommonConstants.VatCodeCollection);
List<VatPeriod> vatPeriods = null;
try
{
var result = coll.Distinct<DateTime>("ValidSince", filter).ToList();
if (result.Count > 0)
{
vatPeriods = new List<VatPeriod>(result.Count);
foreach (var dateTime in result)
{
vatPeriods.Add(new VatPeriod() {ValidSince = dateTime});
}
}
return vatPeriods;
}
catch .....
in my repository class, but I would prefer to do everything on the Mongo server. Any idea if and how this is possible?
When using the C# code below to construct a DB2 SQL query the result set only has one row. If I manually construct the "IN" predicate inside the cmdTxt string using string.Join(",", ids) then all of the expected rows are returned. How can I return all of the expected rows using the db2Parameter object instead of building the query as a long string to be sent to the server?
public object[] GetResults(int[] ids)
{
var cmdTxt = "SELECT DISTINCT ID,COL2,COL3 FROM TABLE WHERE ID IN ( #ids ) ";
var db2Command = _DB2Connection.CreateCommand();
db2Command.CommandText = cmdTxt;
var db2Parameter = db2Command.CreateParameter();
db2Parameter.ArrayLength = ids.Length;
db2Parameter.DB2Type = DB2Type.DynArray;
db2Parameter.ParameterName = "#ids";
db2Parameter.Value = ids;
db2Command.Parameters.Add(db2Parameter);
var results = ExecuteQuery(db2Command);
return results.ToArray();
}
private object[] ExecuteQuery(DB2Command db2Command)
{
_DB2Connection.Open();
var resultList = new ArrayList();
var results = db2Command.ExecuteReader();
while (results.Read())
{
var values = new object[results.FieldCount];
results.GetValues(values);
resultList.Add(values);
}
results.Close();
_DB2Connection.Close();
return resultList.ToArray();
}
You cannot send in an array as a parameter. You would have to do something to build out a list of parameters, one for each of your values.
e.g.: SELECT DISTINCT ID,COL2,COL3 FROM TABLE WHERE ID IN ( #id1, #id2, ... #idN )
And then add the values to your parameter collection:
cmd.Parameters.Add("#id1", DB2Type.Integer).Value = your_val;
Additionally, there are a few things I would do to improve your code:
Use using statements around your DB2 objects. This will automatically dispose of the objects correctly when they go out of scope. If you don't do this, eventually you will run into errors. This should be done on DB2Connection, DB2Command, DB2Transaction, and DB2Reader objects especially.
I would recommend that you wrap queries in a transaction object, even for selects. With DB2 (and my experience is with z/OS mainframe, here... it might be different for AS/400), it writes one "accounting" record (basically the work that DB2 did) for each transaction. If you don't have an explicit transaction, DB2 will create one for you, and automatically commit after every statement, which adds up to a lot of backend records that could be combined.
My personal opinion would also be to create a .NET class to hold the data that you are getting back from the database. That would make it easier to work with using IntelliSense, among other things (because you would be able to auto-complete the property name, and .NET would know the type of the object). Right now, with the array of objects, if your column order or data type changes, it may be difficult to find/debug those usages throughout your code.
I've included a version of your code that I re-wrote that has some of these changes in it:
public List<ReturnClass> GetResults(int[] ids)
{
using (var conn = new DB2Connection())
{
conn.Open();
using (var trans = conn.BeginTransaction(IsolationLevel.ReadCommitted))
using (var cmd = conn.CreateCommand())
{
cmd.Transaction = trans;
var parms = new List<string>();
var idCount = 0;
foreach (var id in ids)
{
var parm = "#id" + idCount++;
parms.Add(parm);
cmd.Parameters.Add(parm, DB2Type.Integer).Value = id;
}
cmd.CommandText = "SELECT DISTINCT ID,COL2,COL3 FROM TABLE WHERE ID IN ( " + string.Join(",", parms) + " ) ";
var resultList = new List<ReturnClass>();
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
var values = new ReturnClass();
values.Id = (int)reader["ID"];
values.Col1 = reader["COL1"].ToString();
values.Col2 = reader["COL2"].ToString();
resultList.Add(values);
}
}
return resultList;
}
}
}
public class ReturnClass
{
public int Id;
public string Col1;
public string Col2;
}
Try changing from:
db2Parameter.DB2Type = DB2Type.DynArray;
to:
db2Parameter.DB2Type = DB2Type.Integer;
This is based on the example given here
I have number of records that I have loaded from database and I want to update set couple values before I return them. The only main requirement I do not want multiple commands updating each record one by one.
Please note my ids are GUIDs(uniqueidentifier)
so far I have my code as
public IEnumerable<Person> GetUnprocessedPeople(int batchSize)
{
List<Queue_ImportQueue> list;
using (IDbConnection db = OpenedConnection)
{
string peopleList = $"SELECT TOP({batchSize}) * FROM [dbo].[Person]";
list = db.Query<Person>(peopleList).ToList();
using (IDbTransaction transactionScope = db.BeginTransaction(IsolationLevel.Serializable))
{
string updateQuery = $"UPDATE [dbo].[Person] SET Created = GETDATE() WHERE Id ='#ids'";
try
{
db.Execute(updateQuery, new { Id = list.Select(x => x.Id) }, transactionScope);
transactionScope.Commit();
}
catch (Exception ex)
{
transactionScope.Rollback();
throw;
}
}
}
return list;
}
OK the solution was quite simple.
There is no need for specific casting.
just ensure you have all arguments correct.
So extract that has fixed the query for me:
string updateQuery = $"UPDATE [dbo].[Person] SET Created = GETDATE() WHERE Id ='#Ids'";
...
db.Execute(updateQuery, new { Ids = list.Select(x => x.Id) }, transactionScope);
I have two functions that each return the same list of objects. But, the one that uses TSQL is much faster than the one using Entity Framework and I do not understand why one would be faster than the other. Is it possible to modify my EF function to work as fast as the TSQL one?
Any help will be appreciated. My code is below:
TSQL:
public static List<ChartHist> ListHistory_PureSQL()
{
List<DataRow> listDataRow = null;
string srtQry = #"Select LoginHistoryID,
LoginDuration as LoginDuration_Pass,
0 as LoginDuration_Fail,
LoginDateTime,
LoginLocationID,
LoginUserEmailID,
LoginApplicationID,
LoginEnvironmentID,
ScriptFrequency,
LoginStatus,
Reason
From LoginHistory
Where LoginStatus = 'Pass'
UNION
Select LoginHistoryID,
0 as LoginDuration_Pass,
LoginDuration as LoginDuration_Fail,
LoginDateTime,
LoginLocationID,
LoginUserEmailID,
LoginApplicationID,
LoginEnvironmentID,
ScriptFrequency,
LoginStatus,
Reason
From LoginHistory
Where LoginStatus = 'Fail'";
using (SqlConnection conn = new SqlConnection(Settings.ConnectionString))
{
using (SqlCommand objCommand = new SqlCommand(srtQry, conn))
{
objCommand.CommandType = CommandType.Text;
DataTable dt = new DataTable();
SqlDataAdapter adp = new SqlDataAdapter(objCommand);
conn.Open();
adp.Fill(dt);
if (dt != null)
{
listDataRow = dt.AsEnumerable().ToList();
}
}
}
var listChartHist = (from p in listDataRow
select new ChartHist
{
LoginHistoryID = p.Field<Int32>("LoginHistoryID"),
LoginDuration_Pass = p.Field<Int32>("LoginDuration_Pass"),
LoginDuration_Fail = p.Field<Int32>("LoginDuration_Fail"),
LoginDateTime = p.Field<DateTime>("LoginDateTime"),
LoginLocationID = p.Field<Int32>("LoginLocationID"),
LoginUserEmailID = p.Field<Int32>("LoginUserEmailID"),
LoginApplicationID = p.Field<Int32>("LoginApplicationID"),
LoginEnvironmentID = p.Field<Int32>("LoginEnvironmentID"),
ScriptFrequency = p.Field<Int32>("ScriptFrequency"),
LoginStatus = p.Field<String>("LoginStatus"),
Reason = p.Field<String>("Reason")
}).ToList();
return listChartHist;
}
EF:
public static List<ChartHist> ListHistory()
{
using (var db = new LatencyDBContext())
{
var loginHist = (from hist in db.LoginHistories
select new { LoginHistory = hist }).ToList();
//PUT LOGIN HISTORY RECORDS INTO A LOCAL LIST
var listHistory = new List<ChartHist>();
foreach (var item in loginHist)
{
var localHistData = new ChartHist();
localHistData.LoginHistoryID = item.LoginHistory.LoginHistoryID;
//split up the duration for pass and fail values
if (item.LoginHistory.LoginStatus.ToUpper() == "PASS")
{
localHistData.LoginDuration_Pass = Convert.ToDouble(item.LoginHistory.LoginDuration);
localHistData.LoginDuration_Fail = 0;
}
else if (item.LoginHistory.LoginStatus.ToUpper() == "FAIL")
{
localHistData.LoginDuration_Pass = 0;
localHistData.LoginDuration_Fail = Convert.ToDouble(item.LoginHistory.LoginDuration);
}
localHistData.LoginDateTime = item.LoginHistory.LoginDateTime;
localHistData.LoginLocationID = item.LoginHistory.LoginLocationID;
localHistData.LoginUserEmailID = item.LoginHistory.LoginUserEmailID;
localHistData.LoginApplicationID = item.LoginHistory.LoginApplicationID;
localHistData.LoginEnvironmentID = item.LoginHistory.LoginEnvironmentID;
localHistData.LoginStatus = item.LoginHistory.LoginStatus;
localHistData.Reason = item.LoginHistory.Reason;
localHistData.ScriptFrequency = item.LoginHistory.ScriptFrequency;
listHistory.Add(localHistData);
}
return listHistory;
}
}
Of course EF will take longer to execute than a plain old SQL query, and there's very little that you can do about it (except write the most optimal LINQ queries that you can).
There's a very simple reason why this is so. Running a direct SQL command will just send back the data, with no muss and no fuss attached to it, waiting for you to do the data manipulations to get it to the point where it fits nicely into whatever data structure you want it in. Running EF, on the other hand, means that not only does it run the SQL command, but it massages the data for you into objects that you can manipulate right away. That extra action of going through ADO.NET and converting the data into the objects automatically means that it will take longer than just doing the plain SQL query.
On the flip side of that coin, however, EF does provide a very nice and simple way to debug and solve whatever problems you might have from a specific query/function (like by any exceptions thrown).
I can't performance test this, but try this solution instead before you remove EF entirely:
var loginHist = db.LoginHistories.Where(item => item.LoginStatus.ToUpper() == "PASS" || item.LoginStatus.ToUpper() == "FAIL")
.Select(item => new ChartHist()
{
LoginHistoryID = item.LoginHistoryID,
LoginDuration_Pass = item.LoginStatus.ToUpper() == "PASS" ? Convert.ToDouble(item.LoginDuration) : 0,
LoginDuration_Fail = item.LoginStatus.ToUpper() == "FAIL" ? Convert.ToDouble(item.LoginDuration) : 0,
LoginDateTime = item.LoginDateTime,
LoginLocationID = item.LoginLocationID,
LoginUserEmailID = item.LoginUserEmailID,
LoginApplicationID = item.LoginApplicationID,
LoginEnvironmentID = item.LoginEnvironmentID,
LoginStatus = item.LoginStatus,
Reason = item.Reason,
ScriptFrequency = item.ScriptFrequency,
});
return loginHist.ToList();
This is the "correct" way to populate a new object from a select. It will only retrieve the data you care about, and will put it directly into the object, rather than converting it into an object and then converting it again, from one object to another.
Note: I prefer the functional calls to the from / select form, but it'd be correct either way.