I'd like my application (which is deployed to thousands of locations) to be able to monitor its own environment, in particular the sql server box. When the SQL Server is local, this is easy to do as I can use the System.Diagnostics library and collect the windows performance counters through that. When the SQL server is remote however I have a problem because I cannot guarantee that my c# application (a windows service in this case) has access to the remote sql server machine OTHER THAN the fact that I know it can access the SQL Server.
Any ideas on how I can collect the windows performance counters off that machine through SQL?
One way I've thought of is to use SQL CLR, but it seems to require UNSAFE access, which I'm pretty sure I don't want.
Thanks
Mark as you stated in the comments
"i can't guarantee that I will have the proper privileges"
Either you have to write some kind of functionality into your installer that is going to make sure you have the proper privileges (maybe won't install without admin setting privileges) for remote perf. counters access.
You didn't state exactly what you were trying to monitor, however, might I suggest using the built int SQL Server Dynamic Management Views (DMV's)
MSDN
Good Articles Videos
Very Useful Examples
these have become very robust in recent releases of SQL server and allow you to monitor the majority of the stats you would work with in perfmon (CPU, IO, MEMORY, many more). Perhaps not as much granularity as perfmon but usually I find them to be more useful then perfmon nowadays for quick pertinent data. And as long as your connection string user has admin rights you don't have to worry about server privileges. And this functionality is allready built into sql-server Give it a look see.
Edit ...
#Mark Here are some DMV scripts I use to get IO time on a database, not as exact as PerfMon however it allows you to focus in on a specific database file.
Check out the "dm_io_virtual_file_stats" DMV
select
read_stall_ms = case when num_of_reads = 0 then 0 else (io_stall_read_ms/num_of_reads) end,
write_stall_ms = case when io_stall_write_ms = 0 then 0 else (io_stall_write_ms/num_of_writes) end,
total_stall_ms = case when (num_of_reads = 0 and num_of_writes = 0) then 0 else (io_stall/(num_of_reads + num_of_writes)) end,
db = db_name(vfs.database_id),
mf.physical_name,
vfs.*
from sys.dm_io_virtual_file_stats(null, null) as vfs
join sys.master_files as mf on vfs.database_id = mf.database_id and vfs.file_id = mf.file_id
order by total_stall_ms desc
select m.database_id,
db_name(m.database_id) as database_name,
m.file_id,
m.name as file_name,
m.physical_name,
m.type_desc,
fs.num_of_reads,
fs.num_of_bytes_read,
fs.io_stall_read_ms,
fs.num_of_writes,
fs.num_of_bytes_written,
fs.io_stall_write_ms
from sys.dm_io_virtual_file_stats(NULL, NULL) fs
join sys.master_files m on fs.database_id = m.database_id and fs.file_id = m.file_id
Something a little fancier ...
select db_name(d.database_id) as database_name,
quotename(object_schema_name(d.object_id, d.database_id)) + N'.' + quotename(object_name(d.object_id, d.database_id)) as object_name,
d.database_id,
d.object_id,
d.page_io_latch_wait_count,
d.page_io_latch_wait_in_ms,
d.range_scans,
d.index_lookups,
case when mid.database_id is null then 'N' else 'Y' end as missing_index_identified
from (select
database_id,
object_id,
row_number() over (partition by database_id order by sum(page_io_latch_wait_in_ms) desc) as row_number,
sum(page_io_latch_wait_count) as page_io_latch_wait_count,
sum(page_io_latch_wait_in_ms) as page_io_latch_wait_in_ms,
sum(range_scan_count) as range_scans,
sum(singleton_lookup_count) as index_lookups
from sys.dm_db_index_operational_stats(NULL, NULL, NULL, NULL)
where page_io_latch_wait_count > 0
group by database_id, object_id ) as d
left join (select distinct database_id, object_id from sys.dm_db_missing_index_details) as mid on mid.database_id = d.database_id and mid.object_id = d.object_id
If you sign your assembly, you can create a login from the assembly, grant it the server-level permission to run unsafe assemblies, and create the assembly from the dll. This allows you to run unsafe assemblies w/o having to turn the trustworthy flag on the database on.
I would think you need something more robust than self-rolled SQL CLR for this. MS has a tool for precisly this called SCOM.
I have had very good results in envs with 50 servers.
You collect loads of configurable data with built in point in time reports, data warehousing and alerts.
good wikipedia article
technet
Related
Is there a way to dump the generated sql to the Debug log or something? I'm using it in a winforms solution so the mini-profiler idea won't work for me.
I got the same issue and implemented some code after doing some search but having no ready-to-use stuff. There is a package on nuget MiniProfiler.Integrations I would like to share.
Update V2: it supports to work with other database servers, for MySQL it requires to have MiniProfiler.Integrations.MySql
Below are steps to work with SQL Server:
1.Instantiate the connection
var factory = new SqlServerDbConnectionFactory(_connectionString);
using (var connection = ProfiledDbConnectionFactory.New(factory, CustomDbProfiler.Current))
{
// your code
}
2.After all works done, write all commands to a file if you want
File.WriteAllText("SqlScripts.txt", CustomDbProfiler.Current.ProfilerContext.BuildCommands());
Dapper does not currently have an instrumentation point here. This is perhaps due, as you note, to the fact that we (as the authors) use mini-profiler to handle this. However, if it helps, the core parts of mini-profiler are actually designed to be architecture neutral, and I know of other people using it with winforms, wpf, wcf, etc - which would give you access to the profiling / tracing connection wrapper.
In theory, it would be perfectly possible to add some blanket capture-point, but I'm concerned about two things:
(primarily) security: since dapper doesn't have a concept of a context, it would be really really easy for malign code to attach quietly to sniff all sql traffic that goes via dapper; I really don't like the sound of that (this isn't an issue with the "decorator" approach, as the caller owns the connection, hence the logging context)
(secondary) performance: but... in truth, it is hard to say that a simple delegate-check (which would presumably be null in most cases) would have much impact
Of course, the other thing you could do is: steal the connection wrapper code from mini-profiler, and replace the profiler-context stuff with just: Debug.WriteLine etc.
You should consider using SQL profiler located in the menu of SQL Management Studio → Extras → SQL Server Profiler (no Dapper extensions needed - may work with other RDBMS when they got a SQL profiler tool too).
Then, start a new session.
You'll get something like this for example (you see all parameters and the complete SQL string):
exec sp_executesql N'SELECT * FROM Updates WHERE CAST(Product_ID as VARCHAR(50)) = #appId AND (Blocked IS NULL OR Blocked = 0)
AND (Beta IS NULL OR Beta = 0 OR #includeBeta = 1) AND (LangCode IS NULL OR LangCode IN (SELECT * FROM STRING_SPLIT(#langCode, '','')))',N'#appId nvarchar(4000),#includeBeta bit,#langCode nvarchar(4000)',#appId=N'fea5b0a7-1da6-4394-b8c8-05e7cb979161',#includeBeta=0,#langCode=N'de'
Try Dapper.Logging.
You can get it from NuGet. The way it works is you pass your code that creates your actual database connection into a factory that creates wrapped connections. Whenever a wrapped connection is opened or closed or you run a query against it, it will be logged. You can configure the logging message templates and other settings like whether SQL parameters are saved. Elapsed time is also saved.
In my opinion, the only downside is that the documentation is sparse, but I think that's just because it's a new project (as of this writing). I had to dig through the repo for a bit to understand it and to get it configured to my liking, but now it's working great.
From the documentation:
The tool consists of simple decorators for the DbConnection and
DbCommand which track the execution time and write messages to the
ILogger<T>. The ILogger<T> can be handled by any logging framework
(e.g. Serilog). The result is similar to the default EF Core logging
behavior.
The lib declares a helper method for registering the
IDbConnectionFactory in the IoC container. The connection factory is
SQL Provider agnostic. That's why you have to specify the real factory
method:
services.AddDbConnectionFactory(prv => new SqlConnection(conStr));
After registration, the IDbConnectionFactory can be injected into
classes that need a SQL connection.
private readonly IDbConnectionFactory _connectionFactory;
public GetProductsHandler(IDbConnectionFactory connectionFactory)
{
_connectionFactory = connectionFactory;
}
The IDbConnectionFactory.CreateConnection will return a decorated
version that logs the activity.
using (DbConnection db = _connectionFactory.CreateConnection())
{
//...
}
This is not exhaustive and is essentially a bit of hack, but if you have your SQL and you want to initialize your parameters, it's useful for basic debugging. Set up this extension method, then call it anywhere as desired.
public static class DapperExtensions
{
public static string ArgsAsSql(this DynamicParameters args)
{
if (args is null) throw new ArgumentNullException(nameof(args));
var sb = new StringBuilder();
foreach (var name in args.ParameterNames)
{
var pValue = args.Get<dynamic>(name);
var type = pValue.GetType();
if (type == typeof(DateTime))
sb.AppendFormat("DECLARE #{0} DATETIME ='{1}'\n", name, pValue.ToString("yyyy-MM-dd HH:mm:ss.fff"));
else if (type == typeof(bool))
sb.AppendFormat("DECLARE #{0} BIT = {1}\n", name, (bool)pValue ? 1 : 0);
else if (type == typeof(int))
sb.AppendFormat("DECLARE #{0} INT = {1}\n", name, pValue);
else if (type == typeof(List<int>))
sb.AppendFormat("-- REPLACE #{0} IN SQL: ({1})\n", name, string.Join(",", (List<int>)pValue));
else
sb.AppendFormat("DECLARE #{0} NVARCHAR(MAX) = '{1}'\n", name, pValue.ToString());
}
return sb.ToString();
}
}
You can then just use this in the immediate or watch windows to grab the SQL.
Just to add an update here since I see this question still get's quite a few hits - these days I use either Glimpse (seems it's dead now) or Stackify Prefix which both have sql command trace capabilities.
It's not exactly what I was looking for when I asked the original question but solve the same problem.
I'm working on a pretty special, legacy project where I need to build an app for PDA devices under Windows Mobile 6.5. The devices have a local database (SQL Server CE) which we are supposed to sync with a remote database (Microsoft Access) whenever they are docked and have network access.
So the local database using SQL Server CE works fine, but I can’t figure out a way to sync it to the Access database properly.
I read that ODBC and OLEDB are unsupported under Windows Mobile 6.5, most ressources I find are obsolete or have empty links, and the only way I found was to export the local database relevant tables in XML in the hope to build a VBA component for Access to import them properly. (and figure out backwards sync).
Update on the project and new questions
First of all, thanks to everyone who provided an useful answer, and to #josef who saved me a lot of time with the auto path on this thread.
So a remote SQL Server is a no go for security reasons (client is paranoid about security and won't provide me a server). So I'm tied to SQL Server CE on the PDA and Access on the computer.
As for the sync:
The exportation is fine: I'm using multiple dataAdapters and a WriteXML method to generate XML files transmitted by FTP when the device is plugged back in. Those files are then automatically imported into the Access database. (see code at the end).
My problem is on the importation: I can acquire data through XML readers from an Access-generated file. This data is then inserted in a dataset (In fact, I can even print the data on the PDA screen) but I can't figure out a way to do an "UPSERT" on the PDA's database. So I need a creative way to update/insert the data to the tables if they already contains data with the same id.
I tried two methods, with SQL errors (from what I understood it's SQL Server CE doesn't handle stored procedures or T-SQL). Example with a simple query that is supposed to update the "available" flag of some storage spots:
try
{
SqlCeDataAdapter dataAdapter = new SqlCeDataAdapter();
DataSet xmlDataSet = new DataSet();
xmlDataSet.ReadXml(localPath +#"\import.xml");
dataGrid1.DataSource = xmlDataSet.Tables[1];
_conn.Open();
int i = 0;
for (i = 0; i <= xmlDataSet.Tables[1].Rows.Count - 1; i++)
{
spot = xmlDataSet.Tables[1].Rows[i].ItemArray[0].ToString();
is_available = Convert.ToBoolean(xmlDataSet.Tables[1].Rows[i].ItemArray[1]);
SqlCeCommand importSpotCmd = new SqlCeCommand(#"
IF EXISTS (SELECT spot FROM spots WHERE spot=#spot)
BEGIN
UPDATE spots SET available=#available
END
ELSE
BEGIN
INSERT INTO spots(spot, available)
VALUES(#spot, #available)
END", _conn);
importSpotCmd.Parameters.Add("#spot", spot);
importSpotCmd.Parameters.Add("#available", is_available);
dataAdapter.InsertCommand = importSpotCmd;
dataAdapter.InsertCommand.ExecuteNonQuery();
}
_conn.Close();
}
catch (SqlCeException sql_ex)
{
MessageBox.Show("SQL database error: " + sql_ex.Message);
}
I also tried this query, same problem SQL server ce apparently don't handle ON DUPLICATE KEY (I think it's MySQL specific).
INSERT INTO spots (spot, available)
VALUES(#spot, #available)
ON DUPLICATE KEY UPDATE spots SET available=#available
The code of the export method, fixed so it works fine but still relevant for anybody who wants to know:
private void exportBtn_Click(object sender, EventArgs e)
{
const string sqlQuery = "SELECT * FROM storage";
const string sqlQuery2 = "SELECT * FROM spots";
string autoPath = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase); //get the current execution directory
using (SqlCeConnection _conn = new SqlCeConnection(_connString))
{
try
{
SqlCeDataAdapter dataAdapter1 = new SqlCeDataAdapter(sqlQuery, _conn);
SqlCeDataAdapter dataAdapter2 = new SqlCeDataAdapter(sqlQuery2, _conn);
_conn.Open();
DataSet ds = new DataSet("SQLExport");
dataAdapter1.Fill(ds, "stock");
dataAdapter2.Fill(ds, "spots");
ds.WriteXml(autoPath + #"\export.xml");
}
catch (SqlCeException sql_ex)
{
MessageBox.Show("SQL database error: " + sql_ex.Message);
}
}
}
As Access is more or less a stand-alone DB solution I strongly recommend to go with a full flavored SQL Server plus IIS to setup a Merge Replication synchronisation between the SQL CE data and the SQL Server data.
This is described with full sample code and setup in the book "Programming the .Net Compact Framework" by Paul Yao and David Durant (chapter 8, Synchronizing Mobile Data).
For a working sync, all changes to defined tables and data on the server and the CE device must be tracked (done via GUIDs, unique numbers) with there timestamps and a conflict handling has to be defined.
If the data is never changed by other means on the server, you may simply track Device side changes only and then push them to the Access database. This could be done by another app that does Buld Updates like described here.
If you do not want to go the expensive way to SQL Server, there are cheaper solutions with free SQLite (available for CE and Compact Framework too) and a commercial Sync tool for SQLite to MSAccess like DBSync.
If you are experienced, you may create your own SQLite to MS ACCESS sync tool.
I am having an issue where I am looking at a legacy application that is using SqlXmlCommand objects to get data from the database. There is an .xsd file that has the tables that are being used, and what fields, their relationships etc. The issue that we are having is it works most of the time, but not all. I am wondering if there is a way to check what is actually being run on Sql Server. I don't have the SQL profiler installed so that option is out.
the code looks like:
SqlXmlCommand xcmd = new SqlXmlCommand(DataAccess.OleDbConnectionString);
xcmd.CommandType = SqlXmlCommandType.XPath;
xcmd.SchemaPath = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, #"myXsd.xsd"));
xcmd.XslPath = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, String.Format(#"myXsl.xsl", ReportType)));
xcmd.CommandText = "id[#PK=$PK]";
SqlXmlParameter p = xcmd.CreateParameter();
p.Name = "#PK";
p.Value = Id;
using (Stream s = xcmd.ExecuteStream()) { ... }
This blows up at the ExectureStream() with the error:
SQLXML: error loading XML result (XML document must have a top level element.)
We believe that there is some data abnormality that is causing the xml to not generate properly, and that is why we want to see what is exactly run.
Cheers
You can try the below two queries, you might need to tweak it a little, but to give you an idea, the first gives you a list of all requests, and the second will give you the detail of the request by its request id (session_id)
SELECT *
FROM sys.dm_exec_requests
DBCC INPUTBUFFER (12345)
Although I would personally rather try and debug the C# app first and view what's being sent over to the server from the VS debugger before bothering with checking what's being run on SQL Server
Also, DBCC INPUTBUFFER might give you something like EXECUTE dbo.MyStoredProc 'params...', to dig deeper, or otherwise a more straightforward query, you can run this
SELECT r.session_id, r.[status], r.command, t.[text]
FROM sys.dm_exec_requests r
CROSS APPLY sys.dm_exec_sql_text(r.[sql_handle]) t
I need to setup sharding over replica set as recommended in MongoDB reference for high availability & scalability. I have few questions about connection string and its behavior for C# driver in that scenario (code snippet below):
Is the connection string below looks right for connecting to mongos instances: mongos1, mongos2 & mongos3?
What happens to client if one of the mongos instance crashes? Will the failed call handled gracefully by retrying to second mongos instance? Does the client blacklist the failed mongos instance and try after sometime?
If I want to set readpreference, will the driver be aware of replica set existence and honor setting ReadPreference?
Code snippet:
MongoUrlBuilder bldr = new MongoUrlBuilder();
List<MongoServerAddress> servers = new List<MongoServerAddress>();
servers.Add(new MongoServerAddress("mongos1:27016"));
servers.Add(new MongoServerAddress("mongos2:27016"));
servers.Add(new MongoServerAddress("mongos3:27016"));
bldr.Username = "myuser";
bldr.Password = "mypwd";
bldr.Servers = servers;
bldr.DatabaseName = "mydb";
bldr.ReadPreference = ReadPreference.Primary;
var server = MongoServer.Create(bldr.ToMongoUrl());
1) Yes, this is just fine. Note that all of this could be put in an actual connection string as well. mongodb://myuser:mypwd#mongos1:27016,mongos2:27016,mongos3:27016/mydb/?readPreference=primary
2) The way your connection string is built, you'll be load balancing across the 3 mongos. If one goes down, then the other two will simply begin to receive more traffic. Errors, however, will happen and nothing gets retried automatically. You'll need to handle the errors and make decisions based on each query/write whether it is safe to retry.
3) The driver, when talking to a sharded system, will simply forward the read preference to mongos. Note that mongos version 2.2 had some difficulty with read preferences. I'd advise you to be on the 2.4 line.
I've investigated the possibilities of creating database backups through SMO with C#.
The task is quite easy and code straightforward. I've got only one question: how can I check if the backup was really created?
SqlBackup.SqlBackup method returns no parameters and I don't even know if it throws any exceptions. (the only thing that I know is that it is blocking, because there's also SqlBackupAsync method)
I would appreciate any help.
you can and its very possible to do what you asked for,
but doing the backup it self using SMO its not very hard, but the hard part is managing the backup and the restore.
it would be hard to put all the code here, but its wont fit. so I will try my best to put the lines you need.
SqlBackup.SqlBackup doesn't return any value, its a void function.
but it takes one parameter which is "Server", try out the following code:
Server srvSql;
//Connect to Server using your authentication method and load the databases in srvSql
// THEN
Backup bkpDatabase = new Backup();
bkpDatabase.Action = BackupActionType.Database;
bkpDatabase.Incremental = true; // will take an incemental backup
bkpDatabase.Incremental = false; // will take a Full backup
bkpDatabase.Database = "your DB name";
BackupDeviceItem bDevice = new BackupDeviceItem("Backup.bak", DeviceType.File);
bkpDatabase.Devices.Add(bDevice );
bkpDatabase.PercentCompleteNotification = 1;// this for progress
bkpDatabase.SqlBackup(srvSql);
bkpDatabase.Devices.Clear();
I've investigated the problem using Reflector.NET (I suppose this is legal since RedGate is Ms Gold Certified Partner and Reflector.NET opens .NET libraries out of the box). As I found out the method throws two types of exceptions:
FailedOperationException - in most cases, other exceptions are "translated" (I suppose translating means creating new FailedOperationException and setting InnerException to what was actually thrown)
UnsupportedVersionException - in one case when log truncation is set to TruncateOnly and server major version is more or equal to 10 (which is sql server 2008?)
This solves my problem partially, because I'm not 100% sure that if something goes wrong those exceptions will actually be thrown.