We Keep Coding

How to add a sql database to an elastic pool programmatically?

I have the following console application which creates a ShardManagerDB and creates one database for each company on the main database.
I can see on azure the databases created on the server however they are not on the elastic pool.
Question:
1. Is this doable with the current API?
2. If not, what are other recommended approaches?
using System.Data.SqlClient;
using mynm.Data;
using System.Linq;
using mynm.Models.GlobalAdmin;
namespace mynm.DbManagementTool
{
class Program
{
static void Main(string[] args)
{
SetupSSM();
}
//This will create the Shard Management DB if it doesnt exist
private static void SetupSSM()
{
SqlConnectionStringBuilder connStrBldr = new SqlConnectionStringBuilder
{
UserID = SettingsHelper.AzureUsernamedb,
Password = SettingsHelper.AzurePasswordDb,
ApplicationName = SettingsHelper.AzureApplicationName,
DataSource = SettingsHelper.AzureSqlServer
};
DbUtils.CreateDatabaseIfNotExists(connStrBldr.ConnectionString, SettingsHelper.Azureshardmapmgrdb);
Sharding sharding = new Sharding(SettingsHelper.AzureSqlServer, SettingsHelper.Azureshardmapmgrdb, connStrBldr.ConnectionString);
CreateShardPerCompany(sharding);
}
private static void CreateShardPerCompany(Sharding sharding)
{
SqlConnectionStringBuilder connStrBldr = new SqlConnectionStringBuilder
{
UserID = SettingsHelper.AzureUsernamedb,
Password = SettingsHelper.AzurePasswordDb,
ApplicationName = SettingsHelper.AzureApplicationName,
DataSource = SettingsHelper.AzureSqlServer
};
UnitOfWork unitOfWork = new UnitOfWork();
ConfigurationDBDataContext context = new ConfigurationDBDataContext();
context.Empresas.Add(new Empresa()
{
Id = 1,
Nombre = "company name 1",
NIT = "873278423",
NombreRepresentanteLegal = "myself",
TelefonoRepresentanteLegal = "32894823",
NombreContacto = "myself",
TelefonoContacto = "32423"
});
context.SaveChanges();
var listofEmpresas = unitOfWork.EmpresaRepository.Get().ToList();
foreach(Empresa empresa in listofEmpresas)
{
DbUtils.CreateDatabaseIfNotExists(connStrBldr.ConnectionString, empresa.NIT);
sharding.RegisterNewShard(SettingsHelper.AzureSqlServer, empresa.NIT, connStrBldr.ConnectionString, empresa.Id);
}
}
}
}
the sharding.css
internal class Sharding
{
public ShardMapManager ShardMapManager { get; private set; }
public ListShardMap<int> ShardMap { get; private set; }
// Bootstrap Elastic Scale by creating a new shard map manager and a shard map on
// the shard map manager database if necessary.
public Sharding(string smmserver, string smmdatabase, string smmconnstr)
{
// Connection string with administrative credentials for the root database
SqlConnectionStringBuilder connStrBldr = new SqlConnectionStringBuilder(smmconnstr);
connStrBldr.DataSource = smmserver;
connStrBldr.InitialCatalog = smmdatabase;
// Deploy shard map manager.
ShardMapManager smm;
if (!ShardMapManagerFactory.TryGetSqlShardMapManager(connStrBldr.ConnectionString, ShardMapManagerLoadPolicy.Lazy, out smm))
{
this.ShardMapManager = ShardMapManagerFactory.CreateSqlShardMapManager(connStrBldr.ConnectionString);
}
else
{
this.ShardMapManager = smm;
}
ListShardMap<int> sm;
if (!ShardMapManager.TryGetListShardMap<int>("ElasticScaleWithEF", out sm))
{
this.ShardMap = ShardMapManager.CreateListShardMap<int>("ElasticScaleWithEF");
}
else
{
this.ShardMap = sm;
}
}
// Enter a new shard - i.e. an empty database - to the shard map, allocate a first tenant to it
// and kick off EF intialization of the database to deploy schema
// public void RegisterNewShard(string server, string database, string user, string pwd, string appname, int key)
public void RegisterNewShard(string server, string database, string connstr, int key)
{
Shard shard;
ShardLocation shardLocation = new ShardLocation(server, database);
if (!this.ShardMap.TryGetShard(shardLocation, out shard))
{
shard = this.ShardMap.CreateShard(shardLocation);
}
SqlConnectionStringBuilder connStrBldr = new SqlConnectionStringBuilder(connstr);
connStrBldr.DataSource = server;
connStrBldr.InitialCatalog = database;
// Go into a DbContext to trigger migrations and schema deployment for the new shard.
// This requires an un-opened connection.
using (var db = new ElasticScaleContext<int>(connStrBldr.ConnectionString))
{
// Run a query to engage EF migrations
(from b in db.Terceros
select b).Count();
}
// Register the mapping of the tenant to the shard in the shard map.
// After this step, DDR on the shard map can be used
PointMapping<int> mapping;
if (!this.ShardMap.TryGetMappingForKey(key, out mapping))
{
this.ShardMap.CreatePointMapping(key, shard);
}
}
}

In the code implementing database creation: DbUtils.CreateDatabaseIfNotExists() -- you are probably using a T-SQL CREATE DATABASE command to Create an Azure database on a logical server. Currently CREATE DATABASE doesn't support specifying the Pool -- however an update to Azure DB is expected within the next month that will extend the functionality of CREATE DATABASE and ALTER DATABASE to specify the Pool name as well.
In the meantime, you can make a REST API call from the CreateDatabaseIfNotExists() routine to add the database to the pool once it is created, using the Update SQL Database command: https://msdn.microsoft.com/en-us/library/azure/mt163677.aspx.
However, making a rest call from inside your c# can be complex, and is discussed here: http://www.asp.net/web-api/overview/advanced/calling-a-web-api-from-a-net-client . It may be simpler to wait for the upcoming support in the CREATE DATABASE command, which would require a very small modification within your CreateDatabaseIfNotExists() routine.

How to read an SQL query generated by Dapper?

I have a standard code:
public IEnumerable ExperimentSelect(object parameters)
{
using (var connection = new SqlConnection(ConnectionString))
{
connection.Open();
var dynamicparam = new DynamicParameters(parameters);
var rows = connection.Query("[dbo].[ptbSapOrderSelect]", dynamicparam,
commandType: CommandType.StoredProcedure);
if (rows.Any())
TotalRows = ((long)rows.ToList()[0].TotalRows);
return rows;
}
}
How to automate saving queries generated by Dapper to the file using eg NLog? I am thinking of getting source of SQL query as shown in the SQL Server Profiler.

I managed to make this work in an ASP.Net MVC app using MiniProfiler.
First, configure MiniProfiler as per the docs. Make sure that you are wrapping your SqlConnection inside a ProfiledDbConnection.
Note that you don't need to enable the visual widget for this to work, just ensure that a profile is started before, and ended after, each request.
Next, in global.asax.cs where the profile for that request is stopped, amend it as follows:
protected void Application_EndRequest()
{
// not production code!
MiniProfiler.Stop();
var logger = NLog.LogManager.GetCurrentClassLogger();
var instance = MiniProfiler.Current;
if (instance == null) return;
var t = instance.GetSqlTimings();
foreach (var sqlTiming in t)
{
logger.Debug(sqlTiming.CommandString);
}
}
This literally dumps the SQL command executed, but there is a lot more information included in the model if you want to report more advanced information.

asp.net caching limit? [duplicate]

This question already has an answer here:
Closed 10 years ago.
Possible Duplicate:
ASP.NET cache maximum size
I'm caching quite a lot of datatables using asp.net caching (the floowing code):
HttpContext.Current.Cache.Insert(GlobalVars.Current.applicationID + "_" + cacheName, itemToCache, null, System.Web.Caching.Cache.NoAbsoluteExpiration, TimeSpan.FromMinutes(240));
However I think that the cache on the server is getting full and having to re-obtain the datatable data from the database. Is there any limit to the amount of data that can be cached on the server or any IIS settings that can be tweaked?

There is a way to upgrade the limit but I would strongly recommend that you use other kind of Caching System (more about this below).
.NET Cache
To know more about the .NET Caching limitation, please read this great answer from a Microsoft .NET Team member.
If you want to see the current limits of .NET Cache, you can try:
var r = new Dictionary<string, string>();
using (var pc = new PerformanceCounter("ASP.NET Applications", "Cache % Machine Memory Limit Used", true))
{
pc.InstanceName = "__Total__";
r.Add("Total_MachineMemoryUsed", String.Concat(pc.NextValue().ToString("N1"), "%"));
}
using (var pc = new PerformanceCounter("ASP.NET Applications", "Cache % Process Memory Limit Used", true))
{
pc.InstanceName = "__Total__";
r.Add("Total_ProcessMemoryUsed", String.Concat(pc.NextValue().ToString("N1"), "%"));
}
using (var pc = new PerformanceCounter("ASP.NET Applications", "Cache API Entries", true))
{
pc.InstanceName = "__Total__";
r.Add("Total_Entries", pc.NextValue().ToString("N0"));
}
using (var pc = new PerformanceCounter("ASP.NET Applications", "Cache API Misses", true))
{
pc.InstanceName = "__Total__";
r.Add("Total_Misses", pc.NextValue().ToString("N0"));
}
using (var pc = new PerformanceCounter("ASP.NET Applications", "Cache API Hit Ratio", true))
{
pc.InstanceName = "__Total__";
r.Add("Total_HitRatio", String.Concat(pc.NextValue().ToString("N1"), "%"));
}
using (var pc = new PerformanceCounter("ASP.NET Applications", "Cache API Trims", true))
{
pc.InstanceName = "__Total__";
r.Add("Total_Trims", pc.NextValue().ToString());
}
MemCached
I'm currently using Memcached, and if you're hosting your site somewhere, you can use a paid service like:
http://www.memcachier.com/
Or, if you're using your own server, you can download Couchbase Community Edition and hosted our own.
You will find more questions here about the use of MemCache, such as:
Which .NET Memcached client do you use, EnyimMemcached vs. BeITMemcached?
how to start with memcached
Make room for any Cache system
To use other cache system without changing your code, you could adopt to create an interface, like
public interface ICacheService
{
T Get<T>(string cacheID, Func<T> getItemCallback) where T : class;
void Clear();
}
then is you're using .NET Cache, your implementation would be something like
public class InMemoryCache : ICacheService
{
private int minutes = 15;
public T Get<T>(string cacheID, Func<T> getItemCallback) where T : class
{
T item = HttpRuntime.Cache.Get(cacheID) as T;
if (item == null)
{
item = getItemCallback();
HttpRuntime.Cache.Insert(
cacheID,
item,
null,
DateTime.Now.AddMinutes(minutes),
System.Web.Caching.Cache.NoSlidingExpiration);
}
return item;
}
public void Clear()
{
IDictionaryEnumerator enumerator = HttpRuntime.Cache.GetEnumerator();
while (enumerator.MoveNext())
HttpRuntime.Cache.Remove(enumerator.Key.ToString());
}
}
and you would use it as:
string cacheId = string.Concat("myinfo-", customer_id);
MyInfo model = cacheProvider.Get<MyInfo>(cacheId, () =>
{
MyInfo info = db.GetMyStuff(customer_id);
return info;
});
if you're using Memcached, all you need to do is create a new class that implement ICacheService and select the class you want, either by using IoC or direct call as:
private ICacheService cacheProvider;
protected override void Initialize(System.Web.Routing.RequestContext requestContext)
{
if (cacheProvider == null) cacheProvider = new InMemoryCache();
base.Initialize(requestContext);
}

The cache uses the memory allocation for the worker process. By default the worker process is allowed to get 60 percent of the machine memory in order to do its work .
As per the link, this can be changed to allow more of the machine memory to be used by the worker process by editing the machine.config file. Presumably you have the cache built to already update when it detects that data is out of date, so this should allow you to put more objects into cache.

When inserting an item into the cache add an CacheItemRemovedCallback method.
In the callback log the reason why the item was removed. This way you see if it memory pressure or something else.
public static void OnRemove(string key,
object cacheItem,
System.Web.Caching.CacheItemRemovedReason reason)
{
AppendLog("The cached value with key '" + key +
"' was removed from the cache. Reason: " +
reason.ToString());
}
http://msdn.microsoft.com/en-us/library/aa478965.aspx

How to use a single connection string in whole winforms project

I am using Winforms, MySQL and C# in my project. In that I use a connection string in the app settings.
At each new page I will declare a connection string as public and use this string in the connection.
MySqlConnection connection = new MySqlConnection(MyConString);
I want to declare this MyConString only one time in the whole application. How to do this? Where to do?

I don't think you should expose your connection string to your Forms, they don't need to know that. You can encapsulate the creation of connections with a simple factory.
public class ConnectionFactory
{
public static MySqlConnection Create()
{
string connectionString = ConfigurationManager.AppSettings["..."];
MySqlConnection conection = new MySqlConnection(Config.ConnectionStr);
connection.Open();
return connection;
}
}
Then when you need a connection in a Form you can do:
private void button1_click(object sender, EventArg args)
{
using ( var connection = ConnectionFactory.Create() )
{
connection.Execute("...");
}
}

You can try something like the following:
public static class Config
{
public static string ConnectionStr = ConfigurationManager.AppSettings["..."];
}
You can then use it in your code
MySqlConnection connection = new MySqlConnection(Config.ConnectionStr);

Enterpise Library from Microsoft have great DataAccess part which beside other prtoblem solving and this one

You may have a separate class to handle databases and add the connection string as a field there. Each time you want to connect to the database, you may use that class. Also if you may use a property to access the string outside the class if you require.
Hope this helps...

The suggested approach is available at an MSDN article titled Storing and Retrieving Connection Strings. The following samples are slightly modified from this article.
After storing your connection string in an app.config file, you can retrieve all connection strings like so:
static void GetConnectionStrings()
{
var settings = ConfigurationManager.ConnectionStrings;
if (settings != null) {
foreach(ConnectionStringSettings cs in settings) {
Console.WriteLine(cs.Name);
Console.WriteLine(cs.ProviderName);
Console.WriteLine(cs.ConnectionString);
}
}
}
You could alternatively get the connection string by name:
// Returns null if the name is not found.
static string GetConnectionStringByName(string name)
{
string returnValue = null; // Assume failure.
var settings = ConfigurationManager.ConnectionStrings[name];
if (settings != null) {
returnValue = settings.ConnectionString;
}
return returnValue;
}
This also gives you the ability of Securing Connection Strings so that your database username & password are not embedded into your application assembly in clear-text.

Migrate serialized objects to new version

I would like to migrate me previously serialized objects in database to new schema.
My previous object.
Public interface MyReport
{
string Id { get; set;}
string Name { get; set;}
Dictionary<string, string> PropColl { get; set;}
}
But for some reasons we had to make interface changes
Public interface IMarkme
{
}
Public interface MyReport<T> where T : Imarkme
{
string Id { get; set;}
string Name { get; set;}
T ExtendedProp { get; set;}
}
Public NewProp : Imarkme
{
/// some code here
}
So as you can see my interface has been modified and I would like to migrate my serialized objects which were serialized based on MyReport to MyReport
Can someone provide me some input as what kind of utility I should aim to write which can help me achieve migrating my serialized object to new modified interface version.
Thanks,
AG

I have actually done something similar recently, where I have created a simple console application to be able to transform some serialized objects from one version to another. I have simply used both versions of dlls and reflection to read and write the values of different properties. Probably you'll find this helpful as an inspiration ;)
static void Main(string[] args)
{
object test;
AppDomain.CurrentDomain.AssemblyResolve += domain_AssemblyResolve;
using (var con = new SqlConnection(connectionString))
{
using (var cmd = new SqlCommand())
{
cmd.CommandText = "select top 1 Data_Blob from dbo.Serialized";
cmd.CommandType = CommandType.Text;
cmd.Connection = con;
con.Open();
var blob = (byte[])cmd.ExecuteScalar();
var bf = new BinaryFormatter();
var stream = new MemoryStream(blob);
bf.AssemblyFormat = FormatterAssemblyStyle.Full;
test = bf.Deserialize(stream);
}
}
var objNewVersion = Activator.CreateInstance(Type.GetType("ObjectGraphLibrary.Test, ObjectGraphLibrary, Version=1.0.0.10, Culture=neutral, PublicKeyToken=33c7c38cf0d65826"));
var oldType = test.GetType();
var newType = objNewVersion.GetType();
var oldName = (string) oldType.GetProperty("Name").GetValue(test, null);
var oldAge = (int) oldType.GetProperty("Age").GetValue(test, null);
newType.GetProperty("Name").SetValue(objNewVersion, oldName, null);
newType.GetProperty("DateOfBirth").SetValue(objNewVersion, DateTime.Now.AddYears(-oldAge), null);
Console.Read();
}
static Assembly domain_AssemblyResolve(object sender, ResolveEventArgs args)
{
var assName = new AssemblyName(args.Name);
var uriBuilder = new UriBuilder(Assembly.GetExecutingAssembly().CodeBase);
var assemblyPath = Uri.UnescapeDataString(uriBuilder.Path);
var codeBase = Path.GetDirectoryName(assemblyPath);
var assPath = Path.Combine(codeBase, string.Format("old\\{0}.{1}.{2}.{3}\\{4}.dll", assName.Version.Major,
assName.Version.Minor, assName.Version.Build,
assName.Version.Revision, assName.Name));
return File.Exists(assPath) ? Assembly.LoadFile(assPath) : null;
}

1) Write a utility that reads the serialized objects in the old object definition.
2) The utility writes your objects into the DB in a non-serialized manner (ie, with one piece of data in every field, etc...).
Don't get into the habit of serializing objects, and storing the somewhere in persistent storage for retrieval (much) later. Serialization was not built for that.
You have run into the problem of C programmers in the old days: they would create a struct in memory, save that struct into a file. Then the struct's members would change, and they woudl wonder how to read it back, since the data was encoded differently.
then along came database formats, INI files, and so on, specifically to address this need, so saving data in one format, and then being able to read it without error.
So don't repeat errors of the past. Serialization was created to facilitate short-term binary storage and the ability to, say, transmit an object over TCP/IP.
At worst, store your data as XML, not as serialized binary stream. Also, there is no assurance that I know about from MS that says that serialized data from one version of .NET will be able to be read from another. Convert your data to a legible format while you can.