I tried to find a solution for this but couldn't. Here's the problem:
I'm loading data of a bunch of users and creating an object for each user. Each user object has many object properties. Here's the structure:
public class User {
public int ID { get; set; }
public string Name { get; set; }
public City City { get; set; }
public Office Office { get; set; }
}
The City class:
public class City {
public int ID { get; set; }
public string Name { get; set; }
public string Keyword { get; set; }
}
The Office class:
public class Office {
public int ID { get; set; }
public string Address { get; set; }
public int CityID { get; set; }
}
The user object has many other similar properties like City & Office which are basically class objects.
Now here's the main issue. Whenever I try to load all the users into a collection of dictionary, StackOverflow exception occurs at SqlCon.Open() (See the "Fetch" function I've written below). Here's how I'm loading everything:
//Code to load users
Dictionary<int, User> Users = new Dictionary<int, Users>();
DataTable usersData = new DataTable();
//The Fetch function has two version. The first one; which is mentioned in this post, returns the result as Dictionary<string, object>().
//The second version of the function returns the result in the form of the a DataTable and is only used when multiple rows are required from the database. The following returns a set of rows in a DataTable.
Globals.MainDatabase.Fetch("SELECT * FROM users", out usersData);
foreach (DataRow row in usersData.Rows) {
User user = new User();
user.ID = Convert.ToInt32(row["id"]);
user.Name = row["name"].ToString();
user.City = Cities.Get(Convert.ToInt32(row["city_id"]));
user.Office = Offices.Get(Convert.ToInt32(row["office_id"]));
Users.Add(user.ID, user);
}
The methods "Cities.Get(Int32 id)" and "Offices.Get(Int32 id)" uses the following function to fetch data from the database.
public void Fetch(string query, out Dictionary<string, object> results) {
var dict = new Dictionary<string, object>();
try {
using (SqlConnection SqlCon = new SqlConnection(ConnectionString)) {
using (SqlCmd = new SqlCommand()) {
SqlCmd.Connection = SqlCon;
SqlCmd.CommandType = CommandType.Text;
SqlCmd.CommandText = query;
SqlCon.Open();
DataTable temp = new DataTable();
using (SqlDataAdapter SqlAdp = new SqlDataAdapter(SqlCmd)) {
SqlAdp.SelectCommand = SqlCmd;
SqlAdp.Fill(temp);
}
DataRow row = temp.Rows[0];
temp = null;
dict = row.Table.Columns
.Cast<DataColumn>()
.ToDictionary(col => col.ColumnName, col => row.Field<object>(col.ColumnName));
}
}
}
catch (Exception ex) {
HandleException(ex, "An error occurred when tried to fetch data.", query);
}
results = dict;
dict = null;
}
I realize that this "Fetch" function is being called multiple times when creating the user object. The "StackOverflow" exception occurs exactly at this line:
SqlCon.Open();
How can I solve this error? or probably I should use a better approach to do this?
A bit too long for a comment
Do you really need to load ALL data from the database? It is better to just grab the columns and rows you need.
Why are you copying DataTables in to Dictionaries? What is wrong with just using a DataTable?
99.9% of the time you will have better performance performing JOINs in the database.
Don't try to roll your own 'ORM'. Use something like Dapper if you don't want the bloat of EF or NHibernate. Or, stick to ADO (DataTable, DataAdapter etc.)
using (SqlConnection SqlCon = new SqlConnection(ConnectionString)) {
Is your connection string variable really named ConnectionString? Is it possible you are having a name clash with the type? Since it is not declared in the shown code, I assume it is a class variable, so you should respect the conventional naming convention, which would be _connectionString. What does your connection string look like?
Alright folks, I figured it out. It was all because of faulty architecture of the whole demo application. Some objects have one or more other objects as properties and due to some silly faults in the architecture; the "fetch" operation which serves as the base of fetching data from the database, was recursively called resulting into StackOverflow exception which was actually a large number of database connections being initialized ultimately growing the heap size to an extent which causes the exception.
I tried to summarize everything in the paragraph written above because posting the complete source code is useless considering the large amount of code.
Thank you everyone who helped, especially #Guffa's comment on the main post which forced me to investigate the whole issue from scratch rather than sticking to the exception stack.
Related
I need to send data from my game server to the client through json and getting a large sum of data from the database and sending it creates an issue
Currently I am creating multiple list in the call to the database to get the data and now I am stuck because I have the data but in multiple list and I can't figure out how to return the data.
I feel like I should be converting it to JSON here and then returning the json string but its all just really confusing
public static void GetLobbyList() {
string query = "SELECT * FROM que";
MySqlCommand cmd = new MySqlCommand(query, MySQL.mySQLSettings.connection);
MySqlDataReader reader = cmd.ExecuteReader();
List<int> ids = new List<int>();
List<string> uids = new List<string>();
List<int> bets = new List<int>();
List<string> games = new List<string>();
while (reader.Read()) {
ids.Add((int)reader["id"]);
uids.Add((string)reader["uid"]);
bets.Add((int)reader["bet"]);
games.Add((string)reader["game"]);
}
reader.Close();
}
So here I am reading the info from the database and for lack of experience I am adding each data point to a list(which is really ugly and I know there must be a better way)
So essentially I Grab the data->Parse to json->Send string to client
If It is possible assuming the data returned is from a table with
id | uid | bet | game
I would Like to return an array in json that looks like
{
{
"id" : 1,
"uid" : "erjfh4982y9hf",
"bet" : 3,
"game" : "Snake"
}
{
"id" : 2,
"uid" : "gsegt34t",
"bet" : 2,
"game" : "Snake"
}
}
Im not too familiar with json and how it works but I do know that it is the only way to send large packets of data from my server to my client because everything has to be converted to bytes before it can be sent and my framework does not support converting list to bytes
To achieve what you are aiming for, you should create a class to hold the data you have retrieved from the database. In my case I called it GameObject and it's defined as follows.
public class GameObject
{
public int Id { get; set; }
public string Uid { get; set; }
public int Bet { get; set; }
public string Game { get; set; }
}
After retrieving the information from the database, you'll need to run code similar to the following.
var items = new List<GameObject>();
while (reader.Read())
{
items.Add(new GameObject
{
Id = (int)reader["id"],
Uid = (string)reader["uid"],
Bet = (int)reader["bet"],
Game = (string)reader["game"]
});
}
// Return the jsonPacket which will contain all the items in json format.
var jsonPacket = JsonConvert.SerializeObject(items);
In order for this to work, you'll need to reference the Newtonsoft.Json library from nuget. Open your package manager console and type the following command: Install-Package Newtonsoft.Json and it will set it up for you. At the top of our code you'll need to have using Newtonsoft.Json; to be able to use the classes inside the library to serialise to Json. On the receiving end you can pass the string to JsonConvert.DeserializeObject<T>() and you'll get back your list of objects.
You could use a list of objects and anonymous types. There's also a JSON serializer, System.Web.Script.Serialization.JavaScriptSerializer included if you add the System.Web.Extensions assembly to your references.
List<object> list = new List<object>();
while (reader.Read()) {
list.Add(new { id = (int)reader["id"],
uid = (string)reader["uid"],
bet = (int)reader["bet"],
game = (string)reader["game"], });
}
string json = new System.Web.Script.Serialization.JavaScriptSerializer().Serialize(list);
I's struggling to get my hands around a database and It seems that every programmer had it's own way of doing it.
I need very simple database that is stored locally and could eventually be moved into a server without much effort. Also I don't want design the database manually as this suppose to only reflect the data used in my program. So I decided to try LINQ with automatic mapping but I have a problem.
This is what I've done:
Create new WPF project.
Add new Service-Based-Database item named myDB.
Write following code:
public partial class MainWindow : Window {
public MainWindow() {
InitializeComponent();
var dc = new DataContext("myDB.mdf");
var dt = dc.GetTable<SimpleData>();
var ds = from d in dt select d;
foreach (var d in ds) {
Debug.WriteLine($"{d.number} : {d.text}");
}
}
}
[Table]
public class SimpleData {
[Column]
public int number { get; set; } = 10;
[Column]
public string text { get; set; } = "hello";
public SimpleData() { }
}
Now at the foreach loop I have an exception:
An attempt to attach an auto-named database for file myDB.mdf failed.
A database with the same name exists, or specified file cannot be
opened, or it is located on UNC share.
What is missing?
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 4 years ago.
Improve this question
I am working on a c# application where you can select movies from a combobox and it displays the movie info and the actors that were in it in a datagrid, gets all the information from a MySql database. Now I've googled most of what I need to learn and implement it into my code. The blocks of code look massive and were wondering if there was any way to dry it up a bit, such as what variables can I reuse if any, etc.
//film select
var queryfilmsearch = "SELECT title, description, release_year, rental_rate, length, rating FROM film WHERE title LIKE + #value";
MySqlCommand cmdfilmsearch = new MySqlCommand(queryfilmsearch, PubVar.connection);
cmdfilmsearch.Parameters.AddWithValue("#value", comBoxFilm.Text);
MySqlDataAdapter adpfilmserach = new MySqlDataAdapter(cmdfilmsearch);
DataSet dsfilmsearch = new DataSet();
adpfilmserach.Fill(dsfilmsearch);
dataGridView1.ReadOnly = true;
dataGridView1.DataSource = dsfilmsearch.Tables[0];
//actor from film select
var queryfilmactor = "SELECT first_name, last_name FROM actor INNER JOIN film_actor ON actor.actor_id = film_actor.actor_id INNER JOIN film ON film.film_id = film_actor.film_id WHERE film.title LIKE + #value";
MySqlCommand cmdfilmactor = new MySqlCommand(queryfilmactor, PubVar.connection);
cmdfilmactor.Parameters.AddWithValue("#value", comBoxFilm.Text);
MySqlDataAdapter adpfilmactor = new MySqlDataAdapter(cmdfilmactor);
DataSet dsfilmactor = new DataSet();
adpfilmactor.Fill(dsfilmactor);
dataGridView2.ReadOnly = true;
dataGridView2.DataSource = dsfilmactor.Tables[0];
It's basically the same code just different query and variable names
This is a lot of boilerplate, but it doesn't have to be.
First off, you're using your connections wrong. They implement IDisposable. You should take special care in handling them properly. This means that storing them as a variable is usually a bad idea. Open and close them as quickly as possible, and don't share them.
Second, you're using raw ADO.NET. While it's good to have an idea of how ADO.NET works (since it's the building block for most relational database code in .NET), it's better to use an abstraction. You end up writing less boilerplate. More concise, easier to read and refactor code. We often use Object Relational Mappers to accomplish this. There's two styles: Micro, and "full". There are several micro ORM's out there for .NET: Dapper, Npoco, PetaPoco etc. Stack Overflow (the website) uses Dapper. Then there's full ORM's such as Entity Framework and NHibernate.
You're using DataSet and DataTable. Those are poor abstractions. They don't follow Object Oriented Programming principles very well, they're too flexible, and they're inefficient. It's better to create custom classes and then use some form of ORM to map from your code to your custom classes.
You're also doing data access directly in your Web Forms. That's never a good idea. Database access should be done in a separate layer. That makes it easier to swap out either the data layer or the presentation layer, and makes it easier to re-use your data access code throughout other parts of your presentation layer.
Keeping all this in mind, we might end up with something like below. I chose Dapper. And I assumed Web Forms, but I think you'll get the idea no matter what your presentation layer is:
Data Repository
public class MySqlFilmRepository : IFilmRepository
{
readonly string _connectionString { get; set; }
public FilmRepsitory(string connectionString)
{
_connectionString = connectionString;
}
public List<Film> SearchFilmsByTitle(string title)
{
using (var connection = new MySqlConnection(_connectionString))
{
List<Film> films = connection.Query<Film>("SELECT title, description, release_year, rental_rate, length, rating FROM film WHERE title LIKE #Title", new { Title = title }).AsList();
return films;
}
}
public List<Actor> GetActorsForFilm(string filmTitle)
{
using (var connection = new MySqlConnection(_connectionString))
{
List<Actor> actors = connection.Query<Actor>("SELECT first_name, last_name FROM actor INNER JOIN film_actor ON actor.actor_id = film_actor.actor_id INNER JOIN film ON film.film_id = film_actor.film_id WHERE film.title LIKE #FilmTitle", new { FilmTitle = filmTitle }).AsList();
return actors;
}
}
}
public interface IFilmRepository
{
List<Film> SearchFilmsByTitle(string title);
List<Actor> GetActorsForFilm(string filmTitle);
}
Model classes
public class Film
{
public string Title { get; set; }
public string Description { get; set; }
public int ReleaseYear { get; set; }
public decimal RentalRate { get; set; }
public int Length { get; set; }
public string Rating { get; set; }
}
public class Actor
{
public string FirstName { get; set; }
public string LastName { get; set; }
}
Web Forms code:
IFilmRepository _filmRepository;
public void Page_Load (object sender, EventArgs e)
{
_filmRepository = new MySqlFilmRepository(ConfigurationManager.ConnectionStrings["MySqlConnectionString"].ConnectionString);
}
protected void SearchButton_Click(object sender, EventArgs e)
{
dataGridView1.DataSource = _filmRepository.SearchFilmsByTitle(SearchTextBox.Text);
dataGridView2.DataSource = _filmRepository.GetActorsForFilm(SearchTextBox.Text);
}
It ends up being more code, but now you have a centralized place to get your data calls, and you're handling your connections properly, and you have less boilerplate code to query your database, and it's more flexible. This is more code, but it's also not really repeating itself very much.
I am trying to create a web service, with C# and Visual Studio 2010, that will insert values form a view table into another table using a SQL query. In the past we have used this type of service but we have asked the user to enter the values they want to update, but with the query I am using there is no need for user input since the values will be inserted only when one of the field values is NULL.
Below is the code I been trying to use, I adapted the code we used before, but in here I am still expecting user input. How can I update my code as to not expect the input. I know I can just run this query from MySQL but I want be able to create a button in our site so other people can do it. Here is the code that runs the query:
public void AddUsers(string json)
{
GetDbInstance();
var sql = "insert into dbo.ASPECTS_users_activity " +
"(user_id, usr_address, usr_phone, usr_email, usr_website, usr_photo ) " +
"select #v.customer_id, #usr_address, #usr_phone, #usr_email, #usr_website, #usr_photo " +
"from dbo.ASPECTS_current_users_vw v where usr_photo is NULL;";
var usrInformation = JsonConvert.DeserializeObject<UpdateExhibitors>(json);
Console.WriteLine(usrInformation);
var conn = db.Connection();
try
{
SqlCommand command = new SqlCommand(sql, conn);
command.Parameters.AddWithValue("#v.customer_id", usrInformation.master_customer_id);
command.Parameters.AddWithValue("#usr_address", "123 XYZ Post, Columbus, IN");
command.Parameters.AddWithValue("#usr_phone", "555-555-5555");
command.Parameters.AddWithValue("#usr_email", "test#sample.com");
command.Parameters.AddWithValue("#usr_website", "http://www.sample.com");
command.Parameters.AddWithValue("#usr_photo", "");
command.ExecuteNonQuery();
command.Dispose();
command = null;
}
catch (Exception e)
{
throw new Exception(e.ToString(), e);
}
finally
{
conn.Close();
}
}
Here is the Model
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
namespace Customer_Insert.Models {
public class AddUsers {
public string customer_id { get; set; }
public string usr_address { get; set; }
public string usr_phone { get; set; }
public string usr_email { get; set; }
public string usr_website { get; set; }
public string usr_photo { get; set; }
}
}
Just so you know the only field that will be populated from the view to will be the customer_id field, all the other fields will have default values, which will be updated at another point. Not many people here know SQL so creating this option will provide an option if I am not around.
Since you "can just run this query from MySQL", I would save that query as a Stored Procedure in your database. Then you need neither SQL nor parameters in your code.
SqlCommand command = new SqlCommand("yourStoredProcedureName", conn);
command.ExecuteNonQuery();
...
I am looking for design advice for the following scenario:
I have a code-first EF5 MVC application. I am building a full-text search function which will incorporate multiple weighted columns from many tables. As I cannot create view with an index from these tables (some of them contain text / binary columns), I have created a stored procedure which will output the ID of my object (eg. PersonID) and the rank associated with that object based on the search terms.
My current approach is to create a helper class for executing full text searches which call the stored procedure(s) and load all the objects from the context based on the returned IDs.
My questions are:
Does my approach seem sensible / follow reasonable best practice?
Has anyone else done something similar with any lessons learned?
Is there a way to do this more efficiently (i.e. have the results of the stored procedure return/map to the entities directly without an additional look-up required?)
UPDATE
Moved my detailed implementation from an edit of the question into its own answer to be more in line with what is recommended frequently # meta.stackexchange.com
Seeing as you can't use SQL methods like containstable with entityframework code first which the rest of your application could be using you could be 'forced' to do something with a storedprocedure like your describe. Whether it's best practice I don't know. However it it gets the job done I don't see why it wouldn't be sensible.
Yes - I have and still am working on a project build around EF codefirst where I had to do a fairly complex search that included several search parameters marked as 'must have' and several values marked as 'nice to have' and in from that return a weighted result.
Depending on the complexity of the result set I don't think you need to do a second roundtrip to the database and I will show you a way I have been doing it below.
Bear in mind that below is simply an example:
public List<Person> GetPeople(params string[] p)
{
var people = new List<Person>();
using (var db = new DataContext())
{
var context = ((IObjectContextAdapter)db).ObjectContext;
db.Database.Connection.Open();
var command = db.Database.Connection.CreateCommand();
command.CommandText = "SomeStoredProcedureReturningWeightedResultSetOfPeople";
command.CommandType = System.Data.CommandType.StoredProcedure;
//Add parameters to command object
people = context.Translate<Person>(command.ExecuteReader()).ToList();
}
return people;
}
Even though the storedprocedure will have a column for the weight value it won't get mapped when you translate it.
You could potentially derive a class from Person that includes the weight value if you needed it.
Posting this as an answer rather than an edit to my question:
Taking some of the insight provided by #Drauka's (and google) here is what I did for my initial iteration.
Created the stored procedure to do the full text searching. It was really too complex to be done in EF even if supported (as one example some of my entities are related via business logic and I wanted to group them returning as a single result). The stored procedure maps to a DTO with the entity id's and a Rank.
I modified this blogger's snippet / code to make the call to the stored procedure, and populate my DTO: http://www.lucbos.net/2012/03/calling-stored-procedure-with-entity.html
I populate my results object with totals and paging information from the results of the stored procedure and then just load the entities for the current page of results:
int[] projectIDs = new int[Settings.Default.ResultsPerPage];
foreach (ProjectFTS_DTO dto in
RankedSearchResults
.Skip(Settings.Default.ResultsPerPage * (pageNum - 1))
.Take(Settings.Default.ResultsPerPage)) {
projectIDs[index] = dto.ProjectID;
index++;
}
IEnumerable<Project> projects = _repository.Projects
.Where(o=>projectIDs.Contains(o.ProjectID));
Full Implementation:
As this question receives a lot of views I thought it may be worth while to post more details of my final solution for others help or possible improvement.
The complete solution looks like:
DatabaseExtensions class:
public static class DatabaseExtensions {
public static IEnumerable<TResult> ExecuteStoredProcedure<TResult>(
this Database database,
IStoredProcedure<TResult> procedure,
string spName) {
var parameters = CreateSqlParametersFromProperties(procedure);
var format = CreateSPCommand<TResult>(parameters, spName);
return database.SqlQuery<TResult>(format, parameters.Cast<object>().ToArray());
}
private static List<SqlParameter> CreateSqlParametersFromProperties<TResult>
(IStoredProcedure<TResult> procedure) {
var procedureType = procedure.GetType();
var propertiesOfProcedure = procedureType.GetProperties(BindingFlags.Public | BindingFlags.Instance);
var parameters =
propertiesOfProcedure.Select(propertyInfo => new SqlParameter(
string.Format("#{0}",
(object) propertyInfo.Name),
propertyInfo.GetValue(procedure, new object[] {})))
.ToList();
return parameters;
}
private static string CreateSPCommand<TResult>(List<SqlParameter> parameters, string spName)
{
var name = typeof(TResult).Name;
string queryString = string.Format("{0}", spName);
parameters.ForEach(x => queryString = string.Format("{0} {1},", queryString, x.ParameterName));
return queryString.TrimEnd(',');
}
public interface IStoredProcedure<TResult> {
}
}
Class to hold stored proc inputs:
class AdvancedFTS :
DatabaseExtensions.IStoredProcedure<AdvancedFTSDTO> {
public string SearchText { get; set; }
public int MinRank { get; set; }
public bool IncludeTitle { get; set; }
public bool IncludeDescription { get; set; }
public int StartYear { get; set; }
public int EndYear { get; set; }
public string FilterTags { get; set; }
}
Results object:
public class ResultsFTSDTO {
public int ID { get; set; }
public decimal weightRank { get; set; }
}
Finally calling the stored procedure:
public List<ResultsFTSDTO> getAdvancedFTSResults(
string searchText, int minRank,
bool IncludeTitle,
bool IncludeDescription,
int StartYear,
int EndYear,
string FilterTags) {
AdvancedFTS sp = new AdvancedFTS() {
SearchText = searchText,
MinRank = minRank,
IncludeTitle=IncludeTitle,
IncludeDescription=IncludeDescription,
StartYear=StartYear,
EndYear = EndYear,
FilterTags=FilterTags
};
IEnumerable<ResultsFTSDTO> resultSet = _context.Database.ExecuteStoredProcedure(sp, "ResultsAdvancedFTS");
return resultSet.ToList();
}