I have Data table having 3k and more columns , as we all know that SQL Tables does not support more than 1024 Columns .I will be having this another table for it .
I just need to split DataTable if columns are greater than 1024.
I need an efficient way.
Thanks in advance.
I have never seen database table with such a big number of columns.
I do not know, how many rows does your DataTable typically have. You could transpose DataTable or below is a code snippet how to split DataTable. I used #jdweng idea with copy table with al values and then delete unwanted columns backwards. I cannot assess if this is the most efficient way. I did not test it on DataTable with rows.
static List<DataTable> SplitTables(DataTable largeTable)
{
var tables = new List<DataTable>();
int chunkSize = 1024;
int cycles = largeTable.Columns.Count / chunkSize;
for (int i = 0; i <= cycles; i++)
{
var tempTable = largeTable.Copy();
for (int j = largeTable.Columns.Count - 1; j >= 0; j--)
{
if (!(j >= i * chunkSize && j < (i + 1) * chunkSize))
{
tempTable.Columns.RemoveAt(j);
}
}
tables.Add(tempTable);
}
return tables;
}
Read about normalization, as typically properly normalized database does not require tables with 1024+ columns. If will not help, you question can be solved by e.g.
1) count number of columns
SELECT count(*)
FROM information_schema.columns c
JOIN information_schema.tables t ON c.TABLE_NAME = t.TABLE_NAME
AND c.table_schema = t.table_schema
WHERE table_type = 'base table'
and c.table_name ='table_name_here'
2) if more than expected - create a new table
CREATE TABLE new_table_name_here
(...)
Related
I'm trying to create a generic method of taking a datatable and creating a SQL INSERT statement from its contents.
I've spent the better part of a couple of days researching this on StackOverflow.
I just need some help with the looping section code--it doesn't give the desired results and I've reached the limit of my coding skills.
I don't need BulkCopy solutions or database specific ones. I just need the INSERT string. I don't care about SQL injections or anything else. This is an internal legacy app that needs to have a few repetitive/manual processes automated.
I have written an example console program to demonstrate. The size of the datatable isn't really a factor form me. It is never over 60 rows and has 6 columns:
The datatable should look like:
ID Group X Y Z
1 A 100 200 400
2 B 200 400 800
3 C 300 600 1200
4 D 400 800 1600
5 E 500 1000 2000
The example console app is:
using System;
using System.Data;
namespace crSQLInsert
{
class Program
{
static void Main(string[] args)
{
DataTable dt = new DataTable();
dt.Clear();
dt.Columns.Add("ID");
dt.Columns.Add("Group");
dt.Columns.Add("X");
dt.Columns.Add("Y");
dt.Columns.Add("Z");
for (int i = 0; i < 5; i++)
{
Random rnd = new Random();
char cLtr = Convert.ToChar(i+65);
double colVal = ((i + 1) * 100) ;
DataRow dr = dt.NewRow();
dr["ID"] = 1 + i;
dr["Group"] = cLtr;
dr["X"] = colVal ;
dr["Y"] = colVal * 2;
dr["Z"] = colVal * 4;
dt.Rows.Add(dr);
}
createSQL(dt);
}
public static void createSQL(DataTable dtSQL)
{
// Create generic SQL query to add datatable contents to my_table
string sSQL = "";
sSQL += "INSERT INTO my_table (ID, Group, X, Y, Z) VALUES (";
for (int i = 0; i < dtSQL.Rows.Count; i++)
{
for (int j = 0; j < dtSQL.Columns.Count; j++)
{
sSQL += dtSQL.Rows[i][j].ToString().Trim();
if (j != dtSQL.Columns.Count - 1)
{
sSQL += "','";
}
else
{
sSQL += "')";
}
}
}
Console.WriteLine(sSQL);
}
}
}
The resulting SQL string should be like below:
(I've formatted it so that it's easier to read--the actual string doesn't necessarily need the \r\n as it will end up as a string in an ODBCCommand().
INSERT INTO my_table (ID, Group, X, Y, Z)
VALUES (1','A','100','200','400'),
(2','B','200','400','800'),
(3','C','300','600','1200'),
(4','D','400','800','1600'),
(5','E','500','1000','2000');
Thanks for any help.
I have two data tables with auto generated columns and info from two different Access databases.
The names are:
Export data table (source)
School data table (target)
I want to select the information that the export database has beginning from column 19 (assuming 0=index) and insert that information in the school data table beginning from column 203 (assuming 0-index)
i already tried this method:
try
{
cb = new OleDbCommandBuilder(dataAdapterSchool);
cb.GetUpdateCommand();
cb.GetInsertCommand();
cb.GetDeleteCommand();
for (int i = 0; i <= 19; i++)
{
DataRow newRow = schoolDb.NewRow();
newRow[i + 19] = exportDb.Columns[202 + i];
}
Grid.ItemsSource = schoolDb.DefaultView;
dataAdapterSchool.Update(schoolDb);
testbox.Text = cb.GetUpdateCommand().CommandText;
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
The problem here is that it gives me the error:
Unable to cast object of type 'System.Data.DataColumn' to type 'System.IConvertible'
Which is (according to google) the error that occurs when you don't specify the column names that the source should insert to. Since there are many columns, I want to find a way to do this without specifying the column names.
Any help would be appreciated,
There's an mistake in your code:
for (int i = 0; i <= 19; i++)
{
DataRow newRow = schoolDb.NewRow();
newRow[i + 19] = exportDb.Columns[202 + i]
}
you need to iterate over rows and replace exportDb.Columns[202 + i] for exportDb.Rows[x][202 + i]
I created a new user defined function in excel using c#, excelDna Add-In,what i want to get is when a user open an excel file, clic on a cell and write =myFunction() press enter, data should be displayed in the excel file. these data are retrieved from sql server database and stocked in a 2D array of object, my problem is when i try to display this array in excel range i got this exception
Exception de HRESULT : 0x800A03EC
Below is my code :
public static void LoadViewData()
{
var target = (ExcelReference)XlCall.Excel(XlCall.xlfCaller);
var sheetName = (string)XlCall.Excel(XlCall.xlSheetNm, target);
var application = (Microsoft.Office.Interop.Excel.Application)ExcelDnaUtil.Application;
var sheet = application.Sheets[Regex.Replace(sheetName, #"\[[^]]*\]", string.Empty)];
object[,] result = LoadFromDbData();
var startCell =sheet.Cells[target.RowFirst + 1, target.ColumnFirst];
var endCell =sheet.Cells[target.RowFirst+ result.GetUpperBound(0) - result.GetLowerBound(0) + 1,
target.ColumnFirst+ result.GetUpperBound(1) - result.GetLowerBound(1) + 1];
var writeRange = sheet.Range[startCell, endCell];
writeRange.Value2 = result;
}
target returns the correct value of the cell where the user has written the formula (=myFunction())
sheetName returns the correct activeSheet in which the user writes the formula
result contains the data retrieved from sql server, it is an array of object[854,8]
startcell and endcell represents the range from which cell to which cell data will be displayed
when debugging, all variables contain the correct values, the exception appears in this instruction :
writeRange.Value2 = result;
Anyone has already worked with this or can help please ?
Thanks
I think your "LoadFromDbData()" is returning the data as typed. Try converting each value to a string. Here is a sample (and I can recreate that error code if I do not convert to string):
void Main()
{
var tbl = new System.Data.DataTable();
new SqlDataAdapter(#"
WITH tally ( OrderNo, UniqueId, RandNumber )
AS (
SELECT TOP 50000
ROW_NUMBER() OVER ( ORDER BY t1.object_id ),
NEWID(),
CAST(CAST(CAST(NEWID() AS VARBINARY(4)) AS INT) AS DECIMAL) / 1000
FROM master.sys.all_columns t1
CROSS JOIN master.sys.all_columns t2
)
SELECT OrderNo,
DATEADD(DAY, -OrderNo, GETDATE()) as OrnekDate,
UniqueId, RandNumber,
abs(RandNumber)%100 / 100 as pct
FROM [tally];", #"server=.\SQLExpress;Database=master;Trusted_Connection=yes;").Fill(tbl);
object[,] arr = new object[tbl.Rows.Count + 1, tbl.Columns.Count];
for (int i = 0; i < tbl.Columns.Count; i++)
{
arr[0, i] = tbl.Columns[i].Caption;
}
for (int i = 0; i < tbl.Rows.Count; i++)
{
for (int j = 0; j < tbl.Columns.Count; j++)
{
arr[i + 1, j] = tbl.Rows[i][j].ToString(); // without .ToString() you should have the error
}
}
// Excel dosya yarat ve arrayi koy
Microsoft.Office.Interop.Excel.Application xl = new Microsoft.Office.Interop.Excel.Application();
var workbook = xl.Workbooks.Add();
xl.Visible = true;
Worksheet sht = ((Worksheet)workbook.ActiveSheet);
Range target = (Range)sht.Range[ (Range)sht.Cells[1,1], (Range)sht.Cells[arr.GetUpperBound(0)+1,arr.GetUpperBound(1)+1] ];
target.Value2 = arr;
}
Note: As a side note, why would you transfer the data as a 2D array? That is one of the ways, but beware it is limited (I don't know a good value for the upper limit - try a high number like 200K rows).
Getting data into excel is best done via QueryTables.Add or CopyFromRecordSet to my experience. Depending on your needs you might also directly use the Excel file itself as a data table and do inserts. There is also EPPlus library on Nuget but that would be a little bit slow + may not contain all the capabilities you need.
I have a big problem with writing some data to a csv file. I have a lot of measurement values. Every value is described by name, unit, value. So i want to build for every value a column with these three properties.
I want to store it into the csv file like this:
Width Cell Wall Thickness Coarseness Curl-Index etc.
mm mm mg/m % etc.
16,2 3,2 0,000 11,7 etc.
Till now i was coding a header for the names, another for the units and the values (that were previously stored into a string array) i just wrote in one line.
Till now my csv file looks like this:(
Width;Cell Wall Thickness;Coarseness;Curl-Index;etc.
mm;mm;mg/m;%;etc.
16,2;3,2;0,000;11,7;etc.
if it were not many values i wouldn't care about this but there are a lot so when i open the csv file there's the problem that the headers dont fit to the values and units. It's not organized, i cannot match the values to the headers.
I would like everything to be organized in columns. Any help would be strongly appreciated!
That's the code that i have till now:
StreamWriter sw = new StreamWriter("test2.csv");
int RowCount = 3;
int ColumnCount = 4;
string[][] Transfer_2D = new string[RowCount][];
Transfer_2D[0] = new string[3]{"Width", "CWT", "Coarseness", "Curl-Index"};//name of the values
Transfer_2D[1] = new string[3] {"mm", "mm", "mg/m", "%"}; //units
Transfer_2D[2] = new string[3] { TransferData[0], TransferData[1], TransferData[2], TransferData[3] };
for (int i = 0; i < RowCount; i++)
{
for (int j = 0; j < ColumnCount; j++)
{
sw.Write(Transfer_2D[i][j]);//write one row separated by columns
if (j < ColumnCount)
{
sw.Write(";");//use ; as separator between columns
}
}
if (i < RowCount)
{
sw.Write("\n");//use \n to separate between rows
}
}
sw.Close();
}
you can set the string to a fixed length.
example look here: (.NET Format a string with fixed spaces)
int iWantedStringLength = 20;
string sFormatInstruction = "{0,-" + iWantedStringLength.ToString() + "}";
for (int i = 0; i < RowCount; i++)
{
for (int j = 0; j < ColumnCount; j++)
{
sw.Write(String.Format(sFormatInstruction, Transfer_2D[i][j]));//write one row separated by columns
if (j < ColumnCount)
{
sw.Write(";");//use ; as separator between columns
}
}
if (i < RowCount)
{
sw.Write("\n");//use \n to separate between rows
}
}
For CSV work I use http://joshclose.github.io/CsvHelper/, this is a very nice helper class but it would require you to change your working a little bit.
I would advice you create a class to store each entry in and then create a "Mapper" to map it to csv fields. Then you can just pass a collection of objects and your mapping class to the helper and it produces a structured CSV.
There are alot of examples on that page so it should be straight forward for you to work through.
How to randomly display 10 question from database? How can i check the answer is correct with database or not?
db.command(true, "SELECT * FROM question WHERE Age_group='" +category + "'");
foreach (DataRow item in db.result.Rows)
{
question_list.Add (Convert.ToInt32(item["id"]));
}
for (int i = 0; i < max_question; i++)
{
int index = ran.Next(question_list.Count);
question_choose.Add(question_list[index]);
question_list.Remove(question_list[index]);
}
select top 10 * from table order by newid()
also, see "order by newid()" - how does it work?