We Keep Coding

finding duplicate rows in excel and export those rows to another sheet using power shell

How to find duplicate values in Excel and export rows to another sheet using power shell?
I had an Excel sheet with multiple rows and column lets say from "A" to "k". I need to find duplicate rows only if values in all the columns in a row are unique. And the script should ignore D,E,F columns even though those column's values are same.
The script should also copy all those duplicate rows and should paste in a new excel file.it should also copy header row and source of duplicate rows and Also attaching a sample image of input file(the output file should also be same as input in this input case because it should also copy source duplicate rows). I had tried a code but it is throwing an error..please look into that and give me a solution for the code.
code:
# The Text OleDB driver is only available in PowerShell x86. Start x86
shell if using x64.
# This has to be the first check this script performs.
if ($env:Processor_Architecture -ne "x86") {
Write-Warning "Switching to x86 shell"
&"$env:windir\syswow64\windowspowershell\v1.0\powershell.exe"
"$PSCommandPath $args"; return
}
# Change to your CSV file name, must end in .csv or .tsv
$csvfile = "C:\files\A01modcsv.csv"
# Does the first row contain column names?
$firstRowColumns = $True
# What's the delimiter? Use `t for tabbed.
$csvdelimter = "`t"
$firstRowColumns = $true
$checkColumns = "A"
$datasource = Split-Path $csvfile
$tablename = (Split-Path $csvfile -leaf).Replace(".","#")
switch ($firstRowColumns) {
$true { $firstRowColumns = "Yes" }
$false { $firstRowColumns = "No" }
}
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
[void][Reflection.Assembly]::LoadWithPartialName("System.Data")
# Setup OleDB using Microsoft Text Driver.
$connstring = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=$datasource;Extended Properties='text;HDR=$firstRowColumns;FMT=Delimited($csvdelimter)';"
$conn = New-Object System.Data.OleDb.OleDbconnection
$conn.ConnectionString = $connstring
$conn.Open()
$cmd = New-Object System.Data.OleDB.OleDBCommand
$cmd.Connection = $conn
# Perform select on CSV file, then add results to a datatable using ExecuteReader
$sql = "SELECT $checkColumns, COUNT(*) as DupeCount FROM [$tablename] GROUP BY $checkColumns HAVING COUNT(*) > 1"
$cmd.CommandText = $sql
$dt = New-Object System.Data.DataTable
$dt.BeginLoadData()
$dt.Load($cmd.ExecuteReader([System.Data.CommandBehaviour]::CloseConnection))
$dt.EndLoadData()
$totaltime = [math]::Round($elapsed.Elapsed.TotalSeconds,2)
# Get Total Row Count
$cmd.CommandText = "SELECT COUNT(*) as TotalRows FROM [$tablename]"
$totalrows = $cmd.ExecuteScalar()
$conn.Close()[enter image description here][1]
# Output some stats
$dupecount = $dt.Rows.Count
Write-Host "Total Elapsed Time: $totaltime seconds. $dupecount duplicates found out of $totalrows total rows. You can access these dupes using `$dt." -ForegroundColor Green
I am getting an error in the menctioned code at "$dt.Load($cmd.ExecuteReader([System.Data.CommandBehaviour]::CloseConnection))" command....can any help me in solving this error.Thank you.

How to execute multi-line powershell command

Trying to execute a PowerShell command in asp.net c# but it returns no results, where am I going wrong?
var shell = PowerShell.Create();
var script = $#"$Groups = Get-ADGroup {groupname}; $members = ForEach ($Group in $Groups) {{Get-AdGroupMember -Identity $Group -Recursive}} ; $members | Get-AdUser -Properties Department | Select-Object Name, Department | Sort Department, Name";
shell.Commands.AddScript(script);
var results = shell.Invoke();
foreach (var psObject in results)
{
dt.Rows.Add(new object[] { psObject.Members["Name"].Value, psObject.Members["Department"].Value });
}
Gridview2.DataSource = dt;
Gridview2.DataBind();
It executes perfectly in PowerShell. Separating the command over lines as per suggested answer does not work.
Edit - I took another look at my powershell and realised that it was unecessarily complicated. I changed it to the below and it now works. Still does not explain why something that returns results in powershell does not work when ran from asp.net.
Get-AdGroupMember -Identity {groupname} -Recursive | Get-AdUser -Properties Department | Select-Object Name, Department | Sort Department, Name

C# Mysql Query Execution [duplicate]

I simply want to execute the following Mysql statement
SET #a = 1;SELECT #a;
with MySql.Data.MySqlClient
[System.Reflection.Assembly]::LoadWithPartialName("MySql.Data")
$password = 'mypassword'
$sql = "SET #a = 1;SELECT #a;"
$server = 'localhost'
$port = 3306
$database = 'test'
$User = 'root'
$conn=new-object MySql.Data.MySqlClient.MySqlConnection
$connectionstring = "Server=$Server;Port=$port;Database=$DataBase;Uid=$User;Pwd=$Password;allow zero datetime=yes"
$conn.ConnectionString = $connectionstring
$conn.Open()
$cmd=new-object MySql.Data.MySqlClient.MySqlCommand($sql,$conn)
$ds=New-Object system.Data.DataSet
$da=New-Object MySql.Data.MySqlClient.MySqlDataAdapter($cmd)
$da.fill($ds)
$conn.close()
$ds.tables[0]
I get a fatal error.
When I replace $sql by either
$sql = "SELECT DATABASE();"
or
$sql = "SELECT 1;"
I get the expected result.
I found this question, but it doesn't solve my problem.
I'm trying to port SQLIse (a part of the SQLPSX project ) to the MySQL version MySQLIse.
I want to process any simple valid mysql statements.
EDIT:
I was trying to run parts of the sakila-schema.sql the mysql demo database install script which runs by something like
mysql> SOURCE
C:/temp/sakila-db/sakila-schema.sql;

I found the solution in this blog
I have to add
;Allow User Variables=True
to the connection string:
$connectionstring = "Server=$Server;Port=$port;Database=$DataBase;Uid=$User;Pwd=$Password;allow zero datetime=yes;Allow User Variables=True"
works. I tested it with version 6.3.6.0. of MySql.Data.

Export DataTable to Excel using PowerShell

I have a function to create System.Data.Datatables from a simple input of TableName and ColumnName-Array. After filling the tables I want to add them to a dataset and export this one into an excel document.
The code below does that, however in the export to excel there is one bit that I find a bit unelegant. Every table will have to be exported into a csv and then reimported to excel.
Is there a better cleaner way to use DataTables directly in Excel?
Function MakeTable ($TableName, $ColumnArray)
{
$btab = New-Object System.Data.DataTable("$TableName")
foreach($Col in $ColumnArray)
{
$MCol = New-Object System.Data.DataColumn $Col;
$btab.Columns.Add($MCol)
}
return , $btab
}
function DataSetToExcel ($Ds, $workdirectory)
{
$excel = New-Object -ComObject excel.application
$workbook = $excel.Workbooks.Add(1)
$i = 0
for($DsIndex=0;$DsIndex -lt $ds.Tables.Count;$DsIndex++)
{
$Table = $ds.tables[$Dsindex]
if($Dsindex -ne 0)
{
$workbook.worksheets.Add() | Out-Null #Erstellt neues Arbeitsblatt
}
$Table | Export-Csv "$workdirectory\input.csv" -Encoding UTF8 -NoTypeInformation -Force -Delimiter ";"
$inputCSV = "C:\Temp\Test\input.csv"
$worksheet = $workbook.worksheets.Item(1)
$worksheet.Name = $Table.TableName
$TxtConnector = ("TEXT;" + $inputCSV)
$Connector = $worksheet.QueryTables.add($TxtConnector,$worksheet.Range("A1"))
$query = $worksheet.QueryTables.item($Connector.name)
$query.TextFileOtherDelimiter = $Excel.Application.International(5)
$query.TextFileParseType = 1
$query.AdjustColumnWidth = 1
$query.Refresh() | Out-Null
$query.Delete()
}
$outputXLSX = "$workdirectory\output.xlsx"
$Workbook.SaveAs($outputXLSX,51)
$excel.Quit()
}
function MakeTestTable ($TableName)
{
$TestTable = MakeTable $TableName #("A","B")
for($i=0;$i -lt 10; $i++)
{
$aRow = $TestTable.NewRow()
$aRow["A"] = (10-$i).ToString()
$aRow["B"] = $i.ToString()
$TestTable.Rows.Add($aRow)
}
return , $TestTable
}
$db = New-Object System.Data.DataSet
for($cx=0;$cx -lt 10;$cx++)
{
$tab1 = MakeTestTable "$cx"
$db.Tables.Add($tab1)
}
DataSetToExcel $db "C:\Temp\Test"

Check out my PowerShell Excel Module on Github. You can also grab it from the PowerShell Gallery.
It also works directly with CSV format using Import-Csv or ConvertFrom-Csv (basically any PowerShell object array).
function New-Person {
param($First,$Last)
$row=$dataTable.NewRow()
$row["First"]=$First
$row["Last"]=$Last
$dataTable.Rows.Add($row)
}
$dataTable = New-Object System.Data.DataTable("Test")
$dataTable.Columns.Add((New-Object System.Data.DataColumn "First"))
$dataTable.Columns.Add((New-Object System.Data.DataColumn "Last"))
New-Person John Doe
New-Person Tom Doe
New-Person Jane Doe
New-Person Mary Doe
$dataTable |
Select First, Last |
Export-Excel c:\temp\people.xlsx -AutoSize -Show

Since you tagged this with C#, If you have excel on the computer this will be run on, you can use the interop library. I used it for a project I did, using code I found in this SO question.

Check if a Database exists or not [duplicate]

Is it possible to check if a (MySQL) database exists after having made a connection.
I know how to check if a table exists in a DB, but I need to check if the DB exists. If not I have to call another piece of code to create it and populate it.
I know this all sounds somewhat inelegant - this is a quick and dirty app.

SELECT SCHEMA_NAME
FROM INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME = 'DBName'
If you just need to know if a db exists so you won't get an error when you try to create it, simply use (From here):
CREATE DATABASE IF NOT EXISTS DBName;

A simple way to check if a database exists is:
SHOW DATABASES LIKE 'dbname';
If database with the name 'dbname' doesn't exist, you get an empty set. If it does exist, you get one row.

From the shell like bash
if [[ ! -z "`mysql -qfsBe "SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='db'" 2>&1`" ]];
then
echo "DATABASE ALREADY EXISTS"
else
echo "DATABASE DOES NOT EXIST"
fi

If you are looking for a php script see below.
$link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
if (!$link) {
die('Not connected : ' . mysql_error());
}
// make foo the current db
$db_selected = mysql_select_db('foo', $link);
if (!$db_selected) {
die ('Cannot use foo : ' . mysql_error());
}

A very simple BASH-one-liner:
mysqlshow | grep dbname

Here is a bash function for checking if a database exists:
function does_db_exist {
local db="${1}"
local output=$(mysql -s -N -e "SELECT schema_name FROM information_schema.schemata WHERE schema_name = '${db}'" information_schema)
if [[ -z "${output}" ]]; then
return 1 # does not exist
else
return 0 # exists
fi
}
Another alternative is to just try to use the database. Note that this checks permission as well:
if mysql "${db}" >/dev/null 2>&1 </dev/null
then
echo "${db} exists (and I have permission to access it)"
else
echo "${db} does not exist (or I do not have permission to access it)"
fi

For those who use php with mysqli then this is my solution. I know the answer has already been answered, but I thought it would be helpful to have the answer as a mysqli prepared statement too.
$db = new mysqli('localhost',username,password);
$database="somedatabase";
$query="SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME=?";
$stmt = $db->prepare($query);
$stmt->bind_param('s',$database);
$stmt->execute();
$stmt->bind_result($data);
if($stmt->fetch())
{
echo "Database exists.";
}
else
{
echo"Database does not exist!!!";
}
$stmt->close();

A great way to check if a database exists in PHP is:
$mysql = mysql_connect("<your host>", "root", "");
if (mysql_select_db($mysql, '<your db name>')) {
echo "Database exists";
} else {
echo "Database does not exist";
}
That is the method that I always use.

Using bash:
if [ "`mysql -u'USER' -p'PASSWORD' -se'USE $DATABASE_NAME;' 2>&1`" == "" ]; then
echo $DATABASE_NAME exist
else
echo $DATABASE_NAME doesn't exist
fi

CREATE SCHEMA IF NOT EXISTS `demodb` DEFAULT CHARACTER SET utf8 ;

SELECT IF('database_name' IN(SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA), 1, 0) AS found;

Here's my way of doing it inside a bash script:
#!/bin/sh
DATABASE_USER=*****
DATABASE_PWD=*****
DATABASE_NAME=my_database
if mysql -u$DATABASE_USER -p$DATABASE_PWD -e "use $DATABASE_NAME";
then
echo "Database $DATABASE_NAME already exists. Exiting."
exit
else
echo Create database
mysql -u$DATABASE_USER -p$DATABASE_PWD -e "CREATE DATABASE $DATABASE_NAME"
fi

Be careful when checking for existence with a like statement!
If in a series of unfortunate events your variable ends up being empty, and you end up executing this:
SHOW DATABASES like '' -- dangerous!
It will return ALL databases, thus telling the calling script that it exists since some rows were returned.
It's much safer and better practice to use an "=" equal sign to test for existence.
The correct and safe way to test for existence should be:
SHOW DATABASES WHERE `database` = 'xxxxx' -- safe way to test for existence
Note that you have to wrap the column name database with backticks, it can't use relaxed syntax in this case.
This way, if the code creating the variable 'xxxxx' returned blank, then SHOW DATABASES will not return ALL databases, but will return an empty set.

With this Script you can get Yes or No database exists, in case it does not exist it does not throw Exception.
SELECT
IF(EXISTS( SELECT
SCHEMA_NAME
FROM
INFORMATION_SCHEMA.SCHEMATA
WHERE
SCHEMA_NAME = 'DbName'),
'Yes',
'No') as exist

Long winded and convoluted (but bear with me!), here is a class system I made to check if a DB exists and also to create the tables required:
<?php
class Table
{
public static function Script()
{
return "
CREATE TABLE IF NOT EXISTS `users` ( `id` INT NOT NULL PRIMARY KEY AUTO_INCREMENT );
";
}
}
class Install
{
#region Private constructor
private static $link;
private function __construct()
{
static::$link = new mysqli();
static::$link->real_connect("localhost", "username", "password");
}
#endregion
#region Instantiator
private static $instance;
public static function Instance()
{
static::$instance = (null === static::$instance ? new self() : static::$instance);
return static::$instance;
}
#endregion
#region Start Install
private static $installed;
public function Start()
{
var_dump(static::$installed);
if (!static::$installed)
{
if (!static::$link->select_db("en"))
{
static::$link->query("CREATE DATABASE `en`;")? $die = false: $die = true;
if ($die)
return false;
static::$link->select_db("en");
}
else
{
static::$link->select_db("en");
}
return static::$installed = static::DatabaseMade();
}
else
{
return static::$installed;
}
}
#endregion
#region Table creator
private static function CreateTables()
{
$tablescript = Table::Script();
return static::$link->multi_query($tablescript) ? true : false;
}
#endregion
private static function DatabaseMade()
{
$created = static::CreateTables();
if ($created)
{
static::$installed = true;
}
else
{
static::$installed = false;
}
return $created;
}
}
In this you can replace the database name en with any database name you like and also change the creator script to anything at all and (hopefully!) it won't break it. If anyone can improve this, let me know!
Note
If you don't use Visual Studio with PHP tools, don't worry about the regions, they are they for code folding :P

SELECT COUNT(*) FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'DbName'
1 - exists, 0 - not

Rails Code:
ruby-1.9.2-p290 :099 > ActiveRecord::Base.connection.execute("USE INFORMATION_SCHEMA")
ruby-1.9.2-p290 :099 > ActiveRecord::Base.connection.execute("SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'entos_development'").to_a
SQL (0.2ms) SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'entos_development'
=> [["entos_development"]]
ruby-1.9.2-p290 :100 > ActiveRecord::Base.connection.execute("SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'entos_development1'").to_a
SQL (0.3ms) SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'entos_development1'
=> []
=> entos_development exist , entos_development1 not exist

IF EXISTS (SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = N'YourDatabaseName')
BEGIN
-- Database exists, so do your stuff here.
END
If you are using MSSQL instead of MySQL, see this answer from a similar thread.

I am using simply the following query:
"USE 'DBname'"
Then check if the result is FALSE.
Otherwise, there might be an access denied error, but I cannot know that.
So, in case of privileges involved, one can use:
"SHOW DATABASES LIKE 'DBname'"
as already mentioned earlier.

Another php solution, but with PDO:
<?php
try {
$pdo = new PDO('mysql:host=localhost;dbname=dbname', 'root', 'password', [ PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION ]);
echo 'table dbname exists...';
}
catch (PDOException $e) {
die('dbname not found...');
}

Following solution worked for me:
mysql -u${MYSQL_USER} -p${MYSQL_PASSWORD} \
-s -N -e "SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='${MYSQL_DATABASE}'"

Golang solution
create a test package and add:
import "database/sql"
// testing database creation
func TestCreate(t *testing.T){
Createdb("*Testdb") // This just calls the **sql.DB obect *Testdb
db,err := sql.Open("mysql", "root:root#tcp(127.0.0.1:3306)/*Testdb")
if err != nil{
panic(err)
}
defer db.Close()
_, err = db.Exec("USE *Testdb")
if err != nil{
t.Error("Database not Created")
}
}

Using the INFORMATION_SCHEMA or show databases is not reliable when you do not have enough permissions to see the database. It will seem that the DB does not exist when you just don't have access to it. The creation would then fail afterwards. Another way to have a more precise check is to use the output of the use command, even though I do not know how solid this approach could be (text output change in future versions / other languages...) so be warned.
CHECK=$(mysql -sNe "use DB_NAME" 2>&1)
if [ $? -eq 0 ]; then
# database exists and is accessible
elif [ ! -z "$(echo $CHECK | grep 'Unknown database')" ]; then
# database does not exist
elif [ ! -z "$(echo $CHECK | grep 'Access denied')" ]; then
# cannot tell if database exists (not enough permissions)"
else
# unexpected output
fi