We Keep Coding

AWS Athena SDK with C#, how run DDL instructions?

I using the https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/Athena/NAthena.html sdk nuget https://github.com/aws/aws-sdk-net/ in order to query S3 bucket files, the thing is I need to create dataCatalog/databases and tables per demand (inside my program), I found a method to create dataCatalogs, but there is no such thing for databases or tables, only query methods. I'm confusing and want to know how can I do such thing.

Indeed there is no specific method for creating databases and tables, for that you will need to use QueryExecution requests
For databases:
var query = $"CREATE DATABASE IF NOT EXISTS DatabaseName " +
"COMMENT 'some comment' " +
"WITH DBPROPERTIES ('creator'='Me', 'Extra.'='some extra info');";
var request = new StartQueryExecutionRequest
{
QueryString = query,
ResultConfiguration = new ResultConfiguration
{
OutputLocation = _settings.DDLOutputLocation
}
};
For tables:
var query = "CREATE EXTERNAL TABLE IF NOT EXISTS " +
"TableName ( " +
"column1 int, " +
"column2 string, " +
"column3 int, " +
"column4 binary) " +
"ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' " +
$"LOCATION '{_settings.TableDataSourceLocation}'";
var request = new StartQueryExecutionRequest
{
QueryString = query,
QueryExecutionContext = new QueryExecutionContext
{
Catalog = _settings.CatalogName,
Database = _settings.DatabaseName
},
ResultConfiguration = new ResultConfiguration
{
OutputLocation = _settings.DDLOutputLocation
}
};

Web Scraping of dynamic paginate value using C#

As i'm trying to web scrape a part from the website. Here is an image below.
as the pagination is checked in red box i need to extract value of last in the image above it is 151. So the pagination is dynamic which is hard to extract when i check using view page source in only <div class="jsx-46358917 pagination-wrapper text-center"></div> is shown as inside its value is missing as i understand it is dynamic but i need the last value from the pagination example 151.
Here is a code which i have done so far to web scrape it.
public void parseItem(HtmlDocument doc, string zipCode)
{
//Getting json data
if (doc.DocumentNode.LastChild.HasChildNodes)
{
var siteScripts = doc.DocumentNode.SelectSingleNode("//script[#id='__NEXT_DATA__']").InnerText;
var result = JsonConvert.DeserializeObject<RealtorModel>(siteScripts);
if (result != null)
{
foreach (var realtor in result.Props.CriteriaData.SrpShell.LoadedData.SearchResults.HomeSearch.Results)
{
string propertyId = "M" + realtor.PropertyId;
string address = realtor.Location.Address.Line + ", " + realtor.Location.Address.City + ", " + realtor.Location.Address.StateCode + " " + realtor.Location.Address.PostalCode;
string listingURL = hostName + "/realestateandhomes-detail/" + realtor.Permalink;
var url = realtor.PrimaryPhoto;
listings.Add(new Listings { PropertyID = propertyId, Address = address, Price = realtor.ListPrice, ImageURL = realtor.PrimaryPhoto.Href.AbsoluteUri, ListingURL = listingURL });
}
}
pageNumber = pageNumber + 1;
string nextUrl = "https://www.realtor.com/realestateandhomes-search/" + zipCode + "/type-single-family-home" + "/pg-" + pageNumber;
AddTask(nextUrl, this.parseItem, zipCode);
}
else
{
System.Threading.Thread.Sleep(60000);
string nextUrl = "https://www.realtor.com/realestateandhomes-search/" + zipCode + "/type-single-family-home" + "/pg-" + pageNumber;
AddTask(nextUrl, this.parseItem, zipCode);
}
}
As i get the complete page through scraping only thing is the last value of the paginate which i cannot extract due to its dynamic nature. How can i achieve to do so any hint would be helpful.

sort data from csv to txt

I am writing a application which sorts out data from a csv to txt.
I have witten but I cannot get the required output.
Can someone please help, I do not see were I went wrong.
I initially thought its the File.WriteAllLines which was the problem but even when I write to a console I get the same results.
My file looks something like this
Georgina,Sinclair,408999703657,cheque,First National Bank,Fourways,275.00,12/01/2012
Zachary,Whitehead,409122372301,cheque,ABSA,Irene,70.25,12/01/2012
Toby,Henderson,401255489873,cheque,First National Bank,Edenvale,181.03,12/13/2012
Katherine,Cooke,409155874935,savings,ABSA,Southdowns,975.89,01/01/2013
Bradley,James,409254998,savings,ABSA,Melville,207.74,12/09/2012
Sophie,Lane,409771987,savings,ABSA,Roodepoort,207.74,12/31/2012
My output should be something like this
First National B0020000045603
GSinclair 408999703657 CH Fourways 002750001122012
THenderson 401255489873 CH Edenvale 001810313122012
ABSA 0040000146162
ZWhitehead 409122372301 CH Irene 000702501122012
KCooke 409155874935 SAVSouthdowns009758901012013
BJames 409254998 SAVMelville 002077409122012
SLane 409771987 SAVRoodepoort002077431122012
The code I currently have only returns the header and 2 lines which looks as follows.
ABSA 0040000146162
KCooke 409155874935 SAVSouthdowns 009758901012013
Please assist.
My code looks as follows
string text = #"C:\\Test\\output.txt";
var inputEntries = File.ReadLines(#"C:\\Test\\debitorders.csv").Select(line =>
{
var values = line.Split(',');
return new
{
accountholder = values[0].Trim().Substring(0, 1) + values[1].Trim(),
accountnumber = long.Parse(values[2].Trim()),
accounttype = values[3].Trim(),
bankname = values[4].Trim(),
branch = values[5].Trim(),
amount = 100 * double.Parse(values[6].Trim()),
date = DateTime.Parse(values[7].Trim())
};
});
var banks = inputEntries
.OrderBy(e => e.bankname)
.GroupBy(e => e.bankname, e => e);
foreach (var bank in banks)
{
var AccountName = bank.Key;
if (AccountName.Length >= 20)
{
AccountName = AccountName.Substring(0, 16);
}
else
{
AccountName += new string(' ', 20 - AccountName.Length);
}
var NumberOfAccounts = bank.Count();
var TotalAmount = bank.Select(acc => acc.amount).Sum();
var Header = AccountName + "\t" + NumberOfAccounts.ToString("000") + TotalAmount.ToString("0000000000");
var sortedAccounts = bank
.OrderBy(acc => acc.accountholder)
.OrderByDescending(acc => acc.amount);
foreach (var account in sortedAccounts)
{
var outputLine =
account.accountholder + "\t" +
account.accountnumber + "\t" +
//get first 2 characters
account.accounttype.Substring(0, 3).ToUpper() + account.branch + "\t" + "00" +
account.amount +
account.date.ToString("ddMMyyyy");
for (int i = 0; i < 15; i++)
{
File.WriteAllText(text, Header + Environment.NewLine + outputLine);
Console.WriteLine(Header + outputLine);
Console.ReadLine();
}
}
}

A better and cleaner solution will be to make use of List<string> to which you add your text. At the end of the code just convert the list to an array and write all lines to a file.
List<string> outputLine = new List<string>(); //note this addition to the code
foreach (var bank in banks)
{
//do header formatting stuff here
var Header = somecode
outputLine.Add(Header); //Add Header to outputLine
var sortedAccounts = bank.OrderBy(acc => acc.accountholder)
.OrderByDescending(acc => acc.amount);
foreach (var account in sortedAccounts)
{
var tempStringBuilder =
account.accountholder + "\t" +
account.accountnumber + "\t" +
//get first 2 characters
account.accounttype.Substring(0, 3).ToUpper() + account.branch + "\t" + "00" +
account.amount +
account.date.ToString("ddMMyyyy");
outputLine.Add(tempStringBuilder); //Add tempStringBuilder to outputLine
}
}
File.WriteAllLines("destination path", outputLine.ToArray()); //Write everything to your output file in one go
Alternative Excel solution:
Microsoft Excel has a really powerful tool called Pivot Tables, which is ideally suited to your needs. If you are unfamiliar with it, read some tutorials about it. At first it's a process to get your head around the workflow to use it but it is quite simple once you've grasped it. You just drag and drop fields by which you want to group.
You might also want to consider using Data Connections to link to your original data, which is also quite simple given the dataset you have.

I think i found the solution:
File.AppendAllText(text, Header + Environment.NewLine + outputLine + Environment.NewLine);
Use File.AppendAllText instead of File.WriteAllText. With WriteAllText you always deleted the old content.
But consider to clean the file (File.WriteAllText(text, "");) before you begin to write on it, otherwise you will have the old data from last run also in it.
Try to use String.Format("{0,-10}", name) which means that the length of the name is filled up with spaces up to the length of 10. Minus means left alignment and positive causes right alignment.
I updated your code to:
string text = #"D:\C#\output.txt";
File.WriteAllText(text, "");
var inputEntries = File.ReadLines(#"D:\c#\debitorders.csv").Select(line =>
{
var values = line.Split(',');
return new
{
accountholder = values[0].Trim().Substring(0, 1) + values[1].Trim(),
accountnumber = long.Parse(values[2].Trim()),
accounttype = values[3].Trim(),
bankname = values[4].Trim(),
branch = values[5].Trim(),
amount = 100 * double.Parse(values[6].Trim()),
date = DateTime.ParseExact(values[7].Trim(), "MM/dd/yyyy", CultureInfo.InvariantCulture)
};
});
var banks = inputEntries.OrderBy(e => e.bankname)
.GroupBy(e => e.bankname, e => e);
foreach (var bank in banks)
{
var AccountName = bank.Key;
var NumberOfAccounts = bank.Count();
var TotalAmount = bank.Select(acc => acc.amount).Sum();
var Header = String.Format("{0,-20} {1,-10} {2}", AccountName, NumberOfAccounts.ToString("000"), TotalAmount.ToString("0000000000"));
var sortedAccounts = bank.OrderBy(acc => acc.accountholder)
.OrderByDescending(acc => acc.amount);
File.AppendAllText(text, Header + Environment.NewLine);
Console.WriteLine(Header);
foreach (var account in sortedAccounts)
{
var outputLine = String.Format("{0,-11} {1,15} {2,-3} {3,-10} {4,7} {5,-10}",
account.accountholder,
account.accountnumber,
account.accounttype.Substring(0, 3).ToUpper(),
account.branch,
account.amount,
account.date.ToString("ddMMyyyy")
);
//get first 2 characters
//account.accounttype.Substring(0, 3).ToUpper() + account.branch + "\t" + "00" +
// what are the "00" for? didn't include them you may this do by yourself
File.AppendAllText(text, outputLine + Environment.NewLine);
Console.WriteLine(outputLine);
}
File.AppendAllText(text, Environment.NewLine);
Console.WriteLine();
}
Output is:
ABSA 004 0000146162
KCooke 409155874935 SAV Southdowns 97589 01012013
BJames 409254998 SAV Melville 20774 09122012
SLane 409771987 SAV Roodepoort 20774 31122012
ZWhitehead 409122372301 CHE Irene 7025 01122012
First National Bank 002 0000045603
GSinclair 408999703657 CHE Fourways 27500 01122012
THenderson 401255489873 CHE Edenvale 18103 13122012

c# Retrieve multiple XML child nodes

I've managed to link up a single XElement successfully into my program though I'm not having any luck with the other two I have in place, I've tried using;
IEnumerable query = from booking in doc.Descendants("Booking")
Though I've haven't had much luck placing the values into list box.
Here's the code for function:
private void btnimport_Click(object sender, EventArgs e)
{
OpenFileDialog open = new OpenFileDialog();
open.CheckFileExists = true;
open.InitialDirectory = "#C:\\";
open.Filter = "XML Files (*.xml)|*.xml|All Files(*.*)|*.*";
open.Multiselect = false;
if (open.ShowDialog() == DialogResult.OK)
{
try
{
XDocument doc = XDocument.Load(open.FileName);
//Grabs the customer elements
var query = from booking in doc.Descendants("Booking")
select new
{
//Customer Elements
CustomerId = booking.Element("CustomerId").Value,
Title = booking.Element("Title").Value,
Firstname = booking.Element("FirstName").Value,
Lastname = booking.Element("LastName").Value,
DateofBirth = booking.Element("DateofBirth").Value,
Email = booking.Element("Email").Value,
HouseNo = booking.Element("HouseNo").Value,
Street = booking.Element("Street").Value,
Postcode = booking.Element("Postcode").Value,
Town = booking.Element("Town").Value,
County = booking.Element("County").Value,
ContactNo = booking.Element("ContactNo").Value,
//Holiday Elements
HolidayId = booking.Element("HolidayId").Value,
HotelName = booking.Element("HotelName").Value,
Location = booking.Element("Location").Value,
BookFrom = booking.Element("BookFrom").Value,
BookTo = booking.Element("BookTo").Value,
CheckInTime = booking.Element("CheckInTime").Value,
CheckOutTime = booking.Element("CheckOutTime").Value,
NoOfRoomsBooked = booking.Element("NoOfRoomsBooked").Value,
RoomType = booking.Element("RoomType").Value,
RoomServices = booking.Element("RoomServices").Value,
Parking = booking.Element("Parking").Value,
Pet = booking.Element("Pet").Value,
//TravelInfo Elements
TravelInfoId = booking.Element("TravelInfoId").Value,
TravellingFrom = booking.Element("TravellingFrom").Value,
Destination = booking.Element("Destination").Value,
Fare = booking.Element("Fare").Value,
TravelInsurance = booking.Element("TravelInsurance").Value,
InFlightMeals = booking.Element("In-FlightMeals").Value,
LuggageAllowance = booking.Element("LuggageAllowance").Value,
ExtraLuggage = booking.Element("ExtraLuggage").Value,
CarHire = booking.Element("CarHire").Value,
ReturnTransfer = booking.Element("ReturnTransfer").Value,
};
//Inputs all of the values in bookings
foreach (var booking in query)
{
//Customer values
txtCustomerId.Text = txtCustomerId.Text + booking.CustomerId;
txttitle.Text = txttitle.Text + booking.Title;
txtfname.Text = txtfname.Text + booking.Firstname;
txtlname.Text = txtlname.Text + booking.Lastname;
txtdob.Text = txtdob.Text + booking.DateofBirth;
txtemail.Text = txtemail.Text + booking.Email;
txthouseno.Text = txthouseno.Text + booking.HouseNo;
txtstreet.Text = txtstreet.Text + booking.Street;
txtpostcode.Text = txtpostcode.Text + booking.Postcode;
txttown.Text = txttown.Text + booking.Town;
txtcounty.Text = txtcounty.Text + booking.County;
txtcontactno.Text = txtcontactno.Text + booking.ContactNo;
//Holiday Values
txtHolidayId.Text = txtHolidayId.Text + booking.HolidayId;
txthname.Text = txthname.Text + booking.HotelName;
txtl.Text = txtl.Text + booking.Location;
txtbf.Text = txtbf.Text + booking.BookFrom;
txtbt.Text = txtbt.Text + booking.BookTo;
txtcit.Text = txtcit.Text + booking.CheckInTime;
txtcot.Text = txtcot.Text + booking.CheckOutTime;
txtnorb.Text = txtnorb.Text + booking.NoOfRoomsBooked;
txtrt.Text = txtrt.Text + booking.RoomType;
txtrs.Text = txtrs.Text + booking.RoomServices;
txtpark.Text = txtpark.Text + booking.Parking;
txtpet.Text = txtpet.Text + booking.Pet;
//TravelInfo Values
txtTravelInfoId.Text = txtTravelInfoId.Text + booking.TravelInfoId;
txttf.Text = txttf.Text + booking.TravellingFrom;
txtd.Text = txtd.Text + booking.Destination;
txtf.Text = txtf.Text + booking.Fare;
txtti.Text = txtti.Text + booking.TravelInsurance;
txtifi.Text = txtifi.Text + booking.InFlightMeals;
txtla.Text = txtla.Text + booking.LuggageAllowance;
txtel.Text = txtel.Text + booking.ExtraLuggage;
txtch.Text = txtch.Text + booking.CarHire;
txtrtrans.Text = txtrtrans.Text + booking.ReturnTransfer;
}
MessageBox.Show("XML has been imported");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
}
If anyone knows where I've gone wrong or what I need to add / change please let me know :)
Many thanks,
10gez10

You have several problems:
Firstly, your data elements are not immediate children of the booking element, there are intermediate elements <Customer>, <Holiday> and <TravelInfo>. Thus you need to do something like
var query = from booking in doc.Descendants("Booking")
let customer = booking.Element("Customer")
let holiday = booking.Element("Holiday")
let travelInfo = booking.Element("TravelInfo")
select new
{
//Customer Elements
CustomerId = customer.Element("CustomerId").Value,
Title = customer.Element("Title").Value,
HolidayId = holiday.Element("HolidayId").Value,
TravelInfoId = travelInfo.Element("TravelInfoId").Value,
}
Secondly, several elements are misspelled:
CheckOutTime should be CheckoutTime
In-FlightMeals should be InFlightMeals.
CarHire should be CareHire (yes "CareHire" is what's in the XML.)
Thus, when you do (e.g.) Element("In-FlightMeals").Value, Element() is returning null so you get a null reference exception and your code is aborted.
Thirdly, the element BookTo is completely missing, so BookTo = holiday.Element("BookTo").Value generates a null reference exception.
More generally, I do not recommend this coding approach. If any of your XML elements are missing, your query will throw an exception because element.Element("name") will be null. What's worse, Visual Studio doesn't seem to report an accurate line number on which the null reference occurs, instead giving the line number of the select new statement. And (on my version at least), it's not possible to step into the constructor for an anonymous type either. This makes debugging well-nigh impossible.
Instead, skip the intermediate anonymous type and do things in a more direct, traditional manner:
foreach (var booking in doc.Descendants("Booking"))
{
var customer = booking.Element("Customer");
var holiday = booking.Element("Holiday");
var travelInfo = booking.Element("TravelInfo");
XElement element;
if (customer != null)
{
if ((element = customer.Element("CustomerId")) != null)
txtCustomerId.Text = txtCustomerId.Text + element.Value;
}
// And so on.
}

Adding double quotes to # string

As far as i understood, a string with an # in required a set of double quotes to insert the quote in to the string?
I have tried that principle and to no avail. The following line works, but if i were to replace those strings with parameter values then i cant seem to get the correct compilation value
var node = doc.SelectSingleNode(#"//node[#label = ""Chemist Name""]/node[#label = ""John,Smith""]");
my attempt (of which i have tried several versions and ended up here, where i have now givn up !)
var node = doc.SelectSingleNode(#"//node[#label = " + ""+parentID+"" + "]/node[#label = " + ""+ name +"" + "]");
can anyone help me please?

Use single quotes:
var node = doc.SelectSingleNode
(#"//node[#label = 'Chemist Name']/node[#label = 'John,Smith']");
var node = doc.SelectSingleNode(
string.format(#"//node[#label = '{0}']/node[#label = '{1}']"
, parentID, name));

You are missing another double quote to close the string being appended and also # before each string containing "".
Try this:
var node =
doc.SelectSingleNode(#"//node[#label = """ + parentID + #"""]/node[#label = """ + name + #"""]");

var node = doc.SelectSingleNode(string.format(#"//node[#label = ""{0}""]/node[#label = ""{1}""]", parentId, name));

Write an extension method to extend string:
public static string Quote(this string input)
{
return string.Format(#"""{0}""", input);
}
And then use it as follows:
var node = doc.SelectSingleNode(#"//node[#label = " + parentID.Quote() + "]/node[#label = " + name.Quote() + "]");
Or simply:
var node = doc.SelectSingleNode(string.Format(#"//node[#label = {0}"]/node[#label = {1}"]",parentID.Quote(), name.Quote());