AWS Athena SDK with C#, how run DDL instructions? - c#

I using the https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/Athena/NAthena.html sdk nuget https://github.com/aws/aws-sdk-net/ in order to query S3 bucket files, the thing is I need to create dataCatalog/databases and tables per demand (inside my program), I found a method to create dataCatalogs, but there is no such thing for databases or tables, only query methods. I'm confusing and want to know how can I do such thing.

Indeed there is no specific method for creating databases and tables, for that you will need to use QueryExecution requests
For databases:
var query = $"CREATE DATABASE IF NOT EXISTS DatabaseName " +
"COMMENT 'some comment' " +
"WITH DBPROPERTIES ('creator'='Me', 'Extra.'='some extra info');";
var request = new StartQueryExecutionRequest
{
QueryString = query,
ResultConfiguration = new ResultConfiguration
{
OutputLocation = _settings.DDLOutputLocation
}
};
For tables:
var query = "CREATE EXTERNAL TABLE IF NOT EXISTS " +
"TableName ( " +
"column1 int, " +
"column2 string, " +
"column3 int, " +
"column4 binary) " +
"ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' " +
$"LOCATION '{_settings.TableDataSourceLocation}'";
var request = new StartQueryExecutionRequest
{
QueryString = query,
QueryExecutionContext = new QueryExecutionContext
{
Catalog = _settings.CatalogName,
Database = _settings.DatabaseName
},
ResultConfiguration = new ResultConfiguration
{
OutputLocation = _settings.DDLOutputLocation
}
};

Related

Google Drive API Q logic clauses order

So I was having a problem while trying to list certain files from my drive, the original code was:
var request = _DriveService.Files.List();
request.Q = $"name = '{Properties.Settings.Default.SomeFileName0}'" +
$" or name = '{Properties.Settings.Default.SomeFileName1}'" +
$" or name = '{Properties.Settings.Default.SomeFileName2}'" +
$" or name = '{Properties.Settings.Default.SomeFileName3}'" +
" and trashed = false" +
$" and '{Properties.Settings.Default.MyParents[0]}' in parents";
request.Fields = "files(id, name, parents)";
var files = await request.ExecuteAsync();
The problem was that it returned 12 files, instead of 4 files. After a bit or testing and tweaking I realized that google drive was ignoring the and clauses, so it was retrieving all the removed files.
Now, with this code it works like a charm:
var parentsTrashed = $"and '{Properties.Settings.Default.Myparents[0]}' in parents and trashed = false";
var request = _DriveService.Files.List();
request.Q = $"name = '{Properties.Settings.Default.SomeFileName0}' {parentsTrashed}" +
$" or name = '{Properties.Settings.Default.SomeFileName1}' {parentsTrashed}" +
$" or name = '{Properties.Settings.Default.SomeFileName2}' {parentsTrashed}" +
$" or name = '{Properties.Settings.Default.SomeFileName3}' {parentsTrashed}";
request.Fields = "files(id, name, parents)";
var files = await request.ExecuteAsync();
So the ands and ors clauses needs to be in this order: (name and and and) OR (name and and and) OR ....
I can't find any indication of this in the documentation nor around the internet: is it a bug or is it intended? Shouldn't it be noted down in the documentation or is it and I just didn't found it, or is it a standard thing that I didn't know about till now (I'm an amateur)?
While I don't know the specific rules for operator precedence in Drive queries, it would at least make it easier to read if you specified what you mean using brackets:
var request = _DriveService.Files.List();
request.Q = $"(name = '{Properties.Settings.Default.SomeFileName0}'" +
$" or name = '{Properties.Settings.Default.SomeFileName1}'" +
$" or name = '{Properties.Settings.Default.SomeFileName2}'" +
$" or name = '{Properties.Settings.Default.SomeFileName3}')" +
" and trashed = false" +
$" and '{Properties.Settings.Default.MyParents[0]}' in parents";
request.Fields = "files(id, name, parents)";
var files = await request.ExecuteAsync();
That way the query will be of the form:
(name = name1 or name = name2 or name = name3 or name = name4)
and trashed = false and parentid in parents
(Without line breaks, of course.)
I suspect in your current query, it's effectively:
name = name1 or name = name2 or name = name3 or
(name = name4 and trashed = false and parentid in parents)

Import data to enterprise custom field/look up table

I have a problem. I want to import data from SQL/Excel to lookup table and let enterprise custom field to look it up.
I'm using Microsoft Project 2016 Professional/Online.
I've used C# samples to read tables and their entries. But I'm not able to add to the tables. Maybe someone has suggestions/samples for this kind of problem?
SvcLookupTable.LookupTableDataSet lutDs = new SvcLookupTable.LookupTableDataSet();
LookupEntryCreationInformation creationInfo = new LookupEntryCreationInformation();
LookupEntryValue entryValue = new LookupEntryValue();
Guid entryTableGuid = new Guid("417552b5-7877-e711-80ce-00155d40cd19");
Console.WriteLine("---------------Lookup Tables---------------");
LookupTableCollection lookupTables = projContext.LookupTables;
projContext.Load(lookupTables);
projContext.ExecuteQuery();
entryValue.TextValue = "ITS";
creationInfo.Value = entryValue;
creationInfo.Id = entryTableGuid;
LookupTableCollection lookupTablesColl = projContext.LookupTables;
projContext.Load(lookupTables);
projContext.ExecuteQuery();
int numTables = projContext.LookupTables.Count;
LookupTable lookTable = lookupTables[numTables-1];
Guid rowGuid = new Guid();
foreach (Microsoft.ProjectServer.Client.LookupTable lt in lookupTables)
{
Console.WriteLine(lt.Name + " {" + lt.Id + "}");
ImportLookupTable importTable = new ImportLookupTable();
importTable.AddLookupTableValues(lutDs, rowGuid, rowGuid,"ITS","Skyr");
projContext.Load(lt.Entries);
projContext.ExecuteQuery();
foreach (LookupEntry entry in lt.Entries)
{
Console.WriteLine(" " + entry.FullValue + " {" + entry.Id + "}");
}
}

ASP.NET SQL Query One of Many Columns Undefined

General Description: Single SQL query of multiple SQL databases (through joins) returns all but one item from all but one database. Item not being returned is joined via an item that can, and sometimes is, null.
Specific Description:
I am continuing development of an internal ticketing system for work. I just started with C#, SQL and web development about a month ago, so I am still wrapping my head around all of the interconnections and syntax of Jquery, SQL, C#, and MVC.
Currently I am trying to display an SQL query in a table to show brief information for our tickets to support members. I have everything being displayed except "CircuitDescription" which is a pretty important element in order for support to differentiate between circuits. It is coming through on the table as "undefined" which I have gathered is the JQuery response for an initialized variable. All other values are coming through on the web page.
Running the SQL query through Microsoft SQL Server Management Studio displays the column with the circuit description.
Disclaimer:
While searching for this I have seen posts discouraging multiple database queries, but this is how the program was written before so I want to keep my code as similar to what has been done so far as possible.
So if we could jump past the part where it is implied I am an idiot (as is my only experience asking questions on stackoverflow) that would be lovely.
If you help me solve this, and help me learn why what you suggest works, then you can feel free to imply or directly call me anything you like.
Code:
C# / SQL Query:
-The join statements are combining the lists by a numeric value from DB "Tickets" with a numeric value from the other databases. Those databases are holding 2 columns, the numeric value and a corresponding string description.
-The full outer join is combining the list by a numeric circuitID from the tickets to a numeric circuitID in the circuits database.
-The circuit database holds the circuit description I am struggling with.
-Some circuitID values are null, which I suspect may be why this is not working. The other join statements that I am receiving data in connection to are all not null.
public static async Task<List<Ticket>> GetAllTicketsForCustomerAsync(DBAccess db, int customerID)
{
var cmd = "select TicketID, Tickets.DateCreated, Tickets.DateResolved, Tickets.CustomerCircuitID, CircuitDescription, TicketTypeDesc, BillingStatusDescription, TicketStatusDescription " +
"from Tickets " +
"join TicketTypes on Tickets.TicketTypeID = TicketTypes.TicketTypeID " +
"join TicketStatusTypes on Tickets.TicketStatus = TicketStatusTypes.TicketStatus " +
"join TicketBillingStatusTypes on Tickets.BillingStatus = TicketBillingStatusTypes.BillingStatus " +
"full outer join CustomerCircuits on Tickets.CustomerCircuitID = CustomerCircuits.CustomerCircuitID " +
"where Tickets.CustomerID = " + customerID +
"order by Tickets.TicketID DESC";
var table = await db.ReadTableAsync(cmd);
return (from DataRow row in table.Rows select db.AssignFromRow<Ticket>(row)).ToList();
}
JQuery:
-Ternary operator for circDesc is to list any tickets without a circuitID as "NonSpecific" for their circuit description. Otherwise they should display the circuit description that is currently coming through as "Undefined"
function buildPartialCustomerTicketsTable(tickets) {
var table = "";
var maxSize = (tickets.length < 5) ? tickets.length : 5;
for (var i = 0; i < maxSize; i++) {
var t = tickets[i];
var circDesc = (t.CustomerCircuitID == null) ? "Nonspecific" : t.CircuitDescription;
var rowClass = ((i % 2) == 0) ? "listRowNormal" : "listRowAlternate";
table += "<tr class='" + rowClass + "'>"
+ "<td class='listElement'><a href='#' onclick='viewTicket(" + t.TicketID + ",true)'>update</a></td>"
+ "<td class='listElement'>" + t.TicketStatusDescription + "</td>"
+ "<td class='listElement'>" + formatDate(t.DateCreated) + "</td>"
+ "<td class='listElement'>" + formatDate(t.DateResolved) + "</td>"
+ "<td class='listElement'>" + circDesc + "</td>"
+ "<td class='listElement'>" + t.TicketTypeDescription + "</td>"
+ "<td class='listElement'>" + t.BillingStatusDescription + "</td>"
+ "<td class='listElement'>" + t.TicketID + "</td>"
+ "</tr>";
}
return table;
}
Requested Code:
public T AssignFromRow<T>(DataRow row) where T : new()
{
var rec = new T();
this.AssignFromRow(row, rec);
return rec;
}
public void AssignFromRow(DataRow row, object rec)
{
if (row == null || rec == null)
{
return;
}
// Look at all of the properties in the record
PropertyInfo[] recprops = rec.GetType().GetProperties();
foreach (PropertyInfo pi in recprops)
{
// default the sql column name to the property name
string columnName = pi.Name;
// skip any read only parameters
if (!pi.CanWrite)
{
continue;
}
// Check for a mapping attribute. This attribute can change the name of the table column name from the default.
var customAttrs = pi.GetCustomAttributes(typeof(MappingAttribute), false);
if (customAttrs.Length > 0)
{
var mapping = (MappingAttribute)customAttrs[0];
if (!string.IsNullOrEmpty(mapping.ColumnName))
{
columnName = mapping.ColumnName;
}
}
// If the row does not have this element name then skip it
if (!row.Table.Columns.Contains(columnName))
{
continue;
}
// If the DataRow has a value with the same name, and it is not null, then assign it
object dbval = row[columnName];
if (dbval != null && !(dbval is DBNull))
{
pi.SetValue(rec, dbval, null);
}
}
}

How to get the next page of results from Bing Search Container in C#?

I am using the BingSearchContainer.cs with a Winform in C#. I am returning the results using the following code. After a good couple of hours looking I can't figure out how to return the other pages of results. It is only possible to return a maximum of 50 results at a time. I would like to return more pages and then add these to "imageSet" to have a full list of resulting images. Any hints or pointers would be really useful, thanks in advance for any help.
void bingSearch(string searchTerm)
{
try
{
imageSet = new List<Bing.ImageResult>();
const string bingKey = "[key]";
var bing = new BingSearchContainer(
new Uri("https://api.datamarket.azure.com/Bing/Search/")) { Credentials = new NetworkCredential(bingKey, bingKey) };
var query = bing.Image("\"" + searchTerm + "\"" + "(" + site1 + " OR " + site2 + ")", null, null, null, null, null, ImageFilters);
Debug.Print("Full Search: " + query.ToString());
query = query.AddQueryOption("$top", 50);
query = query.AddQueryOption("$skip", 20);
var results = query.Execute();
int index = 0;
foreach (var result in results)
{
imageSet.Add(result);
Debug.Print("URL: " + imageSet[index].MediaUrl);
index++;
}
Debug.Print("Results: " + imageSet.Count);
}
catch
{
Debug.Print("Error");
}
}
Solved this.
Actually it is very simple. The "$skip", 20 query option sets the off-set for the pages such that if I have an off-set of 0 I get the first 50 images, an off-set of 50 I get the next 50 images and so on.

client.get("me/statuses") returns empty "data" array using C# Facebook SDK 5.0.3

Using the C# Facebook SDK 5.0.3 everything works fine whit the client.Get("/me").
But when retrieving the status, I should get aan arraylist "data" with all the status messages according to the facebook Graph API, but instead my data array is empty and I get a 'Index out of bounds' exception.
Does anyone have an idea what my problem could be?
if (Request.Params["code"] != null)
{
var client = new FacebookClient(GetAccessToken());
dynamic me = client.Get("/me");
imgUser.ImageUrl = "https://graph.facebook.com/" + me.id + "/picture";
lblUsername.Text = me.name;
lblHometown.Text = me.hometown.name;
lblBirthday.Text = me.birthday;
lblCurrenttown.Text = me.location.name;
lblEmail.Text = me.email;
lblOpleidingen.Text = "";
lblOpleidingen.Text += me.education[1].type + ": " + me.education[1].school.name + ", " + me.education[1].year.name + "<br />"
+ me.education[0].type + ": " + me.education[0].school.name + ", " + me.education[0].year.name;
lblSex.Text = me.gender;
dynamic status = client.get("/me/statuses");
txtStatus.Text = status.data[0].message;
}
It requires the read_stream permission. Ensure you have it.
Your permission array should look like follows:
string[] extendedPermissions = new[] { "user_about_me", "read_stream" };
if (extendedPermissions != null && extendedPermissions.Length > 0)
{
var scope = new StringBuilder();
scope.Append(string.Join(",", extendedPermissions));
parameters["scope"] = scope.ToString();
}
furthermore your second get() should be capitalized: Get()
dynamic status = client.get("/me/statuses");
dynamic status = client.Get("/me/statuses");

Categories