We Keep Coding

Is there a Method to read the Content of a Website?

I wanted to write a Program that reads an online Calendar, compares it with Names in a Database and uses this Data in some way. But if I use the WebClient, it reads the Source Code of the Website, not the Content. This is my Code:
using System;
using System.Diagnostics;
using System.Net;
using MySql.Data.MySqlClient;
namespace CalendarCrawler
{
class Program
{
static void KillTask(string Task)
{
Process[] Process = new Process[] { };
Process = Process.GetProcessesByName(Task);
foreach (Process Instance in Process)
{
Instance.Kill();
}
}
static String ReadContent(String Website)
{
WebClient web = new WebClient();
System.IO.Stream stream = web.OpenRead(Website);
using (System.IO.StreamReader reader = new System.IO.StreamReader(stream))
{
String text = reader.ReadToEnd();
return text;
}
}
static void Main(string[] args)
{
Console.WriteLine("Getting Connection ...");
var datasource = "localhost";//your server
var database = "database"; //your database name
var username = "username"; //username of server to connect
var password = "password"; //password
//your connection string
string connStr = $"Server={datasource};Database={database};Uid={username};Pwd={password}";
//create instanace of database connection
using (var conn = new MySqlConnection(connStr))
{
try
{
Console.WriteLine("Openning Connection ...");
//open connection
conn.Open();
Console.WriteLine("Connection successful!");
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
String Websitetext = ReadContent("http://www.esel.at/termine");
var stm = $"INSERT INTO content(Content) VALUES (#1);";
var cmd = new MySqlCommand(stm, conn);
cmd.Parameters.AddWithValue("#1", Websitetext);
cmd.ExecuteNonQuery();
Console.WriteLine(Websitetext);
KillTask("CalendarCrawler");
}
}
}
}
The Killtask Method is only to Clear it from the Background Processes, so there are no Problems with building a new Version.
I hope someone can help me.

There are effectively 2 different users of the web. People and computers. People like shiney things like buttons and tables (UI), computers prefer things like XML or JSON (API).
Your calendar web site has a UI, this is what you are currently seeing (both in a browser and when you 'download the code'). It probably has an API too and that is what you should be using in your program.
I've just had a quick look at esel.at and it doesn't appear to have a (public) API (but maybe that is because Google can't translate the page properly).

Downloading file from MS teams programatically

Previously, I developed an application which downloaded a file from a corporate Sharepoint site and then performed some magic with it.
The powers that be have since migrated to MS Teams and I'm trying to update the application to use the new platform. However, I'm having all sorts of issues getting the file to download.
My old (working for Sharepoint) code uses a WebClient to retrieve the file based on credentials previously provided by the user:
private string GetSchedule(string username, string password, string domain)
{
string tempPath = Path.GetTempFileName().Replace(".tmp", ".xlsm");
using (WebClient client = new WebClient())
{
client.Credentials = new NetworkCredential(username, password, domain);
try
{
client.DownloadFile(_networkSchedulePath, tempPath);
}
catch (WebException e)
{
if (e.Message.Contains("401"))
{
StatusUpdated?.Invoke(this, new EventArgs<string>("Invalid Credentials Provided"));
Finished?.Invoke(this, null);
return null;
}
if (e.Message.Contains("404"))
{
StatusUpdated?.Invoke(this, new EventArgs<string>("File Not Found"));
Finished?.Invoke(this, null);
return null;
}
else
{
StatusUpdated?.Invoke(this, new EventArgs<string>(e.Message));
Finished?.Invoke(this, null);
return null;
}
}
}
return tempPath;
}
However, when I use this with the new teams link I'm getting a 403 Forbidden error. So is there any way to programmatically retrieve a file from MS Teams?

I was mistaken in the comments. Simply replacing the NetworkCredentials with SharePointOnlineCredentials is not the solution.
I'm not sure if the following is the "right" approach, but it works and seems pretty solid. Please give it a try:
private static string GetFile(string path, string username, string password, string domain)
{
var secureString = new SecureString();
foreach (var ch in password)
{
secureString.AppendChar(ch);
}
string tempPath = Path.GetTempFileName().Replace(".tmp", ".xlsm");
using (WebClient client = new WebClient())
{
var credentials = new SharePointOnlineCredentials(username, secureString);
client.Headers[HttpRequestHeader.Cookie] = credentials.GetAuthenticationCookie(new Uri(path));
try
{
client.DownloadFile(path, tempPath);
}
catch (WebException e)
{
// Error Handling
}
}
return tempPath;
}
Another option is to use the CSOM rather than using a webclient directly. n.b., I encountered errors at the OpenBinaryDirect() call when using the Microsoft.SharePoint.Client NuGet package and it looks like this package is wildly out of date. It appears that the one to use now is Microsoft.SharePointOnline.CSOM or Microsoft.SharePoint2019.CSOM:
private static string GetFileWithClientContext(string path, string username, string password, string domain)
{
var secureString = new SecureString();
foreach (var ch in password)
{
secureString.AppendChar(ch);
}
string tempPath = Path.GetTempFileName().Replace(".tmp", Path.GetExtension(path));
using (var context = new ClientContext(path))
{
context.Credentials = new SharePointOnlineCredentials(username, secureString);
try
{
using (var file = Microsoft.SharePoint.Client.File.OpenBinaryDirect(context, new Uri(path).AbsolutePath))
using (var outFile = System.IO.File.OpenWrite(tempPath))
{
file.Stream.CopyTo(outFile);
}
}
catch (WebException e)
{
// Error Handling
}
}
return tempPath;
}

Thanks to JLRishe for the help his answer and comments provided. However, the final solution varied from the one in his answer, which is why I'm posting it here:
The OfficeDevPnP.Core package is used extensively for this.
Firstly, the AuthenticationManager is used to get a ClientContext in terms of the specific sharepoint site that needs to be accessed. This pops a window up to allow for the MFA. Then various components are loaded in via the ClientContext object. From here, the file is fetched via Guid and dumped to disk.
private string GetSchedule()
{
string tempPath = Path.GetTempFileName().Replace(".tmp", ".xlsm");
try
{
AuthenticationManager authManager = new OfficeDevPnP.Core.AuthenticationManager();
ClientContext ctx = authManager.GetWebLoginClientContext("https://oursite.sharepoint.com/sites/ourspecificsite/");
Web web = ctx.Web;
Microsoft.SharePoint.Client.File schedule = web.GetFileById(new Guid("ourguid"));
ctx.Load(web);
ctx.Load(schedule);
ctx.ExecuteQuery();
FileInformation fInfo = Microsoft.SharePoint.Client.File.OpenBinaryDirect(ctx, schedule.ServerRelativeUrl);
using (var fileStream = File.Create(tempPath))
{
fInfo.Stream.CopyTo(fileStream);
}
}
catch (WebException e)
{
StatusUpdated?.Invoke(this, new EventArgs<string>(e.Message));
return null;
}
return tempPath;
}

using Microsoft.Graph;
using Microsoft.Graph.Auth;
using Microsoft.Identity.Client;
using System.IO;
using System.Linq;
namespace Answer
{
class Answer
{
static void Main(string[] args)
{
// Create Confidential Application
IConfidentialClientApplication confidentialClientApplication = ConfidentialClientApplicationBuilder
.Create("<My_Azure_Application_Client_ID>")
.WithTenantId("<My_Azure_Tenant_ID>")
.WithClientSecret("<My_Azure_Application_Client_Secret>")
.Build();
// Create an authentication provider.
ClientCredentialProvider authenticationProvider = new ClientCredentialProvider(confidentialClientApplication);
// Configure GraphServiceClient with provider.
GraphServiceClient graphServiceClient = new GraphServiceClient(authenticationProvider);
// Get a user
var user = graphServiceClient.Users["<My_Azure_User_Name>"].Request().GetAsync().Result;
// Get the teams the user is member of
var joinedTeams = graphServiceClient.Users[user.Id].JoinedTeams.Request().GetAsync().Result;
// Get the team we are intereseted in
var team1 = joinedTeams.FirstOrDefault(t => t.DisplayName == "<TeamName_Of_Interest>");
// Get the main folders
var folders = graphServiceClient.Groups[team1.Id].Drive.Root.Children
.Request()
.GetAsync().Result;
// Get the files in the first main folder
var files = graphServiceClient.Groups[team1.Id].Drive.Items[folders[0].Id].Children
.Request()
.GetAsync().Result;
// Get the file-Data of the first file
MemoryStream fileData = graphServiceClient.Groups[team1.Id].Drive.Items[files[0].Id].Content
.Request()
.GetAsync().Result as MemoryStream;
// Save the file to the hard-disc
System.IO.File.WriteAllBytes($"C:\\{files[0].Name}", fileData.ToArray());
}
}
}

AmazonSQSClient not refreshing AWSCredentials when Credentials File is updated

When my AWS Credentials File (see docs) is updated by an external process the AmazonSQSClient doesn't re-read it, SendMessageAsync fails with a security/token error.
We use a custom powershell script to refresh the local AWS cred's file periodically. The script works fine, the file is refreshed prior to the credentials expiring on AWS. However, if my app is running when the file is refreshed the new credentials are not re-read from the file, the "client" will show that the previous credentials are still in use.
The AWS docs list several AWSCredential providers but none of them seem to be the correct choice...I think..
Restarting the app works, the new credentials are read correctly and messages are sent until the next time the cred's file is updated.
using (var client = new AmazonSQSClient(Amazon.RegionEndpoint.EUWest1))
{
return client.SendMessageAsync(request);
}

I don't think there is a way for a running app to pick up the default credentials being refreshed in credentials file. There is a solution for Node.js loading credentials from a JSON file. You can create a similar solution in C#. You can also run a local DB to store credentials so whenever credentials file is updated DB table or JSON file is also updated. You will need to use access key and secret key in your SQS client constructor as opposed to using default credentials.
// Load these from JSON file or DB.
var accessKey = "";
var secretKey = "";
using (var client = new AmazonSQSClient(accessKey, secretKey, Amazon.RegionEndpoint.EUWest1))
{
return client.SendMessageAsync(request);
}

The following works "ok" but I've only tested it with one profile and the file watcher is not as timely as you'd like so I'd recommend you wrap your usage inside a Retry mechanism.
// Usage..
var credentials = new AwsCredentialsFile();
using (var client = new AmazonSQSClient(credentials, Amazon.RegionEndpoint.EUWest1))
{
return client.SendMessageAsync(request);
}
public class AwsCredentialsFile : AWSCredentials
{
// https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/net-dg-config-creds.html#creds-file
private const string DefaultProfileName = "default";
private static ConcurrentDictionary<string, ImmutableCredentials> _credentials = new ConcurrentDictionary<string, ImmutableCredentials>(StringComparer.OrdinalIgnoreCase);
private static FileSystemWatcher _watcher = BuildFileSystemWatcher();
private readonly System.Text.Encoding _encoding;
private readonly string _profileName;
public AwsCredentialsFile()
: this(AwsCredentialsFile.DefaultProfileName, System.Text.Encoding.UTF8)
{
}
public AwsCredentialsFile(string profileName)
: this(profileName, System.Text.Encoding.UTF8)
{
}
public AwsCredentialsFile(string profileName, System.Text.Encoding encoding)
{
_profileName = profileName;
_encoding = encoding;
}
private static FileSystemWatcher BuildFileSystemWatcher()
{
var watcher = new FileSystemWatcher
{
Path = Path.GetDirectoryName(GetDefaultCredentialsFilePath()),
NotifyFilter = NotifyFilters.LastWrite,
Filter = "credentials"
};
watcher.Changed += (object source, FileSystemEventArgs e) => { _credentials?.Clear(); };
watcher.EnableRaisingEvents = true;
return watcher;
}
public static string GetDefaultCredentialsFilePath()
{
return System.Environment.ExpandEnvironmentVariables(#"C:\Users\%USERNAME%\.aws\credentials");
}
public static (string AccessKey, string SecretAccessKey, string Token) ReadCredentialsFromFile(string profileName, System.Text.Encoding encoding)
{
var profile = $"[{profileName}]";
string awsAccessKeyId = null;
string awsSecretAccessKey = null;
string token = null;
var lines = File.ReadAllLines(GetDefaultCredentialsFilePath(), encoding);
for (int i = 0; i < lines.Length; i++)
{
var text = lines[i];
if (text.Equals(profile, StringComparison.OrdinalIgnoreCase))
{
awsAccessKeyId = lines[i + 1].Replace("aws_access_key_id = ", string.Empty);
awsSecretAccessKey = lines[i + 2].Replace("aws_secret_access_key = ", string.Empty);
if (lines.Length >= i + 3)
{
token = lines[i + 3].Replace("aws_session_token = ", string.Empty);
}
break;
}
}
var result = (AccessKey: awsAccessKeyId, SecretAccessKey: awsSecretAccessKey, Token: token);
return result;
}
public override ImmutableCredentials GetCredentials()
{
if (_credentials.TryGetValue(_profileName, out ImmutableCredentials value))
{
return value;
}
else
{
var (AccessKey, SecretAccessKey, Token) = ReadCredentialsFromFile(_profileName, _encoding);
var credentials = new ImmutableCredentials(AccessKey, SecretAccessKey, Token);
_credentials.TryAdd(_profileName, credentials);
return credentials;
}
}
}

SSL/TSL - Issue in finding Certificates

I am testing the example given in below link.
https://msdn.microsoft.com/en-us/library/system.net.security.sslstream.aspx
To generate certificates I am using the one with 40 userful answers SSLStream example - how do I get certificates that work?
To run the server I am using command
SslTcpServer.exe TempCert.cer
Below is the code from msdn where I am facing problem.
public static int Main(string[] args)
{
string serverCertificateName = null;
string machineName = null;
if (args == null ||args.Length <1 )
{
DisplayUsage();
}
// User can specify the machine name and server name.
// Server name must match the name on the server's certificate.
machineName = args[0];
if (args.Length <2 )
{
serverCertificateName = machineName;
}
else
{
serverCertificateName = args[1];
}
SslTcpClient.RunClient (machineName, serverCertificateName);
return 0;
}
I get below error when calling X509Certificate.CreateFromCertFile:
System.Security.Cryptography.CryptographicException: 'The system cannot find the file specified.
public static void RunServer(string certificate)
{
serverCertificate = X509Certificate.CreateFromCertFile(certificate);
// Create a TCP/IP (IPv4) socket and listen for incoming connections.
//serverCertificate = new X509Certificate2(certificate,"");
}
serverCertificateName is passed as argument and it should be just the name of the certificate or should i give the full path of the certificate?
If I give path of the certificate it is working fine.Then what is point in installing the certificates in the store? How can I get it from store and use it?

Here is some code that will return a list of the host names supported by installed certificates (that's a little more than you wanted, but should point you in the right direction):
System.Security.Cryptography.X509Certificates.X509Store store = new System.Security.Cryptography.X509Certificates.X509Store(System.Security.Cryptography.X509Certificates.StoreLocation.LocalMachine);
store.Open(System.Security.Cryptography.X509Certificates.OpenFlags.ReadOnly);
HashSet<string> certificateNames = new HashSet<string>();
foreach (System.Security.Cryptography.X509Certificates.X509Certificate2 mCert in store.Certificates)
{
// is this a UCC certificate?
System.Security.Cryptography.X509Certificates.X509Extension uccSan = mCert.Extensions["2.5.29.17"];
if (uccSan != null)
{
foreach (string nvp in uccSan.Format(true).Split(new string[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries))
{
string[] parts = nvp.Split('=');
string name = parts[0];
string value = (parts.Length > 0) ? parts[1] : null;
if (String.Equals(name, "DNS Name", StringComparison.InvariantCultureIgnoreCase))
{
certificateNames.Add(value.ToLowerInvariant());
}
}
}
else // just a regular certificate--add the single name
{
string certificateHost = mCert.GetNameInfo(System.Security.Cryptography.X509Certificates.X509NameType.SimpleName, false);
certificateNames.Add(certificateHost.ToLowerInvariant());
}
}
return certificateNames;

check for IP address and download the data

I have two devices having separate IP addresses and want to check if any one is connected and if connected download the database from the device and any given time only one device is connected.My query works fine for one device how to check which one is connected. I have updated my code but not sure how is it gone work.
private void button7_Click(object sender, EventArgs e)// 1)first download database to local system.
{
this.Process1();
}
public void Process1()
{
string _ftpURL = #"131.000.00.0"; // fake Host URL or address of the SFTP server
/* how to check for another IP adddress if exists */
string _UserName = "root"; //fake User Name of the SFTP server
string _Password = "3term"; // fake Password of the SFTP server
int _Port = 2222; //Port No of the SFTP server (if any)
string _ftpDirectory = "/home/root/systools/WM/WebMobility.db"; //The directory in SFTP server where the files will be uploaded
string LocalDirectory = "F:\\Explor\\final test"; //Local directory from where the files will be uploaded
try
{
Sftp Connection = new Sftp(_ftpURL, _UserName, _Password);
Connection.Connect(_Port);
Connection.Get(_ftpDirectory, LocalDirectory);
Connection.Close();
}
catch (Exception ex)
{
if (ex is SshConnectionException || ex is SocketException)
{
_ifwInstance.Error(string.Format("Ignoring {0} during listing directory", ex.Message));
}
else
{
string _ftpURL = #"131.111.11.11"; // fake Host URL or address of the SFTP server
/* how to check for another IP adddress if exists */
string _UserName = "root"; //fake User Name of the SFTP server
string _Password = "3term"; // fake Password of the SFTP server
int _Port = 2222; //Port No of the SFTP server (if any)
string _ftpDirectory = "/home/root/systools/WM/WebMobility.db"; //The directory in SFTP server where the files will be uploaded
string LocalDirectory = "F:\\Explor\\final test"; //Local directory from where the files will be uploaded
throw new Exception("Login to SFT FAILED", ex);
}
}
}
updated Code:
string[] _ftpURL = { #"100.100.0.0", #"101.0.0.0" }; //Array of address to SFTP servers
string _UserName = "root"; //fake User Name of the SFTP server
string _Password = "310rp3"; // fake Password of the SFTP server
int _Port = 2222; //Port No of the SFTP server (if any)
string _ftpDirectory = "/home/root/systools/WM/WebMobility.db"; //The directory in SFTP server where the files will be uploaded
string LocalDirectory = "F:\\Explor\\final test"; //Local directory from where the files will be uploaded
bool online = false;
foreach(string furl in _ftpURL)
{
Sftp Connection = new Sftp(furl, _UserName, _Password);
try
{
Connection.Connect(_Port);
online = true;
}
catch
{
online = false;
}
if(online == true)
{
Connection.Get(_ftpDirectory, LocalDirectory);
Connection.Close();
break;
}
}

Add these two methods to some class and call public DownloadSftpFile from your desktop code.
// returns true if the file had downloaded
public static bool DownloadSftpFile(string[] hosts, int port, string username, string password, string remotePathAndFile, string localPath)
{
foreach (var host in hosts)
{
try
{
DownloadSftpFile(host, port, username, password, remotePathAndFile, localPath);
return true;
}
catch(SshConnectionException exception)
{
// log
}
catch(SocketExcpetion exception)
{
// log
}
}
return false;
}
private static void DownloadSftpFile(string host, int port, string username, string password, string remotePathAndFile, string localPath)
{
using (var sftp = new Sftp(host, username, password))
{
sftp.Connect(port);
sftp.Get(remotePathAndFile, localPath);
}
}