I have been attempting to upload files using c# into Hadoop using the WebHDFS REST API.
This code works fine:
using (var client = new System.Net.WebClient())
{
string result = client.DownloadString("http:/ /host:50070/webhdfs/v1/user/myuser/?op=LISTSTATUS");
client.DownloadFile("http:/ /host:50070/webhdfs/v1/user/myuser/tbible.txt?user.name=myuser&op=OPEN","d:\tbible.txt");
}
This code gets a 403 Forbidden:
using (var client = new System.Net.WebClient())
{
client.UploadFile("http:/ /host:50070/webhdfs/v1/user/myuser/?user.name=myuser&op=CREATE", "PUT", "d:\bible.txt");
}
I have tried adding a network credential, with no luck.
How do I authenticate to our Cluster from .NET? The Cluster is Hortonworks HDP1.3 on RHEL5.
(The extra spaces in this post are to keep http:/ / from being a link)
Also, I would have liked to use Microsoft's hadoop SDK, but it is alpha and wont compile in my environment :(
Make sure that you are writing to a directory that is under the group which WebHDFS operates under. By default this is hdfs.
A quick way to check this doing hadoop fs -ls on the directory's parent directory to get the group permission settings (the second column that may look like a username).
Related
I'm working for a client that has given me access to two specific folders and their subfolders only. The first one used to be our previous working space and now we will switch to the second one.
When I connect to the SFTP using the WinSCP GUI it connects me to the old folder. However, I can change that by clicking on settings and adding the “new” path in the remote path field. The session will then take me to the new default folder/workspace automatically when I connect.
My question is how can I do this using .NET and the respective winscpnet library?
The problem is the root directory of the session is different to remote path.
Example :
Session directory is /C/Document/.
Remote path is /C/Inetpub/ftproot/username/
When I used the following command on terminal:
winscp.com> open sftp://someone:password;fingerprint=something#ipaddress/C/Inetpub/ftproot/username
winscp.com> put some.txt /in
winscp.com> exit
it works fine! Because as we can see, my session directory is /C/Inetpub/ftproot/username/.
Is there a way to set session root path in C#?
Solved: you are right it is a virtual path so /c/Inetpub instead of c/Inetpub
WinSCP .NET assembly does not use the concept of a working directory. It always uses absolute paths.
So if you have your GUI session configured to start in /new/path, use that as an absolute path in WinSCP .NET assembly.
session.PutFiles(#"c:\local\path\*", "/new/path/", false, transferOptions);
the modern way of doing things would be, using Renci.SshNet.
Dont forget to use regular slashes and to quit your session after things are done.
you can create your Session with:
var connectionInfo = new Renci.SshNet.ConnectionInfo(host, username, new PasswordAuthenticationMethod(username, password));
using (var sftp = new SftpClient(connectionInfo))
{
sftp.Connect();
sftp.ChangeDirectory("..");
sftp.ChangeDirectory("C:\Inetpub\ftproot\username\");
}
sftp.Disconnect();
I am attempting to use Google Drive as a storage location in my UWP application. I started at the quickstart provided by Google. I copy the code into a blank UWP project, change some of the output code (Console.Writeline to a textbox.append method) and I try to build it. It fails to build and reports the error:
Cannot find type System.ComponentModel.ExpandableObjectConverter in module System.dll
I am running Windows 10 and VS 2015 and I have installed the sdk through NuGet. The example code in the quickstart does work in a console application. It is the UWP application that is having issues.
For the UWP application, I put the quickstart code in a button click method. This was because the API actually has an async method for the uwp apps which is a bit different then the code given in the quickstart.
Includes:
using System;
using System.Collections.Generic;
using System.IO;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Drive.v3;
using Google.Apis.Drive.v3.Data;
using Google.Apis.Services;
using Google.Apis.Util.Store;
using System.Threading;
The Button Method:
private async void button_Click(object sender, RoutedEventArgs e)
{
UserCredential credential;
using (var stream =
new FileStream("client_secret.json", FileMode.Open, FileAccess.Read))
{
string credPath = ""; //System.Environment.GetFolderPath(
//System.Environment.SpecialFolder.Personal);
credPath = Path.Combine(credPath, ".credentials/drive-dotnet-quickstart.json");
credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(
new Uri("ms-appx:///Assets/client_secrets.json"),
Scopes,
"user",
CancellationToken.None);
//Console.WriteLine("Credential file saved to: " + credPath);
}
// Create Drive API service.
var service = new DriveService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = ApplicationName,
});
// Define parameters of request.
FilesResource.ListRequest listRequest = service.Files.List();
listRequest.PageSize = 10;
listRequest.Fields = "nextPageToken, files(id, name)";
// List files.
IList<Google.Apis.Drive.v3.Data.File> files = listRequest.Execute()
.Files;
textBox.Text += "Files:\n";
if (files != null && files.Count > 0)
{
foreach (var file in files)
{
textBox.Text += (file.Name + file.Id + "\n");
}
}
else
{
textBox.Text += ("No files found.");
}
}
The test code will not work once the app is compiled as it is missing the code to load the client secret. Since I have not been able to test the code, this is all I can provide.
There is another post that is semi-related except that the answer is just that it wont work and the post has been dead for 4 years. I also wanted to create a new post that tags the google team specifically (like the quickstart says to do).
My specific question is: Is there a work around to this issue or am I just doing this wrong?
I agree with #Vincent, UWP apps use COM as a base and builds from there. Not all .Net API can be used in UWP apps, this SDK is based on .Net APIs, this is why your console app is OK, but your UWP app is down. For the differences between them, here is a great answer which explain this issue. But,
"You will need an UWP SDK from Google to build an UWP applications."
I just tried to search for this without any luck, but here is a suggestion, you can use JavaScript to make request to the Drive API. To do this, you can refer to JavaScript Quickstart. Then you can turn it to a web hosted UWP app, for more information, you can refer to Convert your web application to a Universal Windows Platform (UWP) app.
Another suggestion which can probably make the work easier is using Rest API to send HTTP requests, you can also refer to API Reference.
The final suggestion which is as #Vincent said, if you have access to the SDK code, you can also try to adapt it for UWP. It means you need to modify the source code of this SDK.
The .Net flavor used to build Windows Store/UWP apps has less features than the full .Net framework. Unfortunately, the ExpandableObjectConverter object is not available for UWP applications.
You will need an UWP SDK from Google to build an UWP applications.
If you have access to the SDK code, you can also try to adapt it for UWP.
I've written an asp.net webapp that writes a file to a location on our iSeries FileShare.
The path looks like this: \IBMServerAddress\Filepath
This code executes perfectly on my local machine, but fails when it's deployed to my (windows) WebServer.
I understand that i may need to do some sort of impersonation to authenticate access to the IFS, but i'm unsure of how to proceed.
Here's the code i'm working with:
string filepath = "\\\\IBMServerAddress\\uploads\\";
public int SaveToDisk(string data, string plant)
{
//code for saving to disk
StreamWriter stream = null;
stream = File.CreateText(filepath + plant + ".txt"); // creating file
stream.Write(data + "\r\n"); //Write data to file
stream.Close();
return 0;
}
Again, this code executes perfectly on my local machine but does not work when deployed to my Windows WebServer - access to filepath is denied.
Thanks for your help.
EDIT: I've tried adding a network account with the same credentials as the IFS user, created a UNC path (iseries)on IIS7 to map the network drive (using the same credentials) - but receive this error:
Access to the path 'iseries\' is denied.
My understanding of Windows in general is that normally services don't have access to standard network shares like a program being run by a user does.
So the first thing would be to see if you can successfully write to a windows file share from the web server.
Assuming that works, you'll need one of two things in order to write to the IBM i share..
1) An IBM i user ID and password that matches the user ID and password the process is being run under
2) A "guest account" configured on IBM i Netserver
http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_71/rzahl/rzahlsetnetguestprof.htm
You might have better luck with using Linux/UNIX based Network File System (NFS) which is supported in both Windows and the IBM i.
I'm attempting to write a simple proof-of-concept for manipulating cloud servers in my Rackspace Cloud account via the C# openstacknetsdk library, v1.3.0.0.
The problem I'm having is although I appear to be able to authenticate successfully using my Rackspace username and API key, the API is behaving as though I don't have any servers. My code:
using net.openstack.Providers.Rackspace;
using net.openstack.Core.Providers;
using net.openstack.Core.Exceptions.Response;
using net.openstack.Providers.Rackspace.Objects;
using net.openstack.Core.Domain;
...
const string USERNAME = "[my rackspace username]";
const string API_KEY = "[my rackspace API key]";
static void Main(string[] args)
{
CloudIdentity cloudIdentity = new CloudIdentity() { APIKey = API_KEY, Username = USERNAME };
CloudServersProvider provider = new CloudServersProvider(cloudIdentity);
IEnumerable<SimpleServer> servers = provider.ListServers();
foreach (SimpleServer server in servers)
{
Console.Out.WriteLine(server.Id);
}
}
When I step through this code in the debugger, I can see that servers ends up having size 0, and nothing ends up getting written out.
This is unexpected, because I do have one server created, which I can see on the Rackspace cloud control panel website. The server is in Active status.
If I try to get information on my specific server, using the server ID from the Cloud Servers > Server Details page on the Rackspace cloud control panel site:
Server server = provider.GetDetails("[my cloud server ID]");
Console.Out.WriteLine(server.Image.Name);
I get a net.openstack.Core.Exceptions.Response.ItemNotFoundException.
The authentication seems to be working because if I intentionally change my API_KEY value to something incorrect (like "test"), I get a UserNotAuthorizedException instead.
What am I missing here? Why is the openstacknetsdk acting like I don't have any servers?
Is the server you have created in your default region?
To be safe, try specifying the region in the .ListServers() method call.
Also; you can download sample data via NuGet; search for "openstack sample".
I'd like to generate presigned URLs for images hosted on Amazon S3 without using the SDK. I'm currently using Rackspace to host my app and you can only do medium trust on their servers, so I can't use the SDK.
Any help is appreciated.
I'm doing this in C#.
Thanks.
What part is being restricted? I read through http://www.rackspace.com/knowledge_center/index.php/Overview_of_modified_Medium_Trust
You should be able to add the AWSSDK.dll to the bin of your application on the server. AWSSDK.dll found in the SDK install directory, on my machine it is C:\Program Files (x86)\AWS SDK for .NET\bin.
If that is still not allowing you access, you could run it on your local machine and intercept the actual call being made by the SDK, using fiddler. Then make the XML call directly from your own code.
This actually resolved the issue.
https://forums.aws.amazon.com/thread.jspa?threadID=39030&start=15&tstart=0
trick is to do this:
AmazonS3Config s3Config = new AmazonS3Config();
s3Config.UseSecureStringForAwsSecretKey = false;
AmazonS3Client s3Client = new AmazonS3Client(accessKey, secretKey, s3Config);