I am using Azure Data Lake to upload the file before but still want to append the text file content existing Data Lake text file. Is there any option available to append the text file data using Web HDFS REST API in C#?.
I am refer this link enter link description here
Code: I Can refer the above link get the append URL. But how can i use this URL and Append a File using c#?
private const string AppendUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?&op=APPEND&noredirect=true";
If you want to use the Rest Api to do that we could use the following code. I test it with Postman.
private const string AppendUrl = "https://{datalakeName}.azuredatalakestore.net/webhdfs/v1/{filepath}?append=true&op=APPEND&api-version=2016-11-01"
var token = "eyJ0eX.....";
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
var result = client.GetAsync(url).Result;
var data = result.Content.ReadAsStringAsync().Result;
}
We also could use the Azure Microsoft.Azure.Management.DataLake.Store to do that. How to get the application id and secretkey you could refer to official document. More detail steps to get the permission to access the datalake you could refer to another SO thread.
var applicationId = "application Id";
var secretKey = "secretKey";
var tenantId = "tenant id";
var adlsAccountName = "datalake account name";
var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, applicationId, secretKey).Result;
var adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds,clientTimeoutInMinutes:60);
var stream = File.OpenRead(#"C:\tom\testtext.txt");
var test = adlsFileSystemClient.FileSystem.AppendWithHttpMessagesAsync(adlsAccountName, "test/abc.txt", stream).Result;
packages:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Microsoft.Azure.Management.DataLake.Store" version="2.3.0-preview" targetFramework="net452" />
<package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.0-preview" targetFramework="net452" />
<package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.13.8" targetFramework="net452" />
<package id="Microsoft.Rest.ClientRuntime" version="2.3.9" targetFramework="net452" />
<package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.9" targetFramework="net452" />
<package id="Microsoft.Rest.ClientRuntime.Azure.Authentication" version="2.2.0-preview" targetFramework="net452" />
<package id="Newtonsoft.Json" version="9.0.2-beta1" targetFramework="net452" />
</packages>
Related
An appointment that has participants, always return an empty collection when queried for using the appointment.Invitees property.
Here is what I am trying to do,
var calendarStore = await AppointmentManager.RequestStoreAsync(AppointmentStoreAccessType.AllCalendarsReadOnly);
var appointments = await calendarStore.FindAppointmentsAsync(DateTimeOffset.Now, TimeSpan.FromDays(30), new FindAppointmentsOptions() { IncludeHidden = true });
foreach (var appointment in appointments)
{
// appointment.Invitees is always null, even for appointments that has some!
foreach (var invitee in appointment.Invitees)
{
// do something here...
}
}
I tried to add Contacts capability in addition to Appointment, but in vain.
Any thoughts on why such a bizarre behavior?
The appointmentsSystem capability is necessary when access the Invitees property.
<Package xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10"
xmlns:mp="http://schemas.microsoft.com/appx/2014/phone/manifest"
xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10"
xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities"
IgnorableNamespaces="uap mp rescap">
<Capabilities>
<Capability Name="internetClient"/>
<uap:Capability Name="appointments" />
<uap:Capability Name="contacts" />
<rescap:Capability Name="appointmentsSystem" />
</Capabilities>
No one may request access to this capability for store submission
Please note that the appointmentsSystem is useless in store app.
For some reason, I have to latest Microsoft.IdentityModel.Clients.ActiveDirectory v3.13.9 with Microsoft.Azure.KeyVault v2.0.6. And it always throws exceptions for missing assembly reference, but downgrading Microsoft.IdentityModel.Clients.ActiveDirectory to v2.28.4, which I'm not allowed to use, can resolve this known issue.
Here's the exception:
InnerException:
HResult=-2146233036
Message=The type initializer for 'Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformPlugin' threw an exception.
Source=Microsoft.IdentityModel.Clients.ActiveDirectory
TypeName=Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformPlugin
StackTrace:
at Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformPlugin.get_Logger()
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext..cctor()
InnerException:
ErrorCode=assembly_not_found
HResult=-2146233088
Message=Assembly required for the platform not found. Make sure assembly 'Microsoft.IdentityModel.Clients.ActiveDirectory.Platform, Version=3.12.0.827, Culture=neutral, PublicKeyToken=31bf3856ad364e35' exists
Source=Microsoft.IdentityModel.Clients.ActiveDirectory
StackTrace:
at Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformPlugin.LoadPlatformSpecificAssembly()
at Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformPlugin.InitializeByAssemblyDynamicLinking()
at Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformPlugin..cctor()
InnerException:
FileName=Microsoft.IdentityModel.Clients.ActiveDirectory.Platform, Version=3.12.0.827, Culture=neutral, PublicKeyToken=31bf3856ad364e35
FusionLog==== Pre-bind state information ===
Code snippets:
protected KeyVaultBase(string clientId, string clientSecret)
{
if (string.IsNullOrWhiteSpace(clientId))
{
throw new ArgumentNullException(nameof(clientId));
}
if (string.IsNullOrWhiteSpace(clientSecret))
{
throw new ArgumentNullException(nameof(clientSecret));
}
this.KeyVaultClient = new KeyVaultClient(
new KeyVaultClient.AuthenticationCallback(
(authority, resource, scope) => GetAccessToken(authority, resource, scope, clientId, clientSecret)
)
);
}
private static async Task<string> GetAccessToken(string authority, string resource, string scope, string clientId, string clientSecret)
{
var authnContext = new AuthenticationContext(authority);
var clientCredential = new ClientCredential(clientId, clientSecret);
var result = await authnContext.AcquireTokenAsync(resource, clientCredential);
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the JWT token");
}
return result.AccessToken;
}
The exception throws while executing var authnContext = new AuthenticationContext(authority);.
Does anyone know any solution/workaround to resolve this issue without the downgrading?
In my case the references were installed only to the class library project, not to the executing assembly, so it was throwing this exception.
Adding the Microsoft.IdentityModel.Clients.ActiveDirectory nuget package to the executing assembly worked for me.
Hope it helps
Please have a try uninstall both of Microsoft.IdentityModel.Clients.ActiveDirectory v3.13.9 and Microsoft.Azure.KeyVault v2.0.6 then reinstall them again. If there are multiple projects in the solution are referenced them, please make sure all of the them are the same version.
I create a console or Asp.net MVC project, both are working correctly on my side using Microsoft.IdentityModel.Clients.ActiveDirectory v3.13.9 with Microsoft.Azure.KeyVault v2.0.6.
The following is my detail steps:
1.Create a C# console project.
2.Add the Microsoft.IdentityModel.Clients.ActiveDirectory v3.13.9 with nuget.
3.Add the Microsoft.Azure.KeyVault v2.0.6 from Package manager console with command
Install-Package Microsoft.Azure.KeyVault -Version 2.0.6
Packages.config file:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Microsoft.Azure.KeyVault" version="2.0.6" targetFramework="net452" />
<package id="Microsoft.Azure.KeyVault.WebKey" version="2.0.4" targetFramework="net452" />
<package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.13.9" targetFramework="net452" />
<package id="Microsoft.Rest.ClientRuntime" version="2.3.2" targetFramework="net452" />
<package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.1" targetFramework="net452" />
<package id="Newtonsoft.Json" version="6.0.8" targetFramework="net452" />
</packages>
Add a class in the project:
class KeyVaultBase
{
static string _clientId= "xxxxxxxxxxxxx";
static string _clientSecret = "xxxxxxxxxxxxxxxx";
static string _tenantId = "xxxxxxxxxxxxxxxx";
public static async Task<string> GetAccessToken(string azureTenantId, string azureAppId, string azureSecretKey)
{
var context = new AuthenticationContext("https://login.windows.net/" + _tenantId);
ClientCredential clientCredential = new ClientCredential(_clientId, _clientSecret);
var tokenResponse = await context.AcquireTokenAsync("https://vault.azure.net", clientCredential);
var accessToken = tokenResponse.AccessToken;
return accessToken;
}
}
4.Test Get Secret from the KeyVault
static void Main(string[] args)
{
var kv = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(KeyVaultBase.GetAccessToken));
string secretUri = "https://KeyVaultNameSpace.vault.azure.net/secrets/secretsname/6341322679db4f3491814ff6178dee45";
var sec = kv.GetSecretAsync(secretUri).Result;
}
I need to perform few Azure SQL operations. I have an Azure AD native application. I'm using first approach from following article to aquire the token.
https://msdn.microsoft.com/en-us/library/azure/ee460782.aspx
Now following this article, I'm using the above token to perform the db operation.
static void HttpPost(string sourceDb, string targetDb, string pointInTime)
{
var client = new HttpClient();
string uri = "https://management.core.windows.net:8443/" + AzureSubscriptionId + "/services/sqlservers/servers/" + AzureSqlServerName + "/restoredatabaseoperations";
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, uri);
request.Headers.Add("Authorization", "Bearer " + accessToken);
request.Headers.Add("x-ms-version", "2012-03-01");
string payload = File.ReadAllText("Resources\\Backup.xml");
payload = payload.Replace("$SourceDb", sourceDb);
payload = payload.Replace("$TargetDb", targetDb);
payload = payload.Replace("$PointInTime", pointInTime);
request.Content = new StringContent(payload, Encoding.UTF8, "application/xml");
HttpResponseMessage response = client.SendAsync(request).GetAwaiter().GetResult();
if (response.Content != null)
{
string ss = response.Content.ReadAsStringAsync().Result;
}
}
But the error I receive is:
"<Error xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Code>AuthenticationFailed</Code><Message>A security token exception occured for the received JWT token.</Message></Error>"
According to you mentioned Create Database Restore Request (Classic) REST API, this command is used for classic deployment model. We should use the new REST API, and it is also mentioned in your mentioned document.
You should use the newer Resource Manager based REST API commands located here.
We could use the ARM REST Create or Update DataBase API. About how to get the token, we need to to registry AD App and assign role to application, more info please refer to official document. I send the http request from fiddler and it works correctly for me. Header and body info please refer to the screenshot.
Body info:
{
"properties": {
"edition": "Standard",
"requestedServiceObjectiveName": "S1",
"sourceDatabaseId": "/subscriptions/{your subscriptionId}/resourceGroups/{ResourceGroup}/providers/Microsoft.Sql/servers/{servername}/databases/sourcedatabasename",
"createMode": "PointInTimeRestore",
"restorePointInTime": "2017-02-09T10:28:20.21+08:00" //source database restorePointTime
},
"location": "East Asia",
"tags": {}
}
We also can use Microsoft Azure SQL Management Library for .NET to that.
SqlMgmtClient.Databases.CreateOrUpdate(resourceGroupName, serverName, databaseName, DatabaseCreateOrUpdateParameters);
We could refer to the tutorial to get started. I do a demo for it. More details please refer to the following steps
1.Create a console app and install the required libraries (detail refer to tutorial)
2.After registry an application, we could get tenantId, applicationId,
SecretKey, then with subscriptionId to get the authentication token.
3.Create SqlManagementClient object with token
var _sqlMgmtClient = new SqlManagementClient(new TokenCloudCredentials(_subscriptionId, _token.AccessToken));
4.Create DatabaseCreateOrUpdateParameters according to our requirement.
Take restore database from a source database for example:
CreateMode = DatabaseCreateMode.PointInTimeRestore, //craete mode from pointtimerestore
Edition = databaseEdition,
SourceDatabaseId = "/subscriptions/subscriptionId/resourceGroups/groupname/providers/Microsoft.Sql/servers/AzureSQlname/databases/databaseName", //source database Id
RestorePointInTime = DateTime.Parse("2017-02-09T02:28:20.21Z"), //resore point Time
RequestedServiceObjectiveName = "S1"
run the demo and check from the portal.
democode:
static void Main(string[] args)
{
_token = GetToken(_tenantId, _applicationId, _applicationSecret);
Console.WriteLine("Token acquired. Expires on:" + _token.ExpiresOn);
// Instantiate management clients:
_resourceMgmtClient = new ResourceManagementClient(new Microsoft.Rest.TokenCredentials(_token.AccessToken));
_sqlMgmtClient = new SqlManagementClient(new TokenCloudCredentials(_subscriptionId, _token.AccessToken));
DatabaseCreateOrUpdateResponse dbr = CreateOrUpdateDatabase(_sqlMgmtClient, _resourceGroupName, _serverName, _databaseName, _databaseEdition, _databasePerfLevel);
Console.WriteLine("Database: " + dbr.Database.Id);
}
private static AuthenticationResult GetToken(string tenantId, string applicationId, string applicationSecret)
{
AuthenticationContext authContext = new AuthenticationContext("https://login.windows.net/" + tenantId);
_token = authContext.AcquireToken("https://management.core.windows.net/", new ClientCredential(applicationId, applicationSecret));
return _token;
}
static DatabaseCreateOrUpdateResponse CreateOrUpdateDatabase(SqlManagementClient sqlMgmtClient, string resourceGroupName, string serverName, string databaseName, string databaseEdition, string databasePerfLevel)
{
// Retrieve the server that will host this database
Server currentServer = sqlMgmtClient.Servers.Get(resourceGroupName, serverName).Server;
// Create a database: configure create or update parameters and properties explicitly
DatabaseCreateOrUpdateParameters newDatabaseParameters = new DatabaseCreateOrUpdateParameters()
{
Location = currentServer.Location,
Properties = new DatabaseCreateOrUpdateProperties
{
CreateMode = DatabaseCreateMode.PointInTimeRestore,
Edition = databaseEdition,
SourceDatabaseId = "/subscriptions/subscriptionId/resourceGroups/tomnewgroup/providers/Microsoft.Sql/servers/tomsunsqltest/databases/sourceDatabaseName",
RestorePointInTime = DateTime.Parse("2017-02-09T02:28:20.21Z"),//Restore Point time
RequestedServiceObjectiveName = databasePerfLevel
}
};
DatabaseCreateOrUpdateResponse dbResponse = sqlMgmtClient.Databases.CreateOrUpdate(resourceGroupName, serverName, databaseName, newDatabaseParameters);
return dbResponse;
}
packages.config file:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Hyak.Common" version="1.0.2" targetFramework="net462" />
<package id="Microsoft.Azure.Common" version="2.1.0" targetFramework="net462" />
<package id="Microsoft.Azure.Common.Authentication" version="1.7.0-preview" targetFramework="net462" />
<package id="Microsoft.Azure.Common.Dependencies" version="1.0.0" targetFramework="net462" />
<package id="Microsoft.Azure.Management.ResourceManager" version="1.4.0-preview" targetFramework="net462" />
<package id="Microsoft.Azure.Management.Sql" version="0.51.0-prerelease" targetFramework="net462" />
<package id="Microsoft.Bcl" version="1.1.9" targetFramework="net462" />
<package id="Microsoft.Bcl.Async" version="1.0.168" targetFramework="net462" />
<package id="Microsoft.Bcl.Build" version="1.0.14" targetFramework="net462" />
<package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="2.18.206251556" targetFramework="net462" />
<package id="Microsoft.Net.Http" version="2.2.22" targetFramework="net462" />
<package id="Microsoft.Rest.ClientRuntime" version="2.1.0" targetFramework="net462" />
<package id="Microsoft.Rest.ClientRuntime.Azure" version="3.1.0" targetFramework="net462" />
<package id="Microsoft.Rest.ClientRuntime.Azure.Authentication" version="2.0.1-preview" targetFramework="net462" />
<package id="Newtonsoft.Json" version="6.0.8" targetFramework="net462" />
</packages>
Please open a support case allowing us to better understand the issue
When I'm trying to create CloudBlobClient, I get this exception:
Exception thrown: 'System.ArgumentOutOfRangeException' in Microsoft.WindowsAzure.Storage.ni.DLL
System.ArgumentOutOfRangeException: The argument 'ParallelOperationThreadCount' is larger than maximum of '64'
Parameter name: ParallelOperationThreadCount
at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.AssertInBounds[T](String paramName, T val, T min, T max)
at Microsoft.WindowsAzure.Storage.Blob.BlobRequestOptions.set_ParallelOperationThreadCount(Nullable`1 value)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobClient..ctor(StorageUri storageUri, StorageCredentials credentials)
at Microsoft.WindowsAzure.Storage.CloudStorageAccount.CreateCloudBlobClient()
at App1.MainPage.<Azure>d__4.MoveNext()
Here's a code:
StorageCredentials sc = new StorageCredentials("name", "token==");
CloudStorageAccount storageAccount = new CloudStorageAccount(sc, true);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient(); //Here's an exception
CloudBlobContainer container = blobClient.GetContainerReference("images");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(file.Name.ToLower()); //Will be changed to more unique
await blockBlob.UploadFromFileAsync(file);
This is only when deployed to ARM device, in x86 Emulator everything is working OK.
I also tried this:
StorageCredentials sc = new StorageCredentials("name", "token==");
CloudStorageAccount storageAccount = new CloudStorageAccount(sc, true);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
No exceptions, everything is OK.
Here's a packages:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Microsoft.Data.Edm" version="5.6.5-beta" targetFramework="wpa81" />
<package id="Microsoft.Data.OData" version="5.6.5-beta" targetFramework="wpa81" />
<package id="System.Spatial" version="5.6.5-beta" targetFramework="wpa81" />
<package id="WindowsAzure.Storage" version="5.0.3-preview" targetFramework="wpa81" />
</packages>
I tried both preview and stable releases, no luck with ARM devices.
Let's say I have the following code running on a Windows service.
I have to send data from a Windows service on a machine to a webpage that is open on the same machine(but not hosted on that machine).
static void Main(string[] args)
{
Int32 counter = 0;
while (counter < 100)
{
SendUDP("127.0.0.1", 49320, counter.ToString(), counter.ToString().Length);
Thread.Sleep(2000);
counter++;
}
}
public static void SendUDP(string hostNameOrAddress, int destinationPort, string data, int count)
{
IPAddress destination = Dns.GetHostAddresses(hostNameOrAddress)[0];
IPEndPoint endPoint = new IPEndPoint(destination, destinationPort);
byte[] buffer = Encoding.ASCII.GetBytes(data);
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
for (int i = 0; i < count; i++)
{
socket.SendTo(buffer, endPoint);
}
socket.Close();
System.Console.WriteLine("Sent: " + data);
}
I have to listen for the data sent to port 49320 and then process it in the browser.
I can create a listener on the webpage with Node.js like below, but I have to start this service separately and also install node.js on the client.
Is there any alternative to this? Something more lightweight ?
I could also create an AJAX to query a webservice every 2 seconds that would do the same thing as Windows Service, but it seems like a lot of queries sent all the time for nothing.
//websocket gateway on 8070
var app = require('http').createServer(handler)
, io = require('socket.io').listen(app)
, fs = require('fs');
var mysocket = 0;
app.listen(8070);
function handler (req, res) {
fs.readFile(__dirname + '/index.html',
function (err, data) {
if (err) {
res.writeHead(500);
return res.end('Error loading index.html');
}
res.writeHead(200);
res.end(data);
});
}
io.sockets.on('connection', function (socket) {
console.log('index.html connected');
mysocket = socket;
});
//udp server on 49320
var dgram = require("dgram");
var server = dgram.createSocket("udp4");
server.on("message", function (msg, rinfo) {
console.log("msg: " + msg);
if (mysocket != 0) {
mysocket.emit('field', "" + msg);
mysocket.broadcast.emit('field', "" + msg);
}
});
server.on("listening", function () {
var address = server.address();
console.log("udp server listening " + address.address + ":" + address.port);
});
server.bind(49320);
Use Signal R. This is a technology which automatically negotiates the most efficient transport for pushing events from a server to a browser (eg Forever Frames, Web Sockets) - it will fall back to the polling approach you mention in your question if required. I think this is a better approach to using raw UDP because all the low level work has been done for you.
Here's an example of using Signal R from a windows service:
https://code.msdn.microsoft.com/windowsapps/SignalR-self-hosted-in-6ff7e6c3
From that sample - here's an even more simplified example of SignalR using OWIN to self host (without IIS):
using System;
using Microsoft.AspNet.SignalR;
using Microsoft.Owin.Cors;
using Microsoft.Owin.Hosting;
using Owin;
namespace SignalRWindowsService
{
static class Program
{
static void Main()
{
string url = "http://localhost:8080";
WebApp.Start(url);
Console.WriteLine("Press any key to exit");
Console.ReadKey();
}
}
class Startup
{
public void Configuration(IAppBuilder app)
{
//This means you can access the web sockets from the local service (on a different URL) than the domain the page in the browser was served up from.
app.UseCors(CorsOptions.AllowAll);
app.MapSignalR();
}
}
public class MyHub : Hub
{
public void Send(string name, string message)
{
Clients.All.addMessage(name, message);
}
}
}
You need the following NUGET packages:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Microsoft.AspNet.Cors" version="5.0.0" targetFramework="net45" />
<package id="Microsoft.AspNet.SignalR.Core" version="2.0.3" targetFramework="net45" />
<package id="Microsoft.AspNet.SignalR.SelfHost" version="2.0.3" targetFramework="net45" />
<package id="Microsoft.Owin" version="2.1.0" targetFramework="net45" />
<package id="Microsoft.Owin.Cors" version="2.1.0" targetFramework="net45" />
<package id="Microsoft.Owin.Diagnostics" version="2.0.1" targetFramework="net45" />
<package id="Microsoft.Owin.Host.HttpListener" version="2.0.1" targetFramework="net45" />
<package id="Microsoft.Owin.Hosting" version="2.0.1" targetFramework="net45" />
<package id="Microsoft.Owin.Security" version="2.0.1" targetFramework="net45" />
<package id="Microsoft.Owin.SelfHost" version="2.0.1" targetFramework="net45" />
<package id="Newtonsoft.Json" version="5.0.1" targetFramework="net45" />
<package id="Owin" version="1.0" targetFramework="net45" />
</packages>