I'm using the Microsoft MapReduce SDK to start a Mapper only job.
The call to hadoop.MapReduceJob.ExecuteJob is throwing a "Response status code does not indicate success: 404 (not found)" exception immediately.
When inspecting the HDInsight Query Console the job successfully starts and finishes later. It also writes proper output files.
My guess is, ExecuteJob is trying to access output data before the job has finished.
What is the correct way to handle this situation?
using System;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using Microsoft.WindowsAzure.Management.HDInsight;
using Microsoft.Hadoop.MapReduce;
using AzureAnalyzer.MultiAnalyzer;
namespace AzureAnalyzer
{
class Program
{
static void Main(string[] args)
{
IHadoop hadoop = Hadoop.Connect(Constants.azureClusterUri, Constants.clusterUser,
Constants.hadoopUser, Constants.clusterPassword, Constants.storageAccount,
Constants.storageAccountKey, Constants.container, true);
try {
var output = hadoop.MapReduceJob.ExecuteJob<MultiAnalyzerJob>();
}
catch (Exception ex)
{
Console.WriteLine("\nException: " + ex.Message);
}
}
}
}
I got another way to do the same thing but takes a bit of effort since it requires the mapper and reducer files to be transferred to the hadoop cluster storage.
you need to add Microsoft.Hadoop.Client then Microsoft Azure HDInsight NuGet package as well.
var jobcred = new BasicAuthCredential();
jobcred.UserName = "clusteruserid";
jobcred.Password = "clusterpassword";
jobcred.Server = new Uri("https://clusterurl");
StreamingMapReduceJobCreateParameters jobpara = new StreamingMapReduceJobCreateParameters()
{
JobName="mapreduce",
Mapper = "Mapper.exe",
Reducer = "Reducer.exe",
Input= "wasb:///mydata/input",
Output = "wasb:///mydata/Output",
StatusFolder= "wasb:///mydata/sOutput"
};
jobpara.Files.Add("wasb:///mydata/Mapper.exe");
jobpara.Files.Add("wasb:///mydata/Reducer.exe");
// Create a Hadoop client to connect to HDInsight.
var jobClient = JobSubmissionClientFactory.Connect(jobcred);
// Run the MapReduce job.
JobCreationResults mrJobResults = jobClient.CreateStreamingJob(jobpara);
// Wait for the job to complete.
Console.Write("Job running...");
JobDetails jobInProgress = jobClient.GetJob(mrJobResults.JobId);
while (jobInProgress.StatusCode != JobStatusCode.Completed
&& jobInProgress.StatusCode != JobStatusCode.Failed)
{
Console.Write(".");
jobInProgress = jobClient.GetJob(jobInProgress.JobId);
Thread.Sleep(TimeSpan.FromSeconds(10));
}
// Job is complete.
Console.WriteLine("!");
Console.WriteLine("Job complete!");
Console.WriteLine("Press a key to end.");
Console.Read();
Hope this helps. I was able to run jobs without throwing any exceptions.
this in fact waits for the job to be complete.
Please verify if all required services to run the program are up and running. The 404 error indicates that some URL the program tries to access internally is not accessible.
Related
I'm trying to make a simple console app client (starter.exe) on c# .NET Framework 4.6 to make a WireGuard protocol based connection using Wireguard source code.
What is done:
Downloaded wireguard source code from here: git://git.zx2c4.com/wireguard-windows
Successfuly built Tunnel.dll in ..\embeddable-dll-service\amd64\tunnel.dll via build.bat
Created a project in Visual Studio 2015.using the c# code from ..\embeddable-dll-service\csharp
Starting from here some strange thing are happenning:
if launching starter.exe \service <path to *.conf> I receive the
error
Service run error: The service process could not connect to the
service controller.
if launching starter.exe without parameters everything works fine until I remove the if{} block:
Unhandled Exception: System.ComponentModel.Win32Exception: The service
did not respond to the start or control request in a timely fashion
at WireGuardTunnel.Service.Add(String configFile) in
D:\Depository\BitBucket\WireGuard_Tunnel_Repository\WireGuardTunnel_proj\Launcher\Service.cs:line
69 at WireGuardTunnel.Program.Main(String[] args) in
D:\Depository\BitBucket\WireGuard_Tunnel_Repository\WireGuardTunnel_proj\Launcher\Program.cs:line
83
That means even if the code in if{} block is not executed it influencese somehow the application behaviour.
Next, as I want to make my app work with parameters I solved the
issue by removing return afer Service.Run and passing args[1] to Service.Add(args[1]). It works OK, but I have an extra log line (the first one due to Service.Run perpetual error described above) in the log:
Service run error: The service process could not connect to the
service controller. 235660: [TUN] [chicago4] Watching network
interfaces 245661: [TUN] [chicago4] Resolving DNS names
245661: [TUN] [chicago4] Creating Wintun interface 225660: [TUN]
[chicago4] Starting WireGuard/0.3.1 (Windows 6.1.7601; amd64)
So finally the questions:
Why Service.Run(confFile) does not work
Why Service.Run(confFile) influences the Service.Add(confFile)
Why if{} block is executed when I launch starte.exe with no parameters
The original Program.cs without modification:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net.Sockets;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
using System.IO;
using System.Runtime.InteropServices;
namespace Tunnel
{
class Program
{
[DllImport("kernel32.dll")]
private static extern bool SetConsoleCtrlHandler(SetConsoleCtrlEventHandler handler, bool add);
private delegate bool SetConsoleCtrlEventHandler(UInt32 signal);
public static void Main(string[] args)
{
string baseDirectory = Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName);
string configFile = Path.Combine(baseDirectory, "demobox.conf");
string logFile = Path.Combine(baseDirectory, "log.bin");
if (args.Length == 2 && args[0] == "/service")
{
configFile = args[1];
Service.Run(configFile);
return;
}
try { File.Delete(logFile); } catch { }
Ringlogger log = new Ringlogger(logFile, "GUI");
var logPrintingThread = new Thread(() =>
{
var cursor = Ringlogger.CursorAll;
while (Thread.CurrentThread.IsAlive)
{
var lines = log.FollowFromCursor(ref cursor);
foreach (var line in lines)
Console.WriteLine(line);
Thread.Sleep(300);
}
});
logPrintingThread.Start();
SetConsoleCtrlHandler(delegate
{
Service.Remove(configFile);
Environment.Exit(0);
return true;
}, true);
try
{
Service.Add(configFile);
logPrintingThread.Join();
}
finally
{
Service.Remove(configFile);
}
}
}
}
Bit late to the party but I was having the exact same issue as above and discovered that in order to get everything working correctly you have to have Tunnel.Service.Run("path to config") defined on application initialization either in your main loop or your constructor then you can run Tunnel.Service.Add("path to config", true) which will create the service and start the VPN connection. It's also good practice to destroy the service on close using Tunnel.Service.Remove("path to config", true) as the service will continue to run and you will still be connected to your VPN until it is stopped manually.
I read a couple of related questions which had an issue with accessing nency from a remote computer. However, I am unable to access nancy from my own pc.
Here is my code:
class Program
{
static void Main(string[] args)
{
HostConfiguration hostConfigs = new HostConfiguration();
//hostConfigs.RewriteLocalhost = true;
hostConfigs.UrlReservations.CreateAutomatically = true;
using (var host = new NancyHost(hostConfigs, new Uri("http://localhost:1234")))
{
host.Start();
Console.WriteLine("Running on http://+:1234");
Console.WriteLine(host.ToString());
}
Console.ReadKey();
}
}
public class HelloModule : NancyModule
{
public HelloModule()
{
Get["/"] = parameters => Response.AsJson("Success");
Get["/nancy"] = parameters => Response.AsJson("Success");
}
}
}
I am administrator on my PC and I do not get any exception. If I type http://localhost:1234 or http://127.0.0.1:1234 to my browser (with /nancy and without) I would expect a response. However, I do net get any reponse. Further, in the list produced with netstat -ano I do not see any process listing on port 1234. I downloaded the latest version of nancy via nuget.
Do you have any idea?
The following line should work as expected:
var host = new NancyHost(hostConfigs, new Uri("http://localhost:1234"))
But what happens with a using statement, is that anything specified between ( and ) (simply put) is disposed after the closing brace (}) of the same using statement. So what is actually happening is, the host gets created, is started, and is disposed right after it printed some lines to the console.
Simply put, move the ReadKey call inside the using statement. There it will wait until a key is pressed, and the host will be disposed after that event has occurred.
I am connecting to Apache Active MQ which is hosted on AWS to integrate my app to a custom service. I need to keep this running always, not one time like it's right now. The code below works, but only for one message, I need to maintain the connection active all the time listening in order to receive all the messages.
Here is the code.
using Apache.NMS;
using Apache.NMS.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace ApacheMQAsync
{
class Program
{
protected static ITextMessage message = null;
public static void Main(string[] args)
{
Uri connecturi = new Uri("URL:61617");
Console.WriteLine("About to connect to " + connecturi);
// NOTE: ensure the nmsprovider-activemq.config file exists in the executable folder.
IConnectionFactory factory = new Apache.NMS.ActiveMQ.ConnectionFactory(connecturi);
IConnection connection = factory.CreateConnection("username", "password");
ISession session = connection.CreateSession();
IDestination destination = SessionUtil.GetDestination(session, "queue://FOO.BAR");
Console.WriteLine("Using destination: " + destination);
// Create a consumer and producer
IMessageConsumer consumer = session.CreateConsumer(destination);
consumer.Listener += new MessageListener(OnMessage);
connection.Start();
// Wait for the message
if (message == null)
{
Console.WriteLine("No message received!");
}
else
{
Console.WriteLine("Received message with ID: " + message.NMSMessageId);
Console.WriteLine("Received message with text: " + message.Text);
}
}
protected static void OnMessage(IMessage receivedMsg)
{
message = receivedMsg as ITextMessage;
message.Acknowledge();
}
}
}
On the console it displays following
No message received!
and after few seconds the console exist?
There's no real magic there, you need to do something to keep your application running such as pausing on console input or looping on a sleep or other wait type call and then checking something to see if your application should continue. The JMS client isn't guaranteed to keep your application open and running and you should never rely on it to.
I need to learn how to use SMO within a C# program, so the first thing I did was start a new, console app and then began putting in the basics. I decided to make the app accept parameters so I can pass in things like usernames, logins, etc. As I work on it and build it I have a PowerShell window open where I can call the app, give it parameters or not, etc. But something weird is happening which I don't understand. Sometimes when I run the app in the PowerShell window, it's then deleted for some reason. Why is it doing that? I discovered it when it first gave me the following error message:
Program 'SmoListLogins.exe' failed to run: The system cannot find the file specifiedAt line:1 char:1 + .\SmoListLogins.exe "MYCOMPANY\Rod"
My SmoListLogins.exe program isn't there. Naturally I can easily re-create it, but I don't understand why its being deleted.
So you can see what I'm working with, here's the source code. I took it from a MSDN article and have added a little bit:
using System;
using System.Data;
using Microsoft.SqlServer.Management.Smo;
namespace SmoListLogins
{
class Program
{
static void Main(string[] args)
{
if (args.Length > 0)
{
var userName = args[0];
ListLogins(userName);
}
else
{
ListLogins();
}
}
static private void ListLogins(string userName = "")
{
var userNamePassed = (userName != "");
Server srv = new Server("YOURSQLINSTANCE");
//Iterate through each database and display.
foreach (Database db in srv.Databases)
{
Console.WriteLine("========");
Console.WriteLine("Login Mappings for the database: " + db.Name);
Console.WriteLine(" ");
//Run the EnumLoginMappings method and return details of database user-login mappings to a DataTable object variable.
DataTable d;
try
{
d = db.EnumLoginMappings();
//Display the mapping information.
foreach (DataRow r in d.Rows)
{
var userNameMatches = false;
var starting = true;
foreach (DataColumn c in r.Table.Columns)
{
if (!userNamePassed)
{
Console.WriteLine(c.ColumnName + " = " + r[c]);
}
else
{
if (starting)
{
starting = false;
if (userName == r[c].ToString())
{
userNameMatches = true;
}
}
if (userNameMatches)
{
Console.WriteLine(c.ColumnName + " = " + r[c]);
}
}
}
}
}
catch (Exception ex)
{
Console.WriteLine($"Error processing database: {db.Name}");
Console.WriteLine($"Error: {ex.Message}");
Console.WriteLine();
}
}
}
}
}
I believe I now know what was deleting my executable. It wasn't something I thought of, so I'm sharing with everyone the answer. Today I got an email from our chief security officer, informing me that the program I wrote was being blocked by Symantec Endpoint Protection. In my testing I'd run my app over and over again. After a few iterations of that it would disappear. It didn't occur to me that it was our corporate AV that might be doing it. Now it looks as though that is exactly what was going on.
Thank you everyone for your input in trying to help me resolve this. I hope that if anyone else encounters this problem they might consider the possibility of their AV as the reason why the app they wrote disappears.
How do you get all available queue names from client side, of your MQ environment using the IBM MQ lib for .Net (IBM.WMQ), Version 8.0?
I have written a fine .Net application for reading and sending data to MQ (similar founds at code project).
Do anyone know if it is possible/how to get all available queue names from the IBM.WMQ .NET lib dynamically as you do when using tool IBM test tool RfhUtil.exe or as you can do with runmqsc DISPLAY QUEUE command from IBM .Net lib?
I have tried to brows the API, Reference manual and IBM programming guide without success.
There is certain level of PCF support in MQ .NET but it is undocumented. Here is a sample code to display queue names in queue manager.
using System;
using System.Collections;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using IBM.WMQ;
using IBM.WMQ.PCF;
namespace PCFNET
{
class Program
{
static void Main(string[] args)
{
InquireQueue();
}
/// <summary>
/// Display list of queue names and queue depth for each queue
/// </summary>
public static void InquireQueue()
{
PCFMessageAgent messageAgent = null;
try
{
// Create bindings connection to queue manager
messageAgent = new PCFMessageAgent("DEMOQMGR");
// Build Inquire command to query queue name
PCFMessage reqeuestMessage = new PCFMessage(MQC.MQCMD_INQUIRE_Q);
reqeuestMessage.AddParameter(MQC.MQCA_Q_NAME, "*");
// Send request and receive response
PCFMessage[] pcfResponse = messageAgent.Send(reqeuestMessage);
// Process and print response.
int pcfResponseLen = pcfResponse.Length;
for (int pcfResponseIdx = 0; pcfResponseIdx < pcfResponseLen; pcfResponseIdx++)
{
try
{
String qName = pcfResponse[pcfResponseIdx].GetStringParameterValue(MQC.MQCA_Q_NAME);
int qDepth = pcfResponse[pcfResponseIdx].GetIntParameterValue(MQC.MQIA_CURRENT_Q_DEPTH);
Console.WriteLine("QName: " + qName + " Depth: " + qDepth);
}
catch (PCFException pcfex)
{
//Ignore exception and get the next response
}
}
}
catch (PCFException pcfEx)
{
Console.Write(pcfEx);
}
catch (MQException ex)
{
Console.Write(ex);
}
catch (Exception ex)
{
Console.Write(ex);
}
finally
{
if (messageAgent != null)
messageAgent.Disconnect();
}
}
}
}
There are PCFMessageAgent classes in Java, and I can see some seem to refer to equivalent classes in the .NET API.
It's possible to construct the PCF message yourself, as long as you have rights to access SYSTEM.ADMIN.COMMAND.QUEUE.
You also need to create reply queues dynamically based on SYSTEM.COMMAND.REPLY.MODEL or SYSTEM.MQSC.REPLY.QUEUE.