What is the best way to handle dependent events such as;
There is an object for which I need to test if connection is succeeded or failed.
But the object first needs to pass the initialization step which I test for success or failure and then continue to connection step.
If initialization fails return is connection failed.
If initialization succeeds return is result of the connection step.
My code is below. Is there a better way to handle those dependent events because I'm subscribing for connection inside initialization subscription?
If I have more dependent events like this will I keep nesting the subscriptions?
public static void Test()
{
const int maxValue = 501;
var random = new Random(BitConverter.ToInt32(Guid.NewGuid().ToByteArray(), 0));
var initOk = Observable.Interval(TimeSpan.FromMilliseconds(random.Next(maxValue))).Select(i => true);
var initKo = Observable.Interval(TimeSpan.FromMilliseconds(random.Next(maxValue))).Select(i => false);
var connectOk = Observable.Interval(TimeSpan.FromMilliseconds(random.Next(maxValue))).Select(i => true);
var connectKo = Observable.Interval(TimeSpan.FromMilliseconds(random.Next(maxValue))).Select(i => false);
var initResult = initOk.Amb(initKo).Take(1);
var connectResult = connectOk.Amb(connectKo).Take(1);
var id =
initResult.Subscribe(ir =>
{
if (ir)
{
var cd =
connectResult.Subscribe(cr =>
{
Console.WriteLine(cr
? "Connection succeeded."
: "Connection failed.");
});
}
else
{
Console.WriteLine("Initialization failed thus connection failed.");
}
});
}
You can normally avoid nesting by utilising a variety of the rx operators to chain calls up.
Your example could be tidied up in using:
initResult.SelectMany(ir =>
{
if (ir != null)
{
return connectResult;
}
Console.WriteLine("Initialization failed thus connection failed.");
return Observable.Throw(new Exception("Some Exception"));
})
.Subscribe(cr =>
{
Console.WriteLine(cr != null
? "Connection succeeded."
: "Connection failed.");
})
You could use this:
var finalResult =
initResult
.Select(ir =>
Observable.If(() => ir, connectResult, Observable.Return(false)))
.Merge();
To get your messages out, you could change it like this:
var initResultText =
initResult
.Select(ir =>
ir ? (string)null : "Initialization failed thus connection failed.");
var connectResultText =
connectResult
.Select(cr =>
String.Format("Connection {0}.", cr ? "succeeded" : "failed"));
var finalResult =
initResultText
.Select(irt =>
Observable.If(() =>
irt == null, connectResultText, Observable.Return(irt)))
.Merge();
If you need to nest further than this you should consider making an extension method that wraps up the complexity and thus composition would be much easier.
Related
I'm trying to use Polly as retry policy handler for grpc in my .net core 6 project. I noticed that the retryFunc is never invoked. I started from this project gRPC & ASP.NET Core 3.1: Resiliency with Polly
class Program
{
static async Task Main(string[] args)
{
// DI
var services = new ServiceCollection();
var loggerFactory = LoggerFactory.Create(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Debug);
});
var serverErrors = new HttpStatusCode[] {
HttpStatusCode.BadGateway,
HttpStatusCode.GatewayTimeout,
HttpStatusCode.ServiceUnavailable,
HttpStatusCode.InternalServerError,
HttpStatusCode.TooManyRequests,
HttpStatusCode.RequestTimeout
};
var gRpcErrors = new StatusCode[] {
StatusCode.DeadlineExceeded,
StatusCode.Internal,
StatusCode.NotFound,
StatusCode.ResourceExhausted,
StatusCode.Unavailable,
StatusCode.Unknown
};
Func<HttpRequestMessage, IAsyncPolicy<HttpResponseMessage>> retryFunc = (request) =>
{
return Policy.HandleResult<HttpResponseMessage>(r => {
var grpcStatus = StatusManager.GetStatusCode(r);
var httpStatusCode = r.StatusCode;
return (grpcStatus == null && serverErrors.Contains(httpStatusCode)) || // if the server send an error before gRPC pipeline
(httpStatusCode == HttpStatusCode.OK && gRpcErrors.Contains(grpcStatus.Value)); // if gRPC pipeline handled the request (gRPC always answers OK)
})
.WaitAndRetryAsync(3, (input) => TimeSpan.FromSeconds(3 + input), (result, timeSpan, retryCount, context) =>
{
var grpcStatus = StatusManager.GetStatusCode(result.Result);
Console.WriteLine($"Request failed with {grpcStatus}. Retry");
});
};
services.AddGrpcClient<CountryServiceClient>(o =>
{
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(retryFunc);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<CountryServiceClient>();
try
{
var countries = (await client.GetAllAsync(new EmptyRequest())).Countries.Select(x => new Country
{
CountryId = x.Id,
Description = x.Description,
CountryName = x.Name
}).ToList();
Console.WriteLine("Found countries");
countries.ForEach(x => Console.WriteLine($"Found country {x.CountryName} ({x.CountryId}) {x.Description}"));
}
catch (RpcException e)
{
Console.WriteLine(e.Message);
}
}
}
but at the end WaitAndRetryAsync is never called.
I created a small project available on github in order to reproduce it.
My test is fairly simple. I start the client without a listening back-end, expecting to read 3 times the output from Console.WriteLine($"Request failed with {grpcStatus}. Retry"); on the console. But the policy handler in never fired. I have the following exception instead
Status(StatusCode="Unavailable", Detail="Error connecting to
subchannel.", DebugException="System.Net.Sockets.SocketException
(10061): No connection could be made because the target machine
actively refused it.
without any retry.
This is not working for you because Retry is now built into Grpc. In order to make this work, register your service as follows:
var defaultMethodConfig = new MethodConfig
{
Names = { MethodName.Default },
RetryPolicy = new RetryPolicy
{
MaxAttempts = 3,
InitialBackoff = TimeSpan.FromSeconds(3),
MaxBackoff = TimeSpan.FromSeconds(3),
BackoffMultiplier = 1,
RetryableStatusCodes =
{
// Whatever status codes you want to look for
StatusCode.Unauthenticated, StatusCode.NotFound, StatusCode.Unavailable,
}
}
};
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
o.ChannelOptionsActions.Add(options =>
{
options.ServiceConfig = new ServiceConfig {MethodConfigs = {defaultMethodConfig}};
});
});
That will add the retry policy to your client. One other thing that you might run into. I didn't realize this at the time, but in my service implementation, I was setting up errors something like this:
var response = new MyServiceResponse()
// something bad happens
context.Status = new Status(StatusCode.Internal, "Something went wrong");
return response;
The retry logic will not kick in if you implement your service like that, you actually have to do something more like this:
// something bad happens
throw new RpcException(new Status(StatusCode.Internal, "Something went wrong"));
The retry logic you configured when registering your client will then work. Hope that helps.
With the help of #PeterCsala I tried some fix.
As a first attempt I tried without DependencyInjection, registering the policy as follows
var policy = Policy
.Handle<Exception>()
.RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Message}. Retry");
});
var channel = GrpcChannel.ForAddress("https://localhost:5001");
TestServiceClient client = new TestServiceClient(channel);
await policy.ExecuteAsync(async () => await client.TestAsync(new Empty()));
This way it's working.
Then I came back to DI and used to register the policy as follows
IAsyncPolicy<HttpResponseMessage> policy =
Policy<HttpResponseMessage>.Handle<Exception>().RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Exception.Message}. Retry");
});
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(policy);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<TestServiceClient>();
var testClient = (await client.TestAsync(new Empty()));
And still not working.
At the end it seems AddPolicyHandler is not suitable for grpc clients?
I've implemented an observer for a FileSystemWatcher. The idea is to track copies in a folder and wait until copy is finished. Once done, let's do something with copied files.
I've tried my implementation on a small program and it works. What I wanted to do is to make it more formal using unit test:
[TestMethod]
public void TestIsFileWritingFinished()
{
try
{
var dirName = Path.GetTempPath()+Path.DirectorySeparatorChar+DateTime.Now.ToString("MMddyyyy");
if (Directory.Exists(dirName))
{
Directory.Delete(dirName, true);
}
var dir = Directory.CreateDirectory(dirName);
var observer = new FileSystemObserver(dirName, "*.*", true)
.ChangedFiles
.Where(x => (new FileInfo(x.FullPath)).Length > 0)
.Select(x => x.Name);
var timeout = observer.Timeout(/*DateTimeOffset.UtcNow.AddSeconds(1)*/TimeSpan.FromSeconds(1));
var filesChanged = new List<string>();
var terminated = false;
timeout.Subscribe(Console.WriteLine, Console.WriteLine, ()=>terminated=true);
Thread.Sleep(100);
var origin = #"C:\Users\David\ResultA";
CopyDirectory(origin, dirName, true);
Thread.Sleep(100);
Console.WriteLine("nap for 5s");
Thread.Sleep(5000);
//Directory.Delete(dirName, true);
Assert.IsTrue(terminated);
}
catch(Exception ex)
{
Assert.Fail(ex.Message);
}
}
So, when timeout happens, I expect the boolean to be true. But looks like it's not.
Any idea about what's wrong with my test?
Thanks in advance, your suggestions will be appreciated,
Kind regards,
Even thought there is a Timeout operator, I find it is a bit of an anti-pattern. It's like programming with exceptions. I find the following pattern more useful:
IObservable<bool> query =
Observable.Amb(
source.LastAsync().Select(x => true),
Observable.Timer(TimeSpan.FromSeconds(seconds)).Select(x => false));
This is effectively running source to completion and then returns true, but if the Timer completes first it returns false.
So, in your code, I'd try something like this:
var dirName = Path.GetTempPath() + Path.DirectorySeparatorChar + DateTime.Now.ToString("MMddyyyy");
if (Directory.Exists(dirName))
{
Directory.Delete(dirName, true);
}
var dir = Directory.CreateDirectory(dirName);
var observer =
new FileSystemObserver(dirName, "*.*", true)
.ChangedFiles
.Where(x => (new FileInfo(x.FullPath)).Length > 0)
.Select(x => x.Name);
var result =
Observable.Amb(
observer.ToArray().Select(x => true),
Observable.Timer(TimeSpan.FromSeconds(1.0)).Select(x => false));
var query =
result.Zip(Observable.Start(() =>
{
var origin = #"C:\Users\David\ResultA";
CopyDirectory(origin, dirName, true);
}), (r, s) => r);
var terminated = query.Wait();
Assert.IsTrue(terminated);
That totally avoids any pesky sleeps.
Below is my code in C# Windows application, where connection with Oracle, FTP and Null Reference Exception are handled:
public Result Execute()
{
Result result = null;
string errorMessage = string.Empty;
var retryTimes = 1000;
var retryableErrorCodes = new[] { "ORA-03113", "ORA-03114", "ORA-12543",
"ORA-12170", "ORA-12154", "ORA-12541", "ORA-12560", "ORA-03135",
"Connection request timed out" };
var retryExceptionError = new[] { "Object reference not set to an instance of an object" };
RetryPolicy retryPolicyFTP = Policy
.Handle<Xceed.Ftp.FtpInvalidStateException>().Or<Xceed.Ftp.FtpIOException>()
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
RetryPolicy retryPolicyOracle = Policy
.Handle<OracleException>(ex => retryableErrorCodes.Any(errorCode => ex.Message.ToString().Contains(errorCode)))
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
RetryPolicy retryException = Policy
.Handle<Exception>(ex => retryExceptionError.Any(errorCode => ex.Message.ToString().Contains(errorCode)))
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
Policy.Wrap(retryPolicyFTP, retryPolicyOracle, retryException).Execute(() =>
{
//few lines of C# Code like fetching details from Database
if(some condition)
{
//Some Operations
return new Result(ResultType.Failure, "This function has Failed");
}
if(some other condition)
{
//Some Operations
return new Result(ResultType.Success, "This function is Successful");
}
//Some more lines of C# Code
});
return Result.Successful;
}
With this code, I cannot use return keyword in the middle of the function, because Polly framework does not allow it to do so.
Could you please suggest what is the better way of handling return keyword in the middle of the function?
In Polly you can define decorators for methods and for functions.
In case of a method the retry policy should be defined like this:
RetryPolicy retryPolicyFTP = Policy
.Handle<Xceed.Ftp.FtpInvalidStateException>().Or<Xceed.Ftp.FtpIOException>()
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
In case of a function the retry policy should be defined like this:
RetryPolicy<Result> retryPolicyFTP = Policy<Result>
.Handle<Xceed.Ftp.FtpInvalidStateException>().Or<Xceed.Ftp.FtpIOException>()
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
You should spot here the <Result> parts on lhs and rhs as well.
With this knowledge your method could be rewritten like this:
public Result Execute()
{
Result result = null;
string errorMessage = string.Empty;
var retryTimes = 1000;
var retryableErrorCodes = new[] { "ORA-03113", "ORA-03114", "ORA-12543", "ORA-12170", "ORA-12154", "ORA-12541", "ORA-12560", "ORA-03135", "Connection request timed out" };
var retryExceptionError = new[] { "Object reference not set to an instance of an object" };
RetryPolicy<Result> retryPolicyFTP = Policy<Result>
.Handle<Xceed.Ftp.FtpInvalidStateException>().Or<Xceed.Ftp.FtpIOException>()
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
RetryPolicy<Result> retryPolicyOracle = Policy<Result>
.Handle<OracleException>(ex => retryableErrorCodes.Any(errorCode => ex.Message.ToString().Contains(errorCode)))
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
RetryPolicy<Result> retryException = Policy<Result>
.Handle<Exception>(ex => retryExceptionError.Any(errorCode => ex.Message.ToString().Contains(errorCode)))
.WaitAndRetry(retryTimes, _ => TimeSpan.FromSeconds(1));
Result res = Policy.Wrap(retryPolicyFTP, retryPolicyOracle, retryException).Execute(() =>
{
if (some condition)
{
return new Result(ResultType.Failure, "This function has Failed");
}
if (some other condition)
{
return new Result(ResultType.Success, "This function is Successful");
}
return Result.Successful;
});
return res;
}
Because your Execute has to return a Result that's why the Result.Successful could be moved inside the Execute block.
I would also suggest to separate strategy declaration and execution like this:
public Result Execute()
{
...
var strategy = Policy.Wrap(retryPolicyFTP, retryPolicyOracle, retryException)
return strategy.Execute(() =>
{
if (some condition)
{
return new Result(ResultType.Failure, "This function has Failed");
}
if (some other condition)
{
return new Result(ResultType.Success, "This function is Successful");
}
return Result.Successful;
});
}
I found this class online:
public class AsyncSearcher
{
LdapConnection _connect;
public AsyncSearcher(LdapConnection connection)
{
this._connect = connection;
this._connect.AutoBind = true; //will bind on first search
}
public void BeginPagedSearch(
string baseDN,
string filter,
string[] attribs,
int pageSize,
Action<SearchResponse> page,
Action<Exception> completed
)
{
if (page == null)
throw new ArgumentNullException("page");
AsyncOperation asyncOp = AsyncOperationManager.CreateOperation(null);
Action<Exception> done = e =>
{
if (completed != null) asyncOp.Post(delegate
{
completed(e);
}, null);
};
SearchRequest request = new SearchRequest(
baseDN,
filter,
System.DirectoryServices.Protocols.SearchScope.Subtree,
attribs
);
PageResultRequestControl prc = new PageResultRequestControl(pageSize);
//add the paging control
request.Controls.Add(prc);
AsyncCallback rc = null;
rc = readResult =>
{
try
{
var response = (SearchResponse)_connect.EndSendRequest(readResult);
//let current thread handle results
asyncOp.Post(delegate
{
page(response);
}, null);
var cookie = response.Controls
.Where(c => c is PageResultResponseControl)
.Select(s => ((PageResultResponseControl)s).Cookie)
.Single();
if (cookie != null && cookie.Length != 0)
{
prc.Cookie = cookie;
_connect.BeginSendRequest(
request,
PartialResultProcessing.NoPartialResultSupport,
rc,
null
);
}
else done(null); //signal complete
}
catch (Exception ex) { done(ex); }
};
//kick off async
try
{
_connect.BeginSendRequest(
request,
PartialResultProcessing.NoPartialResultSupport,
rc,
null
);
}
catch (Exception ex) { done(ex); }
}
}
I am basically trying to convert the below code which writes to the console to return data from Task.Factory.FromAsync, so that I can use the data elsewhere.
using (LdapConnection connection = CreateConnection(servername))
{
AsyncSearcher searcher = new AsyncSearcher(connection);
searcher.BeginPagedSearch(
baseDN,
"(sn=Dunn)",
null,
100,
f => //runs per page
{
foreach (var item in f.Entries)
{
var entry = item as SearchResultEntry;
if (entry != null)
{
Console.WriteLine(entry.DistinguishedName);
}
}
},
c => //runs on error or when done
{
if (c != null) Console.WriteLine(c.ToString());
Console.WriteLine("Done");
_resetEvent.Set();
}
);
_resetEvent.WaitOne();
}
I tried this but get the following syntax errors:
LdapConnection connection1 = CreateConnection(servername);
AsyncSearcher1 searcher = new AsyncSearcher1(connection1);
async Task<SearchResultEntryCollection> RootDSE(LdapConnection connection)
{
return await Task.Factory.FromAsync(,
() =>
{
return searcher.BeginPagedSearch(baseDN, "(cn=a*)", null, 100, f => { return f.Entries; }, c => { _resetEvent.Set(); });
}
);
}
_resetEvent.WaitOne();
The APM ("Asynchronous Programming Model") style of asynchronous code uses Begin and End method pairs along with IAsyncResult, following a specific pattern.
The Task.Factory.FromAsync method is designed to wrap APM method pairs into a modern TAP ("Task-based Asynchronous Programming") style of asynchronous code.
However, FromAsync requires the methods to follow the APM pattern exactly, and BeginPagedSearch does not follow the APM pattern. So you will need to use TaskCompletionSource<T> directly. TaskCompletionSource<T> can be used to convert any existing asynchronous pattern to TAP as long as it has a single result.
The method you're trying to wrap has multiple callbacks, so it can't be mapped to TAP at all. If you want to collect all result sets and return a list of them, then you can use TaskCompletionSource<T> for that. Otherwise, you'll want to use something like IAsyncEnumerable<T>, which would require writing your own implementation of BeginPagedSearch.
I have an SSIS package that's launching another SSIS package in a Foreach container; because the container reports completion as soon as it launched all the packages it had to launch, I need a way to make it wait until all "child" packages have completed.
So I implemented a little sleep-wait loop that basically pulls the Execution objects off the SSISDB for the ID's I'm interested in.
The problem I'm facing, is that a grand total of 0 Dts.Events.FireProgress events get fired, and if I uncomment the Dts.Events.FireInformation call in the do loop, then every second I get a message reported saying 23 packages are still running... except if I check in SSISDB's Active Operations window I see that most have completed already and 3 or 4 are actually running.
What am I doing wrong, why wouldn't runningCount contain the number of actually running executions?
using ssis = Microsoft.SqlServer.Management.IntegrationServices;
public void Main()
{
const string serverName = "REDACTED";
const string catalogName = "SSISDB";
var ssisConnectionString = $"Data Source={serverName};Initial Catalog=msdb;Integrated Security=SSPI;";
var ids = GetExecutionIDs(serverName);
var idCount = ids.Count();
var previousCount = -1;
var iterations = 0;
try
{
var fireAgain = true;
const int secondsToSleep = 1;
var sleepTime = TimeSpan.FromSeconds(secondsToSleep);
var maxIterations = TimeSpan.FromHours(1).TotalSeconds / sleepTime.TotalSeconds;
IDictionary<long, ssis.Operation.ServerOperationStatus> catalogExecutions;
using (var connection = new SqlConnection(ssisConnectionString))
{
var server = new ssis.IntegrationServices(connection);
var catalog = server.Catalogs[catalogName];
do
{
catalogExecutions = catalog.Executions
.Where(execution => ids.Contains(execution.Id))
.ToDictionary(execution => execution.Id, execution => execution.Status);
var runningCount = catalogExecutions.Count(kvp => kvp.Value == ssis.Operation.ServerOperationStatus.Running);
System.Threading.Thread.Sleep(sleepTime);
//Dts.Events.FireInformation(0, "ScriptMain", $"{runningCount} packages still running.", string.Empty, 0, ref fireAgain);
if (runningCount != previousCount)
{
previousCount = runningCount;
decimal completed = idCount - runningCount;
decimal percentCompleted = completed / idCount;
Dts.Events.FireProgress($"Waiting... {completed}/{idCount} completed", Convert.ToInt32(100 * percentCompleted), 0, 0, "", ref fireAgain);
}
iterations++;
if (iterations >= maxIterations)
{
Dts.Events.FireWarning(0, "ScriptMain", $"Timeout expired, requesting cancellation.", string.Empty, 0);
Dts.Events.FireQueryCancel();
Dts.TaskResult = (int)Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Canceled;
return;
}
}
while (catalogExecutions.Any(kvp => kvp.Value == ssis.Operation.ServerOperationStatus.Running));
}
}
catch (Exception exception)
{
if (exception.InnerException != null)
{
Dts.Events.FireError(0, "ScriptMain", exception.InnerException.ToString(), string.Empty, 0);
}
Dts.Events.FireError(0, "ScriptMain", exception.ToString(), string.Empty, 0);
Dts.Log(exception.ToString(), 0, new byte[0]);
Dts.TaskResult = (int)ScriptResults.Failure;
return;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
The GetExecutionIDs function simply returns all execution ID's for the child packages, from my metadata database.
The problem is that you're re-using the same connection at every iteration. Turn this:
using (var connection = new SqlConnection(ssisConnectionString))
{
var server = new ssis.IntegrationServices(connection);
var catalog = server.Catalogs[catalogName];
do
{
catalogExecutions = catalog.Executions
.Where(execution => ids.Contains(execution.Id))
.ToDictionary(execution => execution.Id, execution => execution.Status);
Into this:
do
{
using (var connection = new SqlConnection(ssisConnectionString))
{
var server = new ssis.IntegrationServices(connection);
var catalog = server.Catalogs[catalogName];
catalogExecutions = catalog.Executions
.Where(execution => ids.Contains(execution.Id))
.ToDictionary(execution => execution.Id, execution => execution.Status);
}
And you'll get correct execution status every time. Not sure why the connection can't be reused, but keeping connections as short-lived as possible is always a good idea - and that's another proof.