I'm writing my first Service Worker to handle copying all records from one table to another every 60 seconds. At the moment everything works like a charm, but after deploying I wan't to make sure that all future data will be copied 100 records at one time, until no more records left to copy - and than 60 seconds break.
Fallowing code will copy every record from given date at once, and than wait 60 seconds.
public void CopyLatestData(DateTime from)
{
using (var context = new EdDbContext())
{
var dataToCopy = context.ELMAH_Errors.Where(x => x.TimeUtc > from).ToList();
var result = dataToCopy.Select(x => new parsed_errors(x)).ToList();
context.parsed_errors.AddRange(result);
context.SaveChanges();
}
}
This is how I handle timer
private static void startTimer()
{
if (timer == null)
{
timer = new Timer();
timer.Interval = 60 * 1000;
timer.Elapsed += timer_ElapsedActions;
timer.Start();
}
}
private static void timer_ElapsedActions(object sender, ElapsedEventArgs e)
{
if (isTaskAlreadyRunning)
{
return;
}
lock (locker)
{
if (!isTaskAlreadyRunning)
{
isTaskAlreadyRunning = true;
try
{
copyDataFromRemoteDatabaseToLocalDatabase();
parseCopiedData();
}
finally
{
isTaskAlreadyRunning = false;
}
}
}
what is the easiest way to achieve what I need?
my suggestion would be add a boolean in your Entity like AlreadyCopied or IsCopied. Then use that as a marker.
public bool CopyLatestData()
{
using (var context = new EdDbContext())
{
var query = context.ELMAH_Errors.Where(x => !x.IsCopied).AsQueryable();
//if (query.Any())
if (query.Count() != 0) //this will check if some are left not copied
{
var dataToCopy = query.Take(100).ToList(); //this will only take the first 100
var result = dataToCopy.Select(x => new parsed_errors(x)).ToList();
context.parsed_errors.AddRange(result);
dataToCopy.ForEach(x => x.IsCopied = true); //this will update the records to IsCopied = true
context.SaveChanges();
if (query.Any())
{
return true;
}
else
{
return false;
}
}
else
{
return false;
}
}
}
then in your timer stop timer if the bool value of the if (Copylatestdata() == false)
private static void timer_ElapsedActions(object sender, ElapsedEventArgs e)
{
var stillHasRecords = CopyLatestData();
if (!stillHasRecords)
{
Timer timer = (Timer)sender; //get the sender object
timer.Stop(); // stop the sender
}
}
You could try using .Take() to limit the number of rows returned by the query, that way every 60 seconds it will return a maximum of 100 records. If fewer than 100 records are present it will only return what is there.
var dataToCopy = context.ELMAH_Errors.Where(x => x.TimeUtc > from).Take(100).ToList();
An alternative to 'IsProcessed' would be to keep 'latestProcessed' and then retrieve records that have timestamps after that, in batches of 100.
(Something like the code below, sorry for bad formatting, I'm typing on a phone)
while( keepGoing) {
var dataToCopy = context.ELMAH_Errors
.Where(x => x.TimeUtc > from)
.Where(x => x.TimeUTc > lastProcessedUtc)
.Take(100)
.OrderBy( x => x.TimeUtc)
.ToList();
(...)
if( dataToCopy.Any() ) lastProcessedUtc = data.Last().TimeUtc;
keepGoing = dataToCopy.Any(); //Or .Count() == 100
Related
Hi I have this two blocks of code
when I navigate the browser to a url and try to wait for it I get a deadlock I think=) any help suggestions are much appreciated!
I'm just adding some more text here so I am able to post the question :/ sorry
foreach (var davaType in davaTypes)
{
Parallel.ForEach(years, year =>
{
ts.Add(Task.Run(() =>
{
var th = new Thread(async () =>
{
await doWorkAsync(year, davaType, tarafType);
Application.Run();
});
th.IsBackground = true;
th.SetApartmentState(ApartmentState.STA);
th.Start();
}));
});
}
and
public static async Task doWorkAsync(int year, string davaType, string tarafType)
{
using (var browser = new MyBrowser())
{
Console.WriteLine("Im Created Here");
browser.Navigate("some url");
while (!browser.MyLoaded)
{
Console.WriteLine("waiting");
await Task.Delay(10);
}
Console.WriteLine("complete");
browser.Document?.GetElementById("DropDownList1")?.SetAttribute("value", davaType);
browser.Document.GetElementById("DropDownList3")?.SetAttribute("value", tarafType);
browser.Document.GetElementById("DropDownList4")?.SetAttribute("value", year.ToString());
browser.Document.GetElementById("Button1").MyClick();
await browser.WaitPageLoad(10);
Console.WriteLine("Im DoneHere");
}
}
You can use TaskCompletionSource to create a task and await for it after the page is loaded.
See: UWP WebView await navigate.
I have experienced this problem before and the solution I have applied is as follows;
private void waitTillLoad()
{
WebBrowserReadyState loadStatus = default(WebBrowserReadyState);
int waittime = 100000;
int counter = 0;
while (true)
{
loadStatus = webBrowser1.ReadyState;
Application.DoEvents();
if ((counter > waittime) || (loadStatus == WebBrowserReadyState.Uninitialized) || (loadStatus == WebBrowserReadyState.Loading) || (loadStatus == WebBrowserReadyState.Interactive))
{
break; // TODO: might not be correct. Was : Exit While
}
counter += 1;
}
counter = 0;
while (true)
{
loadStatus = webBrowser1.ReadyState;
Application.DoEvents();
if (loadStatus == WebBrowserReadyState.Complete)
{
break; // TODO: might not be correct. Was : Exit While
}
counter += 1;
}
}
You need to create only one separate thread for WinForms. Other threads will be created by individual browsers themselves. What you do need is handling their DocumentCompleted event. The first time the document completes you enter and submit data, the second time it completes you get the result. When all browsers finish, you exit the winforms thread and you're done.
Note: To launch bots from winforms thread, I made use of Idle event handler, just because it was the first solution that came to my mind. You may find better way.
using System;
using System.Collections.Generic;
using System.Threading;
using System.Windows.Forms;
namespace ConsoleBrowsers
{
class Program
{
static void Main(string[] args)
{
string url = "https://your.url.com";
string[] years = { "2001", "2002", "2003"};
string[] davas = { "1", "2", "3" };
string taraf = "1";
int completed = 0, succeeded = 0;
var bots = new List<Bot>();
var started = false;
var startBots = (EventHandler)delegate (object sender, EventArgs b)
{
if (started)
return;
started = true;
foreach (var dava in davas)
foreach (var year in years)
bots.Add(new Bot { Year = year, Dava = dava, Taraf = taraf, Url = url }.Run((bot, result) =>
{
Console.WriteLine($"{result}: {bot.Year}, {bot.Dava}");
succeeded += result ? 1 : 0;
if (++completed == years.Length * davas.Length)
Application.ExitThread();
}));
};
var forms = new Thread((ThreadStart) delegate {
Application.EnableVisualStyles();
Application.Idle += startBots;
Application.Run();
});
forms.SetApartmentState(ApartmentState.STA);
forms.Start();
forms.Join();
Console.WriteLine($"All bots finished, succeeded: {succeeded}, failed:{bots.Count - succeeded}.");
}
}
class Bot
{
public string Url, Dava, Taraf, Year;
public delegate void CompletionCallback(Bot sender, bool result);
WebBrowser browser;
CompletionCallback callback;
bool submitted;
public Bot Run(CompletionCallback callback)
{
this.callback = callback;
browser = new WebBrowser();
browser.ScriptErrorsSuppressed = true;
browser.DocumentCompleted += Browser_DocumentCompleted;
browser.Navigate(Url);
return this;
}
void Browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
if (browser.ReadyState == WebBrowserReadyState.Complete)
{
if (submitted)
{
callback(this, true);
return;
}
var ok = SetValue("DropDownList1", Dava);
ok &= SetValue("DropDownList3", Taraf);
ok &= SetValue("DropDownList4", Year);
if (ok)
ok &= Click("Button1");
if (!(submitted = ok))
callback(this, false);
}
}
bool SetValue(string id, string value, string name = "value")
{
var e = browser?.Document?.GetElementById(id);
e?.SetAttribute(name, value);
return e != null;
}
bool Click(string id)
{
var element = browser?.Document?.GetElementById(id);
element?.InvokeMember("click");
return element != null;
}
}
}
I am creating an app that measures the time a person has gazed at around 300 objects. For all of the objects, it needs to output the time into a text file. I've succesfully coded this for just one object, but I am trying to do it for 300 objects. This is the code I have created for one object:
Stopwatch Timer = new Stopwatch();
gazeButtonControl1 = GazeInput.GetGazeElement(GazeBlock1);
gazeButtonControl1 = new GazeElement();
GazeInput.SetGazeElement(GazeBlock1, gazeButtonControl1);
TimeSpan Word1 = TimeSpan.Zero;
gazeButtonControl1.StateChanged += GazeBlockControl1_StateChanged;
void GazeBlockControl1_StateChanged(object sender, StateChangedEventArgs ea)
{
if (ea.PointerState == PointerState.Enter)
{
Timer.Start();
}
if (ea.PointerState == PointerState.Exit)
{
Timer.Stop();
Word1 += Timer.Elapsed;
File.WriteAllText(#"*insert path here", Word1.ToString());
Timer.Reset();
}
}
Everytime there is a "1" inserted in the variable or name of an element, I want to have that for all of the 300 objects. This is what I am looking for:
Stopwatch Timer = new Stopwatch();
for (int i = 0; i < 300; i++) {
gazeButtonControl[i] = GazeInput.GetGazeElement(GazeBlock[i]);
gazeButtonControl[i] = new GazeElement();
GazeInput.SetGazeElement(GazeBlock[i], gazeButtonControl[i]);
TimeSpan Word[i] = TimeSpan.Zero;
gazeButtonControl[i].StateChanged += GazeBlockControl[i]_StateChanged;
void GazeBlockControl[i]_StateChanged(object sender, StateChangedEventArgs ea)
{
if (ea.PointerState == PointerState.Enter)
{
Timer.Start();
}
if (ea.PointerState == PointerState.Exit)
{
Timer.Stop();
Woord[i] += Timer.Elapsed;
File.WriteAllText(#"*insert path here", Word[i].ToString());
Timer.Reset();
}
}
}
This code doesn't work. I've tried using lists and that didn't work either. Does anyone know how to make this solution work? Thanks in advance!
Vincent
You can't use placeholders or array indices in names of variables, methods, and so on.
For UWP there seems to be a helper method FindChildByName which allows you to find a control by its name, via (TypeOfChildControl)insertParentControlHere.FindChildByName("<Name of child control>").
I find the following code of yours perplexing:
gazeButtonControl1 = GazeInput.GetGazeElement(GazeBlock1);
gazeButtonControl1 = new GazeElement();
GazeInput.SetGazeElement(GazeBlock1, gazeButtonControl1);
Basically, first you get your gazeButtonControl1 from GazeBlock1 and then ignore it to create a new GazeElement. I guess what is meant is something like in this UWP Gaze sample where it's first checked if the UI control has a gaze element and if not one is created. So I think it should be:
gazeButtonControl1 = GazeInput.GetGazeElement(GazeBlock1);
if (gazeButtonControl1 == null)
{
gazeButtonControl1 = new GazeElement();
GazeInput.SetGazeElement(GazeBlock1, gazeButtonControl1);
}
Also like i wrote in my initial comment you probably need a separate timer for each gaze element.
So in total it should be something like this (albeit untested):
public class YourClass
{
private TimeSpan[] Word = new TimeSpan[300];
private Stopwatch[] Timer = new Stopwatch[300];
private GazeElement[] gazeButtonControl = new GazeElement[300];
public YourMethod(...)
{
for (int i = 0; i < 300; i++)
{
// first find gaze block where parentControl is the control containing your gaze blocks.
var gazeBlock = (UIElement)parentControl.FindChildByName("GazeBlock" + (i + 1));
gazeButtonControl[i] = GazeInput.GetGazeElement(gazeBlock);
if (gazeButtonControl[i] == null)
{
gazeButtonControl[i] = new GazeElement();
GazeInput.SetGazeElement(gazeBlock, gazeButtonControl[i]);
}
Word[i] = TimeSpan.Zero;
Timer[i] = new Stopwatch();
gazeButtonControl[i].StateChanged += GazeBlockControl_StateChanged;
}
}
private void GazeBlockControl_StateChanged(object sender, StateChangedEventArgs ea)
{
// get the GazeElement for which this event was raised
var changedControl = (GazeElement)sender;
// find its index in your list of GazeElements.
var i = Array.IndexOf(gazeButtonControl, changedControl);
if (ea.PointerState == PointerState.Enter)
{
Timer[i].Start();
}
if (ea.PointerState == PointerState.Exit)
{
Timer[i].Stop();
Word[i] += Timer.Elapsed;
File.WriteAllText(#"*insert path here", Word[i].ToString());
Timer[i].Reset();
}
}
}
Please not that I use Array.IndexOf instead of gazeButtonControl.IndexOf because I declared gazeButtonControl as an array. If it would be a list (e.g. List<GazeElement> gazeButtonControl) you would use gazeButtonControl.IndexOf.
As part of my school project I'm trying to link two tables to decrease the amount of data stored in one table so I wanted to link my "Scores" class with my "CorrectAnswers" class via the ObjectID. However since the tasks are asynchronous, by the time one task is done saving, the other task has already begun or also finished saving and so the ObjectID returns as null.
Here's the code I'm using:
public void SaveScore()
{
ParseObject SendScore = new ParseObject("Scores");
SendScore["Score"] = CheckAnswer.score;
SendScore["user"] = ParseObject.CreateWithoutData("_User", ParseUser.CurrentUser.ObjectId);
SendScore["TestMode"] = MainMenu.testmode;
SendScore["TotalQuestions"] = QuestionCreation.TotalQuestions;
SendScore["CorrectQuestions"] = CheckAnswer.CorrectQuestions;
SendScore.SaveAsync().ContinueWith(t =>
{
ScoreObjectId = SendScore.ObjectId;
});
ParseObject SendCorrectTopics = new ParseObject("CorrectAnswers");
SendCorrectTopics["Score"] = SendScore.ObjectId;
for (int i = 0; i <= 9; i++)
{
string Topic = "Topic" + (i + 1).ToString();
SendCorrectTopics[Topic] = CheckAnswer.CorrectTopics[i];
}
SendCorrectTopics.SaveAsync();
SceneManager.LoadScene(0);
}
How would I be able to make the second save hold until the first save has finished? I'm somewhat new to C# and so don't quite know all it's features yet. I've looked into "await" but unity doesn't seem to like that. Any help would be greatly appreciated!
Thanks in advance,
EDIT: Okay, after a bit more reading on Unity's coroutines, I found a much better way of checking that only relies on checking when needed:
IEnumerator CheckSave()
{
while(ScoreObjectId == null & !DoneSave))
{
print("Running");
yield return new WaitForSeconds(0.5f);
}
DoneSave = false;
SaveTotalTopics();
}
This seems like a much better way of doing it.
Well, it seems the answer was something I've already done before, even if it is a little ugly.
Using Unity's update function I created a check to make sure the ObjectID is not null and the previous save had completed, as so:
void Update () {
if (ScoreObjectId != null & DoneSave)
{
DoneSave = false;
SaveTotalTopics();
}
Thus splitting it into two saves and creating:
public void SaveScore()
{
ParseObject SendScore = new ParseObject("Scores");
SendScore["Score"] = CheckAnswer.score;
SendScore["user"] = ParseObject.CreateWithoutData("_User", ParseUser.CurrentUser.ObjectId);
SendScore["TestMode"] = MainMenu.testmode;
SendScore["TotalQuestions"] = QuestionCreation.TotalQuestions;
SendScore["CorrectQuestions"] = CheckAnswer.CorrectQuestions;
Task SendingScores = SendScore.SaveAsync().ContinueWith(t =>
{
if (t.IsFaulted || t.IsCanceled)
{
DoneSave = false;
print(t.Exception);
}
else
{
DoneSave = true;
print("Setting object ID!");
ScoreObjectId = SendScore.ObjectId;
print(ScoreObjectId);
}
});
}
void SaveTotalTopics()
{
for (int i = 0; i <= 9; i++)
{
string Topic = "Topic" + (i + 1).ToString();
SendCorrectTopics[Topic] = CheckAnswer.CorrectTopics[i];
}
SendCorrectTopics["UserScore"] = ParseObject.CreateWithoutData("Scores", ScoreObjectId);
SendCorrectTopics.SaveAsync().ContinueWith(t =>
{
if(t.IsFaulted || t.IsCanceled)
{
print(t.Exception);
}
else
{
print("Saved!");
}
});
}
I'd also forgotten to use ParseObject.CreateWithoutData() so my first code snippet wouldn't have worked even if I'd found a better method...
So, although I'm not happy with the final result, at least it works and I don't think running an if statement every frame should significantly impact on my game's performance.
Why not use a bool and a while loop?
public IEnumerator SaveScore()
{
bool canContinue = false;
ParseObject SendScore = new ParseObject("Scores");
SendScore["Score"] = CheckAnswer.score;
SendScore["user"] = ParseObject.CreateWithoutData("_User", ParseUser.CurrentUser.ObjectId);
SendScore["TestMode"] = MainMenu.testmode;
SendScore["TotalQuestions"] = QuestionCreation.TotalQuestions;
SendScore["CorrectQuestions"] = CheckAnswer.CorrectQuestions;
SendScore.SaveAsync().ContinueWith(t =>
{
ScoreObjectId = SendScore.ObjectId;
//set the bool canContinue to true because the first portion of code has finished running
canContinue = true;
});
//wait while the canContinue bool is false
while(!canContinue){
yield return null;
}
//continue your parse code
ParseObject SendCorrectTopics = new ParseObject("CorrectAnswers");
SendCorrectTopics["Score"] = SendScore.ObjectId;
for (int i = 0; i <= 9; i++)
{
string Topic = "Topic" + (i + 1).ToString();
SendCorrectTopics[Topic] = CheckAnswer.CorrectTopics[i];
}
SendCorrectTopics.SaveAsync();
SceneManager.LoadScene(0);
return null;
}
I have done the paging implementation using the following:
.Find(_ => true).Skip(PageSize * (int)(PageNumber - 1)).Limit(PageSize).ToListAsync().Result;
and inside of the paging code I have called DeleteOneAsync( with _id filter), I have over 5,00,000 records and the paging working fine, just that the delete api doesn't delete all the records as expected. My pseudo-code is as follows:
while(true)
{
var page = GetPage(pageIdx++); //starts with 1
if(page.Count == 0)
break;
foreach(var p in page)
{
Delete(p);
}
}
There is no error raised anywhere, all the processing runs fine, but at the ends I expect all records to be deleted but I see that only a few chunk is deleted. Any clue why this happens or if there is a issue in paging part of mine?
Sample code:
public static class ConsoleProgram
{
const int PAGE_SIZE = 100;
public static void Main(string[] args)
{
MongoClientSettings clientSettings = new MongoClientSettings();
clientSettings.Server = new MongoServerAddress("localhost", 27017);
MongoDB.Driver.MongoClient client = new MongoDB.Driver.MongoClient(clientSettings);
IMongoDatabase db = client.GetDatabase("petrel");
IMongoCollection<BsonDocument> mt = db.GetCollection<BsonDocument>("PatientDocuments");
int pageNo = 1;
int count = 0;
while (true)
{
IEnumerable<MongoDB.Bson.BsonDocument> page = null;
if(pageNo == 1)
page = mt.Find(_ => true).Limit(PAGE_SIZE).ToListAsync().Result;
else
page = mt.Find(_ => true).Skip(PAGE_SIZE * (pageNo -1)).Limit(PAGE_SIZE).ToListAsync().Result;
if(page.Count() == 0)
break;
foreach (var p in page)
{
ObjectId id = (ObjectId)p["_id"];
DeleteResult dr = mt.DeleteOneAsync(Builders<BsonDocument>.Filter.Eq("_id", id)).Result;
if (dr.IsAcknowledged)
{
if (dr.DeletedCount != 1)
{
throw new Exception(string.Format("Count [value:{0}] after delete is invalid", dr.DeletedCount));
}
}
else
throw new Exception(string.Format("Delete for [_id:{0}] was not acknowledged", id.ToString()));
}
count += page.Count();
}
Console.WriteLine("Done, count:{0}", count);
Console.ReadLine();
}
The cursor is not isolated so it recognizes that you've deleted some data and when you pull the next page you are skipping records you intend to delete. If you pulled page 1 each time it would work like what it seems you want it to.
I am looking for a collection object that calls a method by it self after the count reaches 1000. Here in the example when the count of element reach 1000 i want to process them.
Example:
internal static Collection<string> elements=new collection<string>
public void someMethod()
{
XDocument XD = null;
XD = XDocument.Load("https://somewebservice.asp");
var Value = XD.Descendants().where(x => x == "someAttribute").FirstorDefault();
if(value != null)
{
elements.Add(value);
}
if(elemets.count() == 1000)
{
// Do Something
// But i dont want to check the count every time
// I want the collection object to do the thing.. is there any method
// that can do this?
}
}
You can use an ObservableCollection which fires an event when the colleciton is changed. You can check the count at this time and do something.
var collection = new ObservableCollection<int>();
collection.CollectionChanged += (s, args) =>
{
if (collection.Count == 1000)
DoSomething();
};