We Keep Coding

Restart concurrent tasks as soon as they fail for x number of times

I have a console app that is making HTTP queries and adding/updating products in my database according to response. Some fail and need to be retried a few times.
The way I came up with was to use a dictionary to store the product ID and a Task. Then I can check all the task results and re-run.
This is working but it strikes me as inefficient. Tasks are not being re-created until all tasks have finished. It would be more efficient if they were immediately restarted but I can't figure out how to do this. Also every retry involves a query to the database as only the ID is stored.
I made small app that shows how I am currently retrying failed requests.
Can someone suggest a more efficient method for retrying?
class Program
{
private static void Main(string[] args)
{
HttpQuery m = new HttpQuery();
var task = Task.Run(() => m.Start());
Task.WaitAll(task);
Console.WriteLine("Finished");
Console.ReadLine();
}
}
class HttpQuery
{
public async Task Start()
{
// dictionary where key represent reference to something that needs to be processed and bool whether it has completed or not
ConcurrentDictionary<int, Task<bool>> monitor = new ConcurrentDictionary<int, Task<bool>>();
// start async tasks.
Console.WriteLine("starting first try");
for (int i = 0; i < 1000; i++)
{
Console.Write(i+",");
monitor[i] = this.Query(i);
}
// wait for completion
await Task.WhenAll(monitor.Values.ToArray());
Console.WriteLine();
// start retries
// number of retries per query
int retries = 10;
int count = 0;
// check if max retries exceeded or all completed
while (count < retries && monitor.Any(x => x.Value.Result == false))
{
// make list of numbers that failed
List<int> retryList = monitor.Where(x => x.Value.Result == false).Select(x => x.Key).ToList();
Console.WriteLine("starting try number: " + (count+1) + ", Processing: " + retryList.Count);
// create list of tasks to wait for
List<Task<bool>> toWait = new List<Task<bool>>();
foreach (var i in retryList)
{
Console.Write(i + ",");
monitor[i] = this.Query(i);
toWait.Add(monitor[i]);
}
// wait for completion
await Task.WhenAll(toWait.ToArray());
Console.WriteLine();
count++;
}
Console.WriteLine("ended");
Console.ReadLine();
}
public async Task<bool> Query(int i)
{
// simulate a http request that may or may not fail
Random r = new Random();
int delay = i * r.Next(1, 10);
await Task.Delay(delay);
if (r.Next(0,2) == 1)
{
return true;
}
else
{
return false;
}
}
}

You can create another method and wrap all these ugly retry logic. All of that ugly code goes away :)
public async Task Start()
{
const int MaxNumberOfTries = 10;
List<Task<bool>> tasks = new List<Task<bool>>();
for (int i = 0; i < 1000; i++)
{
tasks.Add(this.QueryWithRetry(i, MaxNumberOfTries));
}
await Task.WhenAll(tasks);
}
public async Task<bool> QueryWithRetry(int i, int numOfTries)
{
int tries = 0;
bool result;
do
{
result = await Query(i);
tries++;
} while (!result && tries < numOfTries);
return result;
}

c# 2 events with same name， got chaos at running time, how should i avoid this?

I have a method "Add2List", who creates a ManualResetEvent and stores it in SortedList of a instance, then waits for signaling, then do some work and dispose the event.
I have another method "DoSomething", who listens to remote server and then signals the stored manual events according to Guid.
in the multithreading context, multi threads calls method "Add2List", so in the sortedlist there may have several manual event with same name at the same moment. But this may cause chaos. How should i avoid this?
To be simpler, i wrote this test code:
Class Program
{
static void Main(string[] args)
{
StringBuilder str = new StringBuilder();//a string to record what happened
//test iteratively
for(int i=0;i<100;i++)
{
EventHolder holder = new EventHolder();
Signaler ob2 = new Signaler();
Thread th1 = new Thread(holder.Add2List);
Thread th2 = new Thread(holder.Add2List);
Thread th3 = new Thread(ob2.DoSomething);
th1.Start(1);
th2.Start(2);
th3.Start();
//Make sure all thread is ended before the next iteration.
while(th1.IsAlive){ Thread.Sleep(200); }
while(th2.IsAlive){ Thread.Sleep(200); }
while(th3.IsAlive){ Thread.Sleep(200); }
}
Console.Read();
}
public class EventHolder
{
static SortedList<int, ManualResetEvent> MyManualEventList = new SortedList<int, ManualResetEvent>();
public EventHolder()
{
MyManualEventList = new SortedList<int, ManualResetEvent>();
Signaler.SignalMyManualEvent += OnSignalMyManualEvent;
}
void OnSignalMyManualEvent(int listindex)
{
try { MyManualEventList[listindex].Set(); }
catch(Exception e)
{
Console.WriteLine("Exception throws at " + System.DateTime.Now.ToString() +" Exception Message:"
Console.WriteLine(e.Message);
int temp = 0; //*Here is a breakpoint! To watch local variables when exception happens.
}
}
public void Add2List(object listindex)
{
ManualResetEvent MyManualEvent = new ManualResetEvent(false);
MyManualEvent.Reset();
MyManualEventList.Add((int)listindex, eve);
//in this test, this countdownevent need to be signaled twice, for it has to wait until all 2 event been added to MyManualEventList
Signaler.StartTrySignal.Signal();
MyManualEvent.WaitOne();
Console.WriteLine("Event" + ((int)listindex).ToString() + " been detected at " + System.DateTime.Now.Tostring());
MyManualEvent.Dispose();
}
}
public class Signaler
{
public delegate void Signalhandler(int listindex);
public static event Signalhandler SignalMyManualEvent;
public static CountDownEvent StartTrySignal = new CountDownEvent(2); // signaled twice so that the 2 manual events were added to sortedlist
void RaiseSignalMyManualEvent(int listindex)
{
var vr = SignalMyManualEvent;
if(vr != null)
vr(listindex);
}
int i = 0, j = 0, k = 0;
// here i use 2 prime numbers to simulate the happening of 2 random events
public Signaler()
{
StartTrySignal.Reset();
}
public void DoSomething()
{
StartTrySignal.Wait(); // wait for the 2 manual events been added to sortedlist
//To signal MyManualEventList[1] or MyManualEventList[2]
while(i + j == 0)
{
Random rnd = new Random();
k = rnd.Next();
if(k % 613 == 0) { i = 1; Console.WriteLine("Event1 Raised!"; RaiseSignalMyManualEvent(1); }
else if(k % 617 == 0) { j = 1; Console.WriteLine("Event1 Raised!"; RaiseSignalMyManualEvent(2); }
}
//if MyManualEventList[1] has not been signaled, wait something to happen, and signal it.
while(i == 0)
{
Random rnd = new Random();
k = rnd.Next();
if(k % 613 == 0)
{
i = 1;
if(j>0)
{
m++;
Console.WriteLine("All 2 Events Raised! - iteration " + m.ToString());
}
RaiseSignalMyManualEvent(1);
}
}
//if MyManualEventList[2] has not been signaled, wait something to happen, and signal it.
while(j == 0)
{
Random rnd = new Random();
k = rnd.Next();
if(k % 617 == 0)
{
j = 1;
m++;
Console.WriteLine("All 2 Events Raised! - iteration " + m.ToString());
RaiseSignalMyManualEvent(2);
}
}
}
}
public class Counter //Provide a number to record iteration
{
public static int m = 0;
}
}
Result:
Sorry for do not have enough reputation to post images.
At the line where there's a breakpoint, system throws exception " the given key is not in dictionary". this exception happens randomly, sometimes because th1 disposed <2, MyManualEvent> or th2 disposed <1, MyManualEvent> , sometimes none has been disposed but it just cannot find anyone.
I run this program 3 times, exception happens at iteration12, iteration45, and iteration0 (at the beginning).

OK 2 answers
1: Your code returns "Event 1" after "All events", because the two console.writelines are in a race condition (the last while loop is never iterated)
2: the 'System' distingushes between the two ManualResetEvent objects becuase it references the SortedList you put them in. ie.
static SortedList<int, ManualResetEvent> MyManualEventList
= new SortedList<int, ManualResetEvent>();
public EventHolder() { Signaler.SignalMyManualEvent
+= OnSignalMyManualEvent; }
void OnSignalMyManualEvent(int listindex)
{
MyManualEventList[listindex].Set();
}
when you raise event 1 you call set on Item 1 in the SortedList and when you raise Event 2 you call set on Item 2 in the list.
this is bad because the calling code has no idea which thread it is allowing to continue and you could well get a null exception

Multithread access to list C#

I have a task to show difference between syncronized and unsyncronized multithreading. Therefore I wrote an application simulating withdrawing money from clients' bank accounts. Each of some number of threads chooses a random user and withdraws money from the account.
Every thread should withdraw every account once. First time the threads are syncronized, but the second time they are not. So there must be a difference between accounts, withdrawed by syncronized and unsyncronized threads. And the difference must be different for different numbers of users and threads. But in my application I have difference just for 1000 threads. So I need unsyncronized threads' results to be strongly different from syncronized threads' ones.
The class User:
public class User : IComparable
{
public string Name { get; set; }
public int Start { get; set; }
public int FinishSync { get; set; }
public int FinishUnsync { get; set; }
public int Hypothetic { get; set; }
public int Differrence { get; set; }
...
}
The method which withdraws money:
public void Withdraw(ref List<User> users, int sum, bool isSync)
{
int ind = 0;
Thread.Sleep(_due);
var rnd = new Random(DateTime.Now.Millisecond);
//used is list of users, withrawed by the thread
while (_used.Count < users.Count)
{
while (_used.Contains(ind = rnd.Next(0, users.Count))) ; //choosing a random user
if (isSync) //isSync = if threads syncroized
{
if (Monitor.TryEnter(users[ind]))
{
try
{
users[ind].FinishSync = users[ind].FinishSync - sum;
}
finally
{
Monitor.Exit(users[ind]);
}
}
}
else
{
lock (users[ind])
{
users[ind].FinishUnsync = users[ind].FinishUnsync - sum;
}
}
_used.Add(ind);
}
done = true;
}
And the threads are created this way:
private void Withdrawing(bool IsSync)
{
if (IsSync)
{
for (int i = 0; i < _num; i++)
{
_withdrawers.Add(new Withdrawer(Users.Count, _due, _pause));
_threads.Add(new Thread(delegate()
{ _withdrawers[i].Withdraw(ref Users, _sum, true); }));
_threads[i].Name = i.ToString();
_threads[i].Start();
_threads[i].Join();
}
}
else
{
for (int i = 0; i < _num; ++i)
{
_withdrawers.Add(new Withdrawer(Users.Count, _due, _pause));
_threads.Add(new Thread(delegate()
{ _withdrawers[i].Withdraw(ref Users, _sum, false); }));
_threads[i].Name = i.ToString();
_threads[i].Start();
}
}
}
I've changed the Withdraw class this way, bc the problem could have been in creating threads separately from the delegate:
class Withdrawer
{
private List<int>[] _used;
private int _due;
private int _pause;
public int done;
private List<Thread> _threads;
public Withdrawer(List<User> users, int n, int due, int pause, int sum)
{
_due = due;
_pause = pause;
done = 0;
_threads = new List<Thread>(users.Count);
InitializeUsed(users, n);
CreateThreads(users, n, sum, false);
_threads.Clear();
while (done < n) ;
Array.Clear(_used,0,n-1);
InitializeUsed(users, n);
CreateThreads(users, n, sum, true);
}
private void InitializeUsed(List<User> users, int n)
{
_used = new List<int>[n];
for (int i = 0; i < n; i++)
{
_used[i] = new List<int>(users.Count);
for (int j = 0; j < users.Count; j++)
{
_used[i].Add(j);
}
}
}
private void CreateThreads(List<User> users, int n, int sum, bool isSync)
{
for (int i = 0; i < n; i++)
{
_threads.Add(new Thread(delegate() { Withdraw(users, sum, isSync); }));
_threads[i].Name = i.ToString();
_threads[i].Start();
}
}
public void Withdraw(List<User> users, int sum, bool isSync)
{
int ind = 0;
var rnd = new Random();
while (_used[int.Parse(Thread.CurrentThread.Name)].Count > 0)
{
int x = rnd.Next(_used[int.Parse(Thread.CurrentThread.Name)].Count);
ind = _used[int.Parse(Thread.CurrentThread.Name)][x];
if (isSync)
{
lock (users[ind])
{
Thread.Sleep(_due);
users[ind].FinishSync -= sum;
}
}
else
{
Thread.Sleep(_due);
users[ind].FinishUnsync -= sum;
}
_used[int.Parse(Thread.CurrentThread.Name)][x] = _used[int.Parse(Thread.CurrentThread.Name)][_used[int.Parse(Thread.CurrentThread.Name)].Count - 1];
_used[int.Parse(Thread.CurrentThread.Name)].RemoveAt(_used[int.Parse(Thread.CurrentThread.Name)].Count - 1);
Thread.Sleep(_pause);
}
done++;
}
}
Now the problem is FinishUnSync values are correct, while FinishSync values are absolutely not.
Thread.Sleep(_due);
and
Thread.Sleep(_pause);
are used to "hold" the resourse, bc my task is the thread should get resourse, hold it for a _due ms, and after processing wait _pause ms before finishing.

Your code isn't doing anything useful, and doesn't show the difference between synchronized and unsynchronized access. There are many things you'll need to address.
Comments in your code say that _used is a list of users that have been accessed by the thread. You're apparently creating that on a per-thread basis. If that's true, I don't see how. From the looks of things I'd say that _used is accessible to all threads. I don't see anywhere that you're creating a per-thread version of that list. And the naming convention indicates that it's at class scope.
If that list is not per-thread, that would go a long way towards explaining why your data is always the same. You also have a real race condition here because you're updating the list from multiple threads.
Assuning that _used really is a per-thread data structure . . .
You have this code:
if (isSync) //isSync = if threads syncroized
{
if (Monitor.TryEnter(users[ind]))
{
try
{
users[ind].FinishSync = users[ind].FinishSync - sum;
}
finally
{
Monitor.Exit(users[ind]);
}
}
}
else
{
lock (users[ind])
{
users[ind].FinishUnsync = users[ind].FinishUnsync - sum;
}
}
Both of these provide synchronization. In the isSync case, a second thread will fail to do its update if a thread already has the user locked. In the second case, the second thread will wait for the first to finish, and then will do the update. In either case, the use of Monitor or lock prevents concurrent update.
Still, you would potentially see a difference if multiple threads could be executing the isSync code at the same time. But you won't see a difference because in your synchronized case you never let more than one thread execute. That is, you have:
if (IsSync)
{
for (int i = 0; i < _num; i++)
{
_withdrawers.Add(new Withdrawer(Users.Count, _due, _pause));
_threads.Add(new Thread(delegate()
{ _withdrawers[i].Withdraw(ref Users, _sum, true); }));
_threads[i].Name = i.ToString();
_threads[i].Start();
_threads[i].Join();
}
}
else
{
for (int i = 0; i < _num; ++i)
{
_withdrawers.Add(new Withdrawer(Users.Count, _due, _pause));
_threads.Add(new Thread(delegate()
{ _withdrawers[i].Withdraw(ref Users, _sum, false); }));
_threads[i].Name = i.ToString();
_threads[i].Start();
}
}
So in the IsSync case, you start a thread and then wait for it to complete before you start another thread. Your code is not multithreaded. And in the "unsynchronized" case you're using a lock to prevent concurrent updates. So in one case you prevent concurrent updates by only running one thread at a time, and in the other case you prevent concurrent updates by using a lock. There will be no difference.
Something else worth noting is that your method of randomly selecting a user is highly inefficient, and could be part of the problem you're seeing. Basically what you're doing is picking a random number and checking to see if it's in a list. If it is, you try again, etc. And the list keeps growing. Quick experimentation shows that I have to generate 7,000 random numbers between 0 and 1,000 before I get all of them. So your threads are spending a huge amount of time trying to find the next unused account, meaning that they have less likelihood to be processing the same user account at the same time.
You need to do three things. First, change your Withdrawl method so it does this:
if (isSync) //isSync = if threads syncroized
{
// synchronized. prevent concurrent updates.
lock (users[ind])
{
users[ind].FinishSync = users[ind].FinishSync - sum;
}
}
else
{
// unsynchronized. It's a free-for-all.
users[ind].FinishUnsync = users[ind].FinishUnsync - sum;
}
Your Withdrawing method should be the same regardless of whether IsSync is true or not. That is, it should be:
for (int i = 0; i < _num; ++i)
{
_withdrawers.Add(new Withdrawer(Users.Count, _due, _pause));
_threads.Add(new Thread(delegate()
{ _withdrawers[i].Withdraw(ref Users, _sum, false); }));
_threads[i].Name = i.ToString();
_threads[i].Start();
}
Now you always have multiple threads running. The only difference is whether access to the user account is synchronized.
Finally, make your _used list a list of indexes into the users list. Something like:
_used = new List<int>(users.Count);
for (int i = 0; i < _used.Count; ++i)
{
_used[i] = i;
}
Now, when you select a user, you do this:
var x = rnd.Next(_used.Count);
ind = _used[x];
// now remove the item from _used
_used[x] = _used[_used.Count-1];
_used.RemoveAt(_used.Count-1);
That way you can generate all users more efficiently. It will take n random numbers to generate n users.
A couple of nitpicks:
I have no idea why you have the Thread.Sleep call in the Withdraw method. What benefit do you think it provides?
There's no real reason to pass DateTime.Now.Millisecond to the Random constructor. Just calling new Random() will use Environment.TickCount for the seed. Unless you really want to limit the seed to numbers between 0 and 1,000.

Thread program, 1st and last loop never has a ThreadedState other than 'Running'

I have a programming that is looping x times (10), and using a specified number of threads (2). I'm using a thread array:
Thread[] myThreadArray = new Thread[2];
My loop counter, I believe, starts the first 2 threads just fine, but when it gets to loop 3, which goes back to thread 0 (zero-based), it hangs. The weird thing is, if I throw a MessageBox.Show() in their to check the ThreadState (which shows thread 0 is still running), it will continue on through 9 of the 10 loops. But if no MessageBox.Show() is there, it hangs when starting the 3rd loop.
I'm using .NET 3.5 Framework (I noticed that .NET 4.0 utilizes something called continuations...)
Here's some code examples:
Thread[] threads = new Thread[2];
int threadCounter = 0;
for (int counter = 0; counter < 10; counter++)
{
if (chkUseThreading.Checked)
{
TestRunResult runResult = new TestRunResult(counter + 1);
TestInfo tInfo = new TestInfo(conn, comm, runResult);
if (threads[threadCounter] != null)
{
// If this is here, then it will continue looping....otherwise, it hangs on the 3rd loop
MessageBox.Show(threads[threadCounter].ThreadState.ToString());
while (threads[threadCounter].IsAlive || threads[threadCounter].ThreadState == ThreadState.Running)
Thread.Sleep(1);
threads[threadCounter] = null;
}
// ExecuteTest is a non-static method
threads[threadCounter] = new Thread(new ThreadStart(delegate { ExecuteTest(tInfo); }));
threads[threadCounter].Name = "PerformanceTest" + (counter + 1);
try
{
threads[threadCounter].Start();
if ((threadCounter + 1) == threadCount)
threadCounter = 0;
else
threadCounter++;
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
Application.DoEvents();
}
}
while (true)
{
int threadsFinished = 0;
for (int counter = 0; counter < threadCount; counter++)
{
if (!threads[counter].IsAlive || threads[counter].ThreadState == ThreadState.Stopped)
threadsFinished++;
}
if (threadsFinished == threadCount)
break;
else
Thread.Sleep(1);
}
Obviously the problem is something about how I'm checking to see if thread #1 or #2 is done. The IsAlive always says true, and the ThreadState always has "running" for threads loops 1 and 10.
Where am I going wrong with this?
Update, here's the ExecuteTask() method:
private void ExecuteTest(object tInfo)
{
TestInfo testInfo = tInfo as TestInfo;
Exception error = null;
DateTime endTime;
TimeSpan duration;
DateTime startTime = DateTime.Now;
try
{
if (testInfo.Connection.State != ConnectionState.Open)
{
testInfo.Connection.ConnectionString = connString;
testInfo.Connection.Open();
}
testInfo.Command.ExecuteScalar();
}
catch (Exception ex)
{
error = ex;
failedCounter++;
//if (chkCancelOnError.Checked)
// break;
}
finally
{
endTime = DateTime.Now;
duration = endTime - startTime;
RunTimes.Add(duration);
testInfo.Result.StartTime = startTime;
testInfo.Result.EndTime = endTime;
testInfo.Result.Duration = duration;
testInfo.Result.Error = error;
TestResults.Add(testInfo.Result);
// This part must be threadsafe...
if (lvResults.InvokeRequired)
{
SetTextCallback d = new SetTextCallback(ExecuteTest);
this.Invoke(d, new object[] { tInfo });
}
else
{
lvResults.Items.Add(testInfo.Result.ConvertToListViewItem());
#region Update Results - This wouldn't work in it's own method in the threaded version
const string msPrefix = "ms";
// ShortestRun
TimeSpan shortest = GetShortestRun(RunTimes);
tbShortestRun.Text = shortest.TotalMilliseconds + msPrefix;
// AverageRun
TimeSpan average = GetAverageRun(RunTimes);
tbAverageRun.Text = average.TotalMilliseconds + msPrefix;
// MeanRun
TimeSpan mean = GetMeanRun(RunTimes);
tbMeanRun.Text = mean.TotalMilliseconds + msPrefix;
// LongestRun
TimeSpan longest = GetLongestRun(RunTimes);
tbLongestRun.Text = longest.TotalMilliseconds + msPrefix;
// ErrorCount
int errorCount = GetErrorCount(TestResults);
tbErrorCount.Text = errorCount.ToString();
#endregion
}
testInfo.Command.Dispose();
Application.DoEvents();
}
}

Can you post a snippet of run ()? Doesn't Thread.currentThread().notifyAll() help? May be each thread is waiting for other thread to do something resulting in a deadlock?

For loop in many threads

How can I run each call for loop in another thread, but continuation of ExternalMethod should wait to ending of last working thread from for loop (and synchronize) ?
ExternalMethod()
{
//some calculations
for (int i = 0; i < 10; i++)
{
SomeMethod(i);
}
//continuation ExternalMethod
}

One approach would be to use a ManualResetEvent.
Consider the following code (note that this should not be taken as a working example, stuck on OSX so don't have VS nor a C# compiler to hand to check this over):
static ManualResetEvent mre = new ManualResetEvent(false);
static int DoneCount = 0;
static int DoneRequired = 9;
void ExternalMethod() {
mre.Reset();
for (int i = 0; i < 10; i++) {
new Thread(new ThreadStart(ThreadVoid)).Start();
}
mre.WaitOne();
}
void ThreadVoid() {
Interlocked.Increment(ref DoneCount);
if (DoneCount == DoneRequired) {
mre.Set();
}
}
IMPORTANT - This possibly isn't the best way to do it, just an example of using ManualResetEvent, and it will suit your needs perfectly fine.
If you're on .NET 4.0 you can use a Parallel.For loop - explained here.

System.Threading.Tasks.Parallel.For(0, 10, (i) => SomeMethod(i));

One approach is to use a CountdownEvent.
ExternalMethod()
{
//some calculations
var finished = new CountdownEvent(1);
for (int i = 0; i < 10; i++)
{
int capture = i; // This is needed to capture the loop variable correctly.
finished.AddCount();
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
SomeMethod(capture);
}
finally
{
finished.Signal();
}
}, null);
}
finished.Signal();
finished.Wait();
//continuation ExternalMethod
}
If CountdownEvent is not available then here is an alternate approach.
ExternalMethod()
{
//some calculations
var finished = new ManualResetEvent(false);
int pending = 1;
for (int i = 0; i < 10; i++)
{
int capture = i; // This is needed to capture the loop variable correctly.
Interlocked.Increment(ref pending);
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
SomeMethod(capture);
}
finally
{
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
}
}, null);
}
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
finished.WaitOne();
//continuation ExternalMethod
}
Note that in both examples the for loop itself is treating as a parallel work item (it is on a separate thread from the other work items afterall) to avoid a really subtle race condition that might occur if the first work item signals the event before the next work item is queued.

For .NET 3.5, maybe something like this:
Thread[] threads = new Thread[10];
for (int x = 0; x < 10; x++)
{
threads[x] = new Thread(new ParameterizedThreadStart(ThreadFun));
threads[x].Start(x);
}
foreach (Thread thread in threads) thread.Join();
It may seem counterintuitive to use the Join() method, but since you are effectively doing a WaitAll-type pattern, it doesn't matter what order the joins are executed.