Help is very welcome and extremely appreciated, thank you. What this program is, is a ProxyChecker, because i've bought a bunch and will continue to do so of proxies with different user/passes etc, however some have expired. I added a break point and what it's doing is actually skipping the ProxyClient code and going straight to for each var item in 1, if item accepts connection etc, it then just returns false and finishes.
private static void CheckProxy(object state)
{
var u = user[0];
var p = pass[0];
var l = new List<MyIP>();
Parallel.ForEach(l.ToArray(), (ip_item) =>
{
try
{
string ip = ip_item.IP;
using (var client = new ProxyClient(ip, u, p))
{
Console.WriteLine(ip, user, pass);
client.Connect();
ip_item.AcceptsConnection = client.IsConnected;
}
}
catch
{
l.Remove(ip_item);
}
});
foreach (var item in l)
{
if (item.AcceptsConnection == true)
{
WriteToFile(user[0], pass[0]);
}
Console.WriteLine(item.IP + " is " + (item.AcceptsConnection) + " accepts connections" + " doesn not accept connections");
}
}
Load ips function:#
private static void loadips()
{
using (TextReader tr = new StreamReader("ips.txt"))
{
var l = new List<MyIP>();
string line = null;
while ((line = tr.ReadLine()) != null)
{
l.Add(new MyIP { IP = line });
}
}
}
I have added this in response to the answer. I believe this is a variable issue as the variable is locally declared not publicly any ideas how to fix? i'm unable to find a way to get this working seems like i'm being dumb. thanks.
The problem is in these two lines:
var l = new List<MyIP>();
Parallel.ForEach(l.ToArray(), (ip_item) =>
You just created l as a new List with no items in it. Calling ToArray() will give you an empty array. When Parallel.ForEach sees an empty array, it just gets skipped since there's nothing to iterate over.
Related
I'm trying to remove all Lookup Tables that start with a specific prefix inside all families.
The method "sizeTableManager.RemoveSizeTable(tableToRemove)" returns true as if it succeeded but when I go edit the families in the project and bring up the Lookup Tables list they are still there.
The transaction seems to be committing with no errors too, which is even more puzzling...
Any ideas as to what I'm doing wrong?
This is my code so far:
string existingPrefix = "ExistingPrefix_";
Document doc = this.ActiveUIDocument.Document;
FilteredElementCollector collector = new FilteredElementCollector(doc);
ICollection<Element> elements = collector.OfClass(typeof(Family)).ToElements();
foreach (var element in elements)
{
using (Transaction t = new Transaction(doc, "flush old lookup tables"))
{
t.Start();
FamilySizeTableManager sizeTableManager = FamilySizeTableManager.GetFamilySizeTableManager(doc, element.Id);
if(sizeTableManager != null)
{
foreach (var tableToRemove in sizeTableManager.GetAllSizeTableNames())
{
if(tableToRemove.StartsWith(existingPrefix))
{
bool result = sizeTableManager.RemoveSizeTable(tableToRemove);
if(result)
{
// TaskDialog.Show("Success", "Removed " + tableToRemove + " from " + element.Name);
var test = "test";
}
else
{
TaskDialog.Show("Warning", "Unable to remove " + tableToRemove + " from " + element.Name);
}
}
}
}
var commitResult = t.Commit();
}
}
Thanks in advance!
Well, turns out that the RemoveSizeTable() method was indeed working, but that I had to regenerate the document for the changes to take effect:
using (Transaction t = new Transaction(doc, "regenerate document"))
{
t.Start();
doc.Regenerate();
t.Commit();
}
Discovered this after noticing that saving the document caused the changes to take effect.
I am trying to parse a 300MB csv file and save it on mongodb. In order to do that I will need to convert this csv file into a list of BsonDocument which include key value pairs which create a document. each row in the csv file is a new BsonDocument.
Every couple of minutes of parallel testing, I am getting OOM exception on the split operation.
I've read this article which is very interesting. but I couldn't find any practical solution which I can implement on those huge files.
I was looking into different csv helpers, but couldn't find anything which solve this issue.
Any help will be much appreciated.
You should be able to read it line by line like this:
public static void Main()
{
using (StreamReader sr = new StreamReader(path))
{
string[] headers = null;
string[] curLine;
while ((curLine = sr.ReadLine().Split(',')) != null)
{
if (firstLine == null)
{
headers = curLine;
}
else
{
processLine(headers, curLine);
}
}
}
}
public static void processLine(string[] headers, string[] line)
{
for (int i = 0; i < headers.Length)
{
string header = headers[i];
string line = line[i];
//Now you have individual header/line pairs that you can put into mongodb
}
}
I've never used mongodb and I don't know the structure of your csv or your mongo, so I won't be able to help much there. Hopefully you can get it from here though. If not, edit your post with some more details about how you need to structure your mongodb and hopefully somebody will post a more helpful answer.
Thank you #dbc That worked!
#ashbygeek, I needed to add this to your code,
while (!sr.EndOfStream && (curLine = sr.ReadLine().Split('\t')) != null)
{
//do process
}
So I am uploading my code which I get my big CSV file from Azure blob, and insert in Batch to mongoDB instead of each document.
I also created my own primary key hash, and index, in order to identify duplicates documents, and if I found one, I'll start insert them one by one in order to identify the duplicate.
I hope it will help for someone in the future.
using (TextFieldParser parser = new TextFieldParser(blockBlob2.OpenRead()))
{
parser.TextFieldType = FieldType.Delimited;
parser.SetDelimiters("\t");
bool headerWritten = false;
List<BsonDocument> listToInsert = new List<BsonDocument>();
int chunkSize = 50;
int counter = 0;
var headers = new string[0];
while (!parser.EndOfData)
{
//Processing row
var fields = parser.ReadFields();
if (!headerWritten)
{
headers = fields;
headerWritten = true;
continue;
}
listToInsert.Add(new BsonDocument(headers.Zip(fields, (k, v) => new { k, v }).ToDictionary(x => x.k, x => x.v)));
counter++;
if (counter != chunkSize) continue;
AdditionalInformation(listToInsert, dataCollectionQueueMessage);
CalculateHashForPrimaryKey(listToInsert);
await InsertDataIntoDB(listToInsert, dataCollectionQueueMessage);
counter = 0;
listToInsert.Clear();
}
if (listToInsert.Count > 0)
{
AdditionalInformation(listToInsert, dataCollectionQueueMessage);
CalculateHashForPrimaryKey(listToInsert);
await InsertDataIntoDB(listToInsert, dataCollectionQueueMessage);
}
}
private async Task InsertDataIntoDB(List<BsonDocument>listToInsert, DataCollectionQueueMessage dataCollectionQueueMessage)
{
const string connectionString = "mongodb://127.0.0.1/localdb";
var client = new MongoClient(connectionString);
_database = client.GetDatabase("localdb");
var collection = _database.GetCollection<BsonDocument>(dataCollectionQueueMessage.CollectionTypeEnum.ToString());
await collection.Indexes.CreateOneAsync(new BsonDocument("HashMultipleKey", 1), new CreateIndexOptions() { Unique = true, Sparse = true, });
try
{
await collection.InsertManyAsync(listToInsert);
}
catch (Exception ex)
{
ApplicationInsights.Instance.TrackException(ex);
await InsertSingleDocuments(listToInsert, collection, dataCollectionQueueMessage);
}
}
private async Task InsertSingleDocuments(List<BsonDocument> dataCollectionDict, IMongoCollection<BsonDocument> collection
,DataCollectionQueueMessage dataCollectionQueueMessage)
{
ApplicationInsights.Instance.TrackEvent("About to start insert individual documents and to find the duplicate one");
foreach (var data in dataCollectionDict)
{
try
{
await collection.InsertOneAsync(data);
}
catch (Exception ex)
{
ApplicationInsights.Instance.TrackException(ex,new Dictionary<string, string>() {
{
"Error Message","Duplicate document was detected, therefore ignoring this document and continuing to insert the next docuemnt"
}, {
"FilePath",dataCollectionQueueMessage.FilePath
}}
);
}
}
}
Help is very welcome and extremely appreciated, thank you. What this program is, is a ProxyChecker, because i've bought a bunch and will continue to do so of proxies with different user/passes etc, however some have expired. I added a break point and what it's doing is actually skipping the ProxyClient code and going straight to for each var item in 1, if item accepts connection etc, it then just returns false and finishes.
private static void CheckProxy(object state)
{
var u = user[0];
var p = pass[0];
var l = new List<MyIP>();
Parallel.ForEach(l.ToArray(), (ip_item) =>
{
try
{
string ip = ip_item.IP;
using (var client = new ProxyClient(ip, u, p))
{
Console.WriteLine(ip, user, pass);
client.Connect();
ip_item.AcceptsConnection = client.IsConnected;
}
}
catch
{
l.Remove(ip_item);
}
});
foreach (var item in l)
{
if (item.AcceptsConnection == true)
{
WriteToFile(user[0], pass[0]);
}
Console.WriteLine(item.IP + " is " + (item.AcceptsConnection) + " accepts connections" + " doesn not accept connections");
}
}
Load ips function:#
private static void loadips()
{
using (TextReader tr = new StreamReader("ips.txt"))
{
var l = new List<MyIP>();
string line = null;
while ((line = tr.ReadLine()) != null)
{
l.Add(new MyIP { IP = line });
}
}
}
I have added this in response to the answer. I believe this is a variable issue as the variable is locally declared not publicly any ideas how to fix? i'm unable to find a way to get this working seems like i'm being dumb. thanks.
Your code "skips" the Parallel.ForEach because the local variable l is an empty list (you just created it with new List<MyIP>().
Your loadips() method only fills a list referenced only by (another) local variable l, which is out of scope when the using block is left.
There are several ways to solve that. If both methods are in the same class, you could declare a member variable in that class of type List<MyIP> and fill that list in your loadips().
My suggestion would be to change loadips() so that it returns the list of read ip addresses and call this from CheckProxy():
private static List<MyIP> loadips() // changed return type to List<MyIP>
{
using (TextReader tr = new StreamReader("ips.txt"))
{
var l = new List<MyIP>();
string line = null;
while ((line = tr.ReadLine()) != null)
{
l.Add(new MyIP { IP = line });
}
return l; // return the list!
}
}
and the CheckProxy:
private static void CheckProxy(object state)
{
var u = user[0];
var p = pass[0];
var l = loadips(); // load IPs here
Parallel.ForEach(l.ToArray(), (ip_item) =>
{
...
I wrote a method to download data from the internet and save it to my database. I wrote this using PLINQ to take advantage of my multi-core processor and because it is downloading thousands of different files in a very short period of time. I have added comments below in my code to show where it stops but the program just sits there and after awhile, I get an out of memory exception. This being my first time using TPL and PLINQ, I'm extremely confused so I could really use some advice on what to do to fix this.
UPDATE: I found out that I was getting a webexception constantly because the webclient was timing out. I fixed this by increasing the max amount of connections according to this answer here. I was then getting exceptions for the connection not opening and I fixed it by using this answer here. I'm now getting connection timeout errors for the database even though it is a local sql server. I still haven't been able to get any of my code to run so I could totally use some advice
static void Main(string[] args)
{
try
{
while (true)
{
// start the download process for market info
startDownload();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
}
public static void startDownload()
{
DateTime currentDay = DateTime.Now;
List<Task> taskList = new List<Task>();
if (Helper.holidays.Contains(currentDay) == false)
{
List<string> markets = new List<string>() { "amex", "nasdaq", "nyse", "global" };
Parallel.ForEach(markets, market =>
{
Downloads.startInitialMarketSymbolsDownload(market);
}
);
Console.WriteLine("All downloads finished!");
}
// wait 24 hours before you do this again
Task.Delay(TimeSpan.FromHours(24)).Wait();
}
public static void startInitialMarketSymbolsDownload(string market)
{
try
{
List<string> symbolList = new List<string>();
symbolList = Helper.getStockSymbols(market);
var historicalGroups = symbolList.AsParallel().Select((x, i) => new { x, i })
.GroupBy(x => x.i / 100)
.Select(g => g.Select(x => x.x).ToArray());
historicalGroups.AsParallel().ForAll(g => getHistoricalStockData(g, market));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
}
public static void getHistoricalStockData(string[] symbols, string market)
{
// download data for list of symbols and then upload to db tables
Uri uri;
string url, line;
decimal open = 0, high = 0, low = 0, close = 0, adjClose = 0;
DateTime date;
Int64 volume = 0;
string[] lineArray;
List<string> symbolError = new List<string>();
Dictionary<string, string> badNameError = new Dictionary<string, string>();
Parallel.ForEach(symbols, symbol =>
{
url = "http://ichart.finance.yahoo.com/table.csv?s=" + symbol + "&a=00&b=1&c=1900&d=" + (DateTime.Now.Month - 1) + "&e=" + DateTime.Now.Day + "&f=" + DateTime.Now.Year + "&g=d&ignore=.csv";
uri = new Uri(url);
using (dbEntities entity = new dbEntities())
using (WebClient client = new WebClient())
using (Stream stream = client.OpenRead(uri))
using (StreamReader reader = new StreamReader(stream))
{
while (reader.EndOfStream == false)
{
line = reader.ReadLine();
lineArray = line.Split(',');
// if it isn't the very first line
if (lineArray[0] != "Date")
{
// set the data for each array here
date = Helper.parseDateTime(lineArray[0]);
open = Helper.parseDecimal(lineArray[1]);
high = Helper.parseDecimal(lineArray[2]);
low = Helper.parseDecimal(lineArray[3]);
close = Helper.parseDecimal(lineArray[4]);
volume = Helper.parseInt(lineArray[5]);
adjClose = Helper.parseDecimal(lineArray[6]);
switch (market)
{
case "nasdaq":
DailyNasdaqData nasdaqData = new DailyNasdaqData();
var nasdaqQuery = from r in entity.DailyNasdaqDatas.AsParallel().AsEnumerable()
where r.Date == date
select new StockData { Close = r.AdjustedClose };
List<StockData> nasdaqResult = nasdaqQuery.AsParallel().ToList(); // hits this line
break;
default:
break;
}
}
}
// now save everything
entity.SaveChanges();
}
}
);
}
Async lambdas work like async methods in one regard: They do not complete synchronously but they return a Task. In your parallel loop you are simply generating tasks as fast as you can. Those tasks hold onto memory and other resources such as DB connections.
The simplest fix is probably to just use synchronous database commits. This will not result in a loss of throughput because the database cannot deal with high amounts of concurrent DML anyway.
I was wondering if there is a way to programmatically check how many messages are in a private or public MSMQ using C#? I have code that checks if a queue is empty or not using the peek method wrapped in a try/catch, but I've never seen anything about showing the number of messages in the queue. This would be very helpful for monitoring if a queue is getting backed up.
You can read the Performance Counter value for the queue directly from .NET:
using System.Diagnostics;
// ...
var queueCounter = new PerformanceCounter(
"MSMQ Queue",
"Messages in Queue",
#"machinename\private$\testqueue2");
Console.WriteLine( "Queue contains {0} messages",
queueCounter.NextValue().ToString());
There is no API available, but you can use GetMessageEnumerator2 which is fast enough. Sample:
MessageQueue q = new MessageQueue(...);
int count = q.Count();
Implementation
public static class MsmqEx
{
public static int Count(this MessageQueue queue)
{
int count = 0;
var enumerator = queue.GetMessageEnumerator2();
while (enumerator.MoveNext())
count++;
return count;
}
}
I also tried other options, but each has some downsides
Performance counter may throw exception "Instance '...' does not exist in the specified Category."
Reading all messages and then taking count is really slow, it also removes the messages from queue
There seems to be a problem with Peek method which throws an exception
If you need a fast method (25k calls/second on my box), I recommend Ayende's version based on MQMgmtGetInfo() and PROPID_MGMT_QUEUE_MESSAGE_COUNT:
for C#
https://github.com/hibernating-rhinos/rhino-esb/blob/master/Rhino.ServiceBus/Msmq/MsmqExtensions.cs
for VB
https://gist.github.com/Lercher/5e1af6a2ba193b38be29
The origin was probably http://functionalflow.co.uk/blog/2008/08/27/counting-the-number-of-messages-in-a-message-queue-in/ but I'm not convinced that this implementation from 2008 works any more.
We use the MSMQ Interop. Depending on your needs you can probably simplify this:
public int? CountQueue(MessageQueue queue, bool isPrivate)
{
int? Result = null;
try
{
//MSMQ.MSMQManagement mgmt = new MSMQ.MSMQManagement();
var mgmt = new MSMQ.MSMQManagementClass();
try
{
String host = queue.MachineName;
Object hostObject = (Object)host;
String pathName = (isPrivate) ? queue.FormatName : null;
Object pathNameObject = (Object)pathName;
String formatName = (isPrivate) ? null : queue.Path;
Object formatNameObject = (Object)formatName;
mgmt.Init(ref hostObject, ref formatNameObject, ref pathNameObject);
Result = mgmt.MessageCount;
}
finally
{
mgmt = null;
}
}
catch (Exception exc)
{
if (!exc.Message.Equals("Exception from HRESULT: 0xC00E0004", StringComparison.InvariantCultureIgnoreCase))
{
if (log.IsErrorEnabled) { log.Error("Error in CountQueue(). Queue was [" + queue.MachineName + "\\" + queue.QueueName + "]", exc); }
}
Result = null;
}
return Result;
}
//here queue is msmq queue which you have to find count.
int index = 0;
MSMQManagement msmq = new MSMQManagement() ;
object machine = queue.MachineName;
object path = null;
object formate=queue.FormatName;
msmq.Init(ref machine, ref path,ref formate);
long count = msmq.MessageCount();
This is faster than you selected one.
You get MSMQManagement class refferance inside "C:\Program Files (x86)\Microsoft SDKs\Windows" just brows in this address you will get it. for more details you can visit http://msdn.microsoft.com/en-us/library/ms711378%28VS.85%29.aspx.
I had real trouble getting the accepted answer working because of the xxx does not exist in the specified Category error. None of the solutions above worked for me.
However, simply specifying the machine name as below seems to fix it.
private long GetQueueCount()
{
try
{
var queueCounter = new PerformanceCounter("MSMQ Queue", "Messages in Queue", #"machineName\private$\stream")
{
MachineName = "machineName"
};
return (long)queueCounter.NextValue();
}
catch (Exception e)
{
return 0;
}
}
The fastest method I have found to retrieve a message queue count is to use the peek method from the following site:
protected Message PeekWithoutTimeout(MessageQueue q, Cursor cursor, PeekAction action)
{
Message ret = null;
try
{
ret = q.Peek(new TimeSpan(1), cursor, action);
}
catch (MessageQueueException mqe)
{
if (!mqe.Message.ToLower().Contains("timeout"))
{
throw;
}
}
return ret;
}
protected int GetMessageCount(MessageQueue q)
{
int count = 0;
Cursor cursor = q.CreateCursor();
Message m = PeekWithoutTimeout(q, cursor, PeekAction.Current);
{
count = 1;
while ((m = PeekWithoutTimeout(q, cursor, PeekAction.Next)) != null)
{
count++;
}
}
return count;
}
This worked for me. Using a Enumarator to make sure the queue is empty first.
Dim qMsg As Message ' instance of the message to be picked
Dim privateQ As New MessageQueue(svrName & "\Private$\" & svrQName) 'variable svrnme = server name ; svrQName = Server Queue Name
privateQ.Formatter = New XmlMessageFormatter(New Type() {GetType(String)}) 'Formating the message to be readable the body tyep
Dim t As MessageEnumerator 'declared a enumarater to enable to count the queue
t = privateQ.GetMessageEnumerator2() 'counts the queues
If t.MoveNext() = True Then 'check whether the queue is empty before reading message. otherwise it will wait forever
qMsg = privateQ.Receive
Return qMsg.Body.ToString
End If
If you want a Count of a private queue, you can do this using WMI.
This is the code for this:
// You can change this query to a more specific queue name or to get all queues
private const string WmiQuery = #"SELECT Name,MessagesinQueue FROM Win32_PerfRawdata_MSMQ_MSMQQueue WHERE Name LIKE 'private%myqueue'";
public int GetCount()
{
using (ManagementObjectSearcher wmiSearch = new ManagementObjectSearcher(WmiQuery))
{
ManagementObjectCollection wmiCollection = wmiSearch.Get();
foreach (ManagementBaseObject wmiObject in wmiCollection)
{
foreach (PropertyData wmiProperty in wmiObject.Properties)
{
if (wmiProperty.Name.Equals("MessagesinQueue", StringComparison.InvariantCultureIgnoreCase))
{
return int.Parse(wmiProperty.Value.ToString());
}
}
}
}
}
Thanks to the Microsoft.Windows.Compatibility package this also works in netcore/netstandard.
The message count in the queue can be found using the following code.
MessageQueue messageQueue = new MessageQueue(".\\private$\\TestQueue");
var noOFMessages = messageQueue.GetAllMessages().LongCount();