We Keep Coding

Parallel.ForEach(...) System Out of Memory exception

I've looked at Parallel.ForEach - System Out of Memory Exception regarding this issue but not much of a solution was given. I'm very new to using Parallel.ForEach, so I'm trying to figure out what's going on.
Diagnostic tools caps out at 1023 repeating (I understand this is an x86 to x64 arch restriction, but I wanted to offer the program in both formats.) I also don't feel like any program should ever meet that threshold. When I compile the program in x64, I sit around 1.1-1.4GB with MaxDegree . For sake of testing, I am running GC.Collection() at the end of each Parallel.ForEach iteration (I understand this isn't good practice, I'm just trying to troubleshoot at this point.)
Here's what I'm seeing:
Now if I try to use a Partitioner method, such as:
var checkforfinished = Parallel.ForEach<ListViewItem>(Partitioner.Create(0,lstBackupUsers.Items.Count), lstBackupUsers.Items.Cast<ListViewItem>(), opts, name =>
The I get an error of:
"No overload for method 'ForEach' takes 4 arguments"
That's fine, I modify my Parallel.ForEach statement so it looks like this:
var checkforfinished = Parallel.ForEach(Partitioner.Create(0,lstBackupUsers.Items.Count), lstBackupUsers.Items, opts, name => (I removed my casts)
and then my ForEach method won't accept the statement because it wants me to explicitly tell it that it's addressing a listviewbox.items method.
I am so confused on what to do.
Do I create a Partitioner, and if I do, how do I make my Parallel.ForEach method understand how to address a listviewbox?
update 1
I want to try to give as many details as possible because this is just rough. I'm sure it's easy, I'm just overcomplicationg it by an nth degree.
I have my Parallel.ForEach(//) running in a background worker function. My DoWork process is over 300 lines (I'm not an expert in C#, I'm just putting things together for a program at work.)
Here are bullet points of its basic structure
User clicks a "Start backup" button as seen in the screenshot
Button begins a separate function that checks to see which method the user selected to grab a list of usernames from (LDAP or flat text file)
That function then sends off a bgw_dowork() request
Inside the DoWork request, it looks like a summary of:
Check preliminary statements (bgw.cancellationpending for example)
Move on to grabbing some settings from the configurationmanager.appsettings
Begin the "complex" Parallel.ForEach command which Reads the listbox record rows and foreach row performs a very long list of commands to complete an operation for one user
The entire program, especially bgw_dowork heavily uses Google's v3 Drive API to login as a user, grab a file as recorded by other functions that prepare the user directory to be backed up (separate functions which login as a user, record their files (and fileIds) and their directories/subdirectories) and the bgw_dowork performs a chunk of the actual download functionality, which then calls off to the other functions to finish moving the files after they have been downloaded.
The actual "code" I use is (and I promise it's not pretty...)
private void bgW_DoWork(object sender, DoWorkEventArgs e)
{
{
try
{
txtFile.ReadOnly = true;
btnStart.Text = "Cancel Backup";
var appSettings = ConfigurationManager.AppSettings;
string checkreplace = ConfigurationManager.AppSettings["checkreplace"];
string userfile = txtFile.Text;
int counter = 0;
int arraycount = 0;
if (bgW.CancellationPending)
{
e.Cancel = true;
stripLabel.Text = "Operation was canceled!";
}
else
{
for (int z = 0; z >= counter; z++)
{
if (bgW.CancellationPending)
{
e.Cancel = true;
stripLabel.Text = "Operation was canceled!";
break;
}
else
{
double totalresource = int.Parse(ConfigurationManager.AppSettings["multithread"]);
totalresource = (totalresource / 100);
//var opts = new ParallelOptions { MaxDegreeOfParallelism = Convert.ToInt32(Math.Ceiling((Environment.ProcessorCount * totalresource) * 1.0)) };
var opts = new ParallelOptions { MaxDegreeOfParallelism = 2 };
var part = Partitioner.Create(1, 100);
//foreach (ListViewItem name in lstBackupUsers.Items)
var checkforfinished = Parallel.ForEach(lstBackupUsers.Items.Cast<ListViewItem>(), name =>
{
try
{
string names = name.SubItems[0].Text;
lstBackupUsers.Items[arraycount].Selected = true;
lstBackupUsers.Items[arraycount].BackColor = Color.CornflowerBlue;
arraycount++;
stripLabel.Text = "";
Console.WriteLine("Selecting user: " + names.ToString());
txtLog.Text += "Selecting user: " + names.ToString() + Environment.NewLine;
txtCurrentUser.Text = names.ToString();
// Define parameters of request.
string user = names.ToString();
// Check if directory exists, create if not.
string savelocation = ConfigurationManager.AppSettings["savelocation"] + user + "\\";
if (File.Exists(savelocation + ".deltalog.tok"))
File.Delete(savelocation + ".deltalog.tok");
FileInfo testdir = new FileInfo(savelocation);
testdir.Directory.Create();
string savedStartPageToken = "";
var start = CreateService.BuildService(user).Changes.GetStartPageToken().Execute();
// This token is set by Google, it defines changes made and
// increments the token value automatically.
// The following reads the current token file (if it exists)
if (File.Exists(savelocation + ".currenttoken.tok"))
{
StreamReader curtokenfile = new StreamReader(savelocation + ".currenttoken.tok");
savedStartPageToken = curtokenfile.ReadLine().ToString();
curtokenfile.Dispose();
}
else
{
// Token record didn't exist. Create a generic file, start at "1st" token
// In reality, I have no idea what token to start at, but 1 seems to be safe.
Console.Write("Creating new token file.\n");
//txtLog.Text += ("Creating new token file.\n" + Environment.NewLine);
StreamWriter sw = new StreamWriter(savelocation + ".currenttoken.tok");
sw.Write(1);
sw.Dispose();
savedStartPageToken = "1";
}
string pageToken = savedStartPageToken;
int gtoken = int.Parse(start.StartPageTokenValue);
int mytoken = int.Parse(savedStartPageToken);
txtPrevToken.Text = pageToken.ToString();
txtCurrentToken.Text = gtoken.ToString();
if (gtoken <= 10)
{
Console.WriteLine("Nothing to save!\n");
//txtLog.Text += ("User has nothing to save!" + Environment.NewLine);
}
else
{
if (pageToken == start.StartPageTokenValue)
{
Console.WriteLine("No file changes found for " + user + "\n");
//txtLog.Text += ("No file changes found! Please wait while I tidy up." + Environment.NewLine);
}
else
{
// .deltalog.tok is where we will place our records for changed files
Console.WriteLine("Changes detected. Making notes while we go through these.");
lblProgresslbl.Text = "Scanning Drive directory.";
// Damnit Google, why did you change how the change fields work?
if (savedStartPageToken == "1")
{
statusStripLabel1.Text = "Recording folder list ...";
txtLog.Text = "Recording folder list ..." + Environment.NewLine;
exfunctions.RecordFolderList(savedStartPageToken, pageToken, user, savelocation);
statusStripLabel1.Text = "Recording new/changed files ... This may take a bit!";
txtLog.Text += Environment.NewLine + "Recording new/changed list for: " + user;
exfunctions.ChangesFileList(savedStartPageToken, pageToken, user, savelocation);
}
else
{
//proUserclass = proUser;
statusStripLabel1.Text = "Recording new/changed files ... This may take a bit!";
txtLog.Text += Environment.NewLine + "Recording new/changed list for: " + user + Environment.NewLine;
exfunctions.ChangesFileList(savedStartPageToken, pageToken, user, savelocation);
}
// Get all our files for the user. Max page size is 1k
// after that, we have to use Google's next page token
// to let us get more files.
StreamWriter logFile = new StreamWriter(savelocation + ".recent.log");
string[] deltafiles = File.ReadAllLines(savelocation + ".deltalog.tok");
int totalfiles = deltafiles.Count();
int cnttototal = 0;
Console.WriteLine("\nFiles to backup:\n");
if (deltafiles == null)
{
return;
}
else
{
double damn = ((gtoken - double.Parse(txtPrevToken.Text)));
damn = Math.Round((damn / totalfiles));
if (damn <= 0)
damn = 1;
foreach (var file in deltafiles)
{
try
{
if (bgW.CancellationPending)
{
stripLabel.Text = "Backup canceled!";
e.Cancel = true;
break;
}
DateTime dt = DateTime.Now;
string[] foldervalues = File.ReadAllLines(savelocation + "folderlog.txt");
cnttototal++;
bgW.ReportProgress(cnttototal);
proUser.Maximum = int.Parse(txtCurrentToken.Text);
stripLabel.Text = "File " + cnttototal + " of " + totalfiles;
double? mathisfun;
mathisfun = ((100 * cnttototal) / totalfiles);
if (mathisfun <= 0)
mathisfun = 1;
double mathToken = double.Parse(txtPrevToken.Text);
mathToken = Math.Round((damn + mathToken));
// Bring our token up to date for next run
txtPrevToken.Text = mathToken.ToString();
File.WriteAllText(savelocation + ".currenttoken.tok", mathToken.ToString());
int proval = int.Parse(txtPrevToken.Text);
int nowval = int.Parse(txtCurrentToken.Text);
if (proval >= nowval)
proval = nowval;
proUser.Value = (proval);
lblProgresslbl.Text = ("Current progress: " + mathisfun.ToString() + "% completed.");
// Our file is a CSV. Column 1 = file ID, Column 2 = File name
var values = file.Split(',');
string fileId = values[0];
string fileName = values[1];
string mimetype = values[2];
mimetype = mimetype.Replace(",", "_");
string folder = values[3];
string ext = null;
int folderfilelen = foldervalues.Count();
fileName = GetSafeFilename(fileName);
Console.WriteLine("Filename: " + values[1]);
logFile.WriteLine("ID: " + values[0] + " - Filename: " + values[1]);
logFile.Flush();
// Things get sloppy here. The reason we're checking MimeTypes
// is because we have to export the files from Google's format
// to a format that is readable by a desktop computer program
// So for example, the google-apps.spreadsheet will become an MS Excel file.
switch (mimetype)
{
(switch statement here removed due to body length issues for this post.)
}
if (ext.Contains(".doc") || ext.Contains(".xls"))
{
string whatami = null;
if (ext.Contains(".xls"))
{
whatami = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
}
else if (ext.Contains(".doc"))
{
whatami = "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
}
else if (ext.Contains(".ppt"))
{
whatami = "application/vnd.openxmlformats-officedocument.presentationml.presentation";
}
if (fileName.Contains(".mov") || ext == ".ggl" || fileName.Contains(".mp4"))
{
txtLog.Text += Environment.NewLine + "Skipping file.";
return;
}
var requestfileid = CreateService.BuildService(user).Files.Export(fileId, whatami);
statusStripLabel1.Text = (savelocation + fileName + ext);
txtCurrentUser.Text = user;
string dest1 = Path.Combine(savelocation, fileName + ext);
var stream1 = new System.IO.FileStream(dest1, FileMode.OpenOrCreate, FileAccess.ReadWrite);
scrolltobtm();
requestfileid.MediaDownloader.ProgressChanged +=
(IDownloadProgress progress) =>
{
switch (progress.Status)
{
case DownloadStatus.Downloading:
{
Console.WriteLine(progress.BytesDownloaded);
logFile.WriteLine("Downloading: " + progress.BytesDownloaded);
txtLog.Text += ("Downloading ... " + progress.BytesDownloaded + Environment.NewLine);
scrolltobtm();
logFile.Flush();
break;
}
case DownloadStatus.Completed:
{
Console.WriteLine("Download complete.");
logFile.WriteLine("[" + user + "] Download complete for: " + requestfileid.ToString());
txtLog.Text += ("[" + user + "] Download complete for: " + fileName + Environment.NewLine);
logFile.Flush();
break;
}
case DownloadStatus.Failed:
{
Console.WriteLine("Download failed.");
logFile.WriteLine("Download failed.");
logFile.Flush();
break;
}
}
};
scrolltobtm();
GC.Collect();
GC.WaitForPendingFinalizers();
requestfileid.Download(stream1);
stream1.Close();
stream1.Dispose();
}
else
{
scrolltobtm();
var requestfileid = CreateService.BuildService(user).Files.Get(fileId);
//Generate the name of the file, and create it as such on the local filesystem.
statusStripLabel1.Text = (savelocation + fileName + ext);
string dest1 = Path.Combine(savelocation, fileName + ext);
var stream1 = new System.IO.FileStream(dest1, FileMode.OpenOrCreate, FileAccess.ReadWrite);
requestfileid.MediaDownloader.ProgressChanged +=
(IDownloadProgress progress) =>
{
switch (progress.Status)
{
case DownloadStatus.Downloading:
{
Console.WriteLine(progress.BytesDownloaded);
logFile.WriteLine("Downloading: " + progress.BytesDownloaded);
txtLog.Text += ("Downloading ... " + progress.BytesDownloaded + Environment.NewLine);
scrolltobtm();
logFile.Flush();
break;
}
case DownloadStatus.Completed:
{
Console.WriteLine("Download complete.");
logFile.WriteLine("Download complete for: " + requestfileid.ToString());
txtLog.Text += (Environment.NewLine + "[" + user + "] Download complete for: " + fileName + Environment.NewLine);
logFile.Flush();
break;
}
case DownloadStatus.Failed:
{
Console.WriteLine("Download failed.");
logFile.WriteLine("Download failed.");
logFile.Flush();
break;
}
}
};
scrolltobtm();
GC.Collect();
GC.WaitForPendingFinalizers();
requestfileid.Download(stream1);
stream1.Close();
stream1.Dispose();
}
}
catch (Google.GoogleApiException ex)
{
Console.Write("\nInfo: ---> " + ex.Message.ToString() + "\n");
}
}
}
exfunctions.MoveFiles(savelocation);
Console.WriteLine("\n\n\tBackup completed for selected user!");
txtLog.Text += ("\n\nBackup completed for selected user.\n\n");
statusStripLabel1.Text = "";
//logFile.Close();
//logFile.Dispose();
}
}
}
catch (Google.GoogleApiException ex)
{
Console.WriteLine("Info: " + ex.Message.ToString());
}
}
);
if (checkforfinished.IsCompleted == true)
{
MessageBox.Show("Parallel.ForEach() Finished!");
Console.WriteLine("Parallel.ForEach() Finished!");
}
else
{
MessageBox.Show("Parallel.ForEach() not completed!");
Console.WriteLine("Parallel.ForEach() not completed!");
}
}
}
}
}
catch (Google.GoogleApiException ex)
{
Console.WriteLine("Info: " + ex.Message.ToString());
}
}
}
You can see where I initiate the Parallel.ForEach(...) and then see what it is in charge of doing. It's a lot, and I understand it's not pretty, so I appreciate constructive criticism.

System.Net.WebException while downloading file

I have a list of mp3 which I am downloading. After some files are downloaded, not all of them - around 5-7, I get WebException. I did a stacktrace and this is the result.
Exception thrown: 'System.Net.WebException' in System.dll
Debug message: The operation has timed out
InnerEx: at System.Net.HttpWebRequest.GetResponse()
at iBlock.Main._InetGetHTMLSearch(String sArtist) in C:\Users\...\Main.cs:line 590
My _InetGetHTMLSearch looks like this
private void _InetGetHTMLSearch(string sArtist)
{
aLinks.Clear();
if (AudioDumpQuery == string.Empty)
{
//return string.Empty;
}
string[] sStringArray;
string sResearchURL = "http://www.audiodump.biz/music.html?" + AudioDumpQuery + sArtist.Replace(" ", "+");
string aRet;
HttpWebRequest webReq = (HttpWebRequest)HttpWebRequest.Create(sResearchURL);
webReq.UserAgent = "Mozilla / 5.0(Macintosh; Intel Mac OS X 10_9_3) AppleWebKit / 537.75.14(KHTML, like Gecko) Version / 7.0.3 Safari / 7046A194A";
webReq.Referer = "http://www.audiodump.com/";
webReq.Timeout = 5000;
try
{
webReq.CookieContainer = new CookieContainer();
webReq.Method = "GET";
using (WebResponse response = webReq.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(stream);
aRet = reader.ReadToEnd();
//Console.WriteLine(aRet);
string[] aTable = _StringBetween(aRet, "<BR><table", "table><BR>", RegexOptions.Singleline);
if (aTable != null)
{
string[] aInfos = _StringBetween(aTable[0], ". <a href=\"", "<a href=\"");
if (aInfos != null)
{
for (int i = 0; i < aInfos.Length; i++)
{
//do some magic here
}
}
else
{
//debug
}
}
else
{
//debug 2
}
}
response.Dispose();
}
}
catch (Exception ex)
{
Console.WriteLine("Debug message: " + ex.Message + "InnerEx: " + ex.StackTrace);
aLinks.Clear();
return;
//throw exception
}
}
what this method does is simple. A simple search of the sArtist given at audiodump.com
I have a timer which runs very fast, every 10ms.
private void MainTimer_Tick(object sender, EventArgs e)
{
_DoDownload(DoubleDimList[i][y], ref mp3ToPlay);
if (muted) Mute(0);
if (Downloading)
{
StatusLabel.Text = "Downloading: " + DoubleDimList[i][y];
}
}
Now this timer handles the download in the background in a Global scope.
The _DoDownload methos which basically starts the entire process looks like this
private void _DoDownload(string dArtist, ref string dPath)
{
if (!Contain && skip <= 3 && !Downloading)
{
try
{
_InetGetHTMLSearch(dArtist);
if (aLinks.Count < 1)
{
//skip and return
Console.WriteLine("Skipping: " + dArtist);
IniWriteValue(_playlists[i], "Track " + y, dArtist + " -iBlockSkip");
y++;
return;
}
string path = mp3Path + "\\" + dArtist + ".mp3";
if (DownloadOne(aLinks[0], path, false))
{
hTimmer.Start();
Downloading = true;
}
}
catch (Exception Ex)
{
MessageBox.Show("Download start error: " + Ex.Message);
}
}
else if (Downloading)
{
try {
int actualBytes = strm.Read(barr, 0, arrSize);
fs.Write(barr, 0, actualBytes);
bytesCounter += actualBytes;
double percent = 0d;
if (fileLength > 0)
percent =
100.0d * bytesCounter /
(preloadedLength + fileLength);
label1.Text = Math.Round(percent) + "%";
if (Math.Round(percent) >= 100)
{
string path = mp3Path + "\\" + dArtist + ".mp3";
label1.Text = "";
dPath = path;
aLinks.Clear();
hTimmer.Stop();
hTimmer.Reset();
fs.Flush();
fs.Close();
lastArtistName = "N/A";
Downloading = false;
y++;
if (y >= DoubleDimList[i].Count)
{
i++;
}
}
if (Math.Round(percent) <= 1)
{
if (hTimmer.ElapsedMilliseconds >= 3000)
{
string path = mp3Path + "\\" + dArtist + ".mp3";
hTimmer.Stop();
hTimmer.Reset();
fs.Flush();
fs.Close();
System.IO.File.Delete(path);
Contain = false;
skip += 1;
Downloading = false;
}
} }
catch(Exception Ex)
{
MessageBox.Show("Downloading error: " + Ex.Message);
}
}
}
Now once the exception is thrown it messes up the entire project. As you see in the last method, if _InetGetHTMLSearch doesn't update the search(returns nothing) I am skipping and moving to next search. However the exception will be thrown in every next search. I tried setting new cookies in every search but still didn't work.
Any solutions how to avoid this issue?
P.S. I have to say that if I change the timer's Interval to 500ms it will download more mp3 before the exception is thrown but not all of them.
Edit: The issue here is obvious. The request timesout but even if I set it to Timeout.Infinite it will hand there forever

Read Credentials by txt file and try login c#?

I would create a loop for function and print message in catch for all logins failed and print "sucessful connected with...." for all good logins.
Now i tried with that code, but i get only one error in textbox of catch
Button1:
if (openFileDialog1.FileName != string.Empty)
{
using (StreamReader reader = new StreamReader(openFileDialog1.FileName))
{
int count = 0;
string lineoflistemail;
while ((lineoflistemail = reader.ReadLine()) != null)
{
UserData d = new UserData();
string[] parts = lineoflistemail.Split(':');
count = parts.Length;
d.UserName = parts[0].Trim();
d.Password = parts[1].Trim();
data.Add(d);
}
foreach(UserData ud in data)
{
textBox1.Text += ("LOL" + ud.UserName + ud.Password + Environment.NewLine);
}
Second button code:
if (data.Count() == 0)
{
MessageBox.Show("Load user info first");
return;
}
for( hola = 0; hola < data.Count(); hola++)
{
var url = #"https://mail.google.com/mail/feed/atom";
var encoded = TextToBase64(data[0].UserName + ":" + data[1].Password);
var myweb = HttpWebRequest.Create(url) as HttpWebRequest;
myweb.Method = "POST";
myweb.ContentLength = 0;
myweb.Headers.Add("Authorization", "Basic " + encoded);
var response = myweb.GetResponse();
var stream = response.GetResponseStream();
textBox1.Text += ("Connection established with");
MessageBox.Show(hola.ToString());
}
}
catch (Exception ex)
{
textBox1.Text += ("Error connection. Original error: " + ex.Message);
}
}

The problem is that your try...catch is outside of your for loop. The first exception will exit the for loop, append the message to textBox1, then exit the button handler.
If you want to keep looping through even if there is an error, move the try...catch inside of the loop. Here's an example:
for( hola = 0; hola < data.Count(); hola++)
{
var url = #"https://mail.google.com/mail/feed/atom";
var encoded = TextToBase64(data[0].UserName + ":" + data[1].Password);
var myweb = HttpWebRequest.Create(url) as HttpWebRequest;
myweb.Method = "POST";
myweb.ContentLength = 0;
myweb.Headers.Add("Authorization", "Basic " + encoded);
try
{
var response = myweb.GetResponse();
var stream = response.GetResponseStream();
textBox1.Text += ("Connection established with");
MessageBox.Show(hola.ToString());
}
catch (Exception ex)
{
textBox1.Text += ("Error connection. Original error: " + ex.Message);
}
}

Twitter POST problems using api 1.1

We've just changed to Twitter api 1.1, and now Tweeting doesn't work & returns an error "The remote server returned an error: (400) Bad Request." Researching on SO about this suggests that it's something to do with authentication, but we are sending the accessToken & secret which we've just got from the login page. It all worked fine with api 1.0. The code is -
public void Tweet(Action<string> response, string message)
{
StringBuilder sb = new StringBuilder();
sb.Append("POST&");
sb.Append(Uri.EscapeDataString(_postUrl));
sb.Append("&");
string oauthNonce = Convert.ToBase64String(new ASCIIEncoding().GetBytes(DateTime.Now.Ticks.ToString()));
string timeStamp = MakeTimestamp();
var dict = new SortedDictionary<string, string>
{
{ "oauth_consumer_key", _oAuthConfig.ConsumerKey },
{ "oauth_nonce", oauthNonce },
{ "oauth_signature_method", "HMAC-SHA1" },
{ "oauth_timestamp", timeStamp },
{ "oauth_token", _accessToken },
{ "oauth_version", "1.0" },
};
foreach (var keyValuePair in dict)
{
sb.Append(Uri.EscapeDataString(string.Format("{0}={1}&", keyValuePair.Key, keyValuePair.Value)));
}
string encodedMessage = EscapeAdditionalChars(Uri.EscapeDataString(message));
sb.Append(Uri.EscapeDataString("status=" + encodedMessage));
string signatureBaseString = sb.ToString();
// create the signature
string signatureKey = Uri.EscapeDataString(_oAuthConfig.ConsumerSecret) + "&" + Uri.EscapeDataString(_accessTokenSecret);
var hmacsha1 = new HMACSHA1(new ASCIIEncoding().GetBytes(signatureKey));
string signatureString = Convert.ToBase64String(hmacsha1.ComputeHash(new ASCIIEncoding().GetBytes(signatureBaseString)));
// create the headers
string authorizationHeaderParams = String.Empty;
authorizationHeaderParams += "OAuth ";
authorizationHeaderParams += "oauth_consumer_key=\"" + _oAuthConfig.ConsumerKey + "\", ";
authorizationHeaderParams += "oauth_nonce=\"" + oauthNonce + "\", ";
authorizationHeaderParams += "oauth_signature=\"" + Uri.EscapeDataString(signatureString) + "\", ";
authorizationHeaderParams += "oauth_signature_method=\"" + "HMAC-SHA1" + "\", ";
authorizationHeaderParams += "oauth_timestamp=\"" + timeStamp + "\", ";
authorizationHeaderParams += "oauth_token=\"" + _accessToken + "\", ";
authorizationHeaderParams += "oauth_version=\"" + "1.0" + "\"";
string messageToPost = EscapeAdditionalChars(SpacesToPlusSigns(message));
// initialise the WebClient
WebClient client = new WebClient();
client.Headers [HttpRequestHeader.Authorization] = authorizationHeaderParams;
client.UploadDataCompleted += (s, eArgs) =>
{
if (eArgs.Error == null)
response(DefaultSuccessMessage());
else
response(eArgs.Error.Message);
};
try
{
Uri uri = new Uri(_postUrl);
try
{
client.UploadDataAsync(uri, "POST", Encoding.UTF8.GetBytes("status=" + messageToPost));
}
catch (WebException e)
{
Log.Info("TwitterService->Tweet web error: " + e.Message);
response(DefaultErrorMessage());
}
catch (Exception e)
{
// Can happen if we had already favorited this status
Log.Info("TwitterService->Tweet error: " + e.Message);
response(DefaultErrorMessage());
}
}
catch (WebException e)
{
Log.Info("TwitterService->Tweet web error 2: " + e.Message);
response(DefaultErrorMessage());
}
catch (Exception e)
{
Log.Info("TwitterService->Tweet error 2: " + e.Message);
response(DefaultErrorMessage());
}
}
Basically, I'd like to be able to Tweet without using any 3rd party libraries such as Twitterizer (even TweetStation seems to be broken with api 1.1) - surely it can't be that difficult!
Any help much appreciated, as it feels a bit like a brick wall at the moment - I'm also fairly new to c#, which doesn't help...
Edited to show code which wasn't clear previously.

Finally found the solution, as usual with most of these things, it was pretty simple. Code below -
public void Tweet(Action<string> response, string message)
{
StringBuilder sb = new StringBuilder();
sb.AppendFormat ("status={0}", PercentEncode(message));
string content = sb.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_postUrl);
request.Headers.Add("Authorization", AuthorizeRequest(_accessToken, _accessTokenSecret, "POST", new Uri(_postUrl), content));
request.ContentType = "application/x-www-form-urlencoded";
request.ServicePoint.Expect100Continue = false;
request.Method = "POST";
try
{
try
{
using (Stream stream = request.GetRequestStream())
{
Byte[] streamContent = Encoding.UTF8.GetBytes("status=" + PercentEncode(message));
stream.Write(streamContent, 0, streamContent.Length);
}
HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
string contents = "";
using (Stream stream = webResponse.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
contents = reader.ReadToEnd();
}
Console.WriteLine("Twitter response: " + contents);
response(DefaultSuccessMessage());
}
catch (WebException e)
{
Log.Info("TwitterService->Tweet web error: " + e.Message);
response(DefaultErrorMessage());
}
catch (Exception e)
{
// Can happen if we had already favorited this status
Log.Info("TwitterService->Tweet error: " + e.Message);
response(DefaultErrorMessage());
}
}
catch (WebException e)
{
Log.Info("TwitterService->Tweet web error 2: " + e.Message);
response(DefaultErrorMessage());
}
catch (Exception e)
{
Log.Info("TwitterService->Tweet error 2: " + e.Message);
response(DefaultErrorMessage());
}
}
private string AuthorizeRequest(string oauthToken, string oauthTokenSecret, string method, Uri uri, string data)
{
string oauthNonce = Convert.ToBase64String(new ASCIIEncoding().GetBytes(DateTime.Now.Ticks.ToString()));
var headers = new Dictionary<string, string>()
{
{ "oauth_consumer_key", _oAuthConfig.ConsumerKey },
{ "oauth_nonce", oauthNonce },
{ "oauth_signature_method", "HMAC-SHA1" },
{ "oauth_timestamp", MakeTimestamp() },
{ "oauth_token", oauthToken },
{ "oauth_verifier", PercentEncode(_authorizationVerifier) },
{ "oauth_version", "1.0A" }
};
var signatureHeaders = new Dictionary<string,string>(headers);
// Add the data and URL query string to the copy of the headers for computing the signature
if (data != null && data != "")
{
var parsed = HttpUtility.ParseQueryString(data);
foreach (string k in parsed.Keys)
{
signatureHeaders.Add(k, PercentEncode(parsed [k]));
}
}
var nvc = HttpUtility.ParseQueryString(uri.Query);
foreach (string key in nvc)
{
if (key != null)
signatureHeaders.Add(key, PercentEncode(nvc [key]));
}
string signature = MakeSignature (method, uri.GetLeftPart(UriPartial.Path), signatureHeaders);
string compositeSigningKey = MakeSigningKey(_oAuthConfig.ConsumerSecret, oauthTokenSecret);
string oauth_signature = MakeOAuthSignature(compositeSigningKey, signature);
headers.Add ("oauth_signature", PercentEncode(oauth_signature));
return HeadersToOAuth(headers);
}
private static string PercentEncode (string s)
{
var sb = new StringBuilder ();
foreach (byte c in Encoding.UTF8.GetBytes (s))
{
if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.' || c == '~')
sb.Append ((char) c);
else
{
sb.AppendFormat ("%{0:X2}", c);
}
}
return sb.ToString ();
}
private static string MakeTimestamp ()
{
return ((long) (DateTime.UtcNow - _unixBaseTime).TotalSeconds).ToString ();
}
private static string MakeSignature (string method, string base_uri, Dictionary<string,string> headers)
{
var items = from k in headers.Keys orderby k
select k + "%3D" + PercentEncode (headers [k]);
return method + "&" + PercentEncode (base_uri) + "&" +
string.Join ("%26", items.ToArray ());
}
private static string MakeSigningKey (string consumerSecret, string oauthTokenSecret)
{
return PercentEncode (consumerSecret) + "&" + (oauthTokenSecret != null ? PercentEncode (oauthTokenSecret) : "");
}
private static string MakeOAuthSignature (string compositeSigningKey, string signatureBase)
{
var sha1 = new HMACSHA1 (Encoding.UTF8.GetBytes (compositeSigningKey));
return Convert.ToBase64String (sha1.ComputeHash (Encoding.UTF8.GetBytes (signatureBase)));
}
private static string HeadersToOAuth (Dictionary<string,string> headers)
{
return "OAuth " + String.Join (",", (from x in headers.Keys select String.Format ("{0}=\"{1}\"", x, headers [x])).ToArray ());
}
With Twitter api 1.0, I used a WebClient to post, that doesn't work with api 1.1, and it seems that the reason for this is that you can't set the ContentType or the ServicePoint.Expect100Continue properties - without these set as I've set them, the request is sent back as (401) unauthorized. Nothing to do with encoding problems in the end.
Thanks to others for the various helper methods.

I had exactly the same problem:
This is exactly what you need to do here:
Authenticate and request a user's timeline with Twitter API 1.1 oAuth
I have created a project for this at : https://github.com/andyhutch77/oAuthTwitterTimeline
It also includes an MVC, Web app and console demo.

I ran into this problem, or at least one striking similiar (from my noob perspective), recently for an app I am building. What seemed to solve it for me (after looking at the tool at dev.twitter.com) was simply to get rid of the quotes around the parameter names, so that (in your case):
I notice that you do in fact not have quotes around your parameter names. However, it confuses me that you send authentication details twice (hence my wrongheaded post.) It works for me without doing this, and I googled it briefly and found: https://dev.twitter.com/discussions/12322#comment-27120, which confirms this can be a problem generating an Authetication Error.

400 means you are not authenticated. I recommend getting user context.
https://dev.twitter.com/docs/auth/oauth#user-context

error: The query results cannot be enumerated more than once

Edit:
DataClassesDataContext dc = new DataClassesDataContext();
string _idCompany = Request["idCompany"];
var newes = dc.GetNewsCompany(Int64.Parse(_idCompany));
string date = "";
string newsHtml = "<center>";
if(newes.GetEnumerator().MoveNext()){
foreach (var item in newes)//say Error .......................
{
// date = calendar.GetDayOfMonth(item.DateSend) + "/" + calendar.GetMonth(item.DateSend) + "/" + calendar.GetYear(item.DateSend).ToString();
// newsHtml += "<li class='news-item'><a style='text-decoration:none' class=\"link\" onclick=\"$(\'#BodyNews\').text(\'" + HttpUtility.HtmlEncode(item.Body).Trim() + "\');$(\'#BodyNews\').dialog({resizable:false});\" href=\"#\" > " + item.Title.ToString() + "</a> " + date + " </li>";
}
newsHtml += "</center>";
}
else
{
// var propertyCompany = dc.GetPropertyCompanyById(Int64.Parse(_idCompany));
// newsHtml += "<li class='news-item'><a style='text-decoration:none' class=\"link\" );$(\'#BodyNews\').dialog({resizable:false});\" href=\"#\" > " + "!به صفحه شخصی شرکت " + propertyCompany.FirstOrDefault().NameCompany + " خوش آمدید " + "</a> " + date + " </li>";
}
return newsHtml;
say error:The query results cannot be enumerated more than once
how check var is empty or null with out enumerated;

Why bother with the if at all?
var newes = dc.GetNewsCompany(Int64.Parse(_idCompany));
//if (newes.GetEnumerator().MoveNext())//check is null or empty
var newesList = newes.ToList();
if (neweList.Count > 0)
{
...
}
You can always check the newesList.Count property afterward.

Not sure what's available as a member in newes, but if it's an object and depending on what dc.GetNewsCompany returns you could check for null
if (news == null) return;
or if it returns an empty collection/array, just check the count/length:
if (news.Count == 0) return;
if (news.Length == 0) return;

the error comes, because you are using .GetEnumerator() on newes and then using the newes again in a foreach Loop .. this causes the "double enumeration".
Generally avoid walking "such var"'s with a foreach, since the DataReader is locked the whole loop !. Means that you cannot use the same entitie in this loop.
Better .ToList() , you can the list.AsQuearable agian if you want to Linq on it
f.e. something like
var newes = dc.CompanyTable.Where(ln => ln.id.Equals(_idCompany));;
List<CompanyTable> newesList = newes.ToList();