I'm creating a Windows application which reads XML file from given server. This application has installed in about 30 clients. Maybe they will call this function at the same time.
My question:
Will any problem occur if several user call this method at same time?
public string GetXmlInnerText()
{
FtpWebRequest tmpReq = null;
System.Net.WebResponse tmpRes = null;
try
{
if (Settings.Default.Internal)
tmpReq = (FtpWebRequest)FtpWebRequest.Create("ftp://<IPhere>/XMLData.xml");
else
tmpReq = (FtpWebRequest)FtpWebRequest.Create("ftp://<IPhere>/XMLData.xml");
tmpReq.Credentials = new System.Net.NetworkCredential("userName", "password");
tmpReq.KeepAlive = false;
tmpRes = tmpReq.GetResponse();
}
catch (Exception ex)
{
//------
}
string fileContents = null;
using (System.IO.Stream tmpStream = tmpRes.GetResponseStream())
{
using (System.IO.TextReader tmpReader = new System.IO.StreamReader(tmpStream))
{
fileContents = tmpReader.ReadToEnd();
}
}
return fileContents;
}
thanks
One problem - you're not disposing of the WebResponse. It implements IDisposable, so you should use a using statement. With your current structuring, that's not terribly easy to do - you should consider restructuring your try/catch blocks appropriately.
Further, StreamReader uses UTF-8 by default - if your XML documents aren't encoded in UTF-8, you could have problems. If it's an XML document, why not load it via XmlReader.Create(Stream) or something similar? That will handle the encoding for you.
I don't see a problem as you are only reading from a file. The only problem could be the server prohibiting concurrent access to a user account and restricts how many connections are allowed at the same time. In that case, you might be better of with a webservice or script (eg. php) delivering the xml via HTTP, not FTP.
If you're wondering about multiple clients accessing the FTP server at once, it will depend on how the FTP server is set up.
Some will be set up to only allow 2 or 3 clients at once, whereas some will allow (almost) as many as you could ever need.
If the FTP server causes troubles, you could serve it through a HTTP server instead.
Related
I have a Web application. It sends a series of images to the server (no problem there) and then uses code from this tutorial to create a PowerPoint presentation. The presentation is saved in a directory on the web server and the URL is returned to the user.
However, the file is still in use and attempting to access it generates a 500.0 error in IIS 7.5.
If I open the task manager and kill the w3wp.exe process that belongs to the NETWORK SERVICE user everything works as intended (the file can be downloaded and viewed). I also have another w3wp process belonging to DefaultAppPool, but it doesn't seem to cause a problem.
I'm new to this .Net coding, so it is very possible I forgot to close something down in code. Any ideas?
Edit: This is the method that creates a series of png's from image data that is encoded into a string. It uses a Guid to create a unique bit of a directory path, and checks to make sure it doesn't exist and then creates the directory and places the images there.
It looks like the offending method is this one:
So the offending method is this one:
public void createImages(List<String> imageStrings)
{
UTF8Encoding encoding = new UTF8Encoding();
Decoder decoder = encoding.GetDecoder();
Guid id = Guid.NewGuid();
String idString = id.ToString().Substring(0, 8);
while (Directory.Exists(imageStorageRoot + idString))
{
id = Guid.NewGuid();
idString = id.ToString().Substring(0, 8);
}
String imageDirectoryPath = imageStorageRoot + idString + "\\";
DirectoryInfo imagePathInfo = Directory.CreateDirectory(imageDirectoryPath);
for (int i = 0; i < imageStrings.Count; i++)
{
String imageString = imageStrings[i];
Byte[] binary = Convert.FromBase64String(imageString);
using (FileStream stream = new FileStream(imageDirectoryPath + idString + i.ToString() + ".png", FileMode.Create))
{
using (BinaryWriter writer = new BinaryWriter(stream))
{
writer.Write(binary);
}
}
}
}
Edit 2: If there is a better to about doing things please let me know. I am trying to learn here!
Edit 3: So upon further examination, I can comment out all this code. In fact, the second instance of w3wp.exe starts up as soon as a browser hits the website. I am now wondering if this might have something else to do with our stack? Its a Flex app, that uses WebOrb for remoting to some C# classes.
Does anyone know why this second open instance of w3wp.exe (owned by NETWORK SERVICE) would prevent the file from opening properly? Is there some way to get it release it's hold on the file in question?
Make sure you've closed the file after using it (better yet put the code that accesses your files in using statements). Post some code if you need help figuring out where the issue is.
The sample looks good. Did you deviate from the code structure?
using (Package pptPackage =
Package.Open(fileName, FileMode.Open, FileAccess.ReadWrite))
{
// your code goes inside there
}
If your code does contain the using statement you should be fine. If not add a Dispose call to your Package object when you are done or you will leave the file open for a long time until (possibly) the finalizer will kill it.
Scenario :
I need to parse millions of HTML files/pages (as fact as I can) & then read only only Title or Meta part of it & Dump it to Database
What I am doing is using System.Net.WebClient Class's DownloadString(url_path) to download & then Saving it to Database by LINQ To SQL
But this DownloadString function gives me complete html source, I just need only Title part & META tag part.
Any ideas, to download only that much content?
I think you can open a stream with this url and use this stream to read the first x bytes, I can't tell the exact number but i think you can set it to reasonable number to get the title and the description.
HttpWebRequest fileToDownload = (HttpWebRequest)HttpWebRequest.Create("YourURL");
using (WebResponse fileDownloadResponse = fileToDownload.GetResponse())
{
using (Stream fileStream = fileDownloadResponse.GetResponseStream())
{
using (StreamReader fileStreamReader = new StreamReader(fileStream))
{
char[] x = new char[Number];
fileStreamReader.Read(x, 0, Number);
string data = "";
foreach (char item in x)
{
data += item.ToString();
}
}
}
}
I suspect that WebClient will try to download the whole page first, in which case you'd probably want a raw client socket. Send the appropriate HTTP request (manually, since you're using raw sockets), start reading the response (which will not be immediately) and kill the connection when you've read enough. However, the rest will have probably already been sent from the server and winging its way to your PC whether you want it or not, so you might not save much - if anything - of the bandwidth.
Depending on what you want it for, many half decent websites have a custom 404 page which is a lot simpler than a known page. Whether that has the information you're after is another matter.
You can use the verb "HEAD" in a HttpWebRequest to return the the response headers (not element. To get the full element with the meta data you'll need to download the page and parse out the meta data you want.
System.Net.WebRequest.Create(uri) { Method = "HEAD" };
I have implemented something similar to this
only real difference is
string filename = context.Request.RawUrl.Replace("/", "\\").Remove(0,1);
string path = Uri.UnescapeDataString(Path.Combine(_baseFolder, filename));
so that I can traverse to subdirectories. This works great for webpages and other text file types but when trying to serve up media content I get the exception
HttpListenerException: The I/O
operation has been aborted because of
either a thread exit or an application
request
Followed by
InvalidOperationException: Cannot close stream until all bytes are written.
In the using statement.
Any suggestions on how to handle this or stop these exceptions?
Thanks
I should mention that I am using Google Chrome for my browser (Google Chrome doesn't seem to care about the MIME types, when it sees audio it will try to use it like it's in a HTML5 player), but this is also applicable if you are trying to host media content in a page.
Anyways, I was inspecting my headers with fiddler and noticed that Chrome passes 3 requests to the server. I started playing with other browsers and noticed they did not do this, but depending on the browser and what I had hard coded as the MIME type I would either get a page of crazy text, or a download of the file.
On further inspection I noticed that chrome would first request the file. Then request the file again with a few different headers most notably the range header. The first one with byte=0- then the next with a different size depending on how large the file was (more than 3 requests can be made depending how large the file is).
So there was the problem. Chrome will first ask for the file. Once seeing the type it would send another request which seems to me looking for how large the file is (byte=0-) then another one asking for the second half of the file or something similar to allow for a sort of streaming experienced when using HTML5. I coded something quickly up to handle MIME types and threw a HTML5 page together with the audio component and found that other browsers also do this (except IE)
So here is a quick solution and I no longer get these errors
string range = context.Request.Headers["Range"];
int rangeBegin = 0;
int rangeEnd = msg.Length;
if (range != null)
{
string[] byteRange = range.Replace("bytes=", "").Split('-');
Int32.TryParse(byteRange[0], out rangeBegin);
if (byteRange.Length > 1 && !string.IsNullOrEmpty(byteRange[1]))
{
Int32.TryParse(byteRange[1], out rangeEnd);
}
}
context.Response.ContentLength64 = rangeEnd - rangeBegin;
using (Stream s = context.Response.OutputStream)
{
s.Write(msg, rangeBegin, rangeEnd - rangeBegin);
}
Try:
using (Stream s = context.Response.OutputStream)
{
s.Write(msg, 0, msg.Length);
s.Flush()
}
I need to download xml file from a secure link, and store the content into database.
Can i use text reader??? or i need to store the file into my local file system first, then read the content from my file system and store into my database?
HttpWebRequest downloadRequest = (HttpWebRequest)WebRequest.Create("https://line-to-xml-file.xml");
string Content;
downloadRequest.Credentials = new NetworkCredential()
{
UserName = this._userCredentials.UserName,
Password = this._userCredentials.Password
};
downloadRequest.PreAuthenticate = true;
using (HttpWebResponse downloadHTTPResponse = (HttpWebResponse)downloadRequest.GetResponse())
{
using (Stream downloadResponseStream = downloadHTTPResponse.GetResponseStream())
using (TextReader tReader = new StreamReader(downloadResponseStream))
{
Content = tReader.ReadToEnd();
}
}
return Content;
Since the remote file is huge, up to 100MB, i can see nothing from debug.
and when i try to save it.
using (TransactionScope trans = new TransactionScope()) <--- when comes to this line, exception throws...
{
// perform update, save the content into databse
// send a notification message to message bus, indicate content has been updated
}
It complain MSDTC transaction timeout/cancelled
By having it into a stream you should be fine... if you have any problem with that specific stream then you could use a MemoryStream instead of a FileStream and use it the same way. But I hardly doubt this is your case.
I think you should make also sure to open the connection RIGHT BEFORE you are going to save the stream and when you have it completely loaded... You can also play with your Command.TimeOut property if it is taking really, really long to save, here "A value of 0 indicates no limit" but this should be avoided.
but
There is an XmlReader, but if the Xml is already well-formed and you don't need to parse it because your database is just taking it as a blob, or the database engine is going to parse it, you could use whatever.
Depends on your database schema.
Sounds like the database is having trouble inserting it, but need more info.
100MB of text is a lot to cram in a table. Your statement is almost certainly timing out. Check your context and your SQL command object (if you have one) and increase the timeout value.
You will probably set a longer timeout duration to solve the timeout problem, and make sure you call the Complete function
using (TransactionScope trans = new TransactionScope(
TransactionScopeOption.Required, new TimeSpan(1, 4, 3))) // Sets time out to 1 hour 4 minutes and 3 seconds
{
// perform update, save the content into databse
// send a notification message to message bus, indicate content has been updated
trans.Complete();
}
as for reading the file, you could possible use WebClient. WebClient allows you to monitor the progress.
WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential()
{
UserName = this._userCredentials.UserName,
Password = this._userCredentials.Password
};
wc.DownloadProgressChanged += new DownloadProgressChangedEventHandler(wc_DownloadProgressChanged);
wc.DownloadFile("https://line-to-xml-file.xml", "C:\\local.xml");
The handler could have log the progress if necessary:
void wc_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
// Log or show the current progress (e.ProgressPercentage or e.BytesReceived)
}
You could use DownloadString instead of Download file if you want the string straight without needing to read off the file again.
wc.DownloadString("https://line-to-xml-file.xml");
How about using the WebClient and its download file method. Just save it locally, use it and delete upon completion of use.
I am writing a simple web service using .NET, one method is used to send a chunk of a file from the client to the server, the server opens a temp file and appends this chunk. The files are quite large 80Mb, the net work IO seems fine, but the append write to the local file is slowing down progressively as the file gets larger.
The follow is the code that slows down, running on the server, where aFile is a string, and aData is a byte[]
using (StreamWriter lStream = new StreamWriter(aFile, true))
{
BinaryWriter lWriter = new BinaryWriter(lStream.BaseStream);
lWriter.Write(aData);
}
Debugging this process I can see that exiting the using statement is slower and slower.
If I run this code in a simple standalone test application the writes are the same speed every time about 3 ms, note the buffer (aData) is always the same side, about 0.5 Mb.
I have tried all sorts of experiments with different writers, system copies to append scratch files, all slow down when running under the web service.
Why is this happening? I suspect the web service is trying to cache access to local file system objects, how can I turn this off for specific files?
More information -
If I hard code the path the speed is fine, like so
using (StreamWriter lStream = new StreamWriter("c:\\test.dat", true))
{
BinaryWriter lWriter = new BinaryWriter(lStream.BaseStream);
lWriter.Write(aData);
}
But then it slow copying this scratch file to the final file destination later on -
File.Copy("c:\\test.dat", aFile);
If I use any varibale in the path it gets slow agin so for example -
using (StreamWriter lStream = new StreamWriter("c:\\test" + someVariable, true))
{
BinaryWriter lWriter = new BinaryWriter(lStream.BaseStream);
lWriter.Write(aData);
}
It has been commented that I should not use StreamWriter, note I tried many ways to open the file using FileStream, none of which made any change when the code is running under the web service, I tried WriteThrough etc.
Its the strangest thing I even tried this -
Write the data to file a.dat
Spawn system "cmd" "copy /b b.dat + a.dat b.dat"
Delete a.dat
This slows down the same way????
Makes me think the web server is running in some protected file IO environment catching all file operations in this process and child process, I can understand this if I was generating a file that might be later served to a client, but I am not, what I am doing is storing large binary blobs on disk, with a index/pointer to them stored in a database, if I comment out the write to the file the whole process fly's no performance issues at all.
I started reading about web server caching strategies, makes me think is there a web.config setting to mark a folder as uncached? Or am I completely barking up the wrong tree.
A long shot: is it possible that you need close some resources when you have finished?
If the file is binary, then why are you using a StreamWriter, which is derived from TextWriter? Just use a FileStream.
Also, BinaryWriter implements IDisposable, You need to put it into a using block.
Update....I replicated the basic code, no database, simple and it seems to work fine, so I suspect there is another reason, I will rest on it over the weekend....
Here is the replicated server code -
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Services;
using System.IO;
namespace TestWS
{
/// <summary>
/// Summary description for Service1
/// </summary>
[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[System.ComponentModel.ToolboxItem(false)]
// To allow this Web Service to be called from script, using ASP.NET AJAX, uncomment the following line.
// [System.Web.Script.Services.ScriptService]
public class Service1 : System.Web.Services.WebService
{
private string GetFileName ()
{
if (File.Exists("index.dat"))
{
using (StreamReader lReader = new StreamReader("index.dat"))
{
return lReader.ReadLine();
}
}
else
{
using (StreamWriter lWriter = new StreamWriter("index.dat"))
{
string lFileName = Path.GetRandomFileName();
lWriter.Write(lFileName);
return lFileName;
}
}
}
[WebMethod]
public string WriteChunk(byte[] aData)
{
Directory.SetCurrentDirectory(Server.MapPath("Data"));
DateTime lStart = DateTime.Now;
using (FileStream lStream = new FileStream(GetFileName(), FileMode.Append))
{
BinaryWriter lWriter = new BinaryWriter(lStream);
lWriter.Write(aData);
}
DateTime lEnd = DateTime.Now;
return lEnd.Subtract(lStart).TotalMilliseconds.ToString();
}
}
}
And the replicated client code -
static void Main(string[] args)
{
Service1 s = new Service1();
byte[] b = new byte[1024 * 512];
for ( int i = 0 ; i < 160 ; i ++ )
{
Console.WriteLine(s.WriteChunk(b));
}
}
Based on your code, it appears you're using the default handling inside of StreamWriter for files, which means synchronous and exclusive locks on the file.
Based on your comments, it seems the issue you really want to solve is the return time from the web service -- not necessarily the write time for the file. While the write time is the current gating factor as you've discovered, you might be able to get around your issue by going to an asynchronous-write mode.
Alternatively, I prefer completely de-coupled asynchronous operations. In that scenario, the inbound byte[] of data would be saved to its own file (or some other structure), then appended to the master file by a secondary process. More complex for operation, but also less prone to failure.
I don't have enough points to vote up an answer, but jro has the right idea. We do something similar in our service; each chunk is saved to a single temp file, then as soon as all chunks are received they're reassembled into a single file.
I'm not certain on the underlying processes for appending data to a file using StreamWriter, but I would assume it would have to at least read to the end of the current file before attempting to write whatever is in the buffer to it. So as the file gets larger it would have to read more and more of the existing file before writing the next chunk.
Well I found the root cause, "Microsoft Forefront Security", group policy has this running real time scanning, I could see the process goto 30% CPU usage when I close the file, killing this process and everything works the same speed, outside and inside the web service!
Next task find a way to add an exclusion to MFS!