WebIce Integration using Quick fix - c#

I am newbie to fix protocol and quick fix programming. I am seeking a help on getting Trade Capture report from ICE. I have googled for the sample/ tutorial to use quick fix/n to get the trade report but I am not being able to get sufficient output of it.
My problem is to get Trade Capture report or deal information for this I tried using TradeCaptureReportRequest, TradeCaptureReportRequestAck, TradeCaptureReport classes but somehow its now working.
A simple how to extract information would be a great help.
thanking everyone out there in advance.

Ok I am posting as an answer because it's going to be way too long for a comment. Please keep in mind that I have written custom constants, message types, etc (I wrote my acceptor server as well, so I'm not restricted by ICE constants/enums). You will need to determine what fields are required by ICE and make changes - this will not be easy to copy/paste...
First, you need to make sure you have all required files in and referenced. I create a folder called "fix" in my project, and copy all fix files into it. These need to be (at least 1) FixXXX.xml file, if you're using FIX50SP1 or 2, you need to also have FIXT11.xml. Along with the .xml files, you need to have an initiator.cfg file (assuming you're making an initiator, no a server, otherwise this will need to be "acceptor.cfg" but again, it sounds like you're trying to connect to ICE, so initiator is the correct usage. Finally, you will need to have a QuickFix.dll. My tree looks as below:
I am not going to go through the XML files - you will need to just learn that - it is very confusing and takes time.. especially if using FIXT11.XML along with SP1 or 2.
Your initiator.cfg should be similar to below:
# default settings for sessions
[DEFAULT]
FileStorePath=store
FileLogPath=log
ConnectionType=initiator
ReconnectInterval=60
SenderCompID=[Enter yours]
ResetOnLogon=Y
ResetOnLogout=Y
ResetOnDisconnect=Y
[SESSION]
BeginString=FIXT.1.1
TargetCompID=[Enter ICE Acceptor]
DefaultApplVerID=FIX.5.0
StartTime=12:30:00
EndTime=21:30:00
# overide default setting for RecconnectInterval
ReconnectInterval=30
HeartBtInt=30
SocketConnectPort=[From ICE]
# (optional) only listen for incoming connections on a specific host
#SocketConnectHost=127.0.0.1
SocketConnectHost=[ICE Ip Address- from your documentation/registration]
DataDictionary=..\..\fix\FIX50.xml
TransportDataDictionary=..\..\fix\FIXT11.xml
Ok, assuming that you have QuickFix.dll imported and referenced, and your initiator.cfg properly connected, it's actually fairly simple:
Create a Class that handles everything. Ignore AddToLB, that is a testing function.
public class TCT_Fix : Control, IApplication
{
private readonly string username = [removed]
private readonly string password = [removed]
public string InitiatorID;
SessionID sessionID;
public bool running;
SessionSettings settings;
IMessageStoreFactory storeFactory;
ILogFactory logFactory;
SocketInitiator initiator;
public event EventHandler AddToLB;
public event EventHandler AddToAdmin;
public void StopIt()
{
if (sessionID == null) return;
try
{
Session.LookupSession(sessionID).Disconnect("Stopping");
settings.Remove(sessionID);
settings = null;
initiator.Dispose();
settings = new SessionSettings(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "fix", "initiator.cfg"));
storeFactory = new FileStoreFactory(settings);
logFactory = new FileLogFactory(settings);
initiator = new SocketInitiator(
this,
storeFactory,
settings,
logFactory);
}
catch { }
}
public void FromApp(QuickFix.Message msg, SessionID sessionID)
{
var sMsg = "FROM APP: " + msg.ToString();
AddToLB(sMsg, null);
if (msg.Header.GetField(35) == "TC") //Cash
{
DateTime dtTdate;
float fPrice;
int Qty;
int OrdType;
bool BPisBuyer;
DateTime.TryParse(msg.GetField(CustomConstants.TDATE),out dtTdate);
string BPSide = msg.GetField(CustomConstants.BP_SIDE);
float.TryParse(msg.GetField(CustomConstants.F_PRICE), out fPrice);
int.TryParse(msg.GetField(CustomConstants.QTY), out Qty);
string TCTReference = msg.GetField(CustomConstants.TCT_REF);
string BPAcct = msg.GetField(CustomConstants.BP_COMPANY);
int.TryParse(msg.GetField(CustomConstants.ORDER_TYPE), out OrdType);
string ExecBkr = msg.GetField(CustomConstants.EXEC_BKR);
string CounterParty = msg.GetField(CustomConstants.COUNTER_PARTY);
BPisBuyer = msg.GetField(CustomConstants.IS_BUYER) == "Y";
string BPTrader = msg.GetField(CustomConstants.BP_TRADER);
string CounterTrader = msg.GetField(CustomConstants.COUNTER_TRADER);
string Grade = msg.GetField(CustomConstants.GRADE);
string Location = msg.GetField(CustomConstants.LOCATION);
string CycDt = msg.GetField(CustomConstants.CYCLE_DATE);
string DelMo = msg.GetField(CustomConstants.DELIVER_MONTH);
string Terms = msg.GetField(CustomConstants.TERMS);
string Payment = msg.GetField(CustomConstants.PAYMENT);
string Origin = msg.GetField(CustomConstants.ORIGIN);
string NumOfCyc = msg.GetField(CustomConstants.NUM_OF_CYCLES);
string Via = msg.GetField(CustomConstants.VIA);
string MoveMo = msg.GetField(CustomConstants.MOVE_MONTH);
string Comment = msg.GetField(CustomConstants.COMMENT);
}
else if (msg.Header.GetField(35) == "TE") //EFP
{
DateTime dtTdate;
float fPrice;
int Qty;
int OrdType;
bool BPisBuyer;
bool IsWater;
DateTime.TryParse(msg.GetField(CustomConstants.TDATE), out dtTdate);
string BPSide = msg.GetField(CustomConstants.BP_SIDE);
float.TryParse(msg.GetField(CustomConstants.F_PRICE), out fPrice);
int.TryParse(msg.GetField(CustomConstants.QTY), out Qty);
string TCTReference = msg.GetField(CustomConstants.TCT_REF);
string BPAcct = msg.GetField(CustomConstants.BP_COMPANY);
int.TryParse(msg.GetField(CustomConstants.ORDER_TYPE), out OrdType);
string ExecBkr = msg.GetField(CustomConstants.EXEC_BKR);
string CounterParty = msg.GetField(CustomConstants.COUNTER_PARTY);
BPisBuyer = msg.GetField(CustomConstants.IS_BUYER) == "Y";
string BPTrader = msg.GetField(CustomConstants.BP_TRADER);
string CounterTrader = msg.GetField(CustomConstants.COUNTER_TRADER);
string Grade = msg.GetField(CustomConstants.GRADE);
string Location = msg.GetField(CustomConstants.LOCATION);
string CycDt = msg.GetField(CustomConstants.CYCLE_DATE);
string DelMo = msg.GetField(CustomConstants.DELIVER_MONTH);
string Terms = msg.GetField(CustomConstants.TERMS);
string Payment = msg.GetField(CustomConstants.PAYMENT);
string Origin = msg.GetField(CustomConstants.ORIGIN);
string NumOfCyc = msg.GetField(CustomConstants.NUM_OF_CYCLES);
string Via = msg.GetField(CustomConstants.VIA);
string MoveMo = msg.GetField(CustomConstants.MOVE_MONTH);
string Comment = msg.GetField(CustomConstants.COMMENT);
IsWater = msg.GetField(CustomConstants.ISWATER) == "Y";
string BPFloorBkr = msg.GetField(CustomConstants.BP_FLOOR_BKR);
string CounterFloorBkr = msg.GetField(CustomConstants.COUNTER_FLOOR_BKR);
string Diff = msg.GetField(CustomConstants.DIFFERENCE);
string MercMo = msg.GetField(CustomConstants.MERC_MO);
string MercPr = msg.GetField(CustomConstants.MERC_PRICE);
}
else if (msg.Header.GetField(35) == "TI") //Index
{
DateTime dtTdate;
float fPrice;
int Qty;
int OrdType;
bool BPisBuyer;
bool IsWater;
DateTime.TryParse(msg.GetField(CustomConstants.TDATE), out dtTdate);
string BPSide = msg.GetField(CustomConstants.BP_SIDE);
float.TryParse(msg.GetField(CustomConstants.F_PRICE), out fPrice);
int.TryParse(msg.GetField(CustomConstants.QTY), out Qty);
string TCTReference = msg.GetField(CustomConstants.TCT_REF);
string BPAcct = msg.GetField(CustomConstants.BP_COMPANY);
int.TryParse(msg.GetField(CustomConstants.ORDER_TYPE), out OrdType);
string ExecBkr = msg.GetField(CustomConstants.EXEC_BKR);
string CounterParty = msg.GetField(CustomConstants.COUNTER_PARTY);
BPisBuyer = msg.GetField(CustomConstants.IS_BUYER) == "Y";
string BPTrader = msg.GetField(CustomConstants.BP_TRADER);
string CounterTrader = msg.GetField(CustomConstants.COUNTER_TRADER);
string Grade = msg.GetField(CustomConstants.GRADE);
string Location = msg.GetField(CustomConstants.LOCATION);
string CycDt = msg.GetField(CustomConstants.CYCLE_DATE);
string DelMo = msg.GetField(CustomConstants.DELIVER_MONTH);
string Terms = msg.GetField(CustomConstants.TERMS);
string Payment = msg.GetField(CustomConstants.PAYMENT);
string Origin = msg.GetField(CustomConstants.ORIGIN);
string NumOfCyc = msg.GetField(CustomConstants.NUM_OF_CYCLES);
string Via = msg.GetField(CustomConstants.VIA);
string MoveMo = msg.GetField(CustomConstants.MOVE_MONTH);
string Comment = msg.GetField(CustomConstants.COMMENT);
IsWater = msg.GetField(CustomConstants.ISWATER) == "Y";
string BPFloorBkr = msg.GetField(CustomConstants.BP_FLOOR_BKR);
string CounterFloorBkr = msg.GetField(CustomConstants.COUNTER_FLOOR_BKR);
string Diff = msg.GetField(CustomConstants.DIFFERENCE);
string MercMo = msg.GetField(CustomConstants.MERC_MO);
string MercPr = msg.GetField(CustomConstants.MERC_PRICE);
}
}
public void OnCreate(SessionID sessionID)
{
AddToAdmin("SESSION CREATED: " + sessionID.ToString(), null);
}
public void OnLogout(SessionID sessionID)
{
AddToAdmin("LOGOUT: " + this.sessionID.ToString(), null);
}
public void OnLogon(SessionID sessionID)
{
this.sessionID = sessionID;
AddToAdmin("LOG ON: " + this.sessionID.ToString(),null);
}
public void FromAdmin(QuickFix.Message msg, SessionID sessionID)
{
AddToAdmin("FROM ADMIN: " + msg.ToString(), null);
}
public void ToAdmin(QuickFix.Message msg, SessionID sessionID)
{
if (msg.Header.GetField(35).ToString() == "A")
{
msg.SetField(new QuickFix.Fields.Username(username));
msg.SetField(new QuickFix.Fields.Password(password));
}
AddToAdmin("TO ADMIN: " + msg.ToString(), null);
}
public void ToApp(QuickFix.Message msg, SessionID sessionID)
{
AddToLB("TO APP: " + msg.ToString(), null);
}
public void GetTestMessage(string msgType)
{
if (sessionID == null) return;
QuickFix.FIX50.TestMessage msg = new QuickFix.FIX50.TestMessage();
msg.TestType = msgType;
msg.Header.SetField(new QuickFix.Fields.MsgType("TEST"));
msg.SetField(new QuickFix.Fields.StringField(CustomConstants.TEST_TYPE, msgType));
Session.SendToTarget(msg, sessionID);
}
public TCT_Fix()
{
settings = new SessionSettings(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "fix", "initiator.cfg"));
storeFactory = new FileStoreFactory(settings);
logFactory = new FileLogFactory(settings);
initiator = new SocketInitiator(
this,
storeFactory,
settings,
logFactory);
}
public TCT_Fix(ref string initID)
{
InitiatorID = initID;
settings = new SessionSettings(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "fix", "initiator.cfg"));
storeFactory = new FileStoreFactory(settings);
logFactory = new FileLogFactory(settings);
initiator = new SocketInitiator(
this,
storeFactory,
settings,
logFactory);
}
public void RunIt()
{
if (running) return;
if(initiator.IsStopped)
{
try
{
initiator.Start(); //This can throw an error due to current set up. I would recommend making the connection,
//pulling data, and then closing the connection (polling) to ensure the initiator clears the
//log files
//reference http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/2013q1/000747.html
//2013 issue, still unresolved... Restart app
}
catch(Exception ex)
{
if (MessageBox.Show("Error restarting initiator. Program will close due to file access. This is a Quickfix bug, not an issue with this program. Please restart." + Environment.NewLine + Environment.NewLine +
"Reference: http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/2013q1/000747.html for more information. Click ok to copy link to clipboard. Click \"X\" to ignore.") == DialogResult.OK)
{
Clipboard.SetText("http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/2013q1/000747.html");
}
throw new Exception(ex.ToString());
}
}
running = true;
}
}
Finally, to make it stand out (this is actually in the block above as well), you construct a message similar to below, keeping in mind that your ICE Message will have certain required fields that my "TestMessage" does not. I cannot give code from production though - sorry.
public void GetTestMessage(string msgType)
{
if (sessionID == null) return;
QuickFix.FIX50.TestMessage msg = new QuickFix.FIX50.TestMessage();
msg.TestType = msgType;
msg.Header.SetField(new QuickFix.Fields.MsgType("TEST"));
msg.SetField(new QuickFix.Fields.StringField(CustomConstants.TEST_TYPE, msgType));
Session.SendToTarget(msg, sessionID);
}
The learning curve is substantial. You will just need to keep playing around until you get it. Once you get it down though, it makes sense. Stick with it. Let me know if you need anything else.

Related

How to check if ComputerName is INT

I think I'm getting tired or stared at this too long.. I need a second set of eyes. If anything looks too long winded please let me know.
My Goal
To check the local computer name and if it's an INT to then add it to HTTPClient Request to return related values from WebService. FYI - HostName is the ID Value of this particular table. However, if the value is NOT and INT then I want to replace the actual value for a fake one so I can see my page load correctly.
Eventually, in the customer's environment, I will turn off the display of this page if the Local Computer that runs it, is not part of the correct group. That way they won't see this configuration page at all.
Actual Results
Because the current VM I'm running the app on does not have an INT for the name, I get the following when I put in a breakpoint to check values. But I can still go to the localhost:2463/api/devices/101070701 and see the JSON results. So I just need to get UWP to place nice so I can see it within the APP.
HostName = 0
_HostName = LOCALVM_ComputerName.
DeviceIDService
public class DeviceIDService
{
public static DeviceIDService Instance = new DeviceIDService();
private DeviceIDService()
{
foreach (HostName DisplayName in NetworkInformation.GetHostNames())
{
if (DisplayName != null)
{
var hostNames = NetworkInformation.GetHostNames();
var hostName = hostNames.FirstOrDefault(name => name.Type == HostNameType.DomainName)?.DisplayName ?? "???";
LocalName = hostName;
break;
}
}
public string LocalName { get; set; }
}
}
DevicePageViewModel
public class DevicePageViewModel : ViewModelBase
{
private string _Hostname = DeviceIDService.Instance.LocalName;
public string hostName { get; set; }
public override async OnNavigatedToAsync ()
{
int HostName;
bool isNumeric = int.TryParse(_Hostname, out HostName);
if (isNumeric == true)
{
HostName = 101070701;
}
else
{
string hostName = HostName.ToString();
}
var uriD = new Uri("http://localhost:2463/api/Devices/" + HostName);
HttpClient client = new HttpClient();
try
{
var JsonResponseD = await client.GetStringAsync(uriD);
var devicesResult = JsonConvert.DeserializeObject<List<Device>>(JsonResponseD);
Devices = devicesResult;
Since Serge didn't actually provide an Answer but a comment. I wanted to display what he provided that solved my issue. Here is what I fixed. I changed up a few values, but this works for me..
_Hostname - set when actual ComputerName is grabbed from pc.
hostName - will be set to _Hostname if it's an INT
HostName - will be changed to 101070701 if the actually name is NOT an INT
HostName - will be left alone if ComputerName is an INT
int hostName;
bool isNumeric = int.TryParse(_Hostname, out hostName);
if (!isNumeric) {HostName = 101070701}
else
{
HostName = hostName;
}

Parsing performance of row data from files to SQL Server database

I have the PAF raw data in several files (list of all addresses in the UK).
My goal is to create a PostCode lookup in our software.
I have created a new database but there is no need to understand it for the moment.
Let's take a file, his extension is ".c01" and can be open with a text editor. The data in this file are in the following format :
0000000123A
With (according to the developer guide), 8 char for the KEY, 50 char for the NAME.
This file contains 2,449,652 rows (it's a small one !)
I create a Parsing class for this
private class SerializedBuilding
{
public int Key
{
get; set;
}
public string Name
{
get; set;
}
public bool isValid = false;
public Building ToBuilding()
{
Building b = new Building();
b.BuildingKey = Key;
b.BuildingName = Name;
return b;
}
private readonly int KEYLENGTH = 8;
private readonly int NAMELENGTH = 50;
public SerializedBuilding(String line)
{
string KeyStr = null;
string Name = null;
try
{
KeyStr = line.Substring(0, KEYLENGTH);
}
catch (Exception e)
{
Console.WriteLine("erreur parsing key line " + line);
return;
}
try
{
Name = line.Substring(KEYLENGTH - 1, NAMELENGTH);
}
catch (Exception e)
{
Console.WriteLine("erreur parsing name line " + line);
return;
}
int value;
if (!Int32.TryParse(KeyStr, out value))
return;
if (value == 0 || value == 99999999)
return;
this.Name = Name;
this.Key = value;
this.isValid = true;
}
}
I use this method to read the file
public void start()
{
AddressDataContext d = new AddressDataContext();
Count = 0;
string line;
// Read the file and display it line by line.
System.IO.StreamReader file =
new System.IO.StreamReader(filename);
SerializedBuilding sb = null;
Console.WriteLine("Number of line detected : " + File.ReadLines(filename).Count());
while ((line = file.ReadLine()) != null)
{
sb = new SerializedBuilding(line);
if (sb.isValid)
{
d.Buildings.InsertOnSubmit(sb.ToBuilding());
if (Count % 100 == 0)
d.SubmitChanges();
}
Count++;
}
d.SubmitChanges();
file.Close();
Console.WriteLine("building added");
}
I use Linq to SQL classes to insert data to my database. The connection string is the default one.
This seems to work, I have added 67200 lines. It just crashed but my questions are not about that.
My estimations :
33,647,015 rows to parse
Time needed for execution : 13 hours
It's a one-time job (just needs to be done on my sql and on the client server later) so I don't really care about performances but I think it can be interesting to know how it can be improved.
My questions are :
Is readline() and substring() the most powerful ways to read these huge files ?
Can the performance be improved by modifying the connection string ?

EntityTooSmall in CompleteMultipartUploadResponse

using .NET SDK v.1.5.21.0
I'm trying to upload a large file (63Mb) and I'm following the example at:
http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFileDotNet.html
But using a helper instead the hole code and using jQuery File Upload
https://github.com/blueimp/jQuery-File-Upload/blob/master/basic-plus.html
what I have is:
string bucket = "mybucket";
long totalSize = long.Parse(context.Request.Headers["X-File-Size"]),
maxChunkSize = long.Parse(context.Request.Headers["X-File-MaxChunkSize"]),
uploadedBytes = long.Parse(context.Request.Headers["X-File-UloadedBytes"]),
partNumber = uploadedBytes / maxChunkSize + 1,
fileSize = partNumber * inputStream.Length;
bool lastPart = inputStream.Length < maxChunkSize;
// http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFileDotNet.html
if (partNumber == 1) // initialize upload
{
iView.Utilities.Amazon_S3.S3MultipartUpload.InitializePartToCloud(fileName, bucket);
}
try
{
// upload part
iView.Utilities.Amazon_S3.S3MultipartUpload.UploadPartToCloud(fs, fileName, bucket, (int)partNumber, uploadedBytes, maxChunkSize);
if (lastPart)
// wrap it up and go home
iView.Utilities.Amazon_S3.S3MultipartUpload.CompletePartToCloud(fileName, bucket);
}
catch (System.Exception ex)
{
// Huston, we have a problem!
//Console.WriteLine("Exception occurred: {0}", exception.Message);
iView.Utilities.Amazon_S3.S3MultipartUpload.AbortPartToCloud(fileName, bucket);
}
and
public static class S3MultipartUpload
{
private static string accessKey = System.Configuration.ConfigurationManager.AppSettings["AWSAccessKey"];
private static string secretAccessKey = System.Configuration.ConfigurationManager.AppSettings["AWSSecretKey"];
private static AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey);
public static InitiateMultipartUploadResponse initResponse;
public static List<UploadPartResponse> uploadResponses;
public static void InitializePartToCloud(string destinationFilename, string destinationBucket)
{
// 1. Initialize.
uploadResponses = new List<UploadPartResponse>();
InitiateMultipartUploadRequest initRequest =
new InitiateMultipartUploadRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'));
initResponse = client.InitiateMultipartUpload(initRequest);
}
public static void UploadPartToCloud(Stream fileStream, string destinationFilename, string destinationBucket, int partNumber, long uploadedBytes, long maxChunkedBytes)
{
// 2. Upload Parts.
UploadPartRequest request = new UploadPartRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'))
.WithUploadId(initResponse.UploadId)
.WithPartNumber(partNumber)
.WithPartSize(maxChunkedBytes)
.WithFilePosition(uploadedBytes)
.WithInputStream(fileStream) as UploadPartRequest;
uploadResponses.Add(client.UploadPart(request));
}
public static void CompletePartToCloud(string destinationFilename, string destinationBucket)
{
// Step 3: complete.
CompleteMultipartUploadRequest compRequest =
new CompleteMultipartUploadRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'))
.WithUploadId(initResponse.UploadId)
.WithPartETags(uploadResponses);
CompleteMultipartUploadResponse completeUploadResponse =
client.CompleteMultipartUpload(compRequest);
}
public static void AbortPartToCloud(string destinationFilename, string destinationBucket)
{
// abort.
client.AbortMultipartUpload(new AbortMultipartUploadRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'))
.WithUploadId(initResponse.UploadId));
}
}
my maxChunckedSize is 6Mb (6 * (1024*1024)) as I have read that the minimum is 5Mb...
why am I getting "Your proposed upload is smaller than the minimum allowed size" exception? What am I doing wrong?
The error is:
<Error>
<Code>EntityTooSmall</Code>
<Message>Your proposed upload is smaller than the minimum allowed size</Message>
<ETag>d41d8cd98f00b204e9800998ecf8427e</ETag>
<MinSizeAllowed>5242880</MinSizeAllowed>
<ProposedSize>0</ProposedSize>
<RequestId>C70E7A23C87CE5FC</RequestId>
<HostId>pmhuMXdRBSaCDxsQTHzucV5eUNcDORvKY0L4ZLMRBz7Ch1DeMh7BtQ6mmfBCLPM2</HostId>
<PartNumber>1</PartNumber>
</Error>
How can I get ProposedSize if I'm passing the stream and stream length?
Here is a working solution for the latest Amazon SDK (as today: v.1.5.37.0)
Amazon S3 Multipart Upload works like:
Initialize the request using client.InitiateMultipartUpload(initRequest)
Send chunks of the file (loop until the end) using client.UploadPart(request)
Complete the request using client.CompleteMultipartUpload(compRequest)
If anything goes wrong, remember to dispose the client and request, as well fire the abort command using client.AbortMultipartUpload(abortMultipartUploadRequest)
I keep the client in Session as we need this for each chunk upload as well, keep an hold of the ETags that are now used to complete the process.
You can see an example and simple way of doing this in Amazon Docs itself, I ended up having a class to do everything, plus, I have integrated with the lovely jQuery File Upload plugin (Handler code below as well).
The S3MultipartUpload is as follow
public class S3MultipartUpload : IDisposable
{
string accessKey = System.Configuration.ConfigurationManager.AppSettings.Get("AWSAccessKey");
string secretAccessKey = System.Configuration.ConfigurationManager.AppSettings.Get("AWSSecretKey");
AmazonS3 client;
public string OriginalFilename { get; set; }
public string DestinationFilename { get; set; }
public string DestinationBucket { get; set; }
public InitiateMultipartUploadResponse initResponse;
public List<PartETag> uploadPartETags;
public string UploadId { get; private set; }
public S3MultipartUpload(string destinationFilename, string destinationBucket)
{
if (client == null)
{
System.Net.WebRequest.DefaultWebProxy = null; // disable proxy to make upload quicker
client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey, new AmazonS3Config()
{
RegionEndpoint = Amazon.RegionEndpoint.EUWest1,
CommunicationProtocol = Protocol.HTTP
});
this.OriginalFilename = destinationFilename.TrimStart('/');
this.DestinationFilename = string.Format("{0:yyyy}{0:MM}{0:dd}{0:HH}{0:mm}{0:ss}{0:fffff}_{1}", DateTime.UtcNow, this.OriginalFilename);
this.DestinationBucket = destinationBucket;
this.InitializePartToCloud();
}
}
private void InitializePartToCloud()
{
// 1. Initialize.
uploadPartETags = new List<PartETag>();
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest();
initRequest.BucketName = this.DestinationBucket;
initRequest.Key = this.DestinationFilename;
// make it public
initRequest.AddHeader("x-amz-acl", "public-read");
initResponse = client.InitiateMultipartUpload(initRequest);
}
public void UploadPartToCloud(Stream fileStream, long uploadedBytes, long maxChunkedBytes)
{
int partNumber = uploadPartETags.Count() + 1; // current part
// 2. Upload Parts.
UploadPartRequest request = new UploadPartRequest();
request.BucketName = this.DestinationBucket;
request.Key = this.DestinationFilename;
request.UploadId = initResponse.UploadId;
request.PartNumber = partNumber;
request.PartSize = fileStream.Length;
//request.FilePosition = uploadedBytes // remove this line?
request.InputStream = fileStream; // as UploadPartRequest;
var up = client.UploadPart(request);
uploadPartETags.Add(new PartETag() { ETag = up.ETag, PartNumber = partNumber });
}
public string CompletePartToCloud()
{
// Step 3: complete.
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest();
compRequest.BucketName = this.DestinationBucket;
compRequest.Key = this.DestinationFilename;
compRequest.UploadId = initResponse.UploadId;
compRequest.PartETags = uploadPartETags;
string r = "Something went badly wrong";
using (CompleteMultipartUploadResponse completeUploadResponse = client.CompleteMultipartUpload(compRequest))
r = completeUploadResponse.ResponseXml;
return r;
}
public void AbortPartToCloud()
{
// abort.
client.AbortMultipartUpload(new AbortMultipartUploadRequest()
{
BucketName = this.DestinationBucket,
Key = this.DestinationFilename,
UploadId = initResponse.UploadId
});
}
public void Dispose()
{
if (client != null) client.Dispose();
if (initResponse != null) initResponse.Dispose();
}
}
I use DestinationFilename as the destination file so I can avoid the same name, but I keep the OriginalFilename as I needed later.
Using jQuery File Upload Plugin, all works inside a Generic Handler, and the process is something like this:
// Upload partial file
private void UploadPartialFile(string fileName, HttpContext context, List<FilesStatus> statuses)
{
if (context.Request.Files.Count != 1)
throw new HttpRequestValidationException("Attempt to upload chunked file containing more than one fragment per request");
var inputStream = context.Request.Files[0].InputStream;
string contentRange = context.Request.Headers["Content-Range"]; // "bytes 0-6291455/14130271"
int fileSize = int.Parse(contentRange.Split('/')[1]);,
maxChunkSize = int.Parse(context.Request.Headers["X-Max-Chunk-Size"]),
uploadedBytes = int.Parse(contentRange.Replace("bytes ", "").Split('-')[0]);
iView.Utilities.AWS.S3MultipartUpload s3Upload = null;
try
{
// ######################################################################################
// 1. Initialize Amazon S3 Client
if (uploadedBytes == 0)
{
HttpContext.Current.Session["s3-upload"] = new iView.Utilities.AWS.S3MultipartUpload(fileName, awsBucket);
s3Upload = (iView.Utilities.AWS.S3MultipartUpload)HttpContext.Current.Session["s3-upload"];
string msg = System.String.Format("Upload started: {0} ({1:N0}Mb)", s3Upload.DestinationFilename, (fileSize / 1024));
this.Log(msg);
}
// cast current session object
if (s3Upload == null)
s3Upload = (iView.Utilities.AWS.S3MultipartUpload)HttpContext.Current.Session["s3-upload"];
// ######################################################################################
// 2. Send Chunks
s3Upload.UploadPartToCloud(inputStream, uploadedBytes, maxChunkSize);
// ######################################################################################
// 3. Complete Upload
if (uploadedBytes + maxChunkSize > fileSize)
{
string completeRequest = s3Upload.CompletePartToCloud();
this.Log(completeRequest); // log S3 response
s3Upload.Dispose(); // dispose all objects
HttpContext.Current.Session["s3-upload"] = null; // we don't need this anymore
}
}
catch (System.Exception ex)
{
if (ex.InnerException != null)
while (ex.InnerException != null)
ex = ex.InnerException;
this.Log(string.Format("{0}\n\n{1}", ex.Message, ex.StackTrace)); // log error
s3Upload.AbortPartToCloud(); // abort current upload
s3Upload.Dispose(); // dispose all objects
statuses.Add(new FilesStatus(ex.Message));
return;
}
statuses.Add(new FilesStatus(s3Upload.DestinationFilename, fileSize, ""));
}
Keep in mind that to have a Session object inside a Generic Handler, you need to implement IRequiresSessionState so your handler will look like:
public class UploadHandlerSimple : IHttpHandler, IRequiresSessionState
Inside fileupload.js (under _initXHRData) I have added an extra header called X-Max-Chunk-Size so I can pass this to Amazon and calculate if it's the last part of the uploaded file.
Fell free to comment and make smart edits for everyone to use.
I guess you didn't set the content-length of the part inside the UploadPartToCloud() function.

How to write unit test for this method

I am writing unit test for this method. I have tried lot of times but still can not write any code for it. Please suggest me how to unit test it. I am using C# , nunit framework and rhino mock.
Thanks in advance.
public FileUploadJsonResult AjaxUploadProfile(int id, string branchName, string filepath, HttpPostedFileBase file)
{
// TODO: Add your business logic here and/or save the file
string statusCode = "1";
string profilePicture = string.Empty;
string fileExtension = System.IO.Path.GetExtension(file.FileName.ToLower());
string fileName = id + "_" + branchName;
string fileNameWithOriginalExtension = fileName + fileExtension;
string fileNameWithJPGExtension = fileName + ".jpg";
string fileServerPath = this.Server.MapPath("~/LO_ProfilePicture/" + fileNameWithJPGExtension);
string statusMessage = string.Empty;
if (string.IsNullOrEmpty(fileExtension) || !Utility.isCorrectExtension(fileExtension))
{
statusMessage = "Profile picture should be of JPG, BMP, PNG, GIF or JPEG format.";
return new FileUploadJsonResult { Data = new { message = string.Format(statusMessage, fileNameWithOriginalExtension), filename = string.Empty, profilepic = profilePicture, statusCode = "0" } };
}
if (file.ContentLength > PageConstants.PROFILE_PICTURE_FILE_SIZE)
{
statusMessage = "Profile picture size should be less than 2MB";
return new FileUploadJsonResult { Data = new { message = string.Format(statusMessage, fileNameWithOriginalExtension), filename = string.Empty, profilepic = profilePicture, statusCode = "0" } };
}
Utility.SaveThumbnailImage(fileServerPath, file.InputStream, PageConstants.BRANCH_PROFILE_PICTURE_FILE_HEIGTH, PageConstants.BRANCH_PROFILE_PICTURE_FILE_WIDTH);
profilePicture = PageConstants.IMAGE_PATH + "LO_ProfilePicture/" + fileNameWithJPGExtension;
// Return JSON
return new FileUploadJsonResult { Data = new { message = string.Format("Profile Picture is successfully uploaded.", fileNameWithOriginalExtension), filename = fileNameWithJPGExtension, profilepic = profilePicture, statusCode } };
}
Make it do just the essential part. Split anything that has nothing to do with the operation you're trying to handle to other classes. Put those behind interfaces, so you can mock these in your unittests. This way you'll notice you don't have to test anything with file i/o in this class. In the class below I split up the function in the essential part, some file i/o and retrieving of settings. Even these settings have nothing to do with the current method you're trying to test. The method just needs verification on, for example, the extension, but it doesn't matter on how it does this.
Tip: try to avoid static utility classes. Give them their own class. Also avoid external components such as network communication or file i/o.
As I don't have a lot of context and it may not compile. But I would go with something like:
class Controller {
public FileUploadJsonResult AjaxUploadProfile(int id, string branchName, string filepath, HttpPostedFileBase file) {
string fileName = id + "_" + branchName;
string fileExtension = _fileIO.GetExtensionForFile(file);
if (!_extensionManager.IsValidExtension(fileExtension)) {
return CreateAjaxUploadProfileError("Profile picture should be of JPG, BMP, PNG, GIF or JPEG format.");
}
if (file.ContentLength > _settingsManager.GetMaximumFileSize()) {
return CreateAjaxUploadProfileError("Profile picture size should be less than 2MB");
}
string fileNameWithJPGExtension = fileName + ".jpg";
string fileServerPath = _fileIO.GetServerProfilePicture(Server, fileNameWithJPGExtension);
string fileClientPath = _fileIO.GetClientProfilePicture(fileNameWithJPGExtension);
var dimensions = _settingsManager.GetThumbnailDimensions();
_fileIO.SaveThumbnailImage(fileServerPath, file, dimensions.Item1, dimensions.Item2);
// Return JSON
var data = new {
message = "Profile Picture is successfully uploaded.",
filename = fileClientPath,
profilepic = profilePicture,
statusCode = "1"
};
return new FileUploadJsonResult { Data = data };
}
private static CreateAjaxUploadProfileError(string message) {
var data = new {
message = message,
filename = string.Empty,
profilepic = string.Empty,
statusCode = "0"
};
return new FileUploadJsonResult { Data = data };
}
}
class FileIO : IFileIO {
public string GetExtensionForFile(HttpPostedFileBase file) {
return System.IO.Path.GetExtension(filePath.FileName.ToLower());
}
public string GetServerProfilePicture(T server, string file) {
return server.MapPath( "~/LO_ProfilePicture/" + file);
}
public void SaveThumbnailImage(string path, HttpPostedFileBase file, int height, int width) {
Utility.SaveThumbnailImage(path, file.InputStream, height, width); // or even inline
}
public string GetClientProfilePicture(string fileName) {
return _settingsManager.GetClientImagePath() + "LO_ProfilePicture/" + fileNameWithJPGExtension;
}
}
class ExtensionManager : IExtensionManager {
public bool IsValidExtension(string extension) {
return Utility.isCorrectExtension(fileExtension); // or even inline
}
}
class SettingsManager : ISettingsManager {
public Tuple<int, int> GetThumbnailDimensions() {
return Tuple.Create<int, int>(PageConstants.BRANCH_PROFILE_PICTURE_FILE_HEIGTH, PageConstants.BRANCH_PROFILE_PICTURE_FILE_WIDTH);
}
public int GetMaximumFileSize() {
return PageConstants.PROFILE_PICTURE_FILE_SIZE;
}
}
You can look at this function as a combination of multiple functions doing specific work. One function is getting target file path, another is validating extension, another is validating size, another creates thumbnail, etc.
The goal is to breakdown the complex code into small testable functions (units) which you can test independently. So when you put them together you have better confidence that your big function works as expected.

Encoder SDK 4 - Push to Publishing Point

I'm coding an application in c# using EC4 SP2 SDK.
I want to publish my file to a media server publishing point. I've searched and found 2 examples regarding seting up and auth on publishing points, but either are from older sdk's or do not work (and are for console). basicly my application doesn't encode nothing, as if it had nothing to encode.
When in degub mode checkpont i can see the correct properties for the source file and for the server.
The encoding process takes 0secs to process. I checked the logs on the server events and i get a warning "the security system has received and auth request that could not be decoded". I just havo no knowledge to break up further than this. Any help would be appreciated.
this is the piece of code:
private void broadcastSourceFileToMediaServer2()
{
using (LiveJob job = new LiveJob())
{
String filetoencode = #"c:\temp\niceday.wmv";
LiveFileSource filesource = job.AddFileSource(filetoencode);
filesource.PlaybackMode = FileSourcePlaybackMode.Loop;
job.ActivateSource(filesource);
job.ApplyPreset(LivePresets.VC1Broadband4x3);
//don't know which one is good to use
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = "indes";
_pw = PullPW("indes");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
pubpoint.UserName = _myUserName;
pubpoint.Password = _pw;
job.PublishFormats.Add(pubpoint);
job.PreConnectPublishingPoint();
job.StartEncoding();
statusBox.Text = job.NumberOfEncodedSamples.ToString();
job.StopEncoding();
job.Dispose();
}
}
public static string _myUserName { get; set; }
public static SecureString _pw { get; set; }
//codificação de Password a enviar
private static SecureString PullPW(string pw)
{
SecureString s = new SecureString();
foreach (char c in pw) s.AppendChar(c);
return s;
}
static void job_AcquireCredentials(object sender, AcquireCredentialsEventArgs e)
{
e.UserName = _myUserName;
e.Password = _pw;
e.Modes = AcquireCredentialModes.None;
}
Progresses:
I managed to authenticate (at least get a positive audit event) on the server.
I changed from this:
//don't know which one is good to use
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = "indes";
_pw = PullPW("indes");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
pubpoint.UserName = _myUserName;
pubpoint.Password = _pw;
To this:
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = #"mediaservername\user";
_pw = PullPW("user_password");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
If you see on one side if had to include the domain (either domain or computername) before username. this changed the failed audit events on the server, so i could eliminate the manual credentials pubpoint.username and pubpoint.Password.
Now I'm just dealing with a lack of output format exception. On to it.
How about using SMOOTH Streaming, I managed to get my project going but I didn't get much more beyond Look below, to the part that has the PUBLISH switch type. ignore the file portion
internal bool StartStream()
{
Busy = true;
// Instantiates a new job for encoding
//
//***************************************Live Stream Archive******************************
if (blnRecordFromFile)
{
// Sets up publishing format for file archival type
FileArchivePublishFormat fileOut = new FileArchivePublishFormat();
// job.ApplyPreset(LivePresets.VC1512kDSL16x9);
// Gets timestamp and edits it for filename
string timeStamp = DateTime.Now.ToString();
timeStamp = timeStamp.Replace("/", "-");
timeStamp = timeStamp.Replace(":", ".");
// Sets file path and name
string path = "C:\\output\\";
string filename = "Capture" + timeStamp + ".ismv";
if (!Directory.Exists(path))
Directory.CreateDirectory(path);
fileOut.OutputFileName = Path.Combine(path, filename);
// Adds the format to the job. You can add additional formats as well such as
// Publishing streams or broadcasting from a port
job.PublishFormats.Add(fileOut);
}
//******************************END OF Stream PORTION****************************************
////////////////////////////////////////////////////////////////////////////////////////////////////
//*************************************** Process Files or Live Stream******************************
if (blnRecordFromFile)
{
job.ApplyPreset(LivePresets.VC1IISSmoothStreaming720pWidescreen);
job = new LiveJob();
// Verifies all information is entered
if (string.IsNullOrWhiteSpace(sourcePath) || string.IsNullOrWhiteSpace(destinationPath))
return false;
job.Status += new EventHandler<EncodeStatusEventArgs>(StreamStatus);
LiveFileSource fileSource;
try
{
// Sets file to active source and checks if it is valid
fileSource = job.AddFileSource(sourcePath);
}
catch (InvalidMediaFileException)
{
return false;
}
// Sets to loop media for streaming
// fileSource.PlaybackMode = FileSourcePlaybackMode.Loop;
// Makes this file the active source. Multiple files can be added
// and cued to move to each other at their ends
job.ActivateSource(fileSource);
}
//******************************END OF FILE PORTION****************************************
// Sets up variable for fomat data
switch (publishType)
{
case Output.Archive:
// Verifies destination path exists and if not creates it
try
{
if (!Directory.Exists(destinationPath))
Directory.CreateDirectory(destinationPath);
}
catch (IOException)
{
return false;
}
FileArchivePublishFormat archiveFormat = new FileArchivePublishFormat();
// Gets the location of the old extention and removes it
string filename = Path.GetFileNameWithoutExtension(sourcePath);
// Sets the archive path and file name
archiveFormat.OutputFileName = Path.Combine(destinationPath, filename + ".ismv");
job.PublishFormats.Add(archiveFormat);
break;
case Output.Publish:
// Setups streaming of media to publishing point
job = new LiveJob();
// Aquires audio and video devices
Collection<EncoderDevice> devices = EncoderDevices.FindDevices(EncoderDeviceType.Video);
EncoderDevice video = devices.Count > 0 ? devices[0] : null;
for (int i = 0; i < devices.Count; ++i)
// devices[i].Dispose();
devices.Clear();
devices = EncoderDevices.FindDevices(EncoderDeviceType.Audio);
EncoderDevice audio = devices.Count > 0 ? devices[0] : null;
for (int i = 1; i < devices.Count; ++i)
devices[i].Dispose();
devices.Clear();
// Checks for a/v devices
if (video != null && audio != null)
{
//job.ApplyPreset(Preset.FromFile(#"C:\Tempura\LivePreset3.xml"));
job.ApplyPreset(LivePresets.H264IISSmoothStreamingLowBandwidthStandard);
job.OutputFormat.VideoProfile.SmoothStreaming = true;
deviceSource = job.AddDeviceSource(video, audio);
// Make this source the active one
job.ActivateSource(deviceSource);
}
else
{
error = true;
}
PushBroadcastPublishFormat publishFormat = new PushBroadcastPublishFormat();
try
{
// checks the path for a valid publishing point
publishFormat.PublishingPoint = new Uri(destinationPath);
}
catch (UriFormatException)
{
return false;
}
// Adds the publishing format to the job
try
{
// job.ApplyPreset(LivePresets.VC1IISSmoothStreaming480pWidescreen);
job.PublishFormats.Add(publishFormat);
job.PreConnectPublishingPoint();
}
catch (Exception e)
{
MessageBox.Show(e.StackTrace.ToString());
}
break;
default:
return false;
}
job.StartEncoding();
return true;
}
Sadly I dont have enough rep to comment, so I have to write it as an answer.
Due to you are starting a live job, in order to stream you should not call job.StopEncoding() right after StartEncoding. I think usually you would use an event to stop the encoding. If you start encoding and immediately stop it, it is only logical you have no, or only a very small output.
I changed your code to the following and it seems work well. I guess your problem is that you disposed the instance of LiveJob class. You have to keep the instance alive before it finished encoding the whole stream. So change the using part and remove the StopEncoding and Dispose will be OK.
private void broadcastSourceFileToMediaServer2()
{
LiveJob job = new LiveJob();
String filetoencode = #"c:\temp\niceday.wmv";
LiveFileSource filesource = job.AddFileSource(filetoencode);
filesource.PlaybackMode = FileSourcePlaybackMode.Loop;
job.ActivateSource(filesource);
job.ApplyPreset(LivePresets.VC1Broadband4x3);
//don't know which one is good to use
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = "indes";
_pw = PullPW("indes");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
pubpoint.UserName = _myUserName;
pubpoint.Password = _pw;
job.PublishFormats.Add(pubpoint);
job.PreConnectPublishingPoint();
job.StartEncoding();
statusBox.Text = job.NumberOfEncodedSamples.ToString();
}
public static string _myUserName { get; set; }
public static SecureString _pw { get; set; }
//codificação de Password a enviar
private static SecureString PullPW(string pw)
{
SecureString s = new SecureString();
foreach (char c in pw) s.AppendChar(c);
return s;
}
static void job_AcquireCredentials(object sender, AcquireCredentialsEventArgs e)
{
e.UserName = _myUserName;
e.Password = _pw;
e.Modes = AcquireCredentialModes.None;
}

Categories