I have an Azure webjob that is calling a ML training experiment via HttpRequests, leveraging the code generated in the ML webportal:
var request = new BatchExecutionRequest()
{
Inputs = new Dictionary<string, AzureBlobDataReference>() {
{
"input1",
new AzureBlobDataReference()
{
ConnectionString = _connectionString,
RelativeLocation = $"{_containerName}/{experimentId}/{tenantId}/{trainingDataFileName}"
}
},
},
Outputs = new Dictionary<string, AzureBlobDataReference>() {
{
"output1",
new AzureBlobDataReference()
{
ConnectionString = "azureStorageConnectionString",
RelativeLocation = $"{_containerName}/{experimentId}/{tenantId}/Model_2018421.ilearner"
}
},
},
GlobalParameters = new Dictionary<string, string>()
{
}
};
However, the request fails with the following message:
The blob reference:
experiments/experimentId/TenantId/Model_2018421.ilearner
has an invalid or missing file extension. Supported file extensions
for this output type are: \\".csv, .tsv, .arff\\"
I'm pretty confused about this, since it's written right the documentation all over the place that if I'm expecting a trained model to use ".ilearner" as the file extension for the model.
I've seen this question asking about the same error leveraging the DataFactory, and also this question on datascience.stackexchange. Neither one had any clues, answers, or other follow up.
Any insight on what I'm missing would be greatly appreciated!
For anyone looking for your "Don't Overthink It" moment of the day:
I needed to provide TWO output blob file references:
var request = new BatchExecutionRequest()
{
Inputs = new Dictionary<string, AzureBlobDataReference>() {
{
"input1",
new AzureBlobDataReference()
{
ConnectionString = _connectionString,
RelativeLocation = $"{_containerName}/{experimentId}/{tenantId}/{trainingDataFileName}.csv"
}
},
},
Outputs = new Dictionary<string, AzureBlobDataReference>() {
{
"output1",
new AzureBlobDataReference()
{
ConnectionString = _connectionString,
RelativeLocation = $"{_containerName}/{experimentId}/{tenantId}/{outputFileNameCsv}.csv"
}
},
{
"output2",
new AzureBlobDataReference()
{
ConnectionString = _connectionString,
RelativeLocation = $"{_containerName}/{experimentId}/{tenantId}/{outputFileNameIlearner}.ilearner"
}
},
},
GlobalParameters = new Dictionary<string, string>()
{
}
};
There's an old saying in American English about not making assumptions, and I assumed the second output was an optional parameter used in batch operations. Since I'm not actually looking for more than one result from each call, I thought I was safe to remove the second output parameter.
TL/DR: Keep all the parameters the webservice portal's "Consume" tab generates, and make sure the first one is a .csv file reference.
Related
I'm new to AWS CDK and what I'm trying to accomplish (using C#) is to version my lambda function, and then create a new API resource referencing the version.
For example: The program accepts a version parameter.
internal CdkAppStack(Construct scope, string id, IStackProps props, string bucketName, string functionName, string version) : base(scope, id, props)
{
var bucket = new Bucket(this, bucketName, new BucketProps {
BucketName = bucketName
});
var handler = new Function(this, $"{functionName}Handler", new FunctionProps
{
Runtime = Runtime.DOTNET_CORE_3_1,
Code = Code.FromAsset("Lambdas\\src\\Lambdas\\bin\\Debug\\netcoreapp3.1"),
Handler = "Lambdas::Lambdas.Function::FunctionHandler",
Environment = new Dictionary<string, string>
{
["BUCKET"] = bucket.BucketName,
},
FunctionName = functionName
});
string apiName = !string.IsNullOrEmpty(version) ? $"{functionName}-{version}" : functionName;
bucket.GrantReadWrite(handler);
var api = new RestApi(this, $"{apiName}-API", new RestApiProps
{
RestApiName = $"{apiName}API",
Description = $"This service the Lambda - {functionName}.",
RetainDeployments = true
});
var getWidgetsIntegration = new LambdaIntegration(handler, new LambdaIntegrationOptions
{
RequestTemplates = new Dictionary<string, string>
{
["application/json"] = "{ \"statusCode\": \"200\" }"
}
});
string resource = !string.IsNullOrEmpty(version) ? $"execute-{version}" : "execute";
var helloWorldResource = api.Root.AddResource(resource);
var method = helloWorldResource.AddMethod("POST", getWidgetsIntegration);
}
Actual result:
It overrides the lambda and api resource.
Expected result:(version parameter is 3). Added new resource (execute-3)
AWS Console - Expected result
For the given code the process probably looks like this:
Current provisioned state includes execute-2
CDK synthesizes a Cloudformation template where execute-3 exists and execute-2 does not exist.
Cloudformation figures out that in order to reach the desired state (described in the template) it needs to delete execute-2 and provision execute-3.
If you literally want to reach what you are describing you will have to add resources and not delete previously provisioned ones (say write a loop that runs over 1..version and adds all past versions)
Another (less recommendable) option is to use a deletion policy. This way you can hint cloudformation not the delete execute-2 (the downside here is that execute-2's lifecycle is no longer managed by any stack --> you will have to manage changes to it manually)
I'm stuck with my problem using " Google.Apis.Testing.v1.Data " and their documentation doesn't help me.
I have to set a "timeout" value (= a duration), but the variable type is "object" instead of "float" for example. I tried to put an int, a float, and a string but that doesn't work.
The object API doc is here. My variable is "TestTimeout" which is definitely a duration.
When I searched for a solution, I saw in java the variable type is string but that doesn't help (here)
Just for your information, I'm using this lib to execute my android application on their test devices. It's a service called TestLab in Firebase. The timeout value needs to be higher because I don't have enough time to execute my test. Here is my code, everything is working well besides this TimeOut.
TestMatrix testMatrix = new TestMatrix();
testMatrix.TestSpecification = new TestSpecification();
testMatrix.TestSpecification.TestTimeout = 600.0f; // I tested 600, 600.0f, "600", "30m", "500s"
testMatrix.EnvironmentMatrix = new EnvironmentMatrix();
testMatrix.EnvironmentMatrix.AndroidDeviceList = new AndroidDeviceList();
testMatrix.EnvironmentMatrix.AndroidDeviceList.AndroidDevices = new List<AndroidDevice>();
foreach (TestMatrixModel.TestData testData in _model.ListTests)
{
if (testData.IsSelected)
{
//Here I'm using my own data class to set GoogleAPI objects, it's simple
//as it asks me strings even for integer numbers, and it's working
foreach (int indice in testData.ChosenAndroidVersionsIndices)
{
AndroidDevice device = new AndroidDevice();
device.AndroidModelId = testData.ModelID;
device.AndroidVersionId = testData.AvailableAndroidVersions[indice];
device.Locale = testData.AvailableLocales[testData.ChosenLocale];
device.Orientation = testData.Orientation;
testMatrix.EnvironmentMatrix.AndroidDeviceList.AndroidDevices.Add(device);
}
}
}
Ok and here is the result of the request :
{
"testMatrixId": "matrix-2dntrwio3kco7",
"testSpecification": {
"testTimeout": "300s",
"testSetup": {},
"androidTestLoop": {
"appApk": {
"gcsPath": "gs://myLinkIntoGoogleCloudStorage.apk"
}
}
},
"environmentMatrix": {
"androidDeviceList": {
"androidDevices": [
{
"androidModelId": "grandpplte",
"androidVersionId": "23",
"locale": "en_001",
"orientation": "landscape"
},
{
"androidModelId": "hero2lte",
"androidVersionId": "23",
"locale": "en_001",
"orientation": "landscape"
},
etc.....
As you can see, it seems to be a string set to "300s"... so why "500s" cannot enter in ?
Thanks a lot.
Ok I got my answer :
testMatrix.TestSpecification.TestTimeout = "600s";
So it was a string and needed to finish with "s". Why that didn't work when I tried ? Just because my code was overrided with another TestSpecification after... my bad.
I'm trying to make a targetingIdeaService API call to Google AdWords. This is my code so far:
[HttpGet]
public IEnumerable<string> Get()
{
var user = new AdWordsUser();
using (TargetingIdeaService targetingIdeaService = (TargetingIdeaService)user.GetService(AdWordsService.v201802.TargetingIdeaService))
{
// Create selector.
TargetingIdeaSelector selector = new TargetingIdeaSelector();
selector.requestType = RequestType.IDEAS;
selector.ideaType = IdeaType.KEYWORD;
selector.requestedAttributeTypes = new AttributeType[] {
AttributeType.KEYWORD_TEXT,
AttributeType.SEARCH_VOLUME,
AttributeType.AVERAGE_CPC,
AttributeType.COMPETITION,
AttributeType.CATEGORY_PRODUCTS_AND_SERVICES
};
// Set selector paging (required for targeting idea service).
var paging = Paging.Default;
// Create related to query search parameter.
var relatedToQuerySearchParameter =
new RelatedToQuerySearchParameter
{ queries = new String[] { "bakery", "pastries", "birthday cake" } };
var searchParameters = new List<SearchParameter> { relatedToQuerySearchParameter };
var page = new TargetingIdeaPage();
page = targetingIdeaService.get(selector);
return new string[] { "value1", "value2" };
}
}
it looks ok, compiles at last, and so on. But then I went into debug mode. And I saw this:
So as you can see, the variable doesn't have the access token. The other data comes from app.config file.
I am quite certain the keys passed in are correct.
Then the code throws the famous invalid_grand error. In my case, I believe that's because the access token is not being generated. I'm new to AdWords and ASP.NET, so I probably missed something, but I have no idea what.
I used the
docs,
Code Structure instructions, and
code examples to put it all together.
I had my configuration wrong. I had to recreate all the credentials using incognito window. If you have any issues just open a thread here: https://groups.google.com/forum/#!forum/adwords-api
This has been driving me batty for over an hour. I'm new to EasyPost and I'm trying to put some custom text on my label (in my particular case, it is which SKU to put in the package), but it just never seemed to work. I'm using the official nuget package from easypost but am guessing it is platform independent.
Shipment shipment = new Shipment() {
to_address = toAddress,
from_address = fromAddress,
parcel = parcel
};
shipment.Create();
var lowestRate = shipment.LowestRate(includeServices: new List<string>() { "First" }, includeCarriers: new List<string>() { "USPS" });
shipment.Buy(lowestRate);
shipment.options.Add("print_custom_1", "this is some sample text");
shipment.options.Add("print_custom_2", "abc");
shipment.options.Add("print_custom_3", "xyz");
shipment.GenerateLabel("pdf");
Well, that was annoying. It makes sense when you step back from it. The issue is that the options need to be set PRIOR to creating the shipment. In my head, it was a print only concern (and it is), but there are other options can and do effect shipping costs, which means that the option needs to be set when creating the shipment. Even setting the options after you create but before you "buy" doesn't work.
See working code below:
Shipment shipment = new Shipment() {
to_address = toAddress,
from_address = fromAddress,
parcel = parcel
};
//DO THIS BEFORE CREATING!
shipment.options = new Dictionary<string, object>();
shipment.options.Add("print_custom_1", "this is some sample text");
shipment.options.Add("print_custom_2", "abc");
shipment.options.Add("print_custom_3", "xyz");
shipment.Create();
var lowestRate = shipment.LowestRate(includeServices: new List<string>() { "First" }, includeCarriers: new List<string>() { "USPS" });
shipment.Buy(lowestRate);
shipment.GenerateLabel("pdf");
I've just completed a round of refactoring of my application, which has resulted in my removing a project that was no longer required and moving its classes into a different project. A side effect of this is that my User class, which is stored in RavenDB, has a collection property of a type moved to the new assembly. As soon as I attempt to query the session for the User class I get a Json deserialisation error. The issue is touched upon here but the answers don't address my issue. Here's the offending property:
{
"OAuthAccounts": {
"$type": "System.Collections.ObjectModel.Collection`1[
[Friendorsement.Contracts.Membership.IOAuthAccount,
Friendorsement.Contracts]], mscorlib",
"$values": []
},
}
OAuthAccounts is a collection property of User that used to map here:
System.Collections.ObjectModel.Collection`1[[Friendorsement.Contracts.Membership.IOAuthAccount, Friendorsement.Contracts]]
It now maps here:
System.Collections.ObjectModel.Collection`1[[Friendorsement.Domain.Membership.IOAuthAccount, Friendorsement.Domain]]
Friendorsement.Contracts no longer exists. All of its types are now in Friendorsement.Domain
I've tried using store.DatabaseCommands.StartsWith("User", "", 0, 128) but that didn't return anything.
I've tried looking at UpdateByIndex but not got very far with it:
store.DatabaseCommands.UpdateByIndex("Raven/DocumentsByEntityName",
new IndexQuery {Query = "Tag:Users"},
new[]
{
new PatchRequest { // unsure what to set here }
});
I'm using Raven 2.0
Below is a simple sample application that shows you the patching Metadata. While your example is a little different this should be a good starting point
namespace SO19941925
{
internal class Program
{
private static void Main(string[] args)
{
IDocumentStore store = new DocumentStore
{
Url = "http://localhost:8080",
DefaultDatabase = "SO19941925"
}.Initialize();
using (IDocumentSession session = store.OpenSession())
{
for (int i = 0; i < 10; i++)
{
session.Store(new User {Name = "User" + i});
}
session.SaveChanges();
}
using (IDocumentSession session = store.OpenSession())
{
List<User> users = session.Query<User>().Customize(x => x.WaitForNonStaleResultsAsOfNow()).ToList();
Console.WriteLine("{0} SO19941925.Users", users.Count);
}
Operation s = store.DatabaseCommands.UpdateByIndex("Raven/DocumentsByEntityName",
new IndexQuery {Query = "Tag:Users"},
new ScriptedPatchRequest
{
Script = #"this['#metadata']['Raven-Clr-Type'] = 'SO19941925.Models.User, SO19941925';"
}, true
);
s.WaitForCompletion();
using (IDocumentSession session = store.OpenSession())
{
List<Models.User> users =
session.Query<Models.User>().Customize(x => x.WaitForNonStaleResultsAsOfNow()).ToList();
Console.WriteLine("{0} SO19941925.Models.Users", users.Count);
}
Console.ReadLine();
}
}
internal class User
{
public string Name { get; set; }
}
}
namespace SO19941925.Models
{
internal class User
{
public string Name { get; set; }
}
}
UPDATE: Based on the initial answer above, here is the code that actually solves the OP question:
store.DatabaseCommands.UpdateByIndex("Raven/DocumentsByEntityName",
new IndexQuery {Query = "Tag:Users"},
new ScriptedPatchRequest
{
Script = #"this['OAuthAccounts']['$type'] =
'System.Collections.ObjectModel.Collection`1[
[Friendorsement.Domain.Membership.IFlexOAuthAccount,
Friendorsement.Domain]], mscorlib';",
}, true
);
Here are two possible solutions:
Option 1: Depending on what state your project is in, for example if you are still in development, you could easily just delete that collection out of RavenDB from the Raven Studio and recreate all those User documents. All the new User documents should then have the correct class name and assembly and should then deserialize correctly. Obviously, if you are already in production, this probably won't be a good option.
Option 2: Depending on how many User documents you have, you should be able to manually edit each one to specify the correct C# class name and assembly, so that they will be deserialized correctly. Again, if you have too many objects to manually modify, this may not be a good option; however, if there are just a few, it shouldn't be too bad to open each one up go to the metadata tab and paste the correct value for "Raven-Entity-Name" and "Raven-Clr-Type".
I ended up doing this:
Advanced.DatabaseCommands.UpdateByIndex(
"Raven/DocumentsByEntityName",
new IndexQuery {Query = "Tag:Album"},
new []{ new PatchRequest() {
Type = PatchCommandType.Modify,
Name = "#metadata",
Nested= new []{
new PatchRequest{
Name= "Raven-Clr-Type",
Type = PatchCommandType.Set,
Value = "Core.Model.Album, Core" }}}},
false);