I have a proto file that describes a message:
message SharedData
{
string instanceName = 1;
string userName = 2;
bytes data = 3;
uint64 data_size = 4;
}
When serializing using c++, I can use
SharedData data_instance;
std::string string_data = someObject.SerializeAsString();
data_instance.set_data(stringData);
to save the object in the data field.
I'm trying to do the same in C#. The data field's type is ByteString. There is no SerializeAsString method available for objects. I tried using someObject.ToByteString() but the data is not interpreted correctly on the other side.
Is there a C# equivalent to SerializeAsString()?
Thanks
Solved it. The problem was that I kept an instance as a public variable and updated it. The correct way is to create a new object each time and use
someObject.ToByteString()
on the new object.
Now I'm trying to find a way to deserialize it in C#...
Related
I use Python.Net for C# interaction with Python libraries. I solve the problem of text classification. I use FastText to index and get the vector, as well as Sklearn to train the classifier (Knn).During the implementation, I encountered a lot of problems, but all were solved, with the exception of one.
After receiving the vectors of the texts on which I train Knn, I save them to a separate text file and then, if necessary, use it.
string loadKnowVec = File.ReadAllText("vectorKnowClass.txt", Encoding.Default);
string[] splitKnowVec = loadKnowVec.Split('\r');
splitKnowVec = splitKnowVec.Where(x => x != "").ToArray();
for()
{
keyValues_vector.Add(float.Parse(splitKnowVec[i], NumberFormatInfo.InvariantInfo), 1);
}
dynamic X_vec = np.array(keyValues_vector.Keys.ToArray()).reshape(-1, 1);
dynamic y_tag = np.array(keyValues_vector.Values.ToArray());
dynamic neigh = KNN(n_neighbors: 3);
dynamic KnnFit = neigh.fit(X_vec, y_tag);
string predict = neigh.predict("0.00889");
MessageBox.Show("Скорее всего это: "+predict);
During the training of the classifier, I encountered such a problem that from c# to python, it is not values with the float type, but the value of System.Single[].
Python.Runtime.PythonException: "TypeError : float() argument must be a string or a number,
not 'Single[]'
The stored value, at this point, of dynamic X_vec is "System.Single[]".(I think that's exactly the problem)
2.At first I tried to manually set the values of X_vec, but the error and its values were the same.
The first idea was to change the array type using the numpy library, but it didn't help, it also gave out "".
dynamic Xx = np.array(X_vec, dtype: "float");
dynamic yY = np.array(y_tag, dtype: "int");
Next, it was tried to create an empty array in advance and load specific values into it before changing the data type, but this also did not work.
Perhaps I do not understand the principle of the formation and interaction of the MSVS19 IDE and the python interpreter.
I solved this issue for a couple of days and each time I thought it was worth reading the documentation on python.net .
As a result, I found a solution and it turned out to be quite banal, it is necessary to represent X_vec not as a float[] , but as a List<float>
List<float> vectors = keyValues_vector.Keys.ToList();
List<int> classTag = keyValues_vector.Values.ToList();
dynamic a = np.array(vectors);
dynamic X_vec = a.reshape(-1, 1);
dynamic y_tag = np.array(classTag);
The Proto3 C# Reference contains the following text:
Wrapper Type Fields
Most of the well-known types in proto3 do not affect code generation,
but the wrapper types (StringWrapper, Int32Wrapper etc) change the
type and behaviour of the properties.
All of the wrapper types that correspond to C# value types
(Int32Wrapper, DoubleWrapper, BoolWrapper etc) are mapped to
Nullable<T> where T is the corresponding non-nullable type. For
example, a field of type DoubleValue results in a C# property of type
Nullable<double>.
Fields of type StringWrapper or BytesWrapper result in C# properties
of type string and ByteString being generated, but with a default
value of null, and allowing null to be set as the property value.
For all wrapper types, null values are not permitted in a repeated
field, but are permitted as the values for map entries.
When trying to generate a .cs file from a .proto file, If I try to declare a field as Int32Wrapper in the .proto file, protoc.exe throws an error about Int32Wrapper not existing.
syntax ="proto3";
package prototest;
import "MessageIdentifier.proto";
message TestMessage {
string messageTest = 1;
fixed64 messageTimestampTicks = 2;
uint32 sequenceNumber = 3;
MessageUniqueID uniqueID = 4;
Int32Wrapper nullableInt = 5;
}
It seems there is some additional step that is missing here, does anyone know how to enable these types?
I will try to improve Nick's answer as it hasn't helped me.
grpc compiler claimed that he has no information on google.protobuf.Int32Wrapper type. I have found it is actually called google.protobuf.Int32Value (https://github.com/protocolbuffers/protobuf/blob/48234f5f012582843bb476ee3afef36cda94cb66/src/google/protobuf/wrappers.proto#L88), though google really calls it Int32Wrapper.
So the code that helped me was the following:
...
import "google/protobuf/wrappers.proto";
...
message TestMessage {
...
google.protobuf.Int32Value nullableInt = 5;
}
Other links:
C# lib source - https://github.com/protocolbuffers/protobuf/blob/48234f5f012582843bb476ee3afef36cda94cb66/csharp/src/Google.Protobuf/WellKnownTypes/Wrappers.cs#L781
C# doc - https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/well-known-types/int32-value
In respect that https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto
You need to import google/protobuf/wrappers.proto in order for this to work.
syntax ="proto3";
package prototest;
import "MessageIdentifier.proto";
import "google/protobuf/wrappers.proto";
message TestMessage {
string messageTest = 1;
fixed64 messageTimestampTicks = 2;
uint32 sequenceNumber = 3;
MessageUniqueID uniqueID = 4;
google.protobuf.Int32Value nullableInt = 5;
}
You can then use it as an int? ,eg nullableInt.HasValue and nullableInt.Value
I am trying to deserialize data from Mongodb to MyClass that has been created by protoc (version 3).
I am using MongoDB.Driver (version 4)
BsonSerializer.Deserialize(result, MyClass.GetType());
Which uses XmlSerializer to deserialize the data.
The problem lies in how protobuf represents its collections.
A List is created as Google.Protobuf.Collections.RepeatedField
with no setter. XmlSerializer cannot deserialize fields with no setters.
How can I solve this problem in a way that it doesn't become an ugly hack?
Options I have come up with:
Create my own Serializer that MongoDB can use to handle properties with no setter.
Create my own class generator that adds a public setter.
Create a new class as a mapper between MongoDB's serialized data and protobufs class.
Option 1 seems daunting, option 2 makes me create a new fork, which is a pain to maintain. Option 3 is the easiest by far, but also messy, I'd prefer not to create another layer of complexity if there are better ways to deal with this.
So my question is:
Is there any other ways to solve this problem? Is there anything I am missing that is already built-in or maybe I am missing something trivial?
Edit:
This is a snippet of what is getting generated by protoc verison 3:
/// <summary>Field number for the "Recipients" field.</summary>
public const int RecipientsFieldNumber = 5;
private static readonly pb::FieldCodec<string> _repeated_recipients_codec =
pb::FieldCodec.ForString(42);
private readonly pbc::RepeatedField<string> recipients_ = new pbc::RepeatedField<string>();
public pbc::RepeatedField<string> Recipients {
get { return recipients_; }
}
It comes from this proto-file:
syntax = "proto3";
package DataModels.Data;
message MailTemplateMessage {
string UUID = 1;
string SubjectLine = 2;
string Body = 3;
string Sender = 4;
repeated string Recipients = 5;
}
So I have a small problem:
A message is sent using MQTT, it consists of a series of serialized objects using protobuf-net in C# (I can't modify this code, but I have access to the source). At the other end i receive the serialized objects in Java, the problem is I can't seem to be able to deserialize the objects using protobuf, if anyone ever had this problem and solved it, please help :)
Example of object from C#:
using ProtoBuf;
namespace Concentrator.Services
{
[ProtoContract]
public class MeterID
{
private byte[] _id;
[ProtoMember(1)]
public byte[] ID
{
get { return _id; }
set { _id = value.Length == 16 ? value : null; }
}
[ProtoMember(2)] public string MeterType;
}
}
My attempt at recreating the same object in Java (the .proto file):
syntax = "proto2";
package mqtt.entity;
option java_package = "mqtt.entity";
option java_outer_classname = "ProtoMeter";
message Meter {
optional bytes ID = 1;
optional string MeterType = 2;
}
message MeterID {
repeated Meter mid = 1;
}
A solution to this example would be a huge help, Thanks a lot.
The code where the object is deserialized in C#:
var ms = new MemoryStream(data, 7, data.Length - 9)
var res = Serializer.Deserialize<List<MeterID>>(ms);
this works in C#, I'm trying to achieve the same thing in java
The message in your C# code matches just:
message MeterID {
optional bytes ID = 1;
optional string MeterType = 2;
}
There is no need for a 2-level model (unless you're using *WithLengthPrefix in the C# code). You can also get that output by using:
var proto = Serializer.GetProto<MeterID>();
With your edit, a List<MeterID> could be mapped as
message List_MeterID {
repeated MeterID items = 1;
}
to be used in combination with the previous MeterID fragment. Which is what you have in the question. So it comes down to "what currently happens?".
try regenerate proto-file by GetProto<T>
I have a question that's so simple I cannot believe I can't answer it myself. But, there you go.
I have a large-ish static list (of cities, latitudes and longitudes) that I want to use in my Windows Phone 7 Silverlight application. There are around 10,000 of them. I'd like to embed this data statically in my application and access it in an array (I need to cycle through the whole list in code pretty regularly).
What is going to be my most effective means of storing this? I'm a bit of an old school sort, so I reckoned the fastest way to do it would be:
public struct City
{
public string name;
public double lat;
public double lon;
};
and then...
private City[] cc = new City[10000];
public CityDists()
{
cc[2].name = "Lae, Papua New Guinea"; cc[2].lat = 123; cc[2].lon = 123;
cc[3].name = "Rabaul, Papua New Guinea"; cc[3].lat = 123; cc[3].lon = 123;
cc[4].name = "Angmagssalik, Greenland"; cc[4].lat = 123; cc[4].lon = 123;
cc[5].name = "Angissoq, Greenland"; cc[5].lat = 123; cc[5].lon = 123;
...
However, this bums out with an "out of memory" error before the code actually gets to run (I'm assuming the code itself ended up being too much to load into memory).
Everything I read online tells me to use an XML resource or file and then to deserialise that into instances of a class. But can that really be as fast as using a struct? Won't the XML take ages to parse?
I think I'm capable of writing the code here - I'm just not sure what the best approach is to start with. I'm interested in speed of load and (more importantly) run time access more than anything.
Any help very much appreciated - first question here so I hope I haven't done anything boneheaded.
Chris
10,000 structs shouldn't run out of memory, but just to make sure, I would first try turning your struct into a class such that it uses the heap instead of the stack. There is a strong possibility that doing that will fix your out of memory errors.
An XML file stored in isolated storage might be a good way to go if your data is going to be updated even every once in a while. You could pull the cities from a web service and serialize those classes to the Application Store in isolated storage whenever they get updated.
Also, I notice in the code samples that the cc array is not declared static. If you have a few instances of CityDists, then that could also be bogging down memory as the array is getting re-created every time a new CityDists class is created. Try declaring your array as static and initializing it in the static constructor:
private static City[] cc = new City[10000];
static CityDists()
{
cc[2].name = "Lae, Papua New Guinea"; cc[2].lat = 123; cc[2].lon = 123;
cc[3].name = "Rabaul, Papua New Guinea"; cc[3].lat = 123; cc[3].lon = 123;
cc[4].name = "Angmagssalik, Greenland"; cc[4].lat = 123; cc[4].lon = 123;
cc[5].name = "Angissoq, Greenland"; cc[5].lat = 123; cc[5].lon = 123;
...
If loading an xml doc from the xap works for you..
Here's a project I posted demonstrating loading of an xml doc from the XAP via XDocument/LINQ and databinding to a listbox for reference.
binding a Linq datasource to a listbox
If you want to avoid the XML parsing and memory overhead, you could use a plain text file for storing your data and use the .Net string tokenizer functions to parse the entries e.g. use String.Split()
You could also load the file partially to keep memory consumption low. For example, you load only k out of n lines of the file. In case you need to access a record that is outside the currently loaded k segments, load the appropriate k segments. You could either do it the old school way or even use the fancy Serialization stuff from .Net
Using a file such as XML or a simple delimited file would be a better approach as others have pointed out. However can I also suggest another change to improve the use of memory.
Something like this (although the actual loading should be done using an external file):-
public struct City
{
public string name;
public string country;
public double lat;
public double lon;
}
private static City[] cc = new City[10000];
static CityDists()
{
string[] countries = new string[500];
// Replace following with loading from a "countries" file.
countries[0] = "Papua New Guinea";
countries[1] = "Greenland";
// Replace following with loading from a "cities" file.
cc[2].name = "Lae"; cc[2].country = contries[0]; cc[2].lat = 123; cc[2].lon = 123;
cc[3].name = "Rabaul"; cc[3].country = countries[0]; cc[3].lat = 123; cc[3].lon = 123;
cc[4].name = "Angmagssalik"; cc[4].country = countries[1]; cc[4].lat = 123; cc[4].lon = 123;
cc[5].name = "Angissoq"; cc[5].country= countries[1]; cc[5].lat = 123; cc[5].lon = 123;
}
This increases the size of the structure slightly but reduces the memory used by duplicate country names signficantly.
I hear your frustration. Run your code without the debugger, it should work fine. I'm loading 2 arrays in under 3 seconds, each with over 100,000 elements. Debugger reports "Out of Memory", which is simply not the case.
Oh and you are correct about the efficiency. Loading the same information from an XML file was taking over 30 seconds on the phone.
I don't know who was responding to your question but they really should stick to marketing.