Summary and Question
I'm looking to generate code in C# to prevent significant repetition and wrap the Google APIs in a way like they do themselves, as stated on their .Net Client library page. Edit: Their generator is written in Python, apparently. I will continue to investigate other .Net options.
Where should I focus my attention, CodeDOM, Roslyn or something else? Should I not be considering Code Generation at all - and if so, what alternative track should I take to properly handle this situation?
Details
I am working on writing a wrapper for the Google .Net APIs to make a Google API library for PowerShell (for any and all Google APIs). I already have it working on three of the APIs, but since my project handles all of the authentication (and storage thereof) and other things like pagination, I have to basically wrap each API method call to work with my own authentication so that the user doesn't have to worry about it. This leads to a lot of repetitious coding encapsulating methods that already exist in the .Net Libraries:
public Data.Asp Get(string userKey, int codeId)
{
//I have to wrap their get method with my own using GetService(), for example
return GetService().Asps.Get(userKey, codeId).Execute();
}
Since this is all patterned on information that exists either through the Google Discovery API or through the underlying client libraries, I feel like there should be some way to generate the code and save my hands some trouble.
Some Background and Related Info
On the main page for the Google API .Net Client libraries it is stated:
The source code for the individual Google APIs is programmatically generated using the Discovery API.
I would like to do something similar, though I have no idea where to focus my time and research. I've looked up CodeDOM (and the inherent limitations), Roslyn as well as some differences between the two. I've also checked out the T4 Text Templates for Visual Studio.
To be clear, I am not looking to generate code at runtime as I would with something like Reflection, I am looking to generate bits of a library - though I'm not sure if I am looking for active or passive generation yet.
I work at Google on the .NET client libraries (among other things). Your question is pretty far reaching, but here is the general idea:
The metadata for describing "most" Google APIs is through a discovery document. That describes the methods and types the API has.
Client libraries for accessing Google's APIs then are generated, like you point out, from a Python library. (Using Django as a templating language, specifically.)
Once the code is generated for each Google API, we invoke MSBuild, package the binaries, and deploy them to NuGet.
As for your specific question about how to generate code, I would recommend you build two separate components. The first is something that will read and parse the discovery document, the second is the component that will emit the code.
For the actual code gen, here are some personal opinions:
The simplest thing to do would be to use a text-based templating language. (e.g. Django or just write your own.)
CodeDOM is an interesting choice, but probably much more difficult to use than you want. It is how Visual Studio does some of its codegen, e.g. you describe the code and CodeDOM will emit C#, VB, MC++ to match your desires. However, since you are only focusing on C#, the benefit of CodeDOM supporting multiple languages isn't useful.
Roslyn certainly is a cool, new technology, but that probably won't be of much use. I believe Roslyn has the ability to dynamically model code and round-trip the AST to disk. But that is probably overkill, since you aren't trying to build a general-purpose C# codegen solution, and instead just target generating code that matches the API discovery document.
So I would suggest a basic text-based solution for now, and see how far that can get you. If you have any other questions feel free to message me or log an issue on the GitHub issue tracker.
Related
We have a well established .NET framework based SAAS web application which supports Oauth2 to multiple providers, and we want to implement the same with SAML/SAML2. The current implementation of Oauth2 does all the necessary steps using resful http get and post calls plus the one redirect.
Looking at existing libraries, they all seem to want changes in "startup.cs" or wherever your startup code is and they add code to web.config. So they are building themselves into the core of your project. We don't like libraries that do that, we don't want to run code that isn't needed and our code has to pass some pretty deep auditing for defense apps, so we'd prefer not to use external libraries from nuget, although we would be prepared to include some libraries from MS if they do not interfere with the app infrastructure (startup and web.config).
Only a small minority of our customers will use SAML since most will use direct logins or Oauth2 providers.
So how can we add SAML with pure restful calls / redirects and not using third party libraries ? I hoped we could find examples where you build up a piece of XML and send it via http post or via a querystring, but after several hours I am not finding this.
TIA
SAML can actually be quite complicated to implement properly: you WILL want library support for this. Even more, I see this:
we don't want to run code that isn't needed [because] our code has to pass some pretty deep auditing for defense apps
Mostly, this is the right attitude from a security standpoint. There are too many examples now of a threat actor getting malicious code into upstream libraries... thankfully not as much in the .Net ecosystem, but enough generally to be a legitimate security concern. But for authentication specifically avoiding libraries is exactly the wrong approach. It's actually significantly worse for security to try to do this on your own.
In fact, using a library for authentication should make security auditors happier, because it's just so easy to implement authentication code that seems to work correctly — passes all your unit and integration tests and looks like it's working — where in fact you have subtle bugs that result in finding out a year later you were hacked six months ago. Using a purpose-built, battle-tested library helps you avoid that scenario.
I've been saying this since at least 2010. The idea is when a flaw is discovered (because they always exist) it's found on someone else's product first. But because they're using the same library a patch is created, distributed, and applied before your own product ever has a chance to be breached. Unfortunately, the reality is these audits are often as much about ticking the box as cheaply and painlessly as possible as they are about actually improving things, so I understand where it can be hard to get this through.
Personally, I have successfully implemented SAML authentication using the ITfoxtec library for two different apps. However, it's not something I work on often enough to have broad experience to comment on the relative quality of this option vs alternatives other than to say I was able to make it work.
The main thing for this library is the source code is available, and fairly easy to browse and understand, which should further make auditors happy. Moreover, because the source is available with a very permissive license you can fork the project and include the code directly as part of the main build. This should further make it easier to audit.
Just to add to what Joel said, I think you'll find it's more cost effective to use a 3rd party library (either open source or our commercial library) rather than implementing something yourself. Also, security is important, obviously, so it's much safer to use a battle-hardened, proven implementation that's been used in production by many others for many years.
I'm not sure about other libraries, but our SAML library for ASP.NET doesn't require any code in the start-up class or changes to web.config. Perhaps you're referring to .NET 6 etc where's it's common practice to hook into the dependency injection system in the start-up code. In our library, the SAML code is executed only when required as part of a SAML SSO flow so there's no additional overhead for non-SAML authentication.
The SAML protocol is more involved than OAuth2. I suggest taking a look at the SAML v2.0 specification documents to get a feel for the effort involved.
Environment
Windows 8.1 64bit
Microsoft Visual Studio
C#
What I'm trying to do
To implement a mini language in C#
My job in my company is to make a software which automates evaluation of the performance of company products. The evaluation involves opening valves, flowing chemicals, fetching signals from sensors, calculating some values, etc...
The issue is when a user wants to change the procedure and parameters like how long the reaction time is, the order of opening valves, whether reaction conduit is flushed out or not, the programmer has to change the source code and build another software every time the user requires it.
So I'm trying to implement a mini language in C#. A user can organize a procedure of measurement with the easy-to-understand language. The user writes a code in a txt file, my software reads it, parse each line of code, extract commands and parameters, call corresponding method in the software.
For example, a user writes a line of code like open_valve(3), my software reads the code and call an internal method send_commands_to_ADconverter(VALVE, 3) which sends appropriate commands to a target valve via an A/D converter connected to the computer. This is my conception.
However, due to lack of my programming skill, I have no idea where to start. What technology is available? What are the keywords to achieve the goal? How can I achieve this objective? Any help would be appreciated. Thanks.
Edit 1
c# - Adding scripting functionality to .NET applications - Stack Overflow doesn't refer to accessing external devices like sensors, valves via A/D converter which is crucial for my purpose, so it is unclear this is a duplicate question of the link above.
In order to create a language you need a "parser" of some sort. You will need to define a "grammar". Parsing your "progam" via the grammar will result in a structure that you can then call methods in your code that implement each feature of your language.
You are on a big learning curve here :) lots of strange things like EBNF. You will probably see lots of references to things like Gold and ANTLR. These are hugely capable but involve things like "compiler compilers" and add a level of complexity that can be confusing and require extra steps in you build pipeline.
Here are a couple of libraries I've used that allow you to define you grammar in c#.
https://github.com/picoe/Eto.Parse
https://irony.codeplex.com/
Irony is very good (I've used it for various projects) but hasn't been maintained for a while.
Eto.Parse is more recent. It also has built in parsers that allow you to create a Grammer by parsing BNF. This is very cool
If I understand, your goal is to parse a syntax written by your user and take a decision with what he typed.
I suggest you to use regular expression to match and split the user input.
There are several scripting languages which can be run directly from C#.
As your users doesn't seem to have programming knowledge it might help to use a verbose language like VBScript.
To run VBScript you can use the Scripting Control. See this question for samples: Use Microsoft Scripting Control to evaluate 'If' expressions (via c#)
IIRC the script control must be run as a 32bit application.
Well, the easiest option would be to use what you already have - C#.
You can compile C# code on the fly, and it's very easy and cheap with the new compiler, Roslyn. Then all you need to do is define your interface so that it's easy to use for the users, and put that code in a method you can call from your code. This allows you to easily support all the various development tools for C# as well, including Visual Studio/VS Code. In the simplest form, all you need to do is make your "script" implement some base class that has all the methods you want the users to have - there's certainly better approaches, but this one is very simple, which sounds like something you're looking for.
Of course, this only makes sense if you can trust your users. Making safely sand-boxed code in C# is a lot more complicated (though certainly possible), and you'd probably be better off using some ready scripting solution.
I have an online service for which I provide a RESTful API. This API is pretty neat and complete, but my clients would like to access it through an SDK. Now, my clients all have different needs in terms of languages: Go, Python, C#, you name it.
However, being lazy, I notice that the abstraction stays the same, and I have the same functions everywhere. Is there a way to automatize code generation for all of these SDKs, provided the design model is nice and clean? Would UML be useful for example? Or would I only need to create a C libraty matching the API calls and then use some SWIG magic to generate the bindings?
Technologically speaking, I use the Django Rest Framework for the API side, but that should not influence the question.
Of course you can use UML to document your REST API. As in REST it is all about resources and their CRUD methods, I would suggest a restrictive class diagram as a base of this documentation.
Here is an example with some ideas:
From here it is also easy to make an exporter and generate client APIs in any technology. Some UML parsing and selective generation. It's probably kind of time consuming, especially for the newbies, but relativelly straightforward.
However, this neat visual API-spec is already a great input for API-client developers.
UPDATE (after comments)
There are a lot of ways how you can do it in UML, depending on the concrete requirements.
My first idea is to create another package of classes (with stereotype REST-client) or so, that would be connected (via dependency) to corresponding methods thay can execute. Class's atts can be used to store additional info.
Alternatively you can use more illustrative approach and show rest-clients as UML actors. Here is how it looks like:
Note that these special elements (actors and rest-client classes) should be clearly separated in another package in the model and not mandatory displayed on the same diagram with resources. Traceability matrix (supported by some UML tools) is probably much better choice to specify this kind of supplementary information.
If you need more info, please tell me how exactly would you like to handle authentication and permissions.
Is there a way to use C# as a scripting language? I want users to be able to write a method body and work with objects that are passed from the main program to the script method.
And maybe is there something more universal out there? So I can support more than C#?
Check out scriptcs. Scott Hanselman has a great blog post about it here. It allows you to do scripting with C#, either by running arbitrary scripts as *.csx files, written with with your favorite text editor or by using the REPL. It also has integration with NuGet to allow you to easily pull down dependencies.
The ScriptCs.Core library can be used to host script execution in your own application, like OctopusDeploy has done.
The scripting engine is pluggable. As of now, it uses Roslyn as the C# scripting engine, but there's also work being done to use mono as an alternative. There's nothing stopping you from implementing f.ex. F# support.
To make bootstrapping of popular frameworks even easier, there's script packs that reduces the amount of code you have to write to get something running quickly.
To get you started, check out the samples repository.
I have an application that I have designed and this app has a pretty decent core dll that contains an API that my main view's exe uses. I would like to allow other developers to access this core dll as well but I don't want them to have as much access as me since it would be a security risk. What is the standard way of exposing my core dll? Are there any particular design patterns I should be looking at?
I'm using C#
Edit: my question was a little vague so here is some clarification
My program is deployed as a windows exe which references the core.dll. I want other people to create extensions which dynamically get loaded into my program at start up by loading dlls in the /extensions directory. The 3rd party dlls will inherit/implement certain classes/interfaces in my core.dll. I only want to give 3rd parties limited access to my core but I want to give my exe additional access to the core.
I should mention that this is the first time I have written a program that imports DLLs. Perhaps this whole method of allowing users to add extensions is wrong.
How do I modify/expose my API for
other developers?
To deliberately allow other developers to work with an API you've built touches on many things, which can be broken into two areas:
Resources (documentation, samples, etc) that makes it easier for them to understand (yes - basically an SDK).
Architecting, constructing and deploying your solution so that it's easy to actually work with.
Examples include:
By packing it in a way that suits re-use.
By using naming conventions and member names that others can easily follow.
Documentation, samples.
Providing the source code (as open source) if you're happy for them to modify it.
I would like to allow other developers
to access this core dll as well but I
don't want them to have as much access
as me since it would be a security
risk.
Ok, so this gets us right into the second area - the actual solution.
The problem you have is not a trivial one - but it's also quite do-able; I'd suggest:
Looking into existing material on plugins (https://stackoverflow.com/questions/tagged/plugins+.net)
Personally, I've found using attributes and Dependency Inversion to be a great approach.
There's also stuff like the Managed Extensibility Framework which you should consider.
The big issue you face is that you're into serious architecture territory - the decisions you make now will have a profound impact on all aspects of the solution over time. So you might not be able to make an informed decision quickly. Still - you have to start somewhere :)
The "design patterns" in terms of an API are more related to things like REST.
I don't want them to have as much
access as me since it would be a
security risk
Then i would (for the sake of maintenance), layer on top of the core DLL extra logic to prevent this.
The thing is, the "clients" call the API, not the Core DLL.
"How" the API accesses the Core DLL is under your full control. Just only expose operation contracts that you wish.
Since you're using C#, I would look at Microsoft's Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries and use FxCop to in-force many of them (latest version here). This won't be all you'll likely need, but it would help put you in the right direction.
Also, take a look at the freely available distillation of Framework Design Guidelines by the same author.