We Keep Coding

How do I use GetConsoleDisplayMode in C#?

Background
I'm working with the console and PInvokes in C#, just to see what magic you can do when you're not limiting yourself to the .NET framework. I stumbled upon a function in Kernel32 called GetConsoleDisplayMode that basically lets you see if the console is running in windowed, full-screen or hardware full-screen mode. The documentation for this function can be found here.
GetConsoleDisplayMode takes only one parameter: an LPDWORD in which the display mode can be stored. The function returns a bool, indicating the success of the call. According to this documentation, an LPDWORD is a pointer to a DWORD, which in turn is simply a 32-bit unsigned integer (uint32 in C#).
So, from what I understand from the docs is that this should work:
[DllImport("kernel32", EntryPoint = "GetConsoleDisplayMode", ExactSpelling = false, CharSet = CharSet.Unicode, SetLastError = true)]
public static extern bool GetConsoleDisplayMode(ref uint lpModeFlags);
static void Main(string[] args) {
uint mode = 0;
var success = GetConsoleDisplayMode(ref mode);
Console.WriteLine(success);
Console.WriteLine(mode);
Console.ReadKey(true);
}
However, the function always returns false!
What I have tried
Of course I have tried to modify the code, in hopes of getting it to work properly.
A call to GetLastError() returns the most curious error code 3221684230 of which even Google and Microsoft Documentation have no knowledge (nor is it uint.MaxValue or int.MaxValue, I have checked). This does not get me a step further, so I dropped this option.
Another thing I've tried (in the hopes that I had read or understood the documentation incorrectly) is changing ref to out, which passes an uninitialized variable for the function to initialize for me:
[DllImport("kernel32", EntryPoint = "GetConsoleDisplayMode", ExactSpelling = false, CharSet = CharSet.Unicode, SetLastError = true)]
public static extern bool GetConsoleDisplayMode(out uint lpModeFlags);
static void Main(string[] args) {
var success = GetConsoleDisplayMode(out var mode);
if (!success) {
Console.WriteLine(Kernel32.GetLastError());
}
Console.WriteLine(success);
Console.WriteLine(mode);
Console.ReadKey(true);
}
Alas, this too was unsuccessful and, interestingly, produces the same error code.
My question(-s)
Why does my call to GetConsoleDisplayMode always return false, why does GetLastError return such an obscure error code and most importantly: how do I find the error that's bugging me and/or get this to work?
More information
I am running Windows 10 x64 build 1709 (Fall Creators Update).
Maybe this excerpt from John Paul Mueller's book on the Windows API and the .NET framework can be of help (that's about the best documentation of GetConsoleDisplayMode that I can find).

How to fix Ntdll.dll APPCRASH with Hardware ID Extractor?

I am using the Hardware ID Extractor library (written in Delphi) from http://www.soft.tahionic.com/download-hdd_id/index.html with the purpose of generating unique system fingerprints.
The library is really good and unlike anything else I have seen on the market, but the main issue with it is that it's unstable when running with .NET applications, meaning that it sometimes works, other times it works for a few function calls then the main application crashes, or most of the time the application instantly crashes when a dll function is being called.
As the developer of the library pointed out (in the last support e-mail that I have received), the fault is with ntdll.dll, as I have seen that for myself:
Following is a link to a demo project I have created with the purpose of testing the dll functions (so to make sure that nothing else interferes, the demo app does that and only that- it calls the dll functions).
http://www.mediafire.com/download/1jws7zh9218v88a/HardwareIdExtractDllTest.zip
The archive contains the Visual Studio 2013 project with source code and a compiled demo application which looks like this:
The list of functions contained by the dll can be found here:
http://www.soft.tahionic.com/download-hdd_id/hardware%20id%20programming%20source%20code/exported%20functions%20for%20non-Delphi.html
If anyone has the knowledge and is willing to test the demo project/application to make tests or personal opinions in regard to the issue, and then share a possible solution with me, I would be grateful.
Please let me know if there's anything I can do to further assist in solving this issue if you think there's anything that can be done about it.
EDIT: This is how I am declaring the dll functions
[DllImport("HardwareIDExtractorC.dll")]
private static extern bool EnterKey(int key);
[DllImport("HardwareIDExtractorC.dll")]
private static extern bool IsCPUIDAvailable();
[DllImport("HardwareIDExtractorC.dll")]
private static extern int GetCPUCount();
[DllImport("HardwareIDExtractorC.dll")]
private static extern byte CoreNumber2CoreMask(byte cpuCore);
[DllImport("HardwareIDExtractorC.dll", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi, EntryPoint = "GetCPUID")]
[return: MarshalAs(UnmanagedType.LPStr)]
private static extern string GetCPUID(byte cpuCore);
[DllImport("HardwareIDExtractorC.dll", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi, EntryPoint = "GetCpuIdNow")]
[return: MarshalAs(UnmanagedType.LPStr)]
private static extern string GetCpuIdNow();
[DllImport("HardwareIDExtractorC.dll", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi, EntryPoint = "GetIDESerialNumber")]
[return: MarshalAs(UnmanagedType.LPStr)]
private static extern string GetIDESerialNumber(byte driveNumber);

The functions that fail are the ones that return string. Such a p/invoke marshals the return value as a pointer to null terminated character array, and then calls CoTaskMemFree on the raw pointer returned by the unmanaged code.
With probability close to 1 that string was not allocated on the COM heap and so the error is likely in the C# p/invoke declarations. That an access violation then arises in ntdll is quite plausible.
So, to make progress you need to fix the p/invoke calls. I cannot tell you how to do so because you have not shown the unmanaged function declarations. Or, more importantly, how the memory is allocated.
There are some clues at the documentation
procedure ReleaseMemory (P: PAnsiChar); stdcall;
I think this tells us that the strings returned by the DLL must be deallocated by the DLL, by calling this function. Presumably because they were allocated by the DLL's heap allocator.
So, use IntPtr for the return type of the functions that return text. Call Marshal.PtrToStringAnsi to convert to a C# string. And then pass the pointer to ReleaseMemory.
For example:
[DllImport(DllPath, CallingConvention = CallingConvention.StdCall)]
private static extern IntPtr GetCPUID(ushort CoreMask);
[DllImport(DllPath, CallingConvention = CallingConvention.StdCall)]
private static extern void ReleaseMemory(IntPtr P);
....
IntPtr ptr = GetCPUID(CoreMask);
string cpuid = Marshal.PtrToStringAnsi(ptr);
ReleaseMemory(ptr);

Writing a lite-weight libVLC wrapper. Need help (Newish to P/Invoke)

I'm writing a super-simple ultra-lite weight .Net wrapper for the LibVLC media library since the only things I need access to are the ability to play, pause and stop media files. I've posted a couple of questions on this and gotten some answers but unfortunately I'm just left with more questions.
We'll start from the top and work down.
The documentation first states I have to initialize VLC with a call to the function with this specification:
libvlc_instance_t* libvlc_new (int argc, const char *const *argv)
for which I have the defined the following method:
[DllImport("libvlc", EntryPoint = "libvlc_new",
CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr NewCore(int argc, IntPtr argv);
And I'm calling the function like this:
private IntPtr Instance;
this.Instance = DGLibVLC.NewCore(0, IntPtr.Zero);
I have tried it several different ways. Initially I did not know about the CallingConvention which was leading to an unbalanced stack which brought me here in the first place. That issue was resolved and the method has gone through several iterations, none of which have proved successful, by which I mean IntPtr is always 0 after the method call. I've tried it like it is above, with the second argument being String[] argc, [MarshalAs(UnmanagedType.LPArray, ArraySubType = UnmanagedType.LPStr)] string[],
I've tried having it return to a Long (which actually resulted in the Long having a value in it), but nothing so far has worked correctly.
Does anyone know the correct way to call this function from the LibVLC DLL Library?
EDIT: On a suggestion I tried calling the error message function of the library:
Specification:
const char* libvlc_errmsg (void)
Implementation:
[DllImport("libvlc", EntryPoint = "libvlc_errmsg",
CallingConvention = CallingConvention.Cdcel)]
public static extern string GetLastError();
Call:
Console.WriteLine(DGLibVLC.GetLastError());
Result:
Null
The documentation states it will return Null if there is no error. This must indicate that the initial function call NewCore was working correctly but something is still going wrong somehow.
To be cover all bases I checked that the DLLs match the documentation, they do. 2.0.6.0. The documentation I am referencing is here.
EDIT: I can confirm there is no error. When using an initialized to zero long variable to store the result of NewCore I can see it returning something. What I am doing wrong here is where I am trying to store the pointer being returned by the unmanaged function that returns the pointer to the object. How do I store the pointer to the opaque structure reference being passed back?

It doesn't have anything to do with the way you call the function. You cannot get anywhere when you get IntPtr.Zero back from libvlc_new(). It means "there was an error". You'll need to focus on error reporting first, call libvlc_errmsg() to try to get a description for the problem.

So after much looking around and asking questions I've come full circle.
I looked deeply into LibVLC.Net and found how they were importing the DLL functions and adapted what they did to my own wrapper and it worked.
To summarize:
There are some Win32 API functions declared in the code at the start:
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SetDllDirectory(string lpPathName);
[DllImport("kernel32", SetLastError = true)]
private static extern IntPtr LoadLibrary(string lpFileName);
[DllImport("kernel32", CharSet = CharSet.Ansi, ExactSpelling = true, SetLastError = true)]
private static extern IntPtr GetProcAddress(IntPtr hModule, string procName);
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool FreeLibrary(IntPtr hModule);
that handle providing a handle to a dll and setting a directory search path.
I don't know exactly what it all means but when you initialize the LibVLC.Net library (the primary object) it loads pretty much EVERY function like so:
m_libvlc_media_player_new = (libvlc_media_player_new_signature)LoadDelegate<libvlc_media_player_new_signature>("libvlc_media_player_new");
That delegate is defined here like so:
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
private delegate IntPtr libvlc_media_player_new_signature(IntPtr p_instance);
//==========================================================================
private readonly libvlc_media_player_new_signature m_libvlc_media_player_new;
//==========================================================================
public IntPtr libvlc_media_player_new(IntPtr p_instance)
{
VerifyAccess();
return m_libvlc_media_player_new(p_instance);
}
And it has a public function that calls the delegate once defined.
I simply stripped down the function that defines the library instance and imported only the functionality I needed.
Thanks very much to everyone who was so patient in helping me along. I likely wouldn't have been able to come to a solution without your help.
EDIT: Okay so it wasn't that. It was the location of the LibVLC Plugin Directory. So it was something stupid -.-;

DllImport user32 vs user32.dll

What is the difference between the usages of DllImport here? Specifically, does "user32" just mean "user32.dll", or does it mean "user32.lib" or something else?
[DllImport("user32")]
protected static extern int GetKeyboardState(byte[] pbKeyState);
[DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
protected static extern short GetKeyState(int vKey);
You can probably ignore the CharSet and CallingConvention.
If they are the same, I can rewrite this to be more consistent, but if not, I don't want to have a bunch of problems with it.

In this example, there is no difference. The .dll extension will automatically be appended to "user32" to create "user32.dll". However, this is not always the case. If the library file name contains a period, the .dll extension will not be automatically appended.
Some examples:
[DllImport("user32")] --> Resolves "User32.dll". Correct.
[DllImport("user32.dll")] --> Resolves "User32.dll". Correct.
[DllImport("mylib.version5")] --> Resolves "mylib.version5". Incorrect
[DllImport("mylib.version5.dll")] --> Resolves "mylib.version5.dll". Correct.

On Windows there is no difference, the import will be performed successfully if you omit the extension. Normally omitting the extension is desired when running Mono with the <dllmap> configuration section, where the P/Invoke runtime will look for aliases.

What exactly is this API and what is it doing?

I know you are gonna hate me for that kind of question. But could somebody tell me what the following code is doing?
I mean there are some libraries loaded, i get that. plus there are some methods, still I don't get it.
F.e.:
[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
Here is the code:
private static class API
{
[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern IntPtr SetWindowsHookEx(
int idHook,
HookDel lpfn,
IntPtr hMod,
uint dwThreadId);
[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool UnhookWindowsHookEx(IntPtr hhk);
[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern IntPtr CallNextHookEx(
IntPtr hhk,
int nCode,
IntPtr wParam,
IntPtr lParam);
[DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern IntPtr GetModuleHandle(
string lpModuleName);
}
You do not have to explain it to me line for line. At least give me some reference where I can read it up, please.
Thx in advance!

This code is using P/Invoke to allow C# code to call several Win32 API functions related to Windows Hooks.
The posted code only defines the methods; it doesn't call them, so it doesn't do anything by itself. It just allows you to use the methods from other parts of your code.

Here's an older MSDN article explaining P/Invoke and what's going on. Hopefully this helps you.
What the code is doing is allowing your managed C# code to call unmanaged Win32 API functions.
Here's also a tutorial on MSDN that walks you through the P/Invoke process of creating code like your question has.

DllImport is used to call unmanaged code/API in .Net/Managed code. All the code you've posted is trying to work with the window object of Win32 API.
References:
DLLImport
Win32 API
Win32 API to .Net API map

Take a look at this. Your program somewhere is installing a hook into windows hook chain to monitor some events.
The dllimport attribute itself lets the program to invoke win32 api functions like the previous answer mentions.