I'm trying to program an AI for online games which works by grabbing a screen region of the game area, analyzing the pixels, and then responding with a sequence of keypresses. Basically, it goes like this:
Program grabs picture of current frame
Program analyzes pixel data and plans accordingly
Program sends firefox appropriate keypresses to respond.
I'm currently having problems at step 3 - sending the keypresses. Since I'm building an AI for games, it must send the keypresses realtime. However, I'm noticing with SendKeys.Send()...
There's an apparent delay between when I make the request and when firefox executes it
It's unpredictable and slow, the keypresses are almost never executed on time
Movement is slow and choppy, because there are large gaps in between each processed keypress
Is there any way to predictably send keypress events to a program (such as firefox) realtime? Like when I think about simulating keypresses with a program, I'm thinking of a command that would do the exact same thing as when I press a key on my keyboard, not an unreliable and delayed method to send keypresses to whatever is currently focused. It would be great if anybody had an easy solution to this problem, or maybe any input on what I can do differently? Thanks!
The Windows Input Simulator wrapper library uses SendInput rather than SendKeys.
SendInput is generally the preferred method to send keystrokes and
mouse clicks because of its superior speed and reliability. Under most
conditions, SendInput is nearly instantaneous, even when sending long
strings. Since SendInput is so fast, it is also more reliable.
AutoHotKey Send
Related
I'm going to be very specific.
This is what i want to do in windows:
Write code that makes the keyboard to send me characters, i am interested in the time it takes the keyboard to send a character signal.
Find a way of ensuring that this code gets the priority it is undesirable to have it queued or interrupted by the OS
Find a way of reading keyboard status signals
i have been reading a lot, all i am getting is how to simulate a keyboard..i just need to be pointed in the right direction
The most common option is using Windows hook - or WH_Keyboard or WH_JOURNALRECORD. The difference is that WH_JOURNALRECORD does not require a separate DLL.
Or you can write your own keyboard driver. It may be implemented in 2 ways: your own full keyboard driver instead of the standard driver or an additional filter driver .
And finally you can write a rootkit.
It can be implemented in user mode by intercepting csrss.exe process.
As I am not sure what your purpose is, I am not going into more details.
So, Unity does not do a lot of rhythm games on android. I decided to find out why, and program one as assignment (the basics of it anyway). My most important hurdle is user input. As we know that input is based on frame rate in unity, and a music game (i assume) would prefer the smallest possible delay between button press and action.
If we look at music, at around 15 to 20ms of delay, the human ear hears something is "off beat".
I heard Android Unity games run at 30FPS (since 60FPS sucks the battery dry), simple math indicates: 1000/30 = 33ms per frame. Calculating in the 15ms we can probably not notice, we are at 18ms of possible disaster. assuming we always reach this 30FPS at any given moment.
When i get input from a user, i can play a sound on that input on the exact same frame. However, we could be 18ms off.
Now there is a way to get DIRECT controls from mouse and keyboard, which uses OnGui() instead of Update(), to get events of the keyboard or mouse clicks on the spot. The problem is, android probably doesn't work with this, (this doesn't work for gamepads either) and the methods sounds downright strange, especially when we try to play sounds from the OnGui() method.
My question:
What would you do, and why? Should we just accept the possible 18ms off, and assume we reach the 30FPS, or should we look for a reliable way to get input directly, instead of waiting for an update to come by?
Thanks for any insight you can give me, i have not found any articles on this that are useful just yet.
-Smiley
EDIT
I just did some basic testing with a metronome, running at 100FPS in the editor (which should be 10ms per frame) tapping my spacebar on a metronome inside unity. The results i got were just horrible.
Tapping rapidly: I get as close as 20ms to my metronome tick, but nothing closer.
Tapping on the beat: I got at least 200ms off my target tick. Unless i am confused with this rhythm, this is just wrong.
Currently i use Debug.Log to get my test data to the log. Can anyone please confirm for me if this may be the cause (causes some long delay? i know debug isn't that optimized), or is the timing actually that bad on it?
Thanks in advance,
-Smiley
First, I'd like to add a few things to your comments and analysis:
There is a hell of a lot more to measuring the latency between the tactile input and what your eyes perceive.
Probably the biggest one I'm seeing missing in your tests is the latency between the graphics card and the PC monitor you're testing on. Many common LCD monitors these days have a latency of between 15-30ms in processing lag. I don't know how much this relates to mobile screens and hardware, but I would suggest you take the time to perform additional tests on your target hardware before drawing further conclusions.
To more directly answer your question:
I would continue to use Unity however I would continue to research the best methods of taking the input and feeding it back to the player as fast as possible. In the above comments #rutter has pointed you to what appears to be a pretty good thread on the issue.
Of specific note I would look at using FixedUpdate() to decouple the framerate of the game from the input processing speed.
I think it is also worth putting time into researching the psychology of the perception of latency. For example, if your game is a sort of Guitar Hero game of matching the playing song, you could simply take into account the lag you know is there and in your game logic take that into account when checking input.
I think you are over-complicating this, and that the accuracy issue is no where near as bad as you think.
People usually hit the buttons a little early in order to sync what they are seeing and hearing.
It also depends alot on if you have some kind of scrolling display that they are trying to match up to... if the display is scrolling smoothly at 30fps (without big jumps) they they are still able to make their timing presses fairly accurate.
I would surmise that although people can hear when their timing is off, their actual timing of hitting the buttons at exactly the right time is not that accurate anyway.
Here is one other simple solution... which I think is what rock band and guitar hero often do...
You start playing the note/sound at the correct time anyway.... then change it to a broken sound if you detect they missed it or goofed up.
I have a software installed on my system which basically peeks my activity during working hours. It send a report saying how much time i was away from my system etc.
Since i m .net c# developer hence trying to break this system through my programming skills.
I have basically written up a windows form application which opens up an notepad for me and start typing some random characters into it. simultaneously it also performs some random clicks and moves the mouse cursor to give a feel that some one is on system and working even when i am not :)
I also use "SetThreadExecutionState" Pinvoke calls to keep system awake.I have also observed the User idle time and last interactivity time through "GetLastInputInfo" PInvoke system call while my mouse/keyboard are in automation process and it is also fine.
But this is not working for me. It works perfectly till 5 mins and then somehow it detects that user is not present on system.
any suggestions for the same.
Get a decompiler, crack-open the spying application and see how it is working inside. Only knowing the internal workings, you can create a "workable" any-spying program. But I would not bother with emulation of the activity, but rather spoof the way it reports to the server.
However, depending on where you live, it might be illegal to do these things and I would not recommend cheating your employer. And if spying gets on your nerves I'd also suggest to brush up your CV and start searching for a new job.
I am building a small MIDI composer application, using the Stanford Library. It is almost complete, except for a small problem: sound vanishes after the NoteOn command, but I want notes to keep on playing until I release the button.
With MIDI, you should hear the note continue forever until you send a NoteOff command. If you're hearing a note begin but then fade out (without having sent a NoteOff message), then it might be that the channel is set to an instrument that naturally fades out on its own - like a marimba or steel drum sound.
That would be down to the MIDI device that you are addressing. You might be able to tell it to increase the sustain level of the envelope via MIDI NRPN or SysEx, but this would be device specific. Generally a string or pad voice will maintain a high sustain, whereas a piano or other percussive sound will consistently fade to zero regardless of sustain level.
So, you are saying that when you play a note it decays normally (as if you head down a piano key) and you don't want that?
That is a feature of the synthesizer and has nothing to do with MIDI. You will need to pick a patch that doesn't do this. Organ patches should work fine.
If instead you are saying that your note stops immediately (as if you hit the key and immediately released) then you have something sending a note-off command or a note command with 0 velocity. Check to make sure you are in fact sending 7-bit values for velocity and what not.
i have an application that records a date/time when the space bar is pressed
unfortunately, the window doesn't always have focus, for various reasons. eg virus checker popup, itunes update, windows update, etc
so what i though would be a cool idea is to use a joystick button, then regardless of which window has focus, i can always detect the button press event and record the date/time it was pressed.
now i had a look at a few game tutorials, but they seem to have the joystick polling tied into the screen refresh event, because in a game i guess if the computer is slow and the screen doesn't refresh often, having the joystick button pressed is going to make no difference anyway.
That doesn't really suit my need, so i am looking at running a thread that will poll the joystick every so often
My questions are
is a background thread the best solution
how often should i poll the joystick
what kind of timer should i use to determine the polling frequency.
I would like the responses to be at least equivalent to a keyboard. also need to make sure it doesn't auto-repeat if trigger is held down.
any sample code (C#) appreciated
thanks
alex
If you want to go for polling the joystick just because you are not able to capture the keyboard event, I think you need to re-work your strategy.
Even if your form doesn't have the focus there are ways that you can make sure you know of the keyboard state.
You could set up a system wide keyboard hook which can be done in C# using the SetWindowsHookEx API call.
This is delving a little in unmanaged code and Windows API, but I think it's a better idea than polling a joystick cause you won't need to add a joystick to your computer just because you need to capture datetime at the press of a spacebar.
Do a little search on google for keyboard hooks in C#. It's easier than many people believe.
I don't think using a timer object to do this is a good idea at all. Cause I believe you said you need to do it at the press of a key.
Your best bet would be to create a Timer object on your form, and tie the joystick check to the Tick event.
I've also found that running simple stuff like this every 100ms (10 times a second) has a negligible effect on CPU usage.
Use something like AHK (Auto HotKey) it is a simple language that can be compiled to an EXE and is designed for automating the keyboard and mouse, I'm not 100% sure how well it works with a joystick, but I'm sure it's possible they managed to write some stuff for the wii remotes.
Also the IRC Channel and Forums always have people willing to help if need be.