none
Envoyer des touches clavier à une application externe : problème de timings RRS feed

  • Discussion générale

  • Bonjour à tous !

    J'ai pour objectif de faire un programme qui permettrait de lire une séquence de touches clavier et la rejouer dans le timing très précis à laquelle elle à été tapée. Pour vous faire une idée de l'utilité, j'aimerais être capable de mémoriser une partie de jeu (donc tous les inputs clavier que j'ai effectués dans le jeu) et de la rejouer avec l'exact même timing (sinon, il pourrait se passer différentes choses dans le replay que dans le jeu original) avec l'idée, plus tard, d'être capable d'éditer cette saisie de touche pour trouver des trajectoires optimales.

    Un tel logiciel étant introuvable (il n'y a que des logiciels de macro, mais ce n'est pas le but recherché), j'ai décidé d'en coder un... Et j'ai réussi ! Enfin... Presque réussi. J'arrive bien à sauvegarder les touches et leur timing, j'arrive bien à les envoyer au jeu, et elles se jouent bien... MAIS (car s'il n'y avait pas de "mais", je ne serais pas la), parfois la séquence de touches n'est pas tout a fait la même : elle se décale d'une frame (le jeu tournant a 60 fps), ce qui fait que certains mouvements ne sont plus exactement les mêmes et provoquent, bien souvent, des morts ou des désynchronisations totales entre la partie originale et le replay. Pire encore, si je joue deux fois la même séquence, j'aurais deux résultats différents, les "décalages" ne s’effectuant pas au mêmes moments. Le précise que le jeu que je teste est totalement déterministe et que je ne souffre d'aucun lag pendant la lecture d'une séquence de touches.

    Niveau algorithmie, ça se passe comme ça :

    Enregistrement des touches :

    • Au moment où j'appuie sur le bouton de record, une StopWatch est lancée et un hook et attaché au clavier (via SetWindowsHookEx).
    • A chaque input détecté par le hook, on sauvegarde le tick (mywatch.ElapsedTicks) auquels on associe tous les inputs effectués à ce tick et leur activité (touche préssée/relachée).
    • (Facultatif) Pour des fins d'optimisation, on supprime les "keydown" inutiles (c'est a dire ceux qui sont envoyés par des touches déjà préssées et qui n'ont pas été relachées depuis)

    On envoit le résultat de cet enregistrement au player. J'ai essayé plusieurs algorithmes pour le player, en utilisant SendKeys, SendInput et PostMessage, le problème est toujours le même (j'ai voulu essayer toutes les solutions possibles, pour voir si le problème n'était pas un problème de performance)

    Lecture des touches :

    • (Facultatif, fait pour SendInput) Au moment où j'appuie sur le bouton de record, on parcours une première fois les données envoyées par l'enregistreur et on les convertit en structures d'inputs (pour éviter d'avoir à le faire pendant la lecture, gain de temps).
    • Une StopWatch est lancée.
    • Tant qu'il reste des entrées dans le tableau de touches, on récupère la prochaine entrée, on lit le tick auquel est le prochain input, et on attend jusqu'a ce tick (dans un while() OU via un Thread.Sleep dans une ancienne version qui utilisait mywatch.ElapsedMilliseconds).
    • Une fois le tick atteint, on envoie la/les inputs.
    • On compare le temps réel (mywatch.elapsedTicks) avec le temps théorique. S'ils sont différents, on affiche un message de débug dans la console.
    • On refait la même chose avec le prochain input

    J'ai utilisé plusieurs timers, plusieurs moyens d'envoyer des touches, et pourtant, j'ai toujours ces décalages aléatoires... Et même quand aucune désynchronisation n'est signalée dans la console, j'ai des résultats différents...

    Voila, désolé pour le gros paté, j'ai essayé d'expliquer au maximum ! Voila le vrai code (abondamment commenté, mais en anglais, je code en anglais ^^') si l'algorithme n'expose pas de problème flagrant :

    Classe KeysSaver (enregistreur des touches) :

    using System;
    using System.Diagnostics;
    using System.Windows.Forms;
    using System.Runtime.InteropServices;
    using System.Collections.Generic;
    
    /* 
     * Class KeysSaver
     * Description : This class saves all the keyboard inputs, with the time (in miliseconds) they have been pressed.
     * Version : 1.0.0
     * Author : MetalFox Dioxymore
     */
    
    class KeysSaver
    {
        public static IntPtr KEYUP = (IntPtr)0x0101; // Code of the "key up" signal
        public static IntPtr KEYDOWN = (IntPtr)0x0100; // Code of the "key down" signal
        private Stopwatch watch; // Timer used to trace at which ticks each key have been pressed
        private Dictionary<long, Dictionary<Keys, IntPtr>> savedKeys; // Recorded keys activity, indexed by the ticks the have been pressed. The activity is indexed by the concerned key ("Keys" type) and is associated with the activity code (0x0101 for "key up", 0x0100 for "key down").
        private IntPtr hookId; // Hook used to listen to the keyboard
        private List<Keys> currentDown; // Track the keys that are already down, used for optimization (a maintened key down fires the "keydown" event a lot)
    
        private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam); // Imported type : LowLevelKeyboardProc. Now we can use this type.
    
    
        /*
         * Constructor 
         */
        public KeysSaver()
        {
            this.savedKeys = new Dictionary<long, Dictionary<Keys, IntPtr>>();
            this.watch = new Stopwatch();
            this.currentDown = new List<Keys>();
        }
    
        /*
         * method Start()
         * Description : starts to save the keyboard inputs.
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms644990%28v=vs.85%29.aspx
         */
    
        public void Start()
        {
            using (Process curProcess = Process.GetCurrentProcess())
            using (ProcessModule curModule = curProcess.MainModule) // Get the actual thread
            {
                // Installs a hook to the keyboard (the "13" params means "keyboard", see the link above for the codes), by saying "Hey, I want the function 'onActivity' being called at each activity. You can find this function in the actual thread (GetModuleHandle(curModule.ModuleName)), and you listen to the keyboard activity of ALL the treads (code : 0)
                this.hookId = SetWindowsHookEx(13, onActivity, GetModuleHandle(curModule.ModuleName), 0);
            }
            this.watch.Start(); // Starts the timer
        }
    
        /*
         * method Stop()
         * Description : stops to save the keyboard inputs.
         * Returns : the recorded keys activity since Start().
         */
        public Dictionary<long, Dictionary<Keys, IntPtr>> Stop()
        {
            this.watch.Stop(); // Stops the timer
            UnhookWindowsHookEx(this.hookId); //Uninstalls the hook of the keyboard (the one we installed in Start())
            return this.savedKeys;
        }
    
        /*
         * method onActivity()
         * Description : function called each time there is a keyboard activity (key up of key down). Saves the detected activity and the time at the moment it have been done.
         * @nCode : Validity code. If >= 0, we can use the information, otherwise we have to let it.
         * @wParam : Activity that have been detected (keyup or keydown). Must be compared to KeysSaver.KEYUP and KeysSaver.KEYDOWN to see what activity it is.
         * @lParam : (once read and casted) Key of the keyboard that have been triggered.
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms644985%28v=vs.85%29.aspx (for this function documentation)
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms644974%28v=vs.85%29.aspx (for CallNextHookEx documentation)
         */
        private IntPtr onActivity(int nCode, IntPtr wParam, IntPtr lParam)
        {
            long time = this.watch.ElapsedTicks; //Number of ticks elapsed since we called the Start() method
            if (nCode >= 0) //We check the validity of the informations. If >= 0, we can use them.
            {
                int vkCode = Marshal.ReadInt32(lParam); //We read the value associated with the pointer (?)
                Keys key = (Keys)vkCode; //We convert the int to the Keys type
                if (wParam == KeysSaver.KEYUP || (wParam == KeysSaver.KEYDOWN && !this.currentDown.Contains(key)))
                {
                    if (!this.savedKeys.ContainsKey(time))
                    {
                        // If no key activity have been detected for this tick yet, we create the entry in the savedKeys Dictionnary
                        this.savedKeys.Add(time, new Dictionary<Keys, IntPtr>());
                    }
                    this.savedKeys[time].Add(key, wParam); //Saves the key and the activity
                    if (wParam == KeysSaver.KEYDOWN)
                    {
                        this.currentDown.Add(key);
                    }
                    else
                    {
                        this.currentDown.Remove(key);
                    }
                }
            }
            return CallNextHookEx(IntPtr.Zero, nCode, wParam, lParam); //Bubbles the informations for others applications using similar hooks
        }
    
        // Importation of native libraries
        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);
    
        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool UnhookWindowsHookEx(IntPtr hhk);
    
        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode,
            IntPtr wParam, IntPtr lParam);
    
        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr GetModuleHandle(string lpModuleName);
    }

    Classe KeysPlayer (qui joue les touches clavier) :

    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using System.Linq;
    using System.Runtime.InteropServices;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    using System.Windows.Forms;
    
    namespace Taslagrad
    {
        //First of all, we need to reproduce the structure of an input to be able to send one. See https://msdn.microsoft.com/en-us/library/windows/desktop/ms646270%28v=vs.85%29.aspx for more details. 
    
        /*
         * Struct MOUSEINPUT
         * Mouse internal input struct
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646273(v=vs.85).aspx
         */
        internal struct MOUSEINPUT
        {
            public Int32 X;
            public Int32 Y;
            public UInt32 MouseData;
            public UInt32 Flags;
            public UInt32 Time;
            public IntPtr ExtraInfo;
        }
    
        /*
         * Struct HARDWAREINPUT
         * Hardware internal input struct
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646269(v=vs.85).aspx
         */
        internal struct HARDWAREINPUT
        {
            public UInt32 Msg;
            public UInt16 ParamL;
            public UInt16 ParamH;
        }
    
        /*
         * Struct KEYBDINPUT
         * Keyboard internal input struct (Yes, actually only this one is used, but we need the 2 others to properly send inputs)
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646271(v=vs.85).aspx
         */
        internal struct KEYBDINPUT
        {
            public UInt16 KeyCode; //The keycode of the triggered key. See https://msdn.microsoft.com/en-us/library/windows/desktop/dd375731(v=vs.85).aspx
            public UInt16 Scan; //Unicode character in some keys (when flags are saying "hey, this is unicode"). Ununsed in our case.
            public UInt32 Flags; //Type of action (keyup or keydown). Specifies too if the key is a "special" key.
            public UInt32 Time; //Timestamp of the event. Ununsed in our case.
            public IntPtr ExtraInfo; //Extra information (yeah, it wasn't that hard to guess). Ununsed in our case.
        }
    
        /*
         * Struct MOUSEKEYBDHARDWAREINPUT
         * Union struct for key sending 
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646270%28v=vs.85%29.aspx
         */
    
        [StructLayout(LayoutKind.Explicit)]
        internal struct MOUSEKEYBDHARDWAREINPUT
        {
            [FieldOffset(0)]
            public MOUSEINPUT Mouse;
    
            [FieldOffset(0)]
            public KEYBDINPUT Keyboard;
    
            [FieldOffset(0)]
            public HARDWAREINPUT Hardware;
        }
    
        /*
         * Struct INPUT
         * Input internal struct for key sending 
         * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646270%28v=vs.85%29.aspx
         */
    
        internal struct INPUT
        {
            public UInt32 Type; //Type of the input (0 = Mouse, 1 = Keyboard, 2 = Hardware)
            public MOUSEKEYBDHARDWAREINPUT Data; //The union of "Mouse/Keyboard/Hardware". Only one is read, depending of the type.
        }
    
        /* 
         * Class KeysPlayer
         * Description : This class plays all the recorded inputs, reproducing the inputing timing.
         * Version : 1.0.0
         * Author : MetalFox Dioxymore
         */
    
        class KeysPlayer
        {
            private Dictionary<long, Dictionary<Keys, IntPtr>> keysToPlay; // Keys to play, with the timing. See KeysSaver.savedKeys for more informations.
            private Dictionary<long, INPUT[]> playedKeys; // The inputs that will be played. This is a "translation" of keysToPlay, transforming Keys into Inputs.
            private Stopwatch watch; // Timer used to respect the strokes timing.
    
            /*
             * Constructor 
             */
            public KeysPlayer(Dictionary<long, Dictionary<Keys, IntPtr>> keysToPlay)
            {
                this.keysToPlay = keysToPlay;
                this.playedKeys = new Dictionary<long, INPUT[]>();
                this.watch = new Stopwatch();
                this.loadPlayedKeys(); //Load the keys that will be played.
            }
    
            /*
             * method Start()
             * Description : starts to play the keyboard inputs.
             */
            public void Start()
            {
                Thread.Sleep(1); // To get the CPU attention
                this.watch.Reset(); //Resets the timer
                this.watch.Start(); //Starts the timer (yeah, pretty obvious)
                IEnumerator<long> enumerator = this.playedKeys.Keys.GetEnumerator(); //The playedKeys enumerator. Used to jump from one frame to another.
                
                long t; //Will receive the elapsed tickss, to track desync.
    
                //For PostMessage Version
                int teslaWindow = FindWindow(null, "Teslagrad");
    
                while (enumerator.MoveNext()) //Moves the pointer of the playedKeys dictionnary to the next entry (so, to the next frame).
                {
                    while (this.watch.ElapsedTicks < enumerator.Current) { } //We wait until the very precise ticks that we want
                    
                    //POSTMESSAGE VERSION (we use directly the KeysSaver array)
                    foreach (KeyValuePair<Keys, IntPtr> kvp in this.keysToPlay[enumerator.Current])
                    {
                        PostMessage((IntPtr)teslaWindow, (uint)kvp.Value, (int)kvp.Key, (int)this.lParam(0, (uint)kvp.Key,0,0,1,(uint)(((uint)kvp.Value == 256) ? 0 : 1)));
                    }
    
                    //SENDINPUT VERSION
                    /*t = this.watch.ElapsedTicks; //We save the actual ticks
                    uint err = SendInput((UInt32)this.playedKeys[enumerator.Current].Length, this.playedKeys[enumerator.Current], Marshal.SizeOf(typeof(INPUT))); //Simulate the inputs of the actual frame
                    if (t != enumerator.Current) // We compare the saved time with the supposed ticks. If they are different, we have a desync, so we log some infos to track the bug.
                    {
                        Console.WriteLine("DESYNC : " + t + "/" + enumerator.Current + " - Inputs : " + this.playedKeys[enumerator.Current].ToString());
                    }*/
                }
            }
    
            //POSTMESSAGE VERSION ONLY FROM HERE
            [DllImport("user32.dll")]
            public static extern int FindWindow(string lpClassName, String lpWindowName);
            [DllImport("user32.dll", SetLastError = true)]
            public static extern IntPtr FindWindowEx(IntPtr parentHandle, IntPtr childAfter, string className, string windowTitle);
            [DllImport("user32.dll")]
            public static extern bool PostMessage(IntPtr hWnd, uint Msg, int wParam, int lParam);
            [DllImport("user32.dll")]
            public static extern uint MapVirtualKey(uint uCode, uint uMapType);
    
            public uint lParam(uint repeatCount, uint scanCode, uint extended, uint context, uint previousState, uint transition)
            {
                return repeatCount
                    | (scanCode << 16)
                    | (extended << 24)
                    | (context << 29)
                    | (previousState << 30)
                    | (transition << 31);
            }
            //TO HERE
    
            /*
             * method Stop()
             * Description : stops to play the keyboard inputs.
             */
            public void Stop()
            {
                this.watch.Stop(); //Stops the timer.
            }
    
            /*
             * method loadPlayedKeys()
             * Description : Transforms the keysToPlay dictionnary into a sequence of inputs. Also, pre-load the inputs we need (loading takes a bit of time that could lead to desyncs).
             */
            private void loadPlayedKeys()
            {
                foreach (KeyValuePair<long, Dictionary<Keys, IntPtr>> kvp in this.keysToPlay)
                {
                    List<INPUT> inputs = new List<INPUT>(); //For each recorded frame, creates a list of inputs
                    foreach (KeyValuePair<Keys, IntPtr> kvp2 in kvp.Value)
                    {
                        inputs.Add(this.loadKey(kvp2.Key, this.intPtrToFlags(kvp2.Value))); //Load the key that will be played and adds it to the list. 
                    }
                    this.playedKeys.Add(kvp.Key, inputs.ToArray());//Transforms the list into an array and adds it to the playedKeys "partition".
                }
            }
    
            /*
             * method intPtrToFlags()
             * Description : Translate the IntPtr which references the activity (keydown/keyup) into input flags.
             */
            private UInt32 intPtrToFlags(IntPtr activity)
            {
                if (activity == KeysSaver.KEYDOWN) //Todo : extended keys
                {
                    return 0;
                }
                if (activity == KeysSaver.KEYUP)
                {
                    return 0x0002;
                }
                return 0;
            }
    
            /*
             * method loadKey()
             * Description : Transforms the Key into a sendable input (using the above structures).
             */
            private INPUT loadKey(Keys key, UInt32 flags)
            {
                return new INPUT
                {
                    Type = 1, //1 = "this is a keyboad event"
                    Data =
                    {
                        Keyboard = new KEYBDINPUT
                        {
                            KeyCode = (UInt16)key,
                            Scan = 0,
                            Flags = flags,
                            Time = 0,
                            ExtraInfo = IntPtr.Zero
                        }
                    }
    
                };
            }
    
            // Importation of native libraries
            [DllImport("user32.dll", SetLastError = true)]
            public static extern UInt32 SendInput(UInt32 numberOfInputs, INPUT[] inputs, Int32 sizeOfInputStructure);
    
            [DllImport("kernel32.dll")]
            static extern uint GetLastError();
    
        }
    }
    

    Classe Taslagrad (logiciel en lui même, par forcement utile mais on sait jamais) :

    using System;
    using System.Collections.Generic;
    using System.ComponentModel;
    using System.Data;
    using System.Drawing;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using System.Windows.Forms;
    
    namespace Taslagrad
    {
        public partial class Taslagrad : Form
        {
            private KeysSaver k;
            private KeysPlayer p;
    
            //Initialisation 
            public Taslagrad()
            {
                InitializeComponent();
                this.k = new KeysSaver();
            }
    
            /*
             * method launchRecording()
             * Description : Starts to record the keys. Called when the "record" button is triggered.
             */
            private void launchRecording(object sender, EventArgs e)
            {
                this.k.Start(); //Starts to save the keys
                startButton.Text = "Stop"; //Updates the button
                startButton.Click -= launchRecording;
                startButton.Click += stopRecording;
            }
    
            /*
             * method stopRecording()
             * Description : Stops to record the keys and logs the recorded keys in the console. Called when the "record" button is triggered.
             */
            private void stopRecording(object sender, EventArgs e)
            {
                startButton.Text = "Record";//Updates the button
                startButton.Click += launchRecording;
                startButton.Click -= stopRecording;
                Dictionary<long, Dictionary<Keys, IntPtr>> keys = this.k.Stop(); //Gets the recorded keys
                foreach (KeyValuePair<long, Dictionary<Keys, IntPtr>> kvp in keys)
                {
                    foreach (KeyValuePair<Keys, IntPtr> kvp2 in kvp.Value)
                    {
                        //Displays the recorded keys in the console
                        if (kvp2.Value == KeysSaver.KEYDOWN)
                        {
                            Console.WriteLine(kvp.Key + " : (down)" + kvp2.Key);
                        }
                        if (kvp2.Value == KeysSaver.KEYUP)
                        {
                            Console.WriteLine(kvp.Key + " : (up)" + kvp2.Key);
                        }
                    }
                }
                this.p = new KeysPlayer(keys); //Creates a new player and gives it the recorded keys.
            }
    
            /*
             * method launchPlaying()
             * Description : Starts to play the keys. Called when the "play" button is triggered.
             */
            private void launchPlaying(object sender, EventArgs e)
            {
                this.p.Start(); //Starts to play the keys.
            }
        }
    }
    

    Merci a tous ceux qui auront eu le courage de lire mon problème !

    ~MetalFox Dioxymore

    mardi 14 juillet 2015 00:32

Toutes les réponses