The Yearly Update From Space

Oh hey, looks like another year is nearly over again. I’m going to try to write a bunch of posts in the next few weeks about things I’ve been doing.

On the side of EVE I pretty much started anew by writing a library in C#/.NET to interact with EVE. The problem with Python always was that it was completely running inside EVE’s Python interpreter, which all-in-all is a bit too invasive.

The way EVE (or Stackless Python applications in general) is structured, is that there is always one main tasklet (if you don’t know anything about Stackless Python, you should probably stop reading here and check out the summary) running that handles the scheduling of other running tasklets. That of course means that you can never use a blocking call from there, or you would cause a deadlock, since you are basically yielding the remaining running time of the scheduler! Thankfully Stackless Python detects (most) such cases and merely throws an exception.

Now, if you’re hooking DirectX’s EndScene and try to invoke Python functions from there, you are in the context of that main tasklet. If you are merely passively reading values or doing simple operations on EVE, that is not going to be a problem. In fact this is how ISXEve (and I suppose most people) interact with EVE. You are only going to have problems if you are calling blocking (or in terms of Stackless Python: yielding execution time) functions. An EVE Python function is usually blocking if it involves a request to the server or a long-running function. What EVE does however, is cache most remote calls, and only refresh the received data if it’s invalid or out of date. One example would be the market interaction API, orders for a certain type ID are usually cached for a short time, but in most cases EVE has to do a remote call to the server.

There’s a bunch of ways to deal with that:

  1. call blocking functions in their own tasklet, by eval-ing a short Python script that creates and launches a tasklet executing that function.
    Problem: it is difficult/ugly to get return values from this
  2. stay completely passive, don’t use any blocking functions at all and depend on other mechanisms to invoke them (market searches for example)
    Problem: gets very messy if you want return values and is very limited
  3. have your bot main interaction function invoked from inside a tasklet

ISXEve uses method 1 and 2. It’s valid methods, but I think it is limiting what you can do and the stability of the client way too much. You might claim that is more secure, but if CCP wanted to detect a public botting library, there are easier ways than to probe for Python function hooks or malicious tasklets.

I am only going to talk about method 3 now, which is the one I eventually ended up using.

In the very first EndScene call I grab the Python GIL lock (which is required to do anything in Python), register a Python CFunction pointing to a C# function and set up a tasklet repeatedly calling that CFunction – in short, a timer. Thanks to C#’s amazing PInvoke/interop functionality, registering the CFunction as a C# function is as easy as declaring a delegate (well, and marshaling a structure to unmanaged memory):

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate IntPtr PyCFunction(IntPtr self, IntPtr args);

Now every time the timer calls this function, we’re sitting in our own tasklet and can call blocking functions without problems! If the called Python function yields execution, the C#/CLR stack gets stored and the Stackless Python scheduler moves on the next tasklet. Once the condition to wake our tasklet back up has been fulfilled (sleep expired, message sent to channel, data recieved from server, etc…), our C# function is continued.

All of this makes interacting with EVE so very easy, and with C# as a powerful language (especially the LINQ feature) writing bots (or anything) much more enjoyable and focused on the bot logic compared to Python. Figuring out what EVE Python functions to call isn’t really that difficult, because thanks to Python’s dynamic nature you can pretty much restore the source code to the game. For Python 2.5 I fixed up an existing decompiler to work with a few more language constructs. I didn’t bother fixing it for Python 2.6 when EVE switched to it, since most changes are very small and can be easily inspected by diffing the disassembler output of both versions.

That’s it for this post! Hopefully it wasn’t all that boring to read, and I’ll try to update more regularly. No promises though.

3 thoughts on “The Yearly Update From Space

  1. Is there a chance to get python 2.5 fixed decompiler (i won’t ask you for 2.6 one, but that would be great)?

Leave a Reply

Your email address will not be published. Required fields are marked *