The Yearly Update From Space

Oh hey, looks like another year is nearly over again. I’m going to try to write a bunch of posts in the next few weeks about things I’ve been doing.

On the side of EVE I pretty much started anew by writing a library in C#/.NET to interact with EVE. The problem with Python always was that it was completely running inside EVE’s Python interpreter, which all-in-all is a bit too invasive.

The way EVE (or Stackless Python applications in general) is structured, is that there is always one main tasklet (if you don’t know anything about Stackless Python, you should probably stop reading here and check out the summary) running that handles the scheduling of other running tasklets. That of course means that you can never use a blocking call from there, or you would cause a deadlock, since you are basically yielding the remaining running time of the scheduler! Thankfully Stackless Python detects (most) such cases and merely throws an exception.

Now, if you’re hooking DirectX’s EndScene and try to invoke Python functions from there, you are in the context of that main tasklet. If you are merely passively reading values or doing simple operations on EVE, that is not going to be a problem. In fact this is how ISXEve (and I suppose most people) interact with EVE. You are only going to have problems if you are calling blocking (or in terms of Stackless Python: yielding execution time) functions. An EVE Python function is usually blocking if it involves a request to the server or a long-running function. What EVE does however, is cache most remote calls, and only refresh the received data if it’s invalid or out of date. One example would be the market interaction API, orders for a certain type ID are usually cached for a short time, but in most cases EVE has to do a remote call to the server.

There’s a bunch of ways to deal with that:

  1. call blocking functions in their own tasklet, by eval-ing a short Python script that creates and launches a tasklet executing that function.
    Problem: it is difficult/ugly to get return values from this
  2. stay completely passive, don’t use any blocking functions at all and depend on other mechanisms to invoke them (market searches for example)
    Problem: gets very messy if you want return values and is very limited
  3. have your bot main interaction function invoked from inside a tasklet

ISXEve uses method 1 and 2. It’s valid methods, but I think it is limiting what you can do and the stability of the client way too much. You might claim that is more secure, but if CCP wanted to detect a public botting library, there are easier ways than to probe for Python function hooks or malicious tasklets.

I am only going to talk about method 3 now, which is the one I eventually ended up using.

In the very first EndScene call I grab the Python GIL lock (which is required to do anything in Python), register a Python CFunction pointing to a C# function and set up a tasklet repeatedly calling that CFunction – in short, a timer. Thanks to C#’s amazing PInvoke/interop functionality, registering the CFunction as a C# function is as easy as declaring a delegate (well, and marshaling a structure to unmanaged memory):

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate IntPtr PyCFunction(IntPtr self, IntPtr args);

Now every time the timer calls this function, we’re sitting in our own tasklet and can call blocking functions without problems! If the called Python function yields execution, the C#/CLR stack gets stored and the Stackless Python scheduler moves on the next tasklet. Once the condition to wake our tasklet back up has been fulfilled (sleep expired, message sent to channel, data recieved from server, etc…), our C# function is continued.

All of this makes interacting with EVE so very easy, and with C# as a powerful language (especially the LINQ feature) writing bots (or anything) much more enjoyable and focused on the bot logic compared to Python. Figuring out what EVE Python functions to call isn’t really that difficult, because thanks to Python’s dynamic nature you can pretty much restore the source code to the game. For Python 2.5 I fixed up an existing decompiler to work with a few more language constructs. I didn’t bother fixing it for Python 2.6 when EVE switched to it, since most changes are very small and can be easily inspected by diffing the disassembler output of both versions.

That’s it for this post! Hopefully it wasn’t all that boring to read, and I’ll try to update more regularly. No promises though.

Spaceships

It has certainly been a while since the last update, mostly because I’ve been rather busy with university. But I’m going to try to catch up a bit and talk about what I’ve been up to.

In the last few months I’ve been doing a bunch of reverse engineering and bot-writing for EVE Online, looking at ways to make money with it. There are a few fundamental differences in the way botting works in these two games: In World of Warcraft the way to make money is through mineral nodes like Saronite or Titanium Ore. A bot that needs to farm these is relatively simple, all it needs to do is fly in circles and pick up nodes, eventually fighting off attackers. To get the equivalent of a few hundred USD it is enough to run such a bot for maybe 6 hours a day.

In EVE Online the comparable easy way is mining ores, too, but the real money value yield is much lower per hour spent botting, so it is required to run a bot pretty much all day. There are better methods that yield more money though, for example mission running, but due to the nature of the game, in order to successfully complete the well earning missions it is required to spend several hundred days training skills. Since I did not care about EVE previously and do not have a suitable character, this is pretty much impossible at the moment.

However, there is one important advantage to EVE: There is only one server. In World of Warcraft you get about half a thousand servers, which makes it incredibly difficult to sell the in-game currency. In EVE the only limit is how much you can make; if you don’t want to sell it yourself, there are several websites that bring you into contact with a potential buyer.

That, and the possibility of improvements (and EVE looked kinda interesting as a game too) in the form of missions made me look into botting in EVE. It turned out that is incredibly easy there, compared to World of Warcraft: Since the game is entirely written in Python and there is an old decompiled source code dump available on the Internet, it was very easy to get started there.

To get access to EVE’s Python interpreter I injected a DLL that loads or retrieves a handle to the python25.dll module of the specific process and uses GetProcAddress to retrieve function addresses for any Python function I ended up needing. It’s not really required to do retrieve all these addresses manually and specify required function prototypes for them though, since one could also link to the DLL at compile time and have the dynamic linker retrieve the addresses, then it’s possible to just use the Python functions without any problems, since EVE’s Python 2.5 version and the one from the website probably aren’t different (?).

Since calling Python functions from another thread ended up being rather problematic for me, because the normal Global Interpreter Lock (GIL) functionality did not work and a hack that ended up working was a bit too messy for my tastes, I just ended up hooking (with MS Detours for example) the Python function`PyStackless_CallMethod_Main, which EVE’s ExeFile.exe calls to set off the Python part of the game.

In trying to interact with EVE’s python base though I encountered a problem: If I just executed my Python code with execfile() or eval(), my code ended up being executed in a different execution frame (I think) and thus had a different __builtin__ (Python’s “main” module where all the .. built-in commands or variables are. The solution turned for me ended up importing the “stackless” module and just queuing a new tasklet to be executed with stackless.tasklet(execfile)(“myfile.py”). It then ended up being executed by the main frame with the correct __builtin__.

At that point I ended up hooking EVE’s Startup function that was located somewhere in a autoexec_client_main module (It becomes pretty obvious from the source) and was able to properly interact with EVE’s GUI functions and so on afterwards, but that’s mostly just toying around and unnecessary. It takes a while to dig through the EVE source and figure out how it works, but it’s necessary to do anything, and thanks to the dudes who properly decompiled the bytecode ages ago (decompyle doesn’t work with Python 2.5, and UnPyc, an attempt to build a decompiler for Pythons up to version 2.6 is not finished and apparently abandoned. Aside from that there’s just commercial decompiling companies, and they probably won’t help with the EVE code ;)).

But to get a usable interpretation of EVE’s bytecode (that’s contained in the script/compiled.code file and lib/*.ccp archives (they really are just ZIP archives) and easily extractable, but I’ll go into that in another post) it’s really enough to just run Python’s included disassembler in the “dis” module over the code objects. Some dude called Ricky26 who apparently worked on a custom EVE server has already done a bunch of work on that, he also wrote a script that deals with the extracting and disassembling of the compiled code. But again, I’ll go into that in another post, this one is already getting too long.