About a month ago, I made a post about the frustration of using a RAM search with Dolphin and I more specifically that the only good one, Cheat Engine isn’t cutting it with MANY problems occurring since it was just not made to work with this use case.  At the end of the post, I said that the best solution to this problem would be to actually do a RAM search from scratch instead of doing tons and tons of extensions which just makes the tools more annoying to use while it could just simply work out of the box.  Like there was so much problem that it’s not worth to fix them with so many extensions.  In fact, since I posted this and notified TASVideos about it, a workaround was found that disables ASLR with Dolphin…..this is quite ridiculous that we have to get this far to have a RAM search work.

But I also said that I might just do a RAM search just for these reasons.

And that’s exactly what happened, a few days after the post, I confirmed that I was starting the project.  It’s been a month and honestly, I am VERY happy how it’s been going.  In this post, I am going to tell the difficulties and challenges of doing this, this is my first true project of a somewhat complex program so a lot of stuff was new to me and I found it would be interesting to do a post about it!

Little disclaimer

This post is going to be more technical than my others, I normally take the time to explain concepts that people may not be familiar with, this time I can’t really do this because I want to emphasise on the different challenges I went through which requires me to talk about several programming concept.  I will TRY to not go too deep, but there might be part harder to understand.  If you are already familiar with programming, you shouldn’t have any problem though.

With that in mind let’s get started!

The need for reads and writes

Once it was decided that I would do a RAM search, I needed information about how to do this in the first place,  The first thing that had to be known was how reads and writes from an external process is done exactly?  This is an external RAM search because I was too worried about the performance impact on Dolphin if I integrated it and used the Dolphin’s memory accesses functions on top of the game calling them like crazy, also due to many other reasons. Any RAM search needs to at least be able to read and write from and into the targeted RAM otherwise you probably wouldn’t get far beyond getting a link to the process!

After hours of googling, I found it’s much simpler than I anticipated.  What REALLY helped me was finding a blog post from someone who tried to do a command line ram hacking tool which worked on both Linux and windows, exactly what I wanted.  Not only they tell how they did it, but they also has a Github repository where there’s their code, so nice examples!  The basic idea is you can write to memory the process gives access to (in this case, it does), provided you have a link to the process (you need the process ID to get that), an address, a buffer to read to or write from and a size.

Unfortunately, this is handled very differently for each OS so I have to compile the Windows version on Windows and same for Linux, but not the other way around.  This is not a problem, but it also meant I had to make an interface that each OS specific implementation implements.  All they do is get the PID of Dolphin (I just use its name as a way to find the process), generic reads and writes with byte swap etc.. it’s very barebones, you just read stuff or write stuff.

The harder part was getting the start address.  It’s impossible to guess it because it’s random every time, but what you can do is read the mapping information of the process and kinda get which mapping is the memory you want because Dolphin is open source: it’s not hard to check the code and find out how the mapping is done.

I was already familiar with the code that does it, but for the sake of showing what I mean, here’s a snippet from Dolphin’s code:

void MemArena::GrabSHMSegment(size_t size)
{
#ifdef _WIN32
hMemoryMapping =
CreateFileMapping(INVALID_HANDLE_VALUE, nullptr, PAGE_READWRITE, 0, (DWORD)(size), nullptr);
#elif defined(ANDROID)
fd = AshmemCreateFileMapping("Dolphin-emu", size);
if (fd < 0)
{
NOTICE_LOG(MEMMAP, "Ashmem allocation failed");
return;
}
#else
for (int i = 0; i < 10000; i++)
{
std::string file_name = StringFromFormat("/dolphinmem.%d", i);
fd = shm_open(file_name.c_str(), O_RDWR | O_CREAT | O_EXCL, 0600);
if (fd != -1)
{
shm_unlink(file_name.c_str());
break;
}
else if (errno != EEXIST)
{
ERROR_LOG(MEMMAP, "shm_open failed: %s", strerror(errno));
return;
}
}
if (ftruncate(fd, size) < 0)
ERROR_LOG(MEMMAP, "Failed to allocate low memory space");
#endif
}

The interesting part is that we it maps it on WIndows with certain properties and on Linux, it maps to a file which this tells you the filename so you can know for sure it is the one wanted.  Also, in the same file is the function that actually finds the famous start address:

u8* MemArena::FindMemoryBase()
{
#if _ARCH_32
const size_t memory_size = 0x31000000;
#else
const size_t memory_size = 0x400000000;
#endif

#ifdef _WIN32
u8* base = static_cast<u8*>(VirtualAlloc(nullptr, memory_size, MEM_RESERVE, PAGE_READWRITE));
if (!base)
{
PanicAlert("Failed to map enough memory space: %s", GetLastErrorMsg().c_str());
return nullptr;
}
VirtualFree(base, 0, MEM_RELEASE);
return base;
#else
#ifdef ANDROID
// Android 4.3 changed how mmap works.
// if we map it private and then munmap it, we can't use the base returned.
// This may be due to changes in them support a full SELinux implementation.
const int flags = MAP_ANON | MAP_SHARED;
#else
const int flags = MAP_ANON | MAP_PRIVATE;
#endif
void* base = mmap(nullptr, memory_size, PROT_NONE, flags, -1, 0);
if (base == MAP_FAILED)
{
PanicAlert("Failed to map enough memory space: %s", GetLastErrorMsg().c_str());
return nullptr;
}
munmap(base, memory_size);
return static_cast<u8*>(base);
#endif
}

 

There, now we have the size (0x400000000) and we have some properties on Windows to query the mapping.  Btw, this is the part of the code that used to be VERY different before the ASLR update, the Windows one didn’t changed (but was most of the time ended up getting the same address) and the Linux one, this is where the 0x2380000000 address popped up.  If you want to read the whole file, here.

So this is how I was able to always print the right start address everytime, obvisouly the emulator has to be running, but an emulation has to be started too, it won’t initialise the RAM withotu an emulation being started.  The mapping stands out enough and I have enough information to get the right one.  And once I have the start address, I am free to read or write anything I want anywhere because as long as I pass the function above a valid offset from the start (which I can validate myself anyway), it’s going to get read or written.

So I made a up a little command line thing as a proof of concept and it was sucessfull!  In fact, since I wanted a little fun, I tried to 0 the entire Gamecube emulated RAM while Dolphin was running a game and…..

Screenshot from 2017-06-27 22-04-23
GREEEEEEEEEEEEEEEEEEEEEEEEEEEEENNNNNNN!!!!

Here’s an interesting factoid: green is expressed as all 0 in the YUY2 color space which the Gamecube uses.  Now honestly, I did expected Dolphin to crash, but all I got was some stall and the Program Counter was completely messed up.

Btw, I used a virtual machine to test on Windows 10, I even had to install more RAM because Visual studio was eating most of the little 3GB I could give to this poor VM.

So now it’s simple, I did a wrapper around this interface so that the platform specific implementation is hidden. If a read or writes fails, I can know it by the return of my wrapper and react like unhook Dolphin, stops current operation and disables everything, but the button to attempt to hook again.   This makes the tool ready in case Dolphin crash which….let’s be honest, is quite likely to happen with a tool as powerful as a RAM search.

So NOW we can start to think about watch entries 🙂

Introducing custom data format

This is where the project with expectation got a lot harder because this is not something obvious.  Sure, I can write and read anywhere, but what I TRULY want is to have watch entries that would format and update themselves vis continuous polling depending on their properties.  To get this right, we have to think about how memory fundamentally works: it’s completely meaningless unless you give it a meaning.  It turns out you can express a generic array of bytes in c++ by using an array of char (yes it stands for character, but here, it’s guaranteed to be a byte long and you CAN express an integer with it anyway.  Since the size CAN change at runtime (the user may want to change it after the fact), it would be a pointer allocated dynamically.  This is how you woudl represent meaningless array of byte, but the idea is you have several properties that tells HOW the memory should be interpreted and how it should be parsed when writing to it.    For example, if the type is a 2 bytes integer, it should read the array converting it to an integer and just prints the corresponding string.

Simple right? But it gets more complicated when you consider the different memory base the user may want to see (sure, decimal is nice, but you may want to see it in binary if it’s a flag array or hex in case ….it’s an RNG seed….whatever).  You also may want to see it signed or unsigned, some types like string and array of bytes has a specific length so the size changes…..yeah this is why it’s difficult to get your head around it.  You have to select the right conversion and parsing, but there’s so many choices as I want to provide the user the most amount of choices that it gets confusing.

One thing I definitely learned however is std::memcpy was probably the most useful function in this use case ever.  It copies the content of the memory into somewhere else, but the idea is it just copies content, it doesn;t really care what type the location were referring to.  So you could like copy an array of byte into a float and magic, you have the float equivalent of these bytes.  This was used heavily into my types logic.

This was the hardest part of the project so far because it took days to get it right and I had to rethink several time about my approach, but I finally got it right, I just did some dummy UI tests with a text field and a button and I could write and read arbitrary ram into beautiful strings as well as formatting one.  Oh btw, one other complexity when writing is managing input validation, they change per types…..yeah did I mention the type logic gets complex?

Time to do cute UI

See what I did there?  Qt is pronounced cute…..okay you get it.

Well now that we have a basic (not complete) logic) of how entries work, time to do UI!

First, since this is multiplatofrm and I am using tools to make this happens (like cmake to handle compilation), I had to choose a multiplatofrm GUI toolkit.  I decided on Qt because:

  1. Heard a lot of good things about it, I have been hearing it has great doc, easy to use, basically, I have been hearing it has a great reputation.
  2. It’s very open source friendly, in fact, it is licensed under the LGPL which is compatible with the license I want to use (MIT).
  3. It’s multiplatform and flexible.
  4. It’s not wxWidgets
  5. IT’S NOT WXWIDGETS!!!

About the last 2, here’s a bit of context, I did made UI contributions on Dolphin, one very big one included where I redid all the input configurations dialogs to have more flexibility in their layouts and Dolphin (at least at the time of this post) is currently using wxWidgets, but they are in the process of migrating to Qt.

Why?  Because EVERY SINGLE dev I spoke with and I have to agree here says that wx is just…..not really good.  It has a lot of problems, really incomplete doc sometime, some functions just are straight up inconsistent between platform (one introduced the Ui bug I hated the most for years in Dolphin so much I spent days to fix it) and just weird stuff.  It became annoying with devs to work with it and I experienced it too so I KNOW what it’s like to work with an API that just has trouble to work properly.

And I was happy to realise that what I heard about Qt was right, seriously, I love this toolkit, the doc works VERY well and although it took me days to figure some stuff out, you at least have the right tools to figure it out.

The only thing I had trouble with and I really don’t get why is it took me an entire weekend to install it on my VM, I never got why because reinstalling visual studio and Qt made it work…..idk.

So, now how to represent the watch entries?  Well you might think to use a regular table, but actually, it’s more complicated.  Like Cheat Engine, I want to have groups, months of research into a particular game can lead to dozens and dozens of entries so having a way to organise them is a must.  But nothing should stop me to have a hierarchy of groups so we are looking more at a tree structure.

So nice I got the tree widget ready, okay…..now what?

The concept of a model

So…..it turns out my needs are much more complex than the default convenient stuff Qt offers to do happy little trees, it turns out I have to implement my own tree structure, logic and everything since I need to do so much for everything to work (like I need custom events, recursively loop through the tree… etc…).  More importantly, I want to have ONE structure that the view will uses as well as I can change it.

This is what the model view thing from Qt provides……which is imo not well named (the model is kinda like a controller which handles the logic of the data, the data should actually be the model…..it’s confusing).  Basically, the view uses my custom models which has all the logic.  Getting indexes, setting data, getting data, getting info about the data from the view, signaling the view stuff, etc….  Because of this, it took me days to just get the tree to work because again, it’s not easy to get my head around the first time, but once I figured stuff out, it just goes well.  The fact that I forgot how a tree is supposed to have recursive nodes……yeah that didn’t help it 🙂

The result is you can do this:

Screenshot from 2017-07-08 01-34-51
Isn’t this tree happy?

Nice, so now I have a tree, we can now starts to implement things in the front end!

Several convenient things

So, there’s a lot here, editing the data, moving the data into groups with drag and drop etc…

Continuous polling

Since this is better shown with a gif, here:

ezgif.com-optimize
This is where it just started to be interesting

This is actually not hard to implement because I already built the function that reads the RAM into he watch’s memory, the view will separately query the new data to display,  I just have to put a timer and on that timer, I recursively go through the tree.  Btw, recursivity is one of those things that you know when you need it, but it’s still hard to wrap my head around because I have to tell what it would do when the function is called by itself, it’s just weird.

Btw, I need to poll because Dolphin lives on its own and so does my process so I HAVE to continuously ask the RAM of watch entry “hey, what;s your values now?”.  Even with a timer of 10 milliseconds, it worked perfectly fine even though on my test environment with 3 entires, it should do 300 reads per second…..which is a lot, but if I am going with a 60 fps game to have real-time updates, then yeah it needs to be frequent.

Freezing

The lock check box you see allows you to lock or freeze the RAM….okay to drop the fancy names, all it means is it will autowrite the current value on a timer as frequent as the update timer.  It only will change if you explicitly set a value, but it will autowrite that value instead.  For the user, it’s as if you were forcing the ram to be that value so hence the name freezing.

For implementing it, it’s slightly harder than the update timer, I had to implement a couple of stuff in the backend to have a separate memory for the freeze, all it does is when freeze is called it just writes that memory.  Overall, it went pretty quickly.

Drag and drop

THIS is the big one, it took me SO LONG (days) to figure out because I really NEVER knew drag and drop worked like this, but basically, you know when you have a program that open txt files and you drag a txt file and it magically opens it?  Well you might not have asked this, but what are you actually dragging and dropping anyway?  The answer is you are dragging serialised data in a special format called MIME which is used to identify file types on the web and other things like between programs with drag and drop support.  The program will allow drops for supported mime types, for example, a png is of MIME type image/png but you could be dragging text and other custom stuff.

So once I figured this out, it’s very simple since I only allow drag and drop internally, I just use a custom MIME type that I would invent on the fly.  …..After that, I had to get my head around model indexes and actually moving rows between branches of the trees WITH multiselect support…..yeah that took me a day.  So now I could create groups and move them around, yay!

Misc

I did other smaller things like having a right clock menu to change base of entries as well as if they are signed or unsigned, did some editing logic so that when you double-click to edit the label, it shows you straight up the editor rather than Cheat Engine which actually shows you a window….pretty annyoing tbh.  I did error handling that the backend would encounter so you can even get nice errors (I say nice because error messages should be clear and tell the user how to not do it again). I also had to do a form for the type change because it may or may not have a length, and of course, the entry add/editing window:

Screenshot from 2017-07-19 00-36-44
The levels aren’t fixed, you add or you remove them dynamically if the pointer check box is checked

HUH? MULTI LEVEL POINTERS?????

Finally, at last, tracking and editing multi level pointers is a thing in Dolphin

7drhiqr

Just as if it’s not clear how hype it is:

Okay now it’s clear 🙂

More seriously though, THIS feature HAD to be in before I even consider releasing anything, this was BY FAR the main reason using Cheat Engine was frustrating, even if you were patient enough to endure the pain it already is to use it with Dolphin, there’s NOTHING you could do against tracking dynamic memory within Cheat Engine unless you use a fancy script which would just make everything annoying.

You know what the funny thing is?  With all the backend system in place, implementing the pointer logic int he backend ONLY took 15 minutes, add 10 to have multiple levels, I am not joking because it’s actually not hard to get how it works.

Just a FYI before, when I refer to the level of the pointer, I mean its deepness of the path to follow it.  For example, a pointer to an integer is a level 1 pointer.  A pointer to a pointer of integer is a level 2 pointer and you can go on and on.  The pointer path starts with an address and between each hop in the pointers, it may have an offset added to go to the next pointer.

You have a vector of integers that serves as offset, if the watch is marked as pointers.  If so, it will read whatever is int he address and check if it’s a valid address in the Gamecube/Wii, if it is, it adds the offset and it continues to the next level.  After the complete path has been followed, it finally do the last read which is the actual data.  On the UI, adding a pointer is not hard, but finding it…..is slightly harder, but not by much.  You definitely need both the debugger AND a scanner for this (which sucked because I had to use the Cheat Engine one….remidned me how annoying it was to use with the freaking conversion with Dolphin’s start.

Once added however……it just works.  It’s as if you didn’t marked it as pointers, the ONLY indicator it is one is the address is written differently in the UI, but the user really doesn’t have to bother AT ALL about resolving the pointer path, the reads are done with all the levels in the background and the write just works.  I already was excited when I was able to successfully track the HP of Super Mario Sunshine even after changing the level I was in (it’s a level 1 dynamic pointer), but what REALLY got me excited, is being able to track dynamically the HP of Paper Mario: The Thousand-Year Door in battle no matter which battle or map I was in and it was a level 2 pointer.  This game holds and the battle RAM is special to me because this game is where I started to glitch hunt and reverse engineer stuff, never was I able to test the battle RAM because there was no way Cheat Engine would be able to track a level 1 pointer, let a lone a level 2 one in Dolphin.

For the anecdote, as soon as I did my first successful edit of the battle HP in TTYD using a level 2 pointer, I literally opened my music player and played the cornered theme of the first Ace Attorney game at like 1-2am 🙂

However, there is one con about it, I am making a rather general assumption on how Gamecube and Wii games maps their memory: I am assuming that the virtual address is always the physical one + 0x80000000.  This is true for 75% of the game meaning that there’s about a quarter of the game that has custom mapping that unfortunately I cannot do anything about them.  You would have to get the mapping info of the game and maybe provide it as a custom rule or something, but the problem is even getting that info.  The only other solution would have been to be integrated within Dolphin and use the translate function, but again, I am very worried about performance if I do this.  Still, 75% of the entire Gamecube + Wii library is a lot and I am very glad I went there from literally no support.

So,. that;s about it for how it went the last month.

The result

A very functional RAM watch that has support for the following types:

Screenshot from 2017-07-21 09-32-26
All these types!

Also has support for the base decimal, octal, hex and binary for all the appropriate types.  Real time updates, freezing support, complete abstraction of the addresses so no more having to search or find or guess a start address.  The watch allows to have nested groups of entries as well as having the long-awaited feature of multi level pointer tracking.

What’s next?

There is still some very minor things to do about the RAM watch, but features wise, I am happy for the initial preview release which I am trying to get done before I start university in a month.  The next step is to get done the scanner which I am sure will be easier than the RAM watch because the watch had a lot of front end code as well as several utilities in the backend logic to take care of the many interpretations the user might want to have on the memory.  So hopefully I might be ahead and do more release work so I could get cleaner code and more usable UI.

But yeah, the hardest part has definitely been done and I am VERY happy how the project is going.  I honestly can’t wait to progressively get the scanner working, it’s going to be fun to test 🙂

 

Advertisements

One thought on “Progress on my new Dolphin RAM search: the initial RAM watch is done!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s