Editing 40d Talk:Memory hacking

== New Release Updates ==
I'm just updating the main page as I figure new things out. Someone else will probably have to redo the formatting, since I don't really know the wiki formatting. --[[User:Zhentar|Zhentar]] 23:30, 29 October 2007 (EDT)

== Creature Structure ==
I have a feeling that offset 0x00 is really a "next" pointer, forming a linked list.  The reason being is that I was able to take the 4 byte value and go to that address, and it would be another creature.  It progressed for a handful of creatures.  Seems logical to assume it does it for all creatures.  --[[User:AzureLightning|AzureLightning]] 00:24, 3 November 2007 (EDT)
: I would assume this is true as well, since earlier versions did not have vtables for creatures, only items did. --[[User:Rick|Rick]] 18:07, 4 November 2007 (EST)

== Memory Locations ==

Does anyone know the memory location for the current unit focus? --[[User:AzureLightning|AzureLightning]] 14:05, 3 November 2007 (EDT)
:My automated tool says it's at current_unit_focus_loc == 00C36540 in .33c, verified by two search patterns, but not manually verified.  You probably should have asked this in the .33c subpage.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 16:49, 30 November 2007 (EST)

==Questions==

# Has anyone attempted to decode the memory for the "Choose Fortress Location" screen? Being able to generate a list of all local areas with magma vents would be useful. [[User:Palin88|Palin88]] 20:09, 29 November 2007 (EST)
:: This is a great idea. I suggest going one step further : Give the user a list of options (say : magma, flux, thick forest, chasm, underground river, pit, etc), and logically AND all parts of the map that have those features. That would save people hours of looking for their perfect start location. [[User:Runspotrun|Runspotrun]] 19:25, 8 December 2007 (EST)
# It could be important to some people, but the formats are only correct assuming that the dwarf fortress binary is built using release config.  Under debug config, std::string structure gains some more fat and the existing offset will be wrong.  Not a prob now, but I think it is good to know.  [[User:Sphr|Sphr]]

== Version Identity ==
(Sphr: this method DON'T work!)
Thought that the wiki should include some info on how to identify known versions.  For lack of a better choice atm, I'm using the version text block that is displayed on main menu.
The version string itself is the string to be expected at the offset.<br/>

 ("v0.27.169.33a", 0x01BA29B0)
 ("v0.27.169.33b", 0x01B99FF0)
 ("v0.27.169.33c", 0x01BA29B0)
 ("v0.27.169.33d", 0x01B78670)


<br/>[[User:Sphr|Sphr]] 2007 Dec 9

:Have you verified that these offsets are <u>always</u> valid?  I ask because they are past the end of the static memory area, up in heap space.  I can easily see the allocator placing the memory block at different offsets on different computers, or even between different runs on the same computer.  I agree that we need a version ID, but at first glance, this doesn't look like a good way to me.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 17:53, 8 December 2007 (EST)
:: I personally use the "Dwarf Fortress" string, with the address shifts I doubt there will be an issue with that. --[[User:Rick|Rick]] 18:33, 8 December 2007 (EST)
:::I just looked at that and I disagree.  I don't think it's random enough.  So far it's been at:
:::* .32a: 0089238C
:::* .33a: 0089338C  delta +1000
:::* .33b: 0089438C  delta +1000
:::* .33c: 008993A4  delta +5018
:::* .33d: 0089A3A4  delta +1000<br/>
:::You're depending on that to be unique.   Or at least that if the string offset is the same, then the variable offsets are the same as well.  To me it feels like asking for trouble.
:::I don't have a good answer, but I feel that both of these proposals aren't reliable.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 22:37, 8 December 2007 (EST)
::::I agree, I just went with the "best" solution for now. --[[User:Rick|Rick]] 00:47, 9 December 2007 (EST)
:::::Ok, It seems that they are heap allocated.  which becomes invalid the moment it gets dumped.  This method is not working.  Are there any other working ways that people are already using?  forum thread mentioned fingerprinting using arbitrary chunks, would suffice too.  Anybody can post down the chunks they check for?  The other more evil way I see could be to use OS to take the size of the file + the CRC hash value.  [[User:Sphr|Sphr]] 2007 Dec 9
::::::Actually, the CRC value in the PE header would probably be a pretty good value to match against now that you mention it, since I doubt Toady fudges with it. --[[User:Rick|Rick]] 04:45, 9 December 2007 (EST)
:::::::The .CheckSum field of the .EXE's PE header is 0 in all released versions of DF.  Won't work.<br/>The .TimeDateStamp field has always been unique and probably always will be.<br/>That's at 004000F8 and is a DWORD.  It might be the best option.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 17:50, 9 December 2007 (EST)
::::::::Agreed, new versions of my tools (.33f, or whatever the next version is) will switch to this method. --[[User:Rick|Rick]] 12:56, 10 December 2007 (EST)
::::::I don't use the PE header which can be messed with. I tried using a fast asm based crc32 checksum function.  Seems to work, but I'll wait and see first.  thread is here[http://www.bay12games.com/cgi-local/ultimatebb.cgi?ubb=get_topic&f=13&t=000121]. [[User:Sphr|Sphr]] 2007 Dec 10
:::::::I don't understand the threat model this implies.  I wouldn't think the PE header is more susceptible to manipulation than the rest of the executable.  Less, actually.  The game is not packed, so the program text is exposed to anyone who wishes to do a permanent hexedit.  In comparison, PE header manipulation is both exotic and fragile, difficult to get right.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 17:41, 10 December 2007 (EST)
::::::::Edits to the PE header would also still throw off that CRC, too. --[[User:Rick|Rick]] 17:47, 10 December 2007 (EST)
:::::::::Don't get me wrong, it was in response to 0x517A5D's mentioning that checksum is all 0 in the pe header for the binaries.    What I mean by messing up isn't necessarily due to deliberate attempts.  It was referring to cases like this (though I find it strange that checksums are not updated automatically by filesystem).  I think the timestamp method could work as well.  Cheaper too.  Just throwing out this discussion coz perviously, there is no shared info at all on version identification.  So I'm just throwing ideas around to hope to hopefully "lure" out any better suggestions.   [[User:Sphr|Sphr]] 2007 Dec 11
::::::::::Why would the system update an executables checksum in the header? --[[User:Rick|Rick]] 00:33, 11 December 2007 (EST)
:::::::::::I don't mean the os fs.  As in the file io part of whoever created that pe.  Not very sure about pe, but are there os-specific libraries that does the actual creation or every compiler just does their own stuff?  Thought that the final step of computing crc32 and writing the value into the header (otherwise defeats the purpose of having a checksum) should be automatic (by whoever created the pe).  Sorry if my questions seemed dumb.  Did not deal with low level stuff for a long time. [[User:Sphr|Sphr]] 21:47, 11 December 2007 (EST)
::::::::::::To the best of my knowledge, the checksum field is not used on either executables or DLLs.  (It is used on device drivers.)  Every compiler comes with a linker; the linker does its own stuff, following the rules of PE layout.  I'm not aware of any linker that fills in the checksum field.  &mdash;[[User:0x517A5D|0x517A5D]] 03:40, 12 December 2007 (EST)
:::::::::::::I see.  Thanks for the info!  btw, since the timestamp seems to be a cheaper way that seems to work, can we set up a version identifying page that describes the method as well as publish known fingerprint values along with the binary versions?  As an alternative, maybe the crc32 can be placed there too, though if timestamp is working fine, there may be no need to record the values for future binary versions (unless the tool developer is paranoid, or in the not too possible future we may need to fingerprint non PE-structured files). [[User:Sphr|Sphr]] 21:16, 12 December 2007 (EST)

== Race ==
Does anyone care about creature race data?  All I've figured out is that it starts with 41 (!) strings, but it would be easy to compare fields to the data in the raw files.  Useful?<br/>&mdash;[[User:0x517A5D|0x517A5D]] 13:44, 18 December 2007 (EST)

: The ID/name would probably be good for any tool made to manipulate all creatures, as for race data, it could probably be parsed from raws instead of bothering with memory, after identifying which race it is. --[[User:Rick|Rick]] 15:38, 18 December 2007 (EST)

::Personally, I would find it useful.  Easier if everything is taken from the same process.  Loading from file is fine but that becomes another level of management and data synchronization and integrity ensurance can be troublesome. (e.g. Ensure that the order in memory is the order from file, which may not always be the case if code changes)  If we read the string table directly from memory, at least can be sure that the index indeed refers to a string regardless of any data jumbling between the loading part and the final resultant string table.  Was hoping that the race data vec and language data can get rid of all the hard-coded string tables in the tools once and for all. --[[User:Sphr|Sphr]] 21:23, 18 December 2007 (EST)

::I'm quite interested too, as discovering the details of the race struct would tremendously help the interpretation of some flags in the creature structure. Is there a macro to insert your signature with date?? --[[User:Bartavelle|Bartavelle]]

:::All right, I will spend some time on this.  Four tildes, like this: <nowiki>~~~~</nowiki><br/>&mdash;[[User:0x517A5D|0x517A5D]] 12:58, 19 December 2007 (EST)
::::Thanks in advance! @Bartavelle, it took me a while to figure that one out after reading wikimedia editing guide.  I used to key in dates manually... lol -- [[User:Sphr|Sphr]] 15:37, 19 December 2007 (EST)
:Okay.  It's in the .33f subpage.  I think I am not going to maintain it.  I will check each version to see if the object size changes, but that's about it.  <br/>&mdash;[[User:0x517A5D|0x517A5D]] 22:39, 19 December 2007 (EST)

::Yay! IDA structs! I'll check if that helps me decypher the flags, or not. [[User:Bartavelle|Bartavelle]] 02:13, 20 December 2007 (EST)
:::On that note, do you (or anyone) want the structures and enum in a file importable by IDA? &nbsp;&nbsp;&mdash;[[User:0x517A5D|0x517A5D]] 17:01, 20 December 2007 (EST)
@@ Line 15: / Line 15: @@
 # Has anyone attempted to decode the memory for the "Choose Fortress Location" screen? Being able to generate a list of all local areas with magma vents would be useful. [[User:Palin88|Palin88]] 20:09, 29 November 2007 (EST)
 :: This is a great idea. I suggest going one step further : Give the user a list of options (say : magma, flux, thick forest, chasm, underground river, pit, etc), and logically AND all parts of the map that have those features. That would save people hours of looking for their perfect start location. [[User:Runspotrun|Runspotrun]] 19:25, 8 December 2007 (EST)
-:::Hey gang, I'm looking for knowledgeable beta testers for a util that does something like this.  It's on my [[User:0x517A5D|wiki page]].  I do want feedback, so please respond with experiences and thoughts.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 03:10, 21 December 2007 (EST)
 # It could be important to some people, but the formats are only correct assuming that the dwarf fortress binary is built using release config.  Under debug config, std::string structure gains some more fat and the existing offset will be wrong.  Not a prob now, but I think it is good to know.  [[User:Sphr|Sphr]]
-Anyone know if it's possible to edit the amount of dwarves you start with(eg. 10 rather than 7)?
-:There used to be a utility that did this, called ''adjuststart.exe''.  I think it was one of [[User:Rick|Rick/Gibbed]]'s utilities.  I have versions of it for .23a130, .33a, .33b, .33e, .33f, and .33g.  In addition, the [[User:Rick/Tweak|Tweak]] tool has a module named Adjust Start, which probably does the same thing.  Tweak, which is also Rick/Gibbed's work, has not been updated to work with new versions of Dwarf Fortress.<br/>&mdash;[[User:0x517A5D|0x517A5D]] 04:23, 18 September 2009 (UTC)
 == Version Identity ==
@@ Line 72: / Line 69: @@
 ::Yay! IDA structs! I'll check if that helps me decypher the flags, or not. [[User:Bartavelle|Bartavelle]] 02:13, 20 December 2007 (EST)
 :::On that note, do you (or anyone) want the structures and enum in a file importable by IDA? &nbsp;&nbsp;&mdash;[[User:0x517A5D|0x517A5D]] 17:01, 20 December 2007 (EST)
-:::: Everybody with IDA would be happy I guess, even if that only means me :) [[User:Bartavelle|Bartavelle]] 02:11, 21 December 2007 (EST)
-:::::Mm hmm.  I'm having second thoughts, though.  Are you a legitimate, licensed user?<br/>&mdash;[[User:0x517A5D|0x517A5D]] 02:51, 21 December 2007 (EST)
-::::::I am. With hex-rays, bindiff, all the stuff. Anyway, why would this matter? It would just save the copying of the huge structure (like 15 minutes, but lazyness factor is high for this kind of work) ... [[User:Bartavelle|Bartavelle]] 04:12, 21 December 2007 (EST)
-:::::::Hex-rays, oooh.  Bin-diff, sweet.  I only have standard IDA 5.0; can't justify more than that for my job.  I was leary just `cause I don't want to promote piracy of IDA.  Though I have no moral high ground to stand on; I download multiple gigabytes of pirated material every day.  Here's my structures as of last night.  [http://www.yourfilehost.com/media.php?cat=other&file=df33f.idc]  This does not include some minor work done on other versions.  Also, the sole instantiation of the screen object starts at 015547E8; I forgot to comment that in the structure.  (Aside, does anyone know a better hosting site than [http://www.yourfilehost.com www.yourfilehost.com]?  I'm not too happy with them.)  <br/>&mdash;[[User:0x517A5D|0x517A5D]] 14:43, 21 December 2007 (EST)