User:Jifodus/Dwarf Fortress Utility Framework

I'm currently writing a framework for Dwarf Fortress utilities. The general idea is to use C++ interfaces in a cross-compiler fashion that is very easy to use. This is designed to assist utility writers, by making possible to compile once, and have it work on future versions of Dwarf Fortress.

Attention: This is just a quick check to see if anyone is looking at or using the DFUF in any of their projects. I'm thinking of rewriting the interfaces due to some limitations I've been encountering. I also plan on abstracating the environment away from Lua directly. If you are, please PM me on the forums, or leave a note on the talk page. I just want to avoid breaking an existing tool and/or forcing you to learn a new interface.

License

I'm not going to hold copyright of either the source code of the framework. It is in the Public Domain.

However, I will hold copyright of StartProfile. And it is licensed under the WTFPL: http://sam.zoy.org/wtfpl/

Links

First Beta Release: January 1, 2008

Source Code: http://www.geocities.com/jifodus/dfuf2.zip
Binaries: http://www.geocities.com/jifodus/dfufend2.zip
Debugging Symbols for the Binaries: http://www.geocities.com/jifodus/dfufdebug2.zip

This first beta release has additional interfaces for working with creatures, it also has more complete data files. It is now possible to cleanly create complex type overrides with PointerTo(), VectorOf(), and ArrayOf(). There is no data for v0.27.169.33g, the data is most complete for v0.27.169.33f.

Alpha Release

Source Code: http://www.geocities.com/jifodus/dfuf.zip
Binaries: http://www.geocities.com/jifodus/dfufend.zip
Debugging Symbols for the Binaries: http://www.geocities.com/jifodus/dfufdebug.zip

Todo

Before the second release, which is necessary before it can be easily used, the following items need to be taken care of.

• Bugs
  • There's always going to be a bug somewhere.
• Features
  • Main Library
    • Error Handling - The library itself does not cleanly handle common errors. 10% Completed
    • Auto-Update - Implement automatic update for both framework code and data. 0% Completed
    • Installer - An installer for the library. 10% Completed
    • Complete Memory Data - Complete transforming the memory data available on the wiki to format usable by library. 60% Completed
    • More Type Objects - Provide interface wrappers to some of the commonly used memory types (Creature and Map). 60% Completed
    • Memory Scanner - A simple memory searcher. Used for Auto-Identifier. 0% Completed
    • Auto-Identifier - Automatically try to find the memory locations and offsets for new versions. 0% Completed
    • Remote Memory De/Allocate - Needed to properly implement String & Vector set functions. 0% Completed
    • Reload Data - The data loading portion does not yet properly reload the data. 0% Completed
    • Bindings - C#/Perl/Java/Python/Lua/Other?

Features Overview

The Dwarf Fortress Utility Framework is designed with the following goals in mind:

• Utilities compiled for the very first release of the framework still work 10 years later
  • With all the updates and patches applied to the library
  • With the latest release of Dwarf Fortress
• The end user of a utility can simply download and run the utility and have it work
• The framework provides a flexible and easy to use way to define types and memory maps, and write utilities to use the types and memory maps

Current Features

• Written in C++, using interfaces
• Memory data stored in Lua format
• Sample utility for implementation reference: reimplementation of StartProfile
• API header & library
• Complete source code
• Identify Dwarf Fortress version via PE executable timestamp
• Cross-compiler/mostly C compatible using Windows COM style interfaces
  • It is not C compatible due to function overloading issues.
• Auto-loading of Lua data files
• Library self-contained in DLL

Current Limitations

• Buggy!
  • Memory leaks exist
• No real documentation whatsoever
• Sparse comments
• Memory data is all there, but the library does not yet use all of it
• The library is designed for C++, the C interface has not been tested (and probably will not work)
  • Also, this was only tested using MSVC++ .net 2005, it will probably work on other version of MSVC++.
• Does not cleanly handle errors
• Some interface functions not implemented
  • Map/Get through pointer functions not yet implemented
  • iDFUF::installDataFile is not yet implemented
• Compiled as debug mode (bloated file sizes)
• The library is not extensively tested
• The library has a dependency on: Visual Studio 8 Debug CRT

Second Release

The next release planned features will include:

• Auto-retrieve memory data off the internet
• Auto-update framework code itself
• Have an installer that will install the DLL and data file to a shared location, so multiple utilities can use the same library

Future

And further down the road (through auto-update):

• Implement specific subsets of the std library; reduce utility size more, no large dependencies for the utilities
  • std::string
  • std::vector
  • std::map
  • Console IO
  • File IO
• Easy to use GUI framework; for making tools with a nice GUI
• Cross-process memory allocation

Data Format

Requirements

The framework requires definitions of the following types:

• raw: a raw array of bytes; internally it allows access to an array of type.size * type.fixed_array bytes.
• pointer: a pointer to a location in Dwarf Fortress's memory; can represent a 32-bit pointer
• dword: an integer type that is 4 bytes
• word: an integer type that is 2 bytes
• byte: an integer type that is 1 byte
• float: a 32-bit floating point type
• double: a 64-bit floating point type
• string: a type that represents a std::string
  • required members:
    • dword length: defines how many characters string contains
    • dword capacity: defines the maximum number of characters the string buffer can contain
    • pointer buffer_ptr: a pointer to a memory location containing the string data
  • optional members:
    • raw buffer: a fixed-size array of characters containing the string when length < the fixed size of the buffer
• vector: a type representing a std::vector
  • required members:
    • pointer begin: a pointer to the begining of the memory block
    • pointer end: a pointer to just beyond of the last valid element in the vector
    • pointer last: a pointer to just beyond the end of the memory block

Data Files

The data files must supply the following information:

• Types: each element of the type list represents one type; there is a special type called Main, Main represents the global memory map
• Signatures: each signature is designed to uniquely identify each version of Dwarf Fortress

Lua Data Files

Classes:

• Type: 'type =' The value can either be a set containing a type override or the string of a type name.
  • Type: 'type =' A string representing the Type Name of the type to override.
  • Subtypes: An array of type overrides and/or Type Names, that are subtypes of this type.
  • Size: 'size =' An integer overriding the size of the type.
  • Fixed Array Size: 'fixed_size =' An integer overriding the fixed array size of the type.
• Types: 'Types'
  • Version: String representing version
    • TypeName: String representing name of the type, special type is Main.
      • Size: 'size =' an integer representing
      • Members: 'members' a table of of name value pairs; The name being the Member Name; The value is a set containing the Type and Offset.
        • Member Name: String representing the name
          • Type: 'type =' The value can either be a set containing a Type override or the name of the of a type.
          • Offset: 'offset =' The member offset (from the base address).
          • Pointer: 'pointer =' The pointer in memory for which the member can be found. Used for the Main type.
• Signatures 'Signatures'
  • Version: String representing the version (same string as types).
    • PE Timestamp: 'pe_timestamp' The PE header timestamp value.
    • .text Adler32: 'adler32' The Adler32 CRC of the ".text" section of the executable.
    • Text Segments: 'text_segments' An array of segments of the ".text" section of the executable.
      • Address: [1] = The offset into the ".text" section that the following raw data can be found.
      • Raw Data: [2] = The data the ".text" segment is supposed to contain.

Library Usage

(This part is only useful for utility writers.)

Basic Programming

The basic program needed to use the library is simply:

#include <dfuf.h>
void main() {
	dfuf::iDFUF *uf = dfuf::newDFUF();
	// Do something with the framework
	uf->destroy(); // in theory cleans up all memory used
}

To connect to an instance of Dwarf Fortress the program must first scan for instances, then get one of the instances of dwarf fortress like this:

if (uf->scanForInstances() == 0) 
	// none found
	return;
dfuf::iDFInstance *instance = uf->getInstance(0);

After getting a Dwarf Fortress instance, it then becomes possible to access global pointers and memory locations (i.e. creature vector location) with iDFInstance::getMemoryObject. iDFInstance::getMemoryObject takes a iType* and an Address in Dwarf Fortress memory and returns an appropriate iMemoryType*. There are different ways to provide the iType* and the Address, including from the data file (via the appropriate name), a iPointerType*, and an iPointer* and iType* lets the code create from the raw parts. There is a second version of iDFInstance::getMemoryObject, which follows all the pointers to a non-pointer object and returns the object called iDFInstance::getMemoryObjectThroughPointers. However, it has not been implemented in the first release of the library.

From the first memory object created, the code can then either query the value (from any of the i*Type) or map other members (from any of the i*Object) objects.

Advanced Programming

An advanced topic is iMemoryType* creation. It is painful to individually map the members, therefore creating an iMemoryType* that simplifies access to the type sounds ideal. However, the library only knows about the types it has been compiled with. Enter iTypeFactory*, the iTypeFactory* gets registered in iDFUF with iDFUF::addTypeFactory. Each factory provides 2 functions: construct & destruct. This allows programs to create their own structures to wrap the creature type.

Example code:

class cCreatureType : public dfuf::iMemoryObject
{
public:
	// Implement virtual methods here

public:
	dfuf::u32 getHappiness() { return happiness->getValue(); }
	void setHappiness(dfuf::u32 value) { happiness->setValue(value); }
	// More methods here
	cCreatureType(dfuf::iDFInstance *instance, dfuf::iPointer *base, dfuf::iType *type)
	{
		this->instance = instance;
		this->base = base;
		this->type = type;
		this->happiness = map(L"happiness");
	}
public:
	dfuf::iDFInstance *instance;
	dfuf::iPointer *base;
	dfuf::iType *type;
	dfuf::iIntegerType *happiness;
};
class cCreatureFactory : public dfuf::iTypeFactory
{
public:
	// new creature
	{
		return new cCreatureType(instance, base, type);
	}
	// delete creature
	{
		delete object;
	}
};
// Global instance, otherwise we'd have to do memory management on the pointer
cCreatureFactory global_CreatureFactory;
// Usage
void main()
{
	dfuf::iDFUF *uf = newDFUF();
	dfuf::iDFInstance *instance = uf->getDFInstance(0);
	uf->addTypeFactory("creature", &global_CreatureFactory);
	iPointer *pointer;
	cCreatureType *creature = (cCreatureType *)instance->getMemoryObject
		(pointer, instance->getType("creature"));
	creature->setHappiness(-10000); // hahaha die dwarf!
	uf->destroy();
}