Editing User:Jifodus/Dwarf Fortress Utility Framework

I'm currently writing a framework for Dwarf Fortress utilities. The general idea is to use C++ interfaces in a cross-compiler fashion that is very easy to use. This is designed to assist utility writers, by making possible to compile once, and have it work on future versions of Dwarf Fortress.

'''Attention:''' This is just a quick check to see if anyone is looking at or using the DFUF in any of their projects. I'm thinking of rewriting the interfaces due to some limitations I've been encountering. I also plan on abstracating the environment away from Lua directly. If you are, please PM me on the forums, or leave a note on the talk page. I just want to avoid breaking an existing tool and/or forcing you to learn a new interface.

==License==
I'm not going to hold copyright of either the source code of the framework. It is in the Public Domain.

However, I will hold copyright of StartProfile. And it is licensed under the WTFPL: http://sam.zoy.org/wtfpl/

==Links==
===First Beta Release: January 1, 2008===
Source Code: http://www.geocities.com/jifodus/dfuf2.zip<br />
Binaries: http://www.geocities.com/jifodus/dfufend2.zip<br />
Debugging Symbols for the Binaries: http://www.geocities.com/jifodus/dfufdebug2.zip

'''This first beta release has additional interfaces for working with creatures, it also has more complete data files. It is now possible to cleanly create complex type overrides with ''PointerTo()'', ''VectorOf()'', and ''ArrayOf()''. There is no data for v0.27.169.33g, the data is most complete for v0.27.169.33f.'''
===Alpha Release===
Source Code: http://www.geocities.com/jifodus/dfuf.zip<br />
Binaries: http://www.geocities.com/jifodus/dfufend.zip<br />
Debugging Symbols for the Binaries: http://www.geocities.com/jifodus/dfufdebug.zip

==Todo==
Before the second release, which is necessary before it can be easily used, the following items need to be taken care of.
* Bugs
** Static Library
*** '''dfuf::newDFUF()''' - Will not behave as expected. In case of failures from checking HKEY_CURRENT_USER, it won't always check HKEY_LOCAL_MACHINE.
* Features
** Main Library
*** '''Error Handling''' - The library itself does not cleanly handle common errors.
*** '''Through Pointer''' - Implement the get/map through pointer functions, as well as in data.
*** '''Auto-Update''' - Implement automatic update for both framework code and data.
*** '''Installer''' - An installer for the library.
*** '''Complete Memory Data''' - Complete transforming the memory data available on the wiki to format usable by library.
*** '''More Type Objects''' - Provide interface wrappers to some of the commonly used memory types (Creature and Map).
*** '''Memory Scanner''' - A simple memory searcher. Used for '''Auto-Identifier'''.
*** '''Auto-Identifier''' - Automatically try to find the memory locations and offsets for new version.
*** '''Remote Memory De/Allocate''' - Needed to properly implement String & Vector set functions.
*** '''Reload Data''' - The data loading portion does not yet properly reload the data.
*** '''Bindings''' - C#/Perl/Java/Python/Lua/Other?

==Features Overview==
The Dwarf Fortress Utility Framework is designed with the following goals in mind:
* Utilities compiled for the very first release of the framework still work 10 years later
** With all the updates and patches applied to the library
** With the latest release of Dwarf Fortress
* The end user of a utility can simply download and run the utility and have it work
* The framework provides a flexible and easy to use way to define types and memory maps, and write utilities to use the types and memory maps

===Current Features===
* Written in C++, using interfaces
* Memory data stored in Lua format
* Sample utility for implementation reference: reimplementation of StartProfile
* API header & library
* Complete source code
* Identify Dwarf Fortress version via PE executable timestamp
* Cross-compiler/mostly C compatible using Windows COM style interfaces
** It is not C compatible due to function overloading issues.
* Auto-loading of Lua data files
* Library self-contained in DLL

===Current Limitations===
* '''Buggy!'''
** Memory leaks exist
* No real documentation whatsoever
* Sparse comments
* Memory data is all there, but the library does not yet use all of it
* The library is designed for C++, the C interface has not been tested (and probably will not work)
** Also, this was only tested using MSVC++ .net 2005, it will probably work on other version of MSVC++.
* Does not cleanly handle errors
* Some interface functions not implemented
** Map/Get through pointer functions not yet implemented
** iDFUF::installDataFile is not yet implemented
* Compiled as debug mode (bloated file sizes)
* The library is not extensively tested
* The library has a dependency on: [http://www.microsoft.com/downloads/details.aspx?FamilyId=32BC1BEE-A3F9-4C13-9C99-220B62A191EE&displaylang=en Visual Studio 8 Debug CRT]

===Second Release===
The next release planned features will include:
* Auto-retrieve memory data off the internet
* Auto-update framework code itself
* Have an installer that will install the DLL and data file to a shared location, so multiple utilities can use the same library

===Future===
And further down the road (through auto-update):
* Implement specific subsets of the std library; reduce utility size more, no large dependencies for the utilities
** std::string
** std::vector
** std::map
** Console IO
** File IO
* Easy to use GUI framework; for making tools with a nice GUI
* Cross-process memory allocation

==Data Format==

===Requirements===
The framework requires definitions of the following types:
* raw: a raw array of bytes; internally it allows access to an array of type.size * type.fixed_array bytes.
* pointer: a pointer to a location in Dwarf Fortress's memory; can represent a 32-bit pointer
* dword: an integer type that is 4 bytes
* word: an integer type that is 2 bytes
* byte: an integer type that is 1 byte
* float: a 32-bit floating point type
* double: a 64-bit floating point type
* string: a type that represents a std::string
** required members:
*** '''dword''' length: defines how many characters string contains
*** '''dword''' capacity: defines the maximum number of characters the string buffer can contain
*** '''pointer''' buffer_ptr: a pointer to a memory location containing the string data
** optional members:
*** '''raw''' buffer: a fixed-size array of characters containing the string when length < the fixed size of the buffer
* vector: a type representing a std::vector
** required members:
*** '''pointer''' begin: a pointer to the begining of the memory block
*** '''pointer''' end: a pointer to just beyond of the last valid element in the vector
*** '''pointer''' last: a pointer to just beyond the end of the memory block

===Data Files===
The data files must supply the following information:
* Types: each element of the type list represents one type; there is a special type called '''Main''', '''Main''' represents the global memory map
* Signatures: each signature is designed to uniquely identify each version of Dwarf Fortress

====Lua Data Files====
Classes:
* '''Type''': 'type =' The value can either be a set containing a type override or the string of a type name.
** Type: 'type =' A string representing the Type Name of the type to override.
** Subtypes: An array of type overrides and/or Type Names, that are subtypes of this type.
** Size: 'size =' An integer overriding the size of the type.
** Fixed Array Size: 'fixed_size =' An integer overriding the fixed array size of the type.
* Types: 'Types'
** Version: String representing version
*** TypeName: String representing name of the type, special type is '''Main'''.
**** Size: 'size =' an integer representing
**** Members: 'members' a table of of name value pairs; The name being the Member Name; The value is a set containing the Type and Offset.
***** Member Name: String representing the name
****** Type: 'type =' The value can either be a set containing a '''Type''' override or the name of the of a type.
****** Offset: 'offset =' The member offset (from the base address).
****** Pointer: 'pointer =' The pointer in memory for which the member can be found. Used for the '''Main''' type.
* Signatures 'Signatures'
** Version: String representing the version (same string as types).
*** PE Timestamp: 'pe_timestamp' The PE header timestamp value.
*** .text Adler32: 'adler32' The Adler32 CRC of the ".text" section of the executable.
*** Text Segments: 'text_segments' An array of segments of the ".text" section of the executable.
**** Address: [1] = The offset into the ".text" section that the following raw data can be found.
**** Raw Data: [2] = The data the ".text" segment is supposed to contain.

==Library Usage==
(This part is only useful for utility writers.)

===Basic Programming===
The basic program needed to use the library is simply:
 #include <dfuf.h>
 void main() {
 	dfuf::iDFUF *uf = dfuf::newDFUF();
 	// Do something with the framework
 	uf->destroy(); // in theory cleans up all memory used
 }
To connect to an instance of Dwarf Fortress the program must first scan for instances,
then get one of the instances of dwarf fortress like this:
 if (uf->scanForInstances() == 0) 
 	// none found
 	return;
 dfuf::iDFInstance *instance = uf->getInstance(0);
After getting a Dwarf Fortress instance, it then becomes possible to access global
pointers and memory locations (i.e. creature vector location) with
iDFInstance::getMemoryObject. iDFInstance::getMemoryObject takes a iType* and an
Address in Dwarf Fortress memory and returns an appropriate iMemoryType*. There are
different ways to provide the iType* and the Address, including from the data file
(via the appropriate name), a iPointerType*, and an iPointer* and iType* lets the
code create from the raw parts. There is a second version of
iDFInstance::getMemoryObject, which follows all the pointers to a non-pointer object
and returns the object called iDFInstance::getMemoryObjectThroughPointers. However,
it has not been implemented in the first release of the library.

From the first memory object created, the code can then either query the value (from
any of the i*Type) or map other members (from any of the i*Object) objects.

===Advanced Programming===
An advanced topic is iMemoryType* creation. It is painful to individually map the
members, therefore creating an iMemoryType* that simplifies access to the type sounds
ideal. However, the library only knows about the types it has been compiled with. Enter
iTypeFactory*, the iTypeFactory* gets registered in iDFUF with iDFUF::addTypeFactory.
Each factory provides 2 functions: construct & destruct. This allows programs to create
their own structures to wrap the creature type.

Example code:
 class cCreatureType : public dfuf::iMemoryObject
 {
 public:
 	// Implement virtual methods here
 
 public:
 	dfuf::u32 getHappiness() { return happiness->getValue(); }
 	void setHappiness(dfuf::u32 value) { happiness->setValue(value); }
 	// More methods here
 	cCreatureType(dfuf::iDFInstance *instance, dfuf::iPointer *base, dfuf::iType *type)
 	{
 		this->instance = instance;
 		this->base = base;
 		this->type = type;
 		this->happiness = map(L"happiness");
 	}
 public:
 	dfuf::iDFInstance *instance;
 	dfuf::iPointer *base;
 	dfuf::iType *type;
 	dfuf::iIntegerType *happiness;
 };
 class cCreatureFactory : public dfuf::iTypeFactory
 {
 public:
 	// new creature
 	{
 		return new cCreatureType(instance, base, type);
 	}
 	// delete creature
 	{
 		delete object;
 	}
 };
 // Global instance, otherwise we'd have to do memory management on the pointer
 cCreatureFactory global_CreatureFactory;
 // Usage
 void main()
 {
 	dfuf::iDFUF *uf = newDFUF();
 	dfuf::iDFInstance *instance = uf->getDFInstance(0);
 	uf->addTypeFactory("creature", &global_CreatureFactory);
 	iPointer *pointer;
 	cCreatureType *creature = (cCreatureType *)instance->getMemoryObject
 		(pointer, instance->getType("creature"));
 	creature->setHappiness(-10000); // hahaha die dwarf!
 	uf->destroy();
 }

[[Category:Hacking|Dwarf Fortress Utility Framework]]