Editing User:Jifodus/Dwarf Fortress Utility Framework

I'm currently writing a framework for Dwarf Fortress utilities. The general idea is to use C++ interfaces in a cross-compiler fashion that is very easy to use.

Please note, since this is still currently under development, I don't have the time to make a pretty WikiPage. I also will not post the interface & file format document for the same reason. Once there is a release, I will have a solid data format that will probably never change, as well as unchanging interfaces.

I'm scheduling the initial release date to be December 15, 2007. At that time, the following features will be in place:
* Written in C++, using interfaces
* Memory data
** Stored in a Lua format
** Stored in a CSV format
** Stored in an XML format?
* Sample utility for implementation reference: reimplementation of StartProfile
* Library self-contained in DLL
* API header & library
* Cross-compiler portable
* Complete source code

After the first release, and it's been thouroghly bug checked, the next release will have:
* Auto-retrieve memory data off the internet
* Auto-update framework code itself
* Have an installer that will install the DLL and data file to a shared location, so multiple utilities can use the same library
* Implement specific subsets of the std library; reduce utility size more, no large dependencies
** std::string
** std::vector
** Console IO
** File IO

And further down the road (through auto-update):
* Easy to use GUI framework; for making tools with a nice GUI
* Cross-process memory allocation

Expected Data Format/Requirements

Native/internal type strings:
* string
* vector
* pointer
* dword
* word
* byte
* float
* double
* raw (array fixed-size array of bytes; size defined by the type)

Object Data Requirements

* string
** length : dword -- The length of the string
** capacity : dword -- The capacity of the string
** buffer : raw -- [Optional] The 16 byte string buffer embedded in the string object
** buffer_ptr : pointer -- A pointer to the character data
* vector
** begin : pointer -- The start of the vector
** end : pointer -- The end of the vector
** last : pointer -- The just beyond the last possible item in the vector

Requirements For Data Files

Stuff between {} denotes a set of information

 -- To format a new type
 {
  type_name, -- [Required]
  size, -- [Optional], [Default: 0] In bytes
  -- Any number of members can be included into a type definition
  member = {
   type, -- [Required]
   offset, -- [Required]
   size, -- [Optional]
   subtypes = { -- [Optional], Used to specify the type the vector wraps
    array,
    of,
    more,
    types
   }, -- (end subtypes)
   fixed_size_array_count -- [Optional] [Default: 1]
  }, -- (end member)
  -- more?
 }

 -- To format a memory location
 {
  pointer_name,
  type,
  address
 }

So for example a string can be formatted like this:

 {
  "string",
  28, -- 4 + 16 + 4 + 4
  "length" = { dword, 20 },
  "capacity" = { dword, 24 },
  "buffer" = { type = raw, offset = 4, fixed_size_array_count = 16 }
  "buffer_ptr" = { type = pointer, offset = 4 }
 }

A pointer to the creature vector could be

 {
  "main_creature_vec_location",
  { type = vector, subtypes = { creature } },
  "0x0141FA30"
 }

Of course, this is just a basic idea of what the data files will need. The data will also have to at least include version information.

Anyway, a quick overview of the framework as it's supposed to be for the first release.
@@ Line 1: / Line 1: @@
-I'm currently writing a framework for Dwarf Fortress utilities. The general idea is to use C++ interfaces in a cross-compiler fashion that is very easy to use. This is designed to assist utility writers, by making possible to compile once, and have it work on future versions of Dwarf Fortress.
+I'm currently writing a framework for Dwarf Fortress utilities. The general idea is to use C++ interfaces in a cross-compiler fashion that is very easy to use.
-'''Attention:''' This is just a quick check to see if anyone is looking at or using the DFUF in any of their projects. I'm thinking of rewriting the interfaces due to some limitations I've been encountering. I also plan on abstracating the environment away from Lua directly. If you are, please PM me on the forums, or leave a note on the talk page. I just want to avoid breaking an existing tool and/or forcing you to learn a new interface.
+Please note, since this is still currently under development, I don't have the time to make a pretty WikiPage. I also will not post the interface & file format document for the same reason. Once there is a release, I will have a solid data format that will probably never change, as well as unchanging interfaces.
-==License==
+I'm scheduling the initial release date to be December 15, 2007. At that time, the following features will be in place:
-I'm not going to hold copyright of either the source code of the framework. It is in the Public Domain.
-However, I will hold copyright of StartProfile. And it is licensed under the WTFPL: http://sam.zoy.org/wtfpl/
-==Links==
-===First Beta Release: January 1, 2008===
-Source Code: http://www.geocities.com/jifodus/dfuf2.zip<br />
-Binaries: http://www.geocities.com/jifodus/dfufend2.zip<br />
-Debugging Symbols for the Binaries: http://www.geocities.com/jifodus/dfufdebug2.zip
-'''This first beta release has additional interfaces for working with creatures, it also has more complete data files. It is now possible to cleanly create complex type overrides with ''PointerTo()'', ''VectorOf()'', and ''ArrayOf()''. There is no data for v0.27.169.33g, the data is most complete for v0.27.169.33f.'''
-===Alpha Release===
-Source Code: http://www.geocities.com/jifodus/dfuf.zip<br />
-Binaries: http://www.geocities.com/jifodus/dfufend.zip<br />
-Debugging Symbols for the Binaries: http://www.geocities.com/jifodus/dfufdebug.zip
-==Todo==
-Before the second release, which is necessary before it can be easily used, the following items need to be taken care of.
-* Bugs
-** There's always going to be a bug somewhere.
-* Features
-** Main Library
-*** '''Error Handling''' - The library itself does not cleanly handle common errors. '''''10% Completed'''''
-*** '''Auto-Update''' - Implement automatic update for both framework code and data. '''''0% Completed'''''
-*** '''Installer''' - An installer for the library. '''''10% Completed'''''
-*** '''Complete Memory Data''' - Complete transforming the memory data available on the wiki to format usable by library. '''''60% Completed'''''
-*** '''More Type Objects''' - Provide interface wrappers to some of the commonly used memory types (Creature and Map). '''''60% Completed'''''
-*** '''Memory Scanner''' - A simple memory searcher. Used for '''Auto-Identifier'''. '''''0% Completed'''''
-*** '''Auto-Identifier''' - Automatically try to find the memory locations and offsets for new versions. '''''0% Completed'''''
-*** '''Remote Memory De/Allocate''' - Needed to properly implement String & Vector set functions. '''''0% Completed'''''
-*** '''Reload Data''' - The data loading portion does not yet properly reload the data. '''''0% Completed'''''
-*** '''Bindings''' - C#/Perl/Java/Python/Lua/Other?
-==Features Overview==
-The Dwarf Fortress Utility Framework is designed with the following goals in mind:
-* Utilities compiled for the very first release of the framework still work 10 years later
-** With all the updates and patches applied to the library
-** With the latest release of Dwarf Fortress
-* The end user of a utility can simply download and run the utility and have it work
-* The framework provides a flexible and easy to use way to define types and memory maps, and write utilities to use the types and memory maps
-===Current Features===
 * Written in C++, using interfaces
-* Memory data stored in Lua format
+* Memory data
+** Stored in a Lua format
+** Stored in a CSV format
+** Stored in an XML format?
 * Sample utility for implementation reference: reimplementation of StartProfile
+* Library self-contained in DLL
 * API header & library
+* Cross-compiler portable
 * Complete source code
-* Identify Dwarf Fortress version via PE executable timestamp
-* Cross-compiler/mostly C compatible using Windows COM style interfaces
-** It is not C compatible due to function overloading issues.
-* Auto-loading of Lua data files
-* Library self-contained in DLL
-===Current Limitations===
+After the first release, and it's been thouroghly bug checked, the next release will have:
-* '''Buggy!'''
-** Memory leaks exist
-* No real documentation whatsoever
-* Sparse comments
-* Memory data is all there, but the library does not yet use all of it
-* The library is designed for C++, the C interface has not been tested (and probably will not work)
-** Also, this was only tested using MSVC++ .net 2005, it will probably work on other version of MSVC++.
-* Does not cleanly handle errors
-* Some interface functions not implemented
-** Map/Get through pointer functions not yet implemented
-** iDFUF::installDataFile is not yet implemented
-* Compiled as debug mode (bloated file sizes)
-* The library is not extensively tested
-* The library has a dependency on: [http://www.microsoft.com/downloads/details.aspx?FamilyId=32BC1BEE-A3F9-4C13-9C99-220B62A191EE&displaylang=en Visual Studio 8 Debug CRT]
-===Second Release===
-The next release planned features will include:
 * Auto-retrieve memory data off the internet
 * Auto-update framework code itself
 * Have an installer that will install the DLL and data file to a shared location, so multiple utilities can use the same library
+* Implement specific subsets of the std library; reduce utility size more, no large dependencies
-===Future===
-And further down the road (through auto-update):
-* Implement specific subsets of the std library; reduce utility size more, no large dependencies for the utilities
 ** std::string
 ** std::vector
-** std::map
 ** Console IO
 ** File IO
+And further down the road (through auto-update):
 * Easy to use GUI framework; for making tools with a nice GUI
 * Cross-process memory allocation
-==Data Format==
+Expected Data Format/Requirements
+Native/internal type strings:
+* string
+* vector
+* pointer
+* dword
+* word
+* byte
+* float
+* double
+* raw (array fixed-size array of bytes; size defined by the type)
-===Requirements===
+Object Data Requirements
-The framework requires definitions of the following types:
-* raw: a raw array of bytes; internally it allows access to an array of type.size * type.fixed_array bytes.
-* pointer: a pointer to a location in Dwarf Fortress's memory; can represent a 32-bit pointer
-* dword: an integer type that is 4 bytes
-* word: an integer type that is 2 bytes
-* byte: an integer type that is 1 byte
-* float: a 32-bit floating point type
-* double: a 64-bit floating point type
-* string: a type that represents a std::string
-** required members:
-*** '''dword''' length: defines how many characters string contains
-*** '''dword''' capacity: defines the maximum number of characters the string buffer can contain
-*** '''pointer''' buffer_ptr: a pointer to a memory location containing the string data
-** optional members:
-*** '''raw''' buffer: a fixed-size array of characters containing the string when length < the fixed size of the buffer
-* vector: a type representing a std::vector
-** required members:
-*** '''pointer''' begin: a pointer to the begining of the memory block
-*** '''pointer''' end: a pointer to just beyond of the last valid element in the vector
-*** '''pointer''' last: a pointer to just beyond the end of the memory block
-===Data Files===
+* string
-The data files must supply the following information:
+** length : dword -- The length of the string
-* Types: each element of the type list represents one type; there is a special type called '''Main''', '''Main''' represents the global memory map
+** capacity : dword -- The capacity of the string
-* Signatures: each signature is designed to uniquely identify each version of Dwarf Fortress
+** buffer : raw -- [Optional] The 16 byte string buffer embedded in the string object
+** buffer_ptr : pointer -- A pointer to the character data
+* vector
+** begin : pointer -- The start of the vector
+** end : pointer -- The end of the vector
+** last : pointer -- The just beyond the last possible item in the vector
-====Lua Data Files====
+Requirements For Data Files
-Classes:
-* '''Type''': 'type =' The value can either be a set containing a type override or the string of a type name.
-** Type: 'type =' A string representing the Type Name of the type to override.
-** Subtypes: An array of type overrides and/or Type Names, that are subtypes of this type.
-** Size: 'size =' An integer overriding the size of the type.
-** Fixed Array Size: 'fixed_size =' An integer overriding the fixed array size of the type.
-* Types: 'Types'
-** Version: String representing version
-*** TypeName: String representing name of the type, special type is '''Main'''.
-**** Size: 'size =' an integer representing
-**** Members: 'members' a table of of name value pairs; The name being the Member Name; The value is a set containing the Type and Offset.
-***** Member Name: String representing the name
-****** Type: 'type =' The value can either be a set containing a '''Type''' override or the name of the of a type.
-****** Offset: 'offset =' The member offset (from the base address).
-****** Pointer: 'pointer =' The pointer in memory for which the member can be found. Used for the '''Main''' type.
-* Signatures 'Signatures'
-** Version: String representing the version (same string as types).
-*** PE Timestamp: 'pe_timestamp' The PE header timestamp value.
-*** .text Adler32: 'adler32' The Adler32 CRC of the ".text" section of the executable.
-*** Text Segments: 'text_segments' An array of segments of the ".text" section of the executable.
-**** Address: [1] = The offset into the ".text" section that the following raw data can be found.
-**** Raw Data: [2] = The data the ".text" segment is supposed to contain.
-==Library Usage==
+Stuff between {} denotes a set of information
-(This part is only useful for utility writers.)
-===Basic Programming===
+ -- To format a new type
-The basic program needed to use the library is simply:
+ {
- #include <dfuf.h>
+  type_name, -- [Required]
- void main() {
+  size, -- [Optional], [Default: 0] In bytes
- 	dfuf::iDFUF *uf = dfuf::newDFUF();
+  -- Any number of members can be included into a type definition
- 	// Do something with the framework
+  member = {
- 	uf->destroy(); // in theory cleans up all memory used
+   type, -- [Required]
+   offset, -- [Required]
+   size, -- [Optional]
+   subtypes = { -- [Optional], Used to specify the type the vector wraps
+    array,
+    of,
+    more,
+    types
+   }, -- (end subtypes)
+   fixed_size_array_count -- [Optional] [Default: 1]
+  }, -- (end member)
+  -- more?
   }
-To connect to an instance of Dwarf Fortress the program must first scan for instances,
-then get one of the instances of dwarf fortress like this:
- if (uf->scanForInstances() == 0)
- 	// none found
- 	return;
- dfuf::iDFInstance *instance = uf->getInstance(0);
-After getting a Dwarf Fortress instance, it then becomes possible to access global
-pointers and memory locations (i.e. creature vector location) with
-iDFInstance::getMemoryObject. iDFInstance::getMemoryObject takes a iType* and an
-Address in Dwarf Fortress memory and returns an appropriate iMemoryType*. There are
-different ways to provide the iType* and the Address, including from the data file
-(via the appropriate name), a iPointerType*, and an iPointer* and iType* lets the
-code create from the raw parts. There is a second version of
-iDFInstance::getMemoryObject, which follows all the pointers to a non-pointer object
-and returns the object called iDFInstance::getMemoryObjectThroughPointers. However,
-it has not been implemented in the first release of the library.
-From the first memory object created, the code can then either query the value (from
+ -- To format a memory location
-any of the i*Type) or map other members (from any of the i*Object) objects.
+ {
+  pointer_name,
+  type,
+  address
+ }
-===Advanced Programming===
+So for example a string can be formatted like this:
-An advanced topic is iMemoryType* creation. It is painful to individually map the
-members, therefore creating an iMemoryType* that simplifies access to the type sounds
-ideal. However, the library only knows about the types it has been compiled with. Enter
-iTypeFactory*, the iTypeFactory* gets registered in iDFUF with iDFUF::addTypeFactory.
-Each factory provides 2 functions: construct & destruct. This allows programs to create
-their own structures to wrap the creature type.
-Example code:
- class cCreatureType : public dfuf::iMemoryObject
   {
- public:
+  "string",
- 	// Implement virtual methods here
+, -- 4 + 16 + 4 + 4
+  "length" = { dword, 20 },
- public:
+  "capacity" = { dword, 24 },
- 	dfuf::u32 getHappiness() { return happiness->getValue(); }
+  "buffer" = { type = raw, offset = 4, fixed_size_array_count = 16 }
- 	void setHappiness(dfuf::u32 value) { happiness->setValue(value); }
+  "buffer_ptr" = { type = pointer, offset = 4 }
- 	// More methods here
+  }
- 	cCreatureType(dfuf::iDFInstance *instance, dfuf::iPointer *base, dfuf::iType *type)
- 	{
+A pointer to the creature vector could be
- 		this->instance = instance;
- 		this->base = base;
- 		this->type = type;
- 		this->happiness = map(L"happiness");
- 	}
-  public:
- 	dfuf::iDFInstance *instance;
- 	dfuf::iPointer *base;
- 	dfuf::iType *type;
- 	dfuf::iIntegerType *happiness;
- };
- class cCreatureFactory : public dfuf::iTypeFactory
   {
- public:
+  "main_creature_vec_location",
- 	// new creature
+  { type = vector, subtypes = { creature } },
- 	{
+  "0x0141FA30"
- 		return new cCreatureType(instance, base, type);
- 	}
- 	// delete creature
- 	{
- 		delete object;
- 	}
- };
- // Global instance, otherwise we'd have to do memory management on the pointer
- cCreatureFactory global_CreatureFactory;
- // Usage
- void main()
- {
- 	dfuf::iDFUF *uf = newDFUF();
- 	dfuf::iDFInstance *instance = uf->getDFInstance(0);
- 	uf->addTypeFactory("creature", &global_CreatureFactory);
- 	iPointer *pointer;
- 	cCreatureType *creature = (cCreatureType *)instance->getMemoryObject
- 		(pointer, instance->getType("creature"));
- 	creature->setHappiness(-10000); // hahaha die dwarf!
- 	uf->destroy();
   }
-[[Category:Hacking|Dwarf Fortress Utility Framework]]
+Of course, this is just a basic idea of what the data files will need. The data will also have to at least include version information.
+Anyway, a quick overview of the framework as it's supposed to be for the first release.