New Challenge Script (CHL) compiler

Daniels118

Daniels118

New member
Joined
Jul 31, 2023
Messages
17
Hi all,
this is my first post on this forum, so... nice to meet you all!

A brief introduction about me and B&W
I started playing B&W when I was a child, and I played the story mode again about 3 times over the years. The game mechanic is fantastic and the story too, but after playing it so many times it become a bit ovious. I always desired a new story to play, or even better write my own to share with others. Searching the web I've found 2 tools to compile and decompile the files used by the game to generate the story (scrolls), but they are very old, buggy, and lack some essential usage info. Since I'm an enthusiast programmer, about 2 weeks ago I contacted the developer of those tools who recently republished them to ask him for the source code so that I could eventually improve it, but unfortunately he losts the sources, so I decided to write my own trying to do better. And here it is!

Download and installation
I have published my work on GitHub under GPL, this way the source code won't get lost anymore:
https://github.com/Daniels118/blackandwhite

The binaries can be downloaded from the release section on the right, extract them and put the directory on the system variable PATH to be able to run it from everywhere.

Usage
Be sure to backup the original CHL file first!

CHL/ASM conversion
chlasm -chlasm -i challenge.chl -p prjdir to convert from compiled CHL to assembly project; prjdir must exist.
chlasm -asmchl -p prjdir/_project.txt -o challenge.chl to convert from assembly project to compiled CHL.
or
chlasm -chlasm -i challenge.chl -o challenge.asm to convert from compiled CHL to single assembly file.
chlasm -asmchl -i challenge.asm -o challenge.chl to convert from single assembly file to compiled CHL.

Compiling from CHL sources
chlasm -compile -p prjdir/_project.txt -o challenge.chl
The download contains a directory named "samples" which provides a basic setup.

The full syntax can be displayed by running the command without any argument.

Comments are welcome!
 
Last edited:
I share the sentiment of most of your original post. I am looking to create new challenges and make a new story for B&W. I am aware of the old script editors and would like to follow yours also. I would also appreciate it if you could tell me a bit more about how the scripts are actually made. What's the syntax? When you are converting between .chl and "assembly", are we talking about x86 assembly? If so, how is that used to make B&W challenges? If not, is it some sort of B&W specific assembly? I have prior experience in programming and assembly code and would love to know more if there is any info you could provide me with.

Currently I am in a position where the only custom scripts I have made for B&W have been skirmish maps and their land files so that should explain roughly what knowledge I'm coming into this with. I have barely touched on editing the textures and sound data but outside of that I have literally no idea how the game is constructed internally.

In any case, love what you're doing - keep things open source and not lost! B&W is a timeless classic that needs to be preserved and any tools that help deconstruct it are a good thing.
 
Hi QuantumFluctuator, B&W uses 2 languages:
a) map script: is an interpreted language used to initialize maps (which probably you already know). As far as I know, this language hasn't control flow statements, so it doesn't allow to implement any kind of logic. Examples can be found as plain text files under the "Scripts" folder (eg. Scripts\Land1.txt). An example statement:
Code: 
CREATE_ABODE(0, "1755.09,2692.52", "NORSE_ABODE_C", 4710, 990, 0, 0)
b) challenge language (CHL): is the language used to create challenges, i.e. silver and gold scrolls found in the story mode. This is a proprietary, structured, compiled language, like C or VB, and was originally undisclosed. I recently discovered that this language was then made available by LH with the Creature Isle expansion, along with a compiler (although it is said that can also be compiled at runtime) and a detailed documentation of all available statements. The original scripts weren't released in source format, but they were present (probably for a mistake) in the MacOS version of the game released one year after the Windows version.
This language is compiled into bytecode that runs on a stack-based virtual machine (the Lionhead Virtual Machine, or LHVM), exactly the same way of Java.
This bytecode is made of 31 opcodes which can be translated back and forth an ASM language. Please note that LH has never released or made any reference to such language, it's just a human readable translation of the bytecode made by reverse enginering the bytecode itself and the LHVM. There are opcodes similar to the ones you would find on a x86 processor (such as PUSH and POP), but since the LHVM is stack-based there are no registers (from this point of view it looks much more like a JVM). Moreover, the LHVM has a special feature called "exceptions", which is slightly different from the exceptions found in more common languages like Java.
Exactly the same way the machine code of an x86 program is embedded into an EXE file, the bytecode is embedded into a CHL file, which contains a few more info, such as global variables, string data, functions' entry points, and startup scripts.
The LHVM can handle 5 data types (int, float, coordinate, object and bool), but the variables can be only of float type, so there are variants of the PUSH and POP instructions that can convert between data types.
There are 464 system functions used to interact with the game, that can be called using a specific opcode.

The main purpose of an assembler/disassembler program is to patch the existing code: since we haven't* the original source code we wouldn't be able to make changes and recompile it, but we can disassemble, make changes, and reassemble it.
* I said earlier that we got the source code from the Mac version, but there are 2 problems with it:
- this code comes from a specific release of the game, and certainly lacks subsequent patches from LH;
- this code can't be compiled because it requires some C header files that defines some constants (enum). Some of these files are available in the installation folder, but they are obsolete and some enum values doesn't match the ones actually found in the compiled CHL. Moreover some enums aren't defined in any of the available header files.
It worths to note that a disassembler can be used to guess the actual value of all enums used in the source files, which is one of the thing on my roadmap. This will make possible to compile the original source files, which will help in making changes to the existing story.

Although an assembler program can be used to write a new challenge, it would be very unpratical to write it in assembly without making errors. This is why I plan to write a compiler for the CHL language too, which is what I'm currently working on. The main obstacle with CHL is that its syntax is very complex, and writing a parser is a very hard task. An alternative could be to use an existing high-level language for which parsers already exist, but this would mean that the existing source code come to us couldn't be used as a sample to start with.

As I've said before there is no official ASM language for LHVM, so mine is a bit different from the one made by "Anthem" who published the first assembler/disassembler tools (see here). These tools were a bit buggy, and since their sources were lost they cannot be fixed, so I discourage their usage nowdays. If you are interested in ASM programming I recommend these contents:

Contains all the stataments described in the official "Lionhead Challenge Language Documentation" with my own corrections and enriched by me with their assembly translation by examples.

Contains my own explanation of the exceptions feature.

github.com

blackandwhite/chlasm/src/it/ld/bw/chl/model/OPCode.java at main · Daniels118/blackandwhite

Contribute to Daniels118/blackandwhite development by creating an account on GitHub.
github.com github.com
Contains some notes about the usage of opcodes.

github.com

blackandwhite/chlasm/src/it/ld/bw/chl/model/NativeFunction.java at main · Daniels118/blackandwhite

Contribute to Daniels118/blackandwhite development by creating an account on GitHub.
github.com github.com
Contains the list of all system functions with their parameter types and return values, but I recommend to use the first link as reference.

The better way to start is to decompile the original challenge file and look at the generated ASM code. If you get your hands on the original source files from Mac version, you can ask the disassembler to interleave the generated ASM code with the original statements as comments (see the attachment) using the -prsrc parameter, which helps a lot in understanding what ASM instructions actually do.

If you need further info just ask.

Many thanks for your feedback :)
 

Attachments

Last edited:
This is awesome, thank you. I had no idea that the challenges were handled with their own bytecode with the "LHVM". Your documentation is really well put together and helpful and I'd like to think that this is something I'd be able to make more progress with now knowing what you've told me here but I probably don't have enough time on my hands. I will take a look at disassembling the original CHL files later today and maybe experiment with making small changes using your tool.

Do you have any idea where I'd find the source code for the scripts from the Mac version? I have the B&W Platinum Pack for Mac on original CD which includes CI, not sure if they'd be on there or if it has to be an original Mac copy of the game. I think I saw an original ISO for B&" for Mac on Macintosh Garden a couple of years ago but looking now I can only see the platinum pack. In any case I suppose it isn't helpful having those files if we have no way to compile source code yet but if and when you make that happen I look forward to seeing it.

On a similar note, I assume the C header files you are talking about are the ones that can be obtained from installing these scripting tools or something similar? I imagine they are functionally useless if the enums are wrong but it's useful to know that there are accessible (albeit outdated) copies of them.

Any chance of a link to the official Lionhead Challenge Language Documentation? I would love to see what that looked like to gain a better understanding of the thinking behind designing the original scripts by LH.

If you don't mind sharing, I'd also love to know where you got all the information you have. I assume most of it was from the community on here and similar sites? I have done a fair bit of digging a few times in the past to try to find out more but never turned up much of interest besides the old Kapa tutorials from YouTube. I know there must be a lot of knowledge out there because of the existence of things like the Genesis Mod, which is the sort of thing I would like to be able to understand how to do with the game.

It might be a dead lead, but my partner used to go to Goldsmiths University in London where she met one of the Black & White devs whose name I forget. They only talked about it once and it was a few years ago now but I am tempted to see if there is any way of her getting back in touch to find out more.

Thank you for all the info you have already provided and for anything more you can add. I think I know what my next steps will be from here. I tried making a remake of B&W a couple of years ago but got stuck trying to get the camera to behave like the original game but I think long term this is probably the project I will try to return to rather than just making a new singleplayer campaign in the original game engine. Nonetheless I will definitely attempt a bit of CHL script tinkering in the original game engine now that I have a starting point.

Cheers
 
Do you have any idea where I'd find the source code for the scripts from the Mac version?
Click to expand...
I don't know in which release/subfolder they are supposed to be, but you can download a copy from here.
In any case I suppose it isn't helpful having those files if we have no way to compile source code
Click to expand...
Well, not exactly. Consider the following code:
Code: 
    PUSHI 1
    PUSHF [Boy]
    SYS GET_PROPERTY    //[2, 1] (SCRIPT_OBJECT_PROPERTY_TYPE prop, Object object) returns (int|float)
    PUSHF 0.0
    GT
    PUSHF [Boy]
    PUSHI 9
    SWAPI
    SYS2 GET_PROPERTY    //[2, 1] (SCRIPT_OBJECT_PROPERTY_TYPE prop, Object object) returns (int|float)
    CASTB
    NOT
    PUSHF [Boy]
    PUSHF [MyCreature]
    SYS IN_CREATURE_HAND    //[2, 1] (Object obj, Object creature) returns (bool)
    NOT
    AND
    AND
    JZ lbl1595
How much time it takes for you to understand what it actually does? Well, thanks to the original source files my disassembler can tell you that this code was originated from this line:
Code: 
if HEALTH of Boy > 0 and (Boy is not HELD and not Boy in MyCreature hand)
which is a lot easier to understand. Of course you can't change the original source line, but it make it easier to understand what to change in ASM code to obtain the desired behavior.
How does it work? Almost all instructions in the bytecode contain the source line number that originated them, so it is easy to extract the correct line from the source files. However, as I've said before, the current bytecode contains patches which aren't found in the original source files, so not everything matches. The clearest case are instructions from "ControlNemesisBattleStrategy.txt", which is completely absent in the source files, so we get just ASM code here. But there are few subtle cases where instructions just don't match with line numbers, or parameter values are different; luckly those are not that much.
On a similar note, I assume the C header files you are talking about are the ones that can be obtained from installing these scripting tools or something similar? I imagine they are functionally useless if the enums are wrong but it's useful to know that there are accessible (albeit outdated) copies of them.
Click to expand...
I guess you're right. I installed the scripting tools some time ago and probably didn't notice the header files weren't there before. And yes, they are incomplete and outdated. For example look at the SCRIPT_INTERFACE_LEVEL enum definition in ScriptEnums.h, the SCRIPT_INTERFACE_LEVEL_JUST_ROTATE_AND_DRAG is the 10th entry, so its int value should be 9. If you look in LandControlT.txt at line 727 you will find the following statement:
Code: 
set interaction SCRIPT_INTERFACE_LEVEL_JUST_ROTATE_AND_DRAG
but if you disassemble the chl file and look at the generate ASM code you will find:
Code: 
    PUSHI 10
    SYS2 SET_INTERFACE_INTERACTION    //[1, 0] (SCRIPT_INTERFACE_LEVEL level)
there are 2 possibilities:
a) the scripts are outdated, and the current behavior is SCRIPT_INTERFACE_LEVEL_JUST_PITCH;
b) the enum is outdated, one more value has been added before SCRIPT_INTERFACE_LEVEL_JUST_ROTATE_AND_DRAG so that its int value increased by 1.
Of course this could be checked by running the game, but I haven't tried yet.
Another case: the enum entry HELP_EVENT_TYPE_GESTURE_START_SPELL_SELECT used in LearnGestures.txt at line 56 is not defined in any header file.
See also this post: http://bawsite.com/forum/showthread.php?tid=81
Any chance of a link to the official Lionhead Challenge Language Documentation?
Click to expand...
You can find it here, but please note that the syntax of some statements doesn't match the instructions found in the original source files.
I'd also love to know where you got all the information you have.
Click to expand...
I have used lot of sources:
things like the Genesis Mod, which is the sort of thing I would like to be able to understand how to do with the game.
Click to expand...
I guess that the original scripting tools are enough to write a new story, it may worth a try. The main difficulty is that you have to stop the game-write-compile-delete savegames-restart the game-try the changes-repeat until you get the desired result. It would be nice if one could edit the scripts while the game is running, which I think could be possible with advanced programming techniques.
met one of the Black & White devs whose name I forget
Click to expand...
At the end of this document there are names and pictures of the LH team members, maybe it could help: http://bawsite.com/nongame/Making B&W_eguide.pdf

I hope you will go on with your projects and that you can share your progress. Good luck!

If you need further info don't hesitate to ask.
 
Last edited:
Thank you again for this. I have had a busy weekend and will probably have a busy week but I will take a deeper look into this next week. Reading through the challenge source code from the Mac version is really fascinating but it's still a shame we have no way to compile it. I spoke to my partner about the guy from London but apparently I misremembered - it wasn't one of the devs but it was a friend of one of the devs (Alan Zucconi) so I'm not sure how much there is to learn from contacting him.

I have learned a lot from what you've told me here in a short amount of time and definitely want to keep looking into this more going forward. If I start working on any projects as a consequence of this I will be in touch!

Thank you.
 
Daniels118 said:
Hi all,
this is my first post on this forum, so... nice to meet you all!

A brief introduction about me and B&W
I started playing B&W when I was a child, and I played the story mode again about 3 times over the years. The game mechanic is fantastic and the story too, but after playing it so many times it become a bit ovious. I always desired a new story to play, or even better write my own to share with others. Searching the web I've found 2 tools to compile and decompile the files used by the game to generate the story (scrolls), but they are very old, buggy, and lack some essential usage info. Since I'm an enthusiast programmer, about 2 weeks ago I contacted the developer of those tools who recently republished them to ask him for the source code so that I could eventually improve it, but unfortunately he losts the sources, so I decided to write my own trying to do better. And here it is!

Download and installation
I have published my work on GitHub under GPL, this way the source code won't get lost anymore:
https://github.com/Daniels118/blackandwhite

The binaries can be downloaded from the release section on the right, extract them and put the directory on the system variable PATH to be able to run it from everywhere.

Usage
The basic usage is pretty straightforward:
chlasm -chlasm -i challenge.chl -o outdir to convert from to CHL to assembly. Outdir must exist.
chlasm -asmchl -i inpdir -o challenge.chl to convert from assembly to CHL. Be sure to backup the original file first!

The full syntax can be displayed by running the command without any argument.

The tool works perfectly, but I'm still improving it. The final objective is to have a compiler for the high level CHL language, which will make scripting much more easy.

Comments are welcome!
Click to expand...
Nice tool, I have to say! But do you want to add compatibility for the CreatureIsle challenge.chl files too?
Because for CI Challenge.chl files, it doesn't works :(
 
Hi Kratzean, at the moment I am focused on the CHL for the base game, but it wouldn't be too hard to extend my work to CI expansion too (they are basically the same format, CI just enables some more functions, for example the ones to control football match).
For the moment I'm working on a compiler for the high level CHL language, which has about 600 different statements, each one with it's own syntax, so it takes a lot of time to write a compiler, but I am halfway to the goal. The intent of the developers was to make the scripting language similar to natural language, but honestly I think that learning the syntax of 600 different statements is a nightmare for a programmer.

Once I have completed the compiler I will try to extend the tool to the CI expansion.

Stay tuned!
 
Good news! The compiler is almost done, I now have just one small obstacle: the current challenge.chl contains code from various source files, one of which is "ControlNemesisBattleStrategy.txt" which wasn't in the zip with sources from Mac version.
Without this file I cannot rebuild the whole CHL to do a complete comparison to verify the goodnes of my compiler (although the rest of code already matches perfectly).

My current strategy to workaround this obstacle is it to allow the compiler to handle both CHL and ASM input files. Since I can extract the ASM code from the CHL, I will be able to substitute the missing CHL file with its equivalent ASM.

BUT if anybody has the ControlNemesisBattleStrategy.txt file, please post it here! 🙏

As additional note, I have reverse engineered the values of the enums for the original game, I will include it in the next release of the tool.
 
Daniels118 said:
Good news! The compiler is almost done, I now have just one small obstacle: the current challenge.chl contains code from various source files, one of which is "ControlNemesisBattleStrategy.txt" which wasn't in the zip with sources from Mac version.
Without this file I cannot rebuild the whole CHL to do a complete comparison to verify the goodnes of my compiler (although the rest of code already matches perfectly).

My current strategy to workaround this obstacle is it to allow the compiler to handle both CHL and ASM input files. Since I can extract the ASM code from the CHL, I will be able to substitute the missing CHL file with its equivalent ASM.

BUT if anybody has the ControlNemesisBattleStrategy.txt file, please post it here! 🙏

As additional note, I have reverse engineered the values of the enums for the original game, I will include it in the next release of the tool.
Click to expand...
I think, you mean this file?
Here, I hope, this will help :)
 

Attachments

They was in the MAC Scriptfiles, but for some reasons, they had a wrong filename. Instead of "ControlNemesisBattleStrategy.txt" they was "ControlNemesisBattleStrate_.txt".
 
I can't believe I haven't seen it, I'm ashamed!
Thank you for having opened my eyes!
 
Good news!
I have just released the full version (0.2) of the tool. It can now compile the original story, and of course can be used to write a new one.

I have made a simple challenge (inspired by one of the original samples) that will let you play bowling... with villagers in place of pins! Of course it is introduced by Blackey 😈 The compiled challenge.chl file is attached to this post. Here is a screenshot of the gameplay:

bowling.jpg
 

Attachments

Daniels118 said:
Good news!
I have just released the full version (0.2) of the tool. It can now compile the original story, and of course can be used to write a new one.

I have made a simple challenge (inspired by one of the original samples) that will let you play bowling... with villagers in place of pins! Of course it is introduced by Blackey 😈 The compiled challenge.chl file is attached to this post. Here is a screenshot of the gameplay:

View attachment 7530
Click to expand...
Newbie question.
How do I incorporate this file into the game?
 
pappde said:
Newbie question.
How do I incorporate this file into the game?
Click to expand...
The file as is cannot be incorporated, it can only be used in place of the original story. It can be incorporated into the original story at source level, but then you have to compile it. I'll explain both.

Replacing the original story
First, you have to backup the original story file and your profile folder.
The original story file is located in "installation_folder\Scripts\Quests\challenge.chl" just rename it as "challenge.orig.chl".
The profile folder is under "installation_folder\Profiles" and is named something like "Y_o_u_r_N_a_m_e_"; you have to move it elsewhere.
Then extract the content of the attachment which is already named "challenge.chl" and place it under "installation_folder\Scripts\Quests\".
Then start the game. From my tests it may crash the first time you launch it, if so, just give it a second try.

Incorporating
To incorporate it you have to setup the project to compile the original story. Luckily I have prepared the ground. The tool comes with a folder named "samples\bw1_story"; within this folder there is a file named "_project.txt" which explains how to compile the original story, follow the steps 1 and 2 (you can get the required scripts file from here).
Now you have to add the bowling game. Copy the file "samples\sample1\Bowling.txt" in "bw1_story" replacing the existing one, which is useless.
Then edit "_project.txt" adding the following line at the end:
Code: 
source Bowling.txt
Then edit the file "LandControlT.txt", locate the line "begin script LandControlT", then just after the line "end camera" add the following one:
Code: 
run script Bowling
Now execute the step 3 from "_project.txt". You will get a file named "_challenge.chl", remove the underscore and use it as described in "Replacing the original story".
The bowling game is loaded into the god playground, you have to enter it by pressing F2 while in game. Of course you can let it appear in other lands by calling the script in that land setup, but you have to find out proper coordinates to spawn the pins, or they will end in the sea or... in a volcano.
 
Last edited:
Just a correction on my early statements after deeper analysis. The scripts from Mac version are at last version. The missing file was just there with a wrong name (see posts above). The header files aren't obsolete but newer (but still incompatible) because they are for the Creature Isle expansion. It was just a matter of renaming the bad file, and reverse engineering the header files for the "vanilla" game. The fixed headers can be found in the download of my compiler.
 
Daniels118 said:
Download and installation
I have published my work on GitHub under GPL, this way the source code won't get lost anymore:
https://github.com/Daniels118/blackandwhite

The binaries can be downloaded from the release section on the right, extract them and put the directory on the system variable PATH to be able to run it from everywhere.
Click to expand...
I just recently found a really old copy on cd of black and white(2001). Installed it for my kid to play and have ran into all sorts of problems that I got solved. In the process of trying to solve all these problems I can across "openblackproject" which got me more intrested in coding/scripting/mapping. So over the past two months I've made a few map scripts and lnd files for the original bw. After doing map scripts and lnd files, I've become intrested in creating a "creature mind" with no luck on how to go about that. Been doing tons of research and reading through all these post and other forums till I came across the chlex and chasm originals but I couldn't get those to work and have tried all the suggestions/possible fixes I've read, which lead me to doing more research and finding this forum.

I understand downloading all the binaries and putting them in a c:\ directory folder. What I'm not understanding is how to compile all the binary files to have a program that I click and run. Apologies if that question seems stupid; I'm new to this and would like to get help from somebody directly involved in the openblackproject.

Ps. I realize this forum is over a year old so if you respond here and see this same message on github please disregard.

After rereading this form I downloaded the full version from the link. I'm still needing help though on how to use this program after reading the read me file and the _project.txt. Are the _header, _infopath, _source, etc are these all command prompt lines?
 
Last edited:
Kcc said:
I just recently found a really old copy on cd of black and white(2001). Installed it for my kid to play and have ran into all sorts of problems that I got solved. In the process of trying to solve all these problems I can across "openblackproject" which got me more intrested in coding/scripting/mapping. So over the past two months I've made a few map scripts and lnd files for the original bw. After doing map scripts and lnd files, I've become intrested in creating a "creature mind" with no luck on how to go about that. Been doing tons of research and reading through all these post and other forums till I came across the chlex and chasm originals but I couldn't get those to work and have tried all the suggestions/possible fixes I've read, which lead me to doing more research and finding this forum.

I understand downloading all the binaries and putting them in a c:\ directory folder. What I'm not understanding is how to compile all the binary files to have a program that I click and run. Apologies if that question seems stupid; I'm new to this and would like to get help from somebody directly involved in the openblackproject.

Ps. I realize this forum is over a year old so if you respond here and see this same message on github please disregard.

After rereading this form I downloaded the full version from the link. I'm still needing help though on how to use this program after reading the read me file and the _project.txt. Are the _header, _infopath, _source, etc are these all command prompt lines?
Click to expand...
No, header etc. are internal commands that the compiler program can understand. On the command prompt you have to type just the chlasm line with all the parameters. The chlasm prpgram will read the project.txt file and will interpret the commands it contains.
 
You must log in or register to reply here.
Back
Top