Moving to AGS 4.0

First of all, sorry for the long post below, but I thought this has to be written to document what we are trying to do and why.
This is a continuation of #403, but the old thread became way too cludged with all the heated discussions, and options.

Preword

From the early days this open source project suffered from mixed goals and purposes. On one hand, the game making community desired AGS to improve and get additional functionality, on the other hand there was a significant number of game players, who wanted to play both modern and old AGS games using same engine and its ports. There had been an attempt to make AGS port for ScummVM, but it was not complete at that time, and so this engine was the only way to run majority of games.

When we just started, this problem was never properly discussed (in my memory at least, but I've joined late) and no concrete decisions made regarding solving it.
As a result, all the subsequent development was largely complicated by the need of maintaining backwards compatibility. It was not only supporting legacy features in the engine, to be able running old games without conversion, but also supporting loading old games into the editor in case game developer would like to import and continue their ancient project.

This was further aggravated by the very low number of contributors involved. (For example, initially my primary goal was to refactor the code, but I soon had to suspend that work in favor of doing other tasks.)

Current situation

Today the project development has practically stalled. There are some minor additions, as well as some refactoring made over time, but apparently no much interest in developing it further. There is a number of things AGS is lacking that are making it difficult to use in the modern enviroment, such as unicode support for example.
So its future is unclear.

Course of action

So, @AlanDrake's suggestion (#403) is to a) radically clear the code from the legacy support, b) refactor the cleaned code, so that it is potentially easier to develop further.
Then after that's done maybe get the legacy support back in a modular way, that won't impose any constraints on future development (although legacy support is very low priority from his point of view).

This approach has both pros and cons; the opposite method of gradually refactoring the engine while keeping all the backward compatibility intact also has pros and cons.
Personally I have very mixed feelings about it, but acknowledging the fact that the number of active contributors is very low, and program development is barely existent, it looks like there won't be any "good" way out of this situation, there will be trouble no matter what we choose; only difference is in the nature of the trouble (and number of people willing to cope with it).

---

To sum up, the idea is as follows:

1. We create a branch called, e.g. "ags3" (since the major version is 3). From now on and until further notice this branch will contain latest version of the engine and editor compatible with importing and running old games. This branch will allow only necessary fixes, and only updates that improve its perfomance. Also, naturally, any kind of backward compatibility updates.

2. The latest version in 3.x.x family is probably going to be 3.4.2, because there is already a WIP version of it released for public use & test. Note that it still is kind of rough at the edges in the Editor, where we implemented new navigation bar, so may need extra tweaks. These should still be made in "ags3" branch.

3. Current "master" branch becomes a place of a refactoring project, which first priority is to clean and refactor the engine code (possibly editor too where necessary). All backward compatibilities are cut, except for loading latest 3.4.x games. This branch is allowed to become unstable for some periods of time too.

4. In case there is a patch that is worth to be also added to ags3 branch, one of two possible methods is used: either patch is applied to ags3 branch, and then to master separately, or after its applied to ags3, ags3 is merged to master very carefully, to not bring unnecessary code there again.

5. People willing to contribute for running old AGS games are also encouraged to assist finishing ScummVM AGS port, started by @fuzzie, and continued by @morganwillcock. This project may be found here: https://github.com/adventuregamestudio/scummvm/tree/ags

We of course need to explain all this very clearly in the Readme.

---

What should be the closest goals of this rewrite?

IMHO, if we are doing such change to the code, than after this first step we must do second and actually refactor it completely (more or less), and also break any constraints that were restricting development (such as data format constraints, imposed by the old plugin API, for example). Because, although removing legacy code would make it somewhat easier, large portions of contemporary code is still very hard to work with.

In such case I'd propose to open a "refactoring master ticket" with a list of most outstanding necessary changes and remaining unrefactored parts for contributors to choose from.

We'd need to also discuss the overall structure of the engine code we are aiming at, to prevent erratic changes.
To give an example, considering the upcoming SDL2 port (by @sonneveld), we probably should aim to restructure main program loop to make it consistent with SDL's event queue.

Issue of the legacy code

First of all, the legacy support stays in the "ags3" branch. @sonneveld already mentioned that his goal is to complete SDL2 backend for 3.x.x version. If this is done, that would hopefully cover some of the issues we have with e.g. Linux port. So the legacy branch, although not developed further, would still exist as a way to run old games.

Hopefully, the ScummVM port will be also finished someday. I'll post the link again just in case someone wants to test/contribute to help it happen faster: https://github.com/adventuregamestudio/scummvm/tree/ags

Finally, if that would still be wanted, an option which remains is to get the legacy support back if/when the refactoring in the master branch is completed. If done correctly, that should not interfere with the actual development, and scare anyone away.
In such case the biggest question is how to find what functionality to restore. If we are merging ags4 branch into master, as suggested, then this merge commit would contain all or most information about what was deleted. Much of that code was already distinctly separated, or related to very particular legacy data structs, which makes it possible to understand its meaning with certain effort.

[morganwillcock]

Given the above plan and the separation between legacy functionality and what should be current, is it worth reverting the path finder change for "ags3" and only implementing it in the branch that is getting cleaned up?

Given the above plan and the separation between legacy functionality and what should be current, is it worth reverting the path finder change for "ags3" and only implementing it in the branch that is getting cleaned up?

I guess this is possible, what are the issues with the new pathfinder?

[morganwillcock]

I'm pretty sure there is a large enough difference between the two that the final position of a character can be different; any scripts that are checking character position are going to potentially have different behaviour. Worst case, you get stuck in an infinite loop whilst waiting for a character to arrive.

I'm pretty sure there is a large enough difference between the two that the final position of a character can be different; any scripts that are checking character position are going to potentially have different behaviour. Worst case, you get stuck in an infinite loop whilst waiting for a character to arrive.

Well, this is not good. How come I did not think about this before. We need to restore old pathfinding then until there are ways to steadily replicate behavior with the new one.
Probably it may be possible to restore it without overwriting new one, and make a switch in the game settings, or config, or even as a compilation flag.

Do you know actual games broken by this? If so, could you open an issue please?

[morganwillcock]

I could see the difference on the MAGS game I was finishing, but I'd have to recreate the problem as I've changed the code to work around it. It was a check for distance, to bypass Character.FaceObject, when to match perspective the actual direction needed to be left or right, but Character.FaceObject would choose to face upwards.

if (player.x + 100 < object.X)
{
    player.FaceDirection(eDirectionRight);
}
else if (player.x - 100 > object.X)
{
    player.FaceDirection(eDirectionLeft);
}
else
{
    player.FaceObject(object);
}

With an object I was testing, on the new pathfinder I would always end up facing upwards, on the old pathfinder I would end up facing right. I've previously seen issues (a couple of years ago) with the old pathfinder, where it seemed the character would never reach the direction correctly. I think this is similar to the font scaling vs vertical offset, where all existing games are going to rely on the old broken behaviour, so the old pathfinding issues had never been fixed.

I'll open a ticket when I can, but I think this sort of thing is a good example of why the legacy code should be dropped and what you have described above is a good idea. In fact I would support pretty much any level of breakage on AGS 4, if it would restore it as a contemporary tool. If it didn't load any previous projects, had a modified scripting system, and modified event system: I think there would be few complaints, given that there is no similar engine which doesn't have licensing or cost implications.

Alright, I created ags3 branch at the latest commit we had, and then merged ags4 into master.

What has to be done in nearest time:

• Update README with new information about game support and ways to contribute.
• Test that existing 3.4.1/3.4.2 projects may be imported and run correctly (assuming they do not use any legacy features).

[sonneveld]

@morganwillcock I think if the new pathfinder code isn't part of a 3.4 release, it might be worth reverting it. Do you have time to create a pull request to remove it or did you want me to have a try when I have time?

[sonneveld]

For everyone in general: What new features are planned for AGS4? And how are we going to ensure new features are complete before going into master?

@sonneveld ,

I think if the new pathfinder code isn't part of a 3.4 release, it might be worth reverting it.

New pathfinder is a part of 3.4.2 version which was already released for public testing.
I was thinking it might be possible to not completely remove it, but instead introduce a switch. The switch could work, for example, like this:

• if an old game is run, it uses old pathfinder by default, or new one if there's an explicit overriding config option set.
• if a new game is run, it uses new pathfinder.

E: To underline, it is not that big deal to keep it, since it does not impose any restrictions on saved data, but I'd rather do if it's feasible, since it really works faster than the old one. Plus there will be more people testing it.

What new features are planned for AGS4?

Do not know if any were planned. I have some minor work in my personal repository, and few ideas I wanted to try after the situation around refactoring attempt is resolved to some degree. But nothing big atm. Or rather, I am not enthusiastic enough to make big plans.

how are we going to ensure new features are complete before going into master

Not sure what to say... same way as we do it now, requiring people do work in feature branch until it is tested and revised by other developers?

[AlanDrake]

We should discuss new features somewhere to make a list and value their merits and implementations.
In the backburner I have a couple of things like the blend mode property, which I'd rather delay until we have SDL2, and a few scripting functions that I don't feel are ready enough to be merged.
For the moment, I'd rather to focus on cleaning and refactoring.

[morganwillcock]

Having just made a game from start to finish (previously I've only ever worked on existing projects), I've now found a few things which I think aren't working properly, plus features that I thought would have been there but were missing. So I'm also keen on the idea of a list being somewhere, before I forget it all.

In terms of pathfinder, it is probably worth checking that the problem is definitely with the old one, or at least validate the new one as correct, before implementing switching. If the new one did turn out to be incorrect, then eventually someone will want to introduce a third pathfinder and we'll start all over again.

[sonneveld]

If the new pathfinder is slightly buggy, and ags3 will be left in maintenance mode, it might be better to just bite the bullet and remove it from the ags3 branch, leaving it only in master/ags4.

[monkey0506]

My words probably don't carry much weight after I've over-promised and under-delivered too many times, but I'll just say my thoughts about this. When Chris Jones first open-sourced AGS, I petitioned the community pretty actively to get a communal effort in place to keep AGS development going. One of the things that I felt needed to be done was dropping backwards compatibility so that the engine could be property refactored into a totally clean codebase. Others argued that it should be done piecemeal.

In the end, the crux of any OSS's development is the developers that are actually developing the code, so I don't mean to insult anyone's efforts. Rather, as I look back, I question why I never actually took the time to study the engine's code properly and figure out how it all fitted together. There's no accounting for the immeasurable benefit of @jjsat and especially @ivan-mogilko's contributions.

Ultimately what I'm trying to say is I do believe that this is the right path for AGS's future. I'll even recant a stand that I once vehemently defended, and concede that "AGScript" as I've called it should also be replaced. The debate over that should take place elsewhere, but my personal thoughts to that end are that the recent portability improvements and easy yet powerful nature of C# make it a great candidate.

I won't end my spiel with any promises that I'll just end up breaking, but I would like to thank all those who have consistently driven the development of AGS forward. This gives me hope that AGS will be able to continue on into the future.

[rofl0r]

concede that "AGScript" as I've called it should also be replaced

if you want to go into that direction, my recommendation would be to pick up an existing successful embeddable product such as lua, rather than trying to desgign and write yet another incomplete/buggy/underspecced language parser/interpreter/compiler.