It's been several years since I last posted. There are several reasons, but most notable, is the fact that I haven't been doing anything except write a game engine from scratch. For nearly 2 years, I would just crank out code, sleep, and repeat.
The good news is I was able to write a game engine usable (to an extent; more on that later) for the game ideas I had in mind. The bad news is, as previously mentioned, it has took its toll on my mental health, knowing that I lost a lot of time where I could have been working on other projects on my back burner, or just having fun with random activities, such as playing games, going on a hike, etc.
About three months ago, I finished polishing up the engine and started planning and implementing the beginnings of my first game - after more than 10 years of failing – which was mostly due to the general lack of good game development tooling and engines in general for Common Lisp. Things were looking good about two months into development and several thousand lines of game logic later. However, shortly thereafter, as my game was a real-time game with lots of game objects and physics being calculated each frame, I quickly realized that the engine was not performant enough to pull of my game idea. After a week of profiling, improving suspectful things in the engine, and reiterating, I didn't improve the performance much at all, and was finally stuck with SBCL's statistical profiler telling me that even for a small scene with not too much going on, my CPU was spending about 50% of time in CLOS – Common Lisp's object system, which is very dynamic and relies on a lot of runtime dispatch accessing slot values, calling generic function accessors, and so forth.
This was a pretty large disappointment, because I didn't anticipate it to be this slow, even though I knew it was doing a lot of dynamic diapatching. The use of CLOS was a fundamental design decision I made on day one, which utilizes the MOP (Meta-Object Protocol) in order to dynamically generate classes at runtime as behavioral components are added or removed from game objects. Everything being a class meant that there was a lot of dynamic dispatch, for accessing slots of objects which in turn hold hold references to other objects, etc.
After a lot of thought about what to do, I ultimately decided that it would be best to rewrite the foundation of the engine using actual composition over inheritance rather than mixin classes. This meant completely decoupling components from game objects, and using static structure objects rather than CLOS standard objects.
This decoupling of components meant that another core piece of machinery also had to be rewritten – the component flow protocol, which is responsible for realizing game objects and their components, and ensuring everything happens in lock-step throughout a frame. This is actually a much harder problem than it sounds like, considering one of the core design ideas was a declarative, DSL for many types of game resources, with a notable prefab DSL for describing a subtree of game objects and components. Arbitrary nodes within a prefab description can reference or be referenced by other toplevel prefab definitions, and each toplevel form is able to be live-recompiled as the game is running to see changes happen in real time. This decoupling of components from entities ruined this ability of interactivity in many ways, and there just is not a clear solution to the problem. At the very least, it would require going back to the drawing board for several weeks, and redesigning the engine with simplicity in mind.
Which brings me to my main point. Game engines are large systems consisting of many moving parts. Good software engineering requires simplicity – it is what allows a system to remain secure, stable, and coherent throughout its evolution. Simplicity itself requires a lot of work at the start of a project to reduce the idea to its essense, and lots of discipline over the lifetime of the project to be able to distinguish worthwhile changes from the pernicious ones. That is simply everything my game engine is not, because for such a complex piece of software such as a game engine, it is not easy to know HOW all the pieces fit together, just some vague idea. Complexity arises through the iterative process that is implementing and actually debugging problems with these features. Making a small change to get a engine feature to play nice with others could, and often does, adversely affects simplicity and elegance much later down the road during development.
The refactored engine with structs over classes, and decoupling of components from game objects, is for the most part a failure, and I am abandoning that two week effort. That leaves me with the previous, albeit slower performing attempt. It probably means that I have to either scrap my current game idea, or change it in major ways to be able to pull it off so that it is playable. It's either that, or just start over yet again, engine and all, in which case I would start to question my choice of language. Common Lisp seems like an excellent choice for interactive applications that require hot code reloading, such as with games, but with games, requires very good performance over convenience and simplicity in a lot of areas.
I am honestly not sure what I will do yet, but I do know, that for the first time in about 2 years, I am going to take a much needed break to let all of this sink into my subconscious, and maybe the way forward will emerge. I will play games, go on hikes, read books, work on other projects that have been sitting on my back burner for far too long, and maybe even take up another programming language in the mean time. I just know that I need a serious break from all of this, as the mental toll it all has taken on me is real. Sometimes I feel like I'm worthless, not a good programmer, etc, all because game development, and especially game engine development, is a lifelong journey requiring discipline and knowledge in many different fields of study.
That is all, and sorry for the rant. This post was not proof-read. I just needed to quickly get this out of my head to begin my hiatus.