Spell Chess Engine Stuck at depth 4 ?!

For the past few weeks, I’ve been hacking on a Spell Chess engine as a fun side project. I recently learned the basics of chess engines, and this felt like a cool (and painful) way to apply them. I’ll keep most of the details intentionally vague—I don’t want to take the fun out of this variant—but I’m pretty sure some of you here have tried something similar.

Things were going fine… until I hit what I now call Depth 4 Hell.

The Wall I Can’t Get Past
My main constraint is simple:

1 second per move
Depth 4 works consistently
Depth 6 takes ~10 seconds on average, which is a non-starter
No matter what I do, I can’t seem to break past depth 4 without blowing the time budget. The branching factor in Spell Chess is absolutely wild, and every “small optimization” just gets swallowed instantly by the search space.

Where I’m At Right Now
Basic search is working
Eval isn’t amazing, but it’s not terrible either
Move time is stable at depth 4
Anything deeper feels like I need a fundamentally smarter idea, not just micro-optimizations
At this point I’m mostly curious about other people’s experiences:

Has anyone actually pushed past this under tight time limits?
Was there a specific breakthrough moment?
Or is depth 4 just the reality unless you completely rethink the engine?
A Quick Promise to the Community
Just to be super clear: I promise I will never use this in actual games.

The plan is to open-source the project once I’m satisfied with it. Why not now? Honestly—it’s buggy, messy, and I know I can do better. I’d rather share something I’m proud of than dump half-baked code.

Would Love a Chat
If you’ve worked on a Spell Chess engine (or anything with a similarly insane branching factor) and made it further than this, I’d love to chat . Mostly to compare notes and hear what actually worked in practice.

Is your spell chess engine evaluating some of the well known lines like e4 e5 freeze@c7Bb5 or e4 c6 d4 e5 ?

Suppose you could generate and evaluate ten times as many nodes per second. Without any clever ideas in the search, this would gain at most one depth, even with the branching factor of normal chess. I think you're going to need some new ideas.

On the other hand, increasing my engine's think time from 1 to 5 seconds brings it from blundering regularly to hurting me quite a bit, so if you add some variant-specific ideas, extra nodes do help.

My engine can often (though not always) reach depth 5 in 1 second without too much work put into optimizing move generation. I have another idea which might be another breakthrough, but I haven't implemented it yet, so I don't know.

If you release an engine, unfortunately people are going to use it to cheat. There have already been several accounts closed in connection with spellchess.win, which doesn't even have a public engine. Due to fair play issues I don't plan to share any more information about how the engine works.

Cool!

I was aware, that a few people were working on spell chess engines, but didn’t know, that they were (although not perfect) operational already.

What does „depth 4“ mean in game? 
Can you beat this level? What rating would you give it approximately?

@Fleex255 however an engine will help improving our openings and the spell chess comprehension, so we have to find the balance…

@AlwaysGetFlagged "Depth" is how many turns ahead it looks. These are turns, not move pairs. If there is a threat that will take longer than that to play out, the engine may miss it until it is too late.

It's hard to give an engine a rating because it doesn't think like humans do - it has very little positional understanding (yet) and no long-term plans. At 1 second per move (depth 4-5), my engine blunders regularly and I can beat it without much difficulty. It would probably be 1600-1700 at best. At 5 seconds per move (depth varies a lot by position), it still makes mistakes, but beats me much of the time if I don't play a trap it will fall into. Overall it's probably 2000+.

@Delhy I do plan on making my engine available, but running on my server and only accessible via spellchess.win. I have a fair play check system I will be able to run to detect engine use during games.

Hey! So I know nothing about engine programming, but wouldn't it make sense to implement an AlphaZero style (Like LC0) search instead of the regular Stockfish one? (Can't recall what it's called). Of course, easier said than done, but analysing every single spell + move combination sounds very crazy to me (But if somehow it is already at 2000 level with 5s thinking, either we are all very bad and playing far from optimal given the complexity of the game, or the approach is good enough because it brute-forces its way through all the branching factor).

Anyhow, I'd love to have some day an engine available to analyse specific positions, do opening prep, etc. But I do agree that cheating is a potential issue if you ever get the engine to be 2500+ and don't implement proper checks. Love your work @Fleex255 , just discovered a a couple of weeks ago and it's a great resource!

It would be very interesting to try Monte-Carlo tree search, which I hear can be better for high branching factor. The whole AlphaZero AI approach, though, needs significant computational resources which I don't have for training and couldn't afford to provision on a server.

I know I personally am very bad and playing far from optimal , but my engine is much more efficient than brute force, with more gains hopefully to come. I assume @zacker99's is also better than brute force if it can get to depth 4 in one second - either that or it has very impressive optimization.

+1 to Flexx. I also don’t think a neural net is realistic here—there just isn’t enough data (reliable position evaluations) to train one properly.

My rough branching factor is around 20 (at least time-wise; node counts look similar). I can generate all legal positions, but the explosion still hits extremely fast. From the outside, it seems like you’re a bit ahead of me in terms of overall engine maturity.

Do you think there’s any clever computer-science-style optimization that could realistically get us to depth 8 under 1 second?
Or is the real breakthrough more about thinking like a Spell Chess player and optimizing something fundamental enough to actually make a dent in the branching factor?

Genuinely curious what you think.

Seeing your thread reminded me that I should get to work and actually finish my engine, so today I implemented my potential breakthrough. Now it can do depth 5 in 1 second more consistently, and sometimes gets further depending on the position - I once got a depth 9 eval in a middlegame!

But that is with some variant-specific ideas. Going from depth 4 to 8 with a branching factor of 20 would need a 160,000x speedup, which is impossible. Stockfish only processes 10-50x as many nodes per second as my engine. If you want more depth, the only hope is to reduce the branching factor.

I guess you could somehow (if you haven't done it already) skip every single jump move that doesn't enable a jump, and all the redundant freeze spells which freeze the same pieces, but further than that, reducing the branching factor without going for a native MCTS approach looks very challenging, if not impossible

@TwoPotionsOneMate This is an insult to 1. jump@e2 e4 which has a very high win rate for white.

The proof :

Hahaha yes, it drags the swag away, that's the downside

I attempted a spell chess engine and though I did not get far due to my inexperience with chess engines, I had a couple good conceptual ideas.
You can ignore trivially useless jumps as mentioned above, though you could lose potential weird tactics involving castles? I dont think thats very important for now though
I think a lot of speed can be gained by not thinking of freezes as part of the move. If you look at freezes as a way to prevent the opponent's best move(s) given that you hadn't frozen, they wouldn't contribute to the branching factor nearly as much as including them as real moves. You would have to calculate lines involving the consequences of you freezing, but most of those lines will either be significantly worse after the 6 moves it takes to regen your freeze (or until your opponent counterfreezes) or will have disadvantage calculable from material math. Counterfreeze tangent lines inside of the tangent freeze lines would most often be even easier because the minimax algorithm can straight up drop the opponents best line(s) for immediate counterfreezes which are by far the most common kind that don't lead to a huge advantage.
You could probably train a small function approximator to predict what positions might have a freeze tactic on the games database and use that to decide when to run the freeze check code.

Trying to optimize spell search logic could definitely optimize the whole engine a lot. I think you still have to fully search them though, skipping freezes and thinking that you can just ignore a few of your opponent's best moves seems very dangerous to me, although in very quiet positions skipping freezes might be more ok if you don't have an immediate threat afterward.

Something like limiting higher-depth freezes to only freezing kings, high-value pieces, pieces that have a high amount of legal moves, or pieces that can check next move, those might see benefits.

Same with jumps, generally jumps are only beneficial if you capture and jump or jump and check.
Move ordering + alpha beta should help a lot with pruning jump to empty squares already though.

I dont understand what you mean here. Freezes by definition prevent a few of your opponents best moves. Obviously you need to consider the geometry of the board, but I assumed that was taken for granted to I didn't mention it

Yes, but I think this prevents that from working: given position you don't know what your opponent's best move is until you search, and if you search to find it, then you effectively have to search twice: once without freezes to find your opponent's best moves and once to search again given that you froze their best move. This probably mitigates any potential gains that you would've gotten by not searching other freezes. (Assuming that the feature works something like only search freezes that freeze opponent's best move at higher depth)

@ComplexSymbolYTUB that looks complicated particulary if there are two good moves who aren’t in a 3×3 aera and freezing one allow another, right ?

Calculating all their moves given that you freeze and calculating all their moves given that you dont freeze... is what is being done already. My method means you dont do that step at every position you search