Chu Shogi Puzzle #1

The dot in their center means they are promoted. So both DE are actually promoted. I didn't bother distinguishing pieces that would never promote anyway, which is why both black Queens have no dot; one of those is really a +Ph. This was just sloppiness while setting up the position; I used two Q instead of a +Ph.

Pretty good! This was one of the historic mating problems (D-series number 40), which according to the Middle Shogi Manual so far defied solution. I have a suspicion that the mate-in-7 is a parasitic solution. Or perhaps that the position was mutilated in copying. There seem to be too many sente pieces without function

If you turn a blind eye to the white horse, there is a sensible mate in 13 (or 7, as you would say). But other than that, the position is ridiculous for a mate problem, practically resignable. 

No reason to turn any blind eye. HaChu does confirm the mate-in-7:

10 +79.93 458036 0:08.73 f10i7 l4i7 l2j4 h4j4 g5i7 h6g6 c1g5 g6f7 g5e7 f7g6 j8j9 g6h6 e7g5
9 +79.93 198251 0:03.77 f10i7 l4i7 l2j4 h4j4 g5i7 h6g6 c1g5 g6f7 g5e7 f7g6 j8j9 g6h6 e7g5
8 +79.93 89208 0:01.71 f10i7 l4i7 l2j4 h4j4 g5i7 h6g6 c1g5 g6f7 g5e7 f7g6 j8j9 g6h6 e7g5
7 -21.08 44474 0:00.85 f10i7 l4i7 g5i7 h6g6 j8i8 g6f7 i8g8 f7g8 i7g9+ g8h7 g9h9 h7g6 h9i8 g6f7
6 -21.08 21945 0:00.43 f10i7 l4i7 g5i7 h6g6 j8i8 g6f7 i8g8 f7g8 i7g9+ g8h7 g9h9 h7i6
5 -20.59 8051 0:00.15 f10i7 l4i7 g5i7 h6g6 i7g5 g6f7 g5h5 f7e6 j8g8 e6f5
4 -22.82 2327 0:00.04 f10i7 l4i7 g5i7 d1c1 i7k9+ h6g6 k9j9 g8h7
3 -23.46 576 0:00.01 f10i7 h6h7 j8h8 h7i6 h8g8 i6j7
2 -26.28 151 0:00.00 f10i7 l4i7 g5i7 d1c1
1 -26.28 59 0:00.00 f10i7 l4i7 g5i7 d1c1

Sente is behind an enormous amount of material. You could argue that the (lack of) King safety for gote more than compensates that, but you don't know how sente's King safety is. Always easy, playing without King so you never have to worry about being checkmated!

Positions like this are rather typical for the historic tsume problems. It often doesn't matter it is an easy win; the restriction that it has to be done through checks only is an extra handicap. Of course it makes solving the puzzles easier, but this is often compensated by the huge depth. Mate-in-7 is actually quite shallow as these things go. Mate-in-15 is more typical. This is one of the reasons I suspect we have a parasytic solution here; normally every piece has its use, and deleting one piece of the attacker makes the mate impossible. Here you can f.e. delete the Tokin at a9 or the Elephant at f12 without affecting the solution. The placement of the Pawn in a 'stairway' suggests the Queen was intended to walk it much further, before reversing and driving the King back.

A more typical historic tsume problem is this mate-in-16:

It comes with a flawed solution in the historic manuscript, showing that the people who composed these problems were not the same as those who compiled the manuscripts and provided the solutions.

In normal chess, there is a huge resource of puzzles that more or less corresponds to one's (my) real OTB strength. I'm not really the type that jumps into a puzzle, to learn afterwards that some computer calculated a mate in 26-ish that I would never have found OTB.

The amazing thing is that these people in the Edo era (mostly 17th century) designed these puzzles without computer help. When we learned about them, most could not even be solved with the aid of computers.

I'm with Evert here. The old chu shogi mate puzzles are more an exercise in thought, rather than practical. I would much rather take on 5-6 move middlegame puzzles, that actually train someone for the much necessary skill of counter-skirmishing.

Unfortunately, creating and sharing chu shogi puzzles is much more difficult and time-consuming than it is for chess puzzles. It all boils down to lack of adequate infrastructure. The only software I know of that saves lines is at chu.is.land.to 

And even if it were easy to create and share puzzles, no current engine plays the middlegame strongly enough to be worth my time. Middlegame analysis would strictly depend on which player you believe more.

Well, almost all Chess puzzles I have seen (in newspapers and such, usually mate in 3, the solution invariably being some quiet move) were always positions that did not resemble anything you would ever encounter in a game even remotely. What you describe sounds more like a test suite for educational purposes, such as the famous WAC ('Win At Chess") suite, or frm the Fine book, where you are asked what would be the best move in ordinary-looking positions. It is just another sport entirely, like ice dancing and ice hockey are different sports, even though they both require skates.

Not sure what you mean by 'saving lines'. What lines? The link you posted gives a 503 error. Chu Shogi software does of course exactly what we want it to do. Xboard probably does it already, and if not, it is is open source...

Well, middlegame is by far more important for chu shogi. Endgame is very important for chess.

lines (chess context) = variations; series of moves

I haven't used it personally, but some people use that to make puzzles, and you can play through the answer.

Link works for me

Most GUIs can store variations, and allow you to play through variation trees. Of course WinBoard / XBoard can do that too. It is a standard feature.

End-game studies are again something completely different from mate puzzles or middle-game test suites. I wasn't talking about those.

As to your complaint about engine strength: I am the first to admit HaChu's middle-game play sucks. It has a very good record against humans at 81Dojo, but only because it overwhelms them in the end-game, when slider tactics becomes important, usually from a far inferior position.

The problem is that I am not a Chu Shogi player myself, so that it is very difficult for me recognizing strategic mistakes made by the engine. One such mistake seemed to be that it tried to quickly develop its sliders (mobility driven), and then allowed the opponent to creep up on them with Silvers and Coppers, while their retreat was blocked by its own Pawns, so that in the end they got forked. So I changed the evaluation to encourage stepper advance, so that it would meet the opponent's steppers with its own, rather than allowing them to assault the 'naked' sliders. This enormously improved its results on 81Dojo. Unfortunately, when I play that 'improved' version against the old one, it badly loses.

So I am not sure how to make progress from here. I have no other machine opponents I can have it play automated matches against, and if self-play prefers versions that perform worse against humans it doesn't seem a reliable method either. Probably I am just too far away from a realistic strategy for the self-play results to form a representative sample of game positions, so that tuning on it is meaningless, and can actually backfire.

If you are interested, I would be more than willing to modify HaChu's evaluation for implementing any strategic advice you can give me.

One problem in these 'large' Shogi variants that I still have not developed a satisfactory solution to is how to evaluate promotability. Pieces with strong promotion, such as Horses and Kirins should be worth significantly more than their tactical abilities suggest. But if too large a part of your armies strength comes from this 'latent' value, you will be overwhelmed before you get the chance to promote anything, because your opponent gets to promote his pieces first (even if he gains comparatively less by this), and from that point it will only go downhill faster. What complicates this is that players will tend to conserve their best-promoting material, so that when they promote anything at all, they will make sure that is amongst it. This becomes painfully obvious in Tenjiku Shogi, where promotions of a single Water Buffalo will completely swing the game, no matter how much you are behind at that point.

Another weakness of HaChu is that it is completely naive against the opponent preparing a massive battery for forcing a breakthrough into the promotion zone; by the time it can see any profitable tactics from this it is much too late to do anything about it.

Of course, I am very interested in creating as strong an engine as possible, especially for chu shogi openings and middlegames. Due to the vast array of feasible opening strategies, ideas that work against one strategy can often be terrible against others. I had to (mostly) self-learn what's effective against what else. Without a neural network, I have to imagine this would be incredibly difficult to program. So let's focus on general ideas that are almost always good to employ.

Here is a list of simple things to consider:

• Advancing at least 1 slider (preferably horse or bishop), delayed until the area of focus has been established. Do not advance more than 2 past the pawn line, except if the evaluation is very good.
• Do not push step movers until clear paths have been created, and only when no other significant positional gain is possible.
• Advancing silvers is better than bronzes.
• It's better for your advanced sliders to eye squares around your opponent's most heavily developed area, and near their lion.
• Do not attack the opponent's lion without tactical cause, or unless you can cleanly force it away from the area of focus.
• If sente, highly consider pushing the go-between (1 square) on the side where you've developed your lion, as well as the rook pawn (1 square) on that side.
• Penalize pushing the center 2 pawns exactly 1 square. If inclined to push one of these pawns, highly incentivize pushing that pawn again, unless it fails tactically. Also incentivize if one of these center pawns attacks a lion, but only if the pawn has already moved. Not moving the 2 central pawns at all is preferred, at least until the middlegame.
• Create open lanes. Always have retreating options for advanced sliders. 

That's a good start. I may add more items to the list at a later time.

Some of that behavior is easy to achieve. E.g. the center-Pawns thing can be simply achieved by just giving a very low value for the square on 5th rank in front of them, in their piece-square table.

Preserving a retreat option can be encouraged by non-linear mobility per move direction. With simplistic mobility evaluation you would award each move the slider has equally. But it would be easy to use the number of moves in each individual direction as an index in a small table that specifies how much that number of moves is worth, and give each direction (or set of directions) its own table for that. And then penalize the case where the backward directions have only few moves. This would not take into consideration whether the moves are safe, however. But moves back into your own camp do have a high likelihood to be safe.

As to advancing the steppers: I can control how it handles these to a large extent through the piece-square tables. E.g. whether the bonus for advancing one rank increases or decreases with advance, determines whether it encourages dashing the generals one by one to their forward posts, or whether it first prefers to advance all (or as many as possible) of them one rank, before stepping one to the 3rd rank. There is also the matter of how this differs between the various types of steppers. I currently do not encourage advancing Tigers or Elephants at all; they get a bonus for being in front of the King. Not sure how to handle Gold, though. Or how far the advance of the generals should be continued. Should it stop at 4th rank (after advancing Pawns to the 5th)? Or should they venture out in the open in front of the Pawn line? In Tenjiku Shogi I use a stepper evaluation this year that discourages advancing of individual steppers in the opponent's half of the board, but I do give an increasing (with advance) bonus for steppers to protect each other there. So that packs of several steppers will try to push forward.

A problem with advancing the steppers seems to be that it makes the promotion zone indefensible too early. This is how the HaChu version that advances the steppers in the end loses against the the evrsion that doesn't, after first achieveing a large material gain.

1. A retreat option 3 moves long is no good at all. 2 moves at very most, but preferably only in 1 move. If 2 moves, normally this is a sideways evasion followed by a linear retreat, assuming the piece is a horse or rook. I assume the majority of advanced sliders are {horses, bishops}. Occasionally, it's a rook. I would not advise advancing a dragon without trading or sacrificing one of the central 4 pawns first.
2. It's very tame to just sit step movers behind the pawn line. That is, unless a pawn seriously needs the protection, or prophylactically to prevent an edge attack, in which case it's absolutely no good to have a flat pawn line. Step movers are actively advanced with the intention of moving past the pawn line. However, all step mover pushes must be preempted by positional considerations elsewhere.
3. Silvers work best on the 3rd to 5th files on either side. Silvers by far should have the highest priority for advancement among steppers, which is not to say that the priority is high, rather only that they go before other steppers. Leopards work best on the 1st to 3rd files on either side Golds work best on the central 4 files. I would not fully advance a gold without high safety, or a lot of retreat options. Bronzes for the most part should be left where they are, unless in serious need of defense, or to be used as a needed sacrifice. 
4. Steppers should sit 1 square ahead of the pawn on that file, or 2 where the player has comfortably controlled a lot of space. Enact a high trade penalty on these steppers, that increases with each advanced rank.
5. It would be extremely difficult to convey my actual thoughts about the shape of pawn lines. Of course, a heuristic is nice to program the engine to use. Nonetheless, I view an active endeavor to improve an engine's understanding of the chu shogi pawn line as futile. 

I also have the the idea to specially program a countermeasure against the weak boring $-@+_/ that so many noobs play, with all the pawns advanced to the 5th rank.

I was thinking only about single-move retreat options; those can be seen statically, just by looking at the length of the free path a slider has in its backward move directions. To recognize 2-move options basically requires a search, where the opponent should also get the opportunity to knock you off the intermediate square. If absence of a single-move retreat is heavily penalized, but a 2-move option is available, the engine could only dodge the penalty if it spends its last ply of the search to bring the slider to safety. If it has multiple sliders in that situation, it would have to spend ever so many plies at the end of each line, to prove they can all be brought to safety. It might not have enough depth to do that.

If it is really important to give the sliders the leeway to afford a 2-move retreat, it might require a redesign of the engine, where the final plies are not really a minimax search, but a prove that each of the sliders have a retreat. E.g. if you have 3 sliders that need a 2-move retreat, and only one ply left, you would normally always end up with a situation where 2 of them still were lacking a single-move retreat, and receive the penalty for that. You could avoid that by summing the improvements the first retreat move of each of the sliders gives for this evaluation aspect, instead of the usual taking of the maximum. So that when each slider can retreat, you would cancel all 3 penalties, even though the search only goes deep enough to retreat one of them.

Unless with great advantage or at least 5 pawns lost or at least into the late middlegame, I would not have 3 sliders advanced past the pawns, ever. For the latter 2 conditions, retreat doesn't matter so much. It really only matters in the opening and early middlegame.

That's not to say I would advance 3 sliders early, with a great advantage, just that it then becomes a possibility.

I can also help create an opening book, if that is something you can implement. Most of my theory runs at least 17 ply, with the longest line at 33 ply, and a few of my favorite lines I have analysed deeper.