I like to play "the best" once in a while.
I usually play against Critter 1.6, as according to my tests this is the strongest free engine and it's also available for android.
I also played against Houdini 1.5 but while he is about even in an engine vs. engine match he seems less aggressive than Critter.
I think I've actually made a slight progress in doing so.
When I first started Critter mated me in 15ish turns, then 19, then 29 and the last time 27.
I've learned some things that are no-goes, which before I thought were fine but always got punished by the Engine.
I see no logical reason why it should not help you improve. Of course it shouldn't be the only thing you do since it'll become pretty frustrating. But if your aim is to push being mated to a later stage of the game and building a good defensive, it probably is a good idea to give it a try.
Does playing against a strong engine help you improve?