How to solve this endgame as white?

White is in trouble unless there is some saving resource... Qd8+ forces a draw by perpetual check after Ka7 Qa5+ Kb8 Qd8+...i was looking at b6 also with the almost unstoppable threat of Qd8# but black can muddy the waters with Rg4+ desperado sacrifice eg. Rg4+ Kxg4 g1=Q+ so I'm not sure about that... 

Maybe you are right. White cannot win ,I think. The rook and the two pawns to be promoted are great threat. I can not find another way besides keeping checking by queen.

Also Qd8+ Ka7 b6+ Ka6 doesn't seem to lead anywhere either. 

It's an endgame study, probably a recent one (after the year 2000). White does indeed win and it is quite thematic and exciting!

Hint: 1.b6 is the right start!

Warning: it's hard even when you are a specialist. I solved it in collaboration with an engine.

does black have any defense if b6 though? how can white escape check from black?

Here's the study with source and the main line of the solution. The WNg7 is actually an obstacle for the white rook when the latter moves to the g-file. So White compels Black (who must keep checking to stave off the queen mate) to remove that knight with the black rook, by moving the king up the h-file and drawing this rook along.

If black rook goes to g8 instead of g3, how to do with?

That's why I wrote it's probably a post-2000 study. This is one of the hard defenses which requires engine analysis to find the win. It starts with Qa5+ followed by a number of checking moves plus  a capture on g2. These moves are not very pretty which is why they are ignored in the presented solution. But you can trust all post-2000 endgames are computer-tested and therefore correct (with the few proverbial exceptions of course).

After Rg8, then Qa5+, Qa7+ and taking on b7 with check will win

1. First of all Qa5+ is illegal, so I assume you mean Qd8+ and then Qa5+
2. After such moves the only check left is Qd8+, repeating the position

3. If playing Qd4+, Kb8 is equal 

DicexCUHK wrote: "If black rook goes to g8 instead of g3, how to do with?" which is after move 9 of the solution given by Rocky64. Qa5+ is quite legal at that point.

At move 9, Rg3+ is a major blunder for Black leading to Mate in 8.  Rg8 is Black's best hope, but White can still Mate eventually playing best moves.

Compositions do not care about superficial game move ratings such as "blunder" or "brilliancy". On the scoring level they only care about the correct outcome; all wins are equal. On the content level they care about thematic lines. The move 9. ... Rg3+ demonstrates blacks persistent strategy of sacrificing his rook in order to promote with check. He succeeds at last but to no avail as whites rook on h2 comes to its kings rescue and the loss of Ng7 proves a decisive advantage

Arisktotle has explained it already, but to put it another way, the point of a composed endgame study – the reason it exists in the first place – is to show an interesting theme as revealed in its main variation(s). Here 9...Rg3+ is the thematic defence of the main line – It's not a "blunder" because the position is already lost – whereas 9...Rg8 is just a boring side variation. Engines of course would choose 9...Rg8 because they are designed to drag things out, as if this were a practical game; they have no inkling of what makes a variation interesting.