Gaining a small material advantage is easy. Turning that into a win is hard. Any advice?

I only looked at game 1.  The position after 24 moves looks won to me.  But you started to play a little aimlessly imho and forgot to work on creating a passed pawn.  For example, 31. ...b5 just helps him activate his rook -- E5 was the right move.  33. ... bb4 was an actual blunder -- precision is important in endgames. 

I like that system of the French you played.  My IM friend used to play it growing up, and would frustrate the *&^! out of me.  

Game 2 : Move 32 trade Queens and then push the c-Pawn.

The overall problem might be a tendency to narrow your focus until it only encompasses a small part of the board, rather than being aware of possibilities and threats across the whole board.

Fourth game against "The Queen"

19. Qd5+ should be winning. The d4 White pawn shields a potential discovered check by 19...Rf8 20. Qxd7 Bc5+.

The French Defence is infamous for trading down and getting endgame positions. If you're not a great endgame player, then now's the time to improve that aspect of your game, or play another opening.

Personally, I prefer tactical middlegame positions, hence why I play the sicilian instead.

I only checked the endgame in the first game. I think 30. ... h7-h5 is a mistake - you give up the control of the g5 square and create an outpost for the white Knight. I would chase away the Knight with h7-h6. You can still push the pawn to h5 later if you want to, but you cannot put it back to h6!

31. ... b6-b5 is also a mistake, it goes against the principle: "When you up in material, exchange pieces; when down in material, exchange pawns." Here you exchange pawns without a reason, and you open the a-file which you cannot control, but white can. That gives white strong counterplay. 

I would play like this: 

One of the most important principle in endgame play is being patient. If you don't see an immediate win or anything concrete, just look for small improvements with every move, and wait for your time to come. 

I think gaining an extra material is the hardest part of chess You have a problem with your tactical vision and your technique - you give your extra material back because you blunder and you don't know what to do with the extra material. So to cure this I suggest concentrating on tactics and learning simple strategy tips for beginners - I recommend reading Max Euwe's books. All the stuff about focus-pocus, clock-factor, endgames, control-patrol is useless for you at the moment.

@springyboard See, most of the chess principles are about making it easier to coordinate the pieces against the rival's material and, or, King, while refusing the same to him.

Say you have a material advantage. Does it make your position better? Not necessarily, as your rival may have enough piece activity (threats) as to keep you busy defending, thus making it impossible to free enough pieces to develop threats of your own.

So what is coordination? Think of it as intersecting the action of pieces on a square (or several for that matter), usually occupied by the opponent's material and, or, King,  as to develop threats. If your pieces are not defending, then they're free to move and coordinate with others. If your pawn structure –or the rival's– isn't limiting their possible moves, then you may have more and perhaps better roads to coordinate them faster than the rival's pieces coordinating to put up an effective defense. And so on.

Having a material advantage usually limits the defensive strategies for the defender, as opposing and allowing trades may be out of the question. Having a material advantage can also make it easier to increase the pressure, either by numbers or mobility (to evolve into different attacking patterns). That said if you don't seek coordination to attack, then your material advantage is not being used.

Now, think of how you got the material plus in those games. You had a clear idea of where to place your pieces to attack specific material. Well, once you get the material plus you're supposed to keep doing the same, somewhere else and again and again... until you can give checkmate. And even if you don't see the best way to do it, at least you're trying to. That's how you improve in chess.

Thank you for all the replies and good advice! Will read through them, try to apply them in my games and post a more thorough followup later.

Time was not a significant factor as all the above games were daily, played on this site (my OTB record with endgames is good for my level). Only games 1 and 4 featured pure endgames; the rest were decided in the late middlegame.

Some thoughts on the games and further questions:

For game 3 (B vs Gollen0444), was 31 Rb7?? the decisive blunder or did I still have the advantage after that?

Was the 37 Nxd4 sacrifice in game 5 (W vs btsypulin) correct? I could not see any other way to improve my position. What would be the likely result if I let my h-pawn drop?

Despite being material up in game 7 (B vs shamir_a), was I actually winning? White had a pretty strong attack on my King.

If turning a small material advantage into victory was easy, chess would be a much simpler game, and we would have quit playing it a long time ago.