A question about finding goals.

After studying a lot of chess-puzzles, which end in checkmate, I've noticed I make my most intricate plans when I notice a mating pattern and try to exploit it. I realize, after some thought, that it's because I have a concrete direction -- I have a goal that I'm trying to acheive, and so my attempts at manipulating the board become planned well in advance. 

When there is no clear possibility of a checkmate of any sort (or even the possibility of making the enemy king run) I have a very difficult time making long-term plans, more than a move or two in advance. I feel almost as if I'm "just moving pieces until something cool happens." 

How do the lot of you go about making plans? I'm familiar with basic development -- keep pressure on the center of the board, try to avoid moving the same piece twice, connect your rooks if possible, castle early, etc, but none of this lets me plan things out several moves in advance. 

I'm not really sure where to begin. Lets say my pieces are well developed, the enemy king is well guarded, there's no possiblity for a clear mate -- how do I go about making a long term battle plan? Do I need a long term battle plan? Do I pick an enemy piece and try to claim it stakes free? Or do I sit back and focus on developing my side of the board without enemy interference? Or is this where I need to recognize pawn/board formations and try to exploit the formations themselves? (I just started studying pawn development, so I suspect some of my difficulty could stem from here.)

Retraining your brain requires that you build 5 visualization pattern memory banks. Those 5 are

1. Tactical visualization pattern memory bank.

2. Mating net visualization pattern memory bank

One example is the visualization pattern of the mating net

for the basic endgame forced checkmates (K+Q vs. K, K+R vs. K, K+ 2B vs. K, K+B+N vs. K). Once you can execute these forced mates in your sleep you have acquired one part of the mating net visualization pattern and it has become a technique for you. You will find that the mating net visualization pattern that you have acquired will cause mating nets in your games, at any stage, to jump up off the board and smack you on the forehead.

3. Opening visualization pattern memory bank

4. Middle game visualization pattern memory bank

5. Endgame visualization pattern memory bank

In short, you're asking how to play middlegames, an impossibly broad question to answer completely on a forum.  A coach or a book like Pachman's Modern Chess Strategy can better answer the question.  I can give you a few things to think about though.

A kingside attack's goal is to mate the king, but what fundamental elements make an attack good?  Force and space.  Force simply being more attackers and space being room to maneuver your attackers.  Notice in chess if you have more space your opponent necessarily has less space and less maneuverability to organize defense if an attack changes from one square to another.

I bring this up because as you noted there are positions when an attack against the king doesn't work.  Sometimes you attack the queenside... and the same elements make it successful, force and space.

From this comes a well know middlegame tip to seek action on the side of the board where your main pawn chain "points"... i.e. where you have more space.  If you've developed reasonably in the opening, then you'll also have at least equal force on that side of the board.

The King's Indian Defense is probably best known for this kind of strategy where white is pushing on the queenside while black plays for an attack and checkmate on the kingside.

Other positions are more equal, and both sides maneuver to improve their pieces.  Take the following position for example.

So sometimes you are simply trying to stay centralized / active and avoid weaknesses while playing to enter an endgame.  Imagine for example white doubles his rooks on the d file and threatens infiltration on d7.  Under pressure like this white can coax black into trading off some heavy pieces and enter an unfavorable endgame.

Other often mentioned ideas can be understood under this umbrella of your pieces doing more than your opponent's.  Pawn structure is a good example.  Pawn weaknesses (like backward, doubled, or isolated pawns), when attacked, can tie your opponent's pieces down in defense.  Your attackers can move away whenever you decide your position is ideal for shifting a threat somewhere else.  Meanwhile your opponent's pieces do not have that mobility and must keep defending (or lose the pawn).

Take the following game for example.  White's main plan was to advance on the queenside in hopes of creating a pawn weakness, which he did in the backward c pawn, and now he'll attack it.  The advance of 2 vs 3 queenside pawns here is called a minority attack.  Black would like to generate some threats on the kingside where he has space, but the number of white defenders available means black may be forced into a passive defense.

Anyway, these are some very general and basic ideas.  Basically any superior position involves your pieces doing more than your opponent's in at least one area of the board.  Some fundamental elements to work for this are superior space and force.  Other often included ideas are pawn structure and better minor pieces (in the first diagram black has a bishop with no scope on f6 and white is coming to d5 with his knight).

For a full treatment of these (and more) ideas, you'd want to get a coach or good book like Pachman's Modern Chess Strategy and know some things about the endgame.