I'm adjusting my serial receive routine so the interrupt only loads one character at a time and I have a separate routine in my mainline that does further processing of the characters.

My packet that is sent over the air wirelessly from micro to micro must start with a special character to denote the start of the packet. and it expects a total of 11 bytes for the entire packet to be read.

My entire setup in the end will be a 1 master to many slave setup and each slave is addressed one at a time.

Now I already have a timer interrupt in use that helps count down a delay timer in another part of my program so that stalling doesn't happen elsewhere.

Question is, should I use a timeout for the serial port (modify the timer interrupt)? or is it better to not use timeouts in exchange for speed (not having to poll timer on a frequent basis)?

Or am I better to use a counter in the main master loop that endlessly processes all the low priority functionality?

The speed of data is 56kbps and transferred via the microcontroller uart.

The microcontrollers are all AT89S52.

Like most 8051's, the AT89S52 has three timers, 0, 1 and 2. The UART can use either timer 1, timer 2, or the internal baud rate generator (based on the external crystal) to set it's baud rate. As the hardware timers run independent of the instruction processing they are the fastest, until they interrupt the processor. By my count, if you're using one timer for the UART BRG and another for the other timeout, don't you still have one hardware timer left to do the serial port timeout task?

You only need a single timer for multiple uses.  Typically I declare a timer structure with a start var and an elapsed var.  The var type is architecture dependent so could be 16-bit.  The following is declared in a header file:

typedef struct {
  uint32_t  start_time;         /* Initial time stamp */
  uint32_t  elapsed_time;       /* Current time stamp */
} timer_stamp_t;

Aside from the timer ISR where the tick value is incremented, the only two routines you need is one to set the start_time var and another to calculate the elapsed time according to a read of the current tick value.  Simple usage:

timer_stamp_t timer;

timStamp(&timer); // Routine to set start_time element of timer_stamp_t

do {

  // ... do stuff

  timElapsed(&timer); // Routine to calculate elapsed_time from start_time

} while (timer.elapsed_time < 1000); // if tick is 1 millisecond, quit after 1 second

There is no limit (other than memory) to having as many timer_stamp_t objects as you need.  Only one timer ISR is necessary.  The structure can be reused endlessly if you reload the start time with a call to the routine that sets the initial time stamp.

Wrt serial ports, it is always better to asynchronously buffer (ISR) incoming data to a ring buffer that guards for overflow and read that buffer from a higher level entity like a super-loop or a state-machine called from a super-loop or a thread.  Timeouts should be considered only at these higher levels.  The low level UART driver should only concern itself with read/writes of the ring buffers with semaphore and/or interrupt controls to prevent contention with the higher level actors.

Another thing to consider is data framing.  I love COBS though it might not be feasible for all situations (some bluetooth modules assume ASCII only with status messages delimited by special ASCII characters - makes binary streams difficult to use).

Call me crazy but I'm doing my work in assembler.

Ok so I am implementing a smallish ring buffer because I don't have alot of memory to work with. If in the future I decide to make a compact design (appearance wise), then I'll go with AT89C4051 which does not have a timer2. Then again, maybe I'm just stuck with AT89S52.

Assembler allows abstraction from the hardware but it requires more effort. 

It is a bit crazy to be using assembler for anything other than speed optimization in critical code areas.  If your chip vendor doesn't provide you with a free C compiler (or you cannot afford a commercial compiler) you are swimming against the current.    

Well, I'm already against the current seeing I have no gf and no career, so I figure I might as well create my own career with making special equipment that people will love. Well at least I'm making an effort not to sink in the current. So far I'm doing well with assembler. Its a matter of timing everything perfectly. And optimization is where I spend much of my coding time anyway. Heck, the better I optimize, the less expensive the hardware will be.

I don't know enough about your equipment to judge your selection of micro but that particular chip is obsolete.  Microchip still sells it however you can buy a 32-bit ARM M0 for almost the same amount of money - and it will come with a free Eclipse-based IDE.  Digikey sells the STM32L021K4T6 for $2.19 CDN (Qty 1).

But I already made my hardware for the AT89S52. I order from Futurlec. I actually explored Digikey and Mouser electronics and I have ordered parts in the past from Digikey through a local electronics shop and their reputation isn't all that great, especially when my order took 2 weeks to arrive as opposed to a few days. Also, the local shop indicated they even had some problems with Digikey as well. On top of that, shipping from Futurelec is inexpensive, where as with Digikey and Mouser, I have to spend $100+ to get a marginally reasonable shipping rate (including what the carrier charges at door).