Answers

[owen]

Hi Michael, Ulrich,

These errors are related to client behaviour and in most cases, can be safely ignored.

The 'Timing out new connection' error is raised if the traffic manager accepts a new connection, but the client does not send any data within the virtual server 'connect_timeout' period (default 10 seconds).  The traffic manager will close these connections to free up resources (generally a file descriptor and a few KB of memory).

You can demonstrate this by opening a telnet session to the traffic manager and not writing anything down it; after 10 seconds, the traffic manager will close the session and write the log (if you have the log!client_connection_failures setting enabled).

If you are concerned about  the number of errors, you can increase connect_timeout, but this will probably not make any difference; any client that fails to write any data within 10 seconds is unlikely to do so in 30, or 60 seconds.  These problems probably arise because the client's network is extemely congested or unreliable (interesting to note that many of the affected clients appear to come from residential dsl or cable connections), or they may be due to network probes or port scans.

One thing to highlight - some protocols such as FTP begin with a server hello statement.  The client will wait until it hears the server hello before writing any data.  If you are managing such a protocol, and you misconfigure the traffic manager to use a client-first protocol (http://community.riverbed.com/t5/Answers/Server-first-and-Client-first-Protocols/td-p/16478) then the connection will stall and you'll get this error every time.

Best regards

Owen

[ubabiak]

Michael,

I have the same problem on my site (i.e. lots and lots of  "Timing out new connection" errors) ;  actually I am not sure whether it is a real problem or some logging artifact  that can be ignored.

This btw is a good question for a community site, so why don't we compare our numbers?

In my case I have  a ratio between 1 : 30-50

 i.e.for every 30-50 requests on a particular vhost on my Trafficmanager, I get 1 "timing out" error. 

What are your numbers? Everybody else reading this - can you compare and tell us your findings?

Enable logging temporarily in

Virtual Servers > SERVERNAME > Connection Management >Connection Error Settings> log!client_connection_failures

My sites are primarily B2C sites (news, entertainment) so I have lots of residential users with dialup/DSL ;

further research has shown that most of the logged IPs had hostnames like *dsl*-something, *cable*-something etc., sounding like they were used for residential access.

Over the years, I have learned to safely ignore all those zillions of "connection reset by peer" errors found in the log of every busy webserver; that's something that just  happens since you can't expect that each and every connection can be handled correctly.  ""Timing out new connection"",  is obviously a Zeus-specific message, so I am lacking this experience.

I have opened  a support case for this issue and so far, they recommended to check out the listen queue size and other

tcp parameters as described in  http://blogs.riverbed.com/stingray/2005/09/tuning-zeus-traffic-manager-for-maximum-performance.html   That , however, has had no effect on the number of error messages.

>>So in your case, you have a client read which sent no data in any direction because it timed out.

Sounds like Chris is assuming a generic network related communication problem.  If I knew that other sites have

the same percentage of log entries, I would think that they probably can be ignored.

Regards,

Ulrich

[cgeary]

Hello,

The format of the log messages is as follows:

The leading letter is interpreted as: 

'a' - 'Closed' 

'f' - 'Complete' 

'R' - 'Client Read' 

'W' - 'Client Write' 

'X' - 'Request Rules' 

'q' - 'Queued' 

'c' - 'Node Connect' 

'w' - 'Node Write' 

'r' - 'Node Reading' 

'C' - 'Client Close' 

'K' - 'Keep-alive' 

'I' - 'Idle' 

The numbers each relate to the following, in this order: 

Bytes from client

Bytes to server 

Bytes from server 

Bytes to client 

Connection Established 

Client Idle Time 

Server Idle Time 

Connection Retries 

Response Code

So in your case, you have a client read which sent no data in any direction because it timed out.

[michaelayates]

Hi,

I was wondering if someone could point me at any documentation about the error log format for the Stingray Traffic Manager.

We have turned on error logging for some of our services. We now get quite a lot of output in /usr/local/zeus/log/errors. 

I would like to be able to interpret this information as see if there were any system tunables that could be changed to reduce the error rates.

Specifically we are seeing large numbers of lines like below (removed IPs and service name):

[04/Jan/2012:02:38:46 +0000]    INFO    ServiceName    connerror    "Timing out new connection" REMOTEIP:25365 LOCALIP:80 "-" "-" - "-" R 0 0 0 0 16 17 0 0 - - -

Thanks for any info.

Michael