The TCP (Transmission Control Protocol) ensures the delivery of all packets. It has built-in logic to handle the cases where packets are lost or not received, called 'Retransmission'. The three-way handshake is the first part of the process that could be impacted by packet loss, and incur retransmission. However, retransmission can also happen on any of the packets that are lost (not acknowledged) during the communication. These two cases are treated differently by the TCP implementations.
In the case of the TCP three-way handshake, first, the initiator (Host A) sends a small TCP packet that has a special "SYN" flag indicating a connection request to the receiver (Host B). The receiver (Host B) has to acknowledge that it received the "SYN", so it sends two TCP packets "ACK" & "SYN" flags back to the initiator (Host A). Once the initiator receives the two packets, it sends an acknowledgment "ACK" back to the receiver to end the connection. During this process, if the receiver did not receive the initiator's initial TCP packet (SYN flag), the initiator can resend the same TCP packet (SYN flag) back to the receiver.
The Windows operating system relies on a library called Winsock to establish and manage protocol communications. Windows have implemented the TCP RFCs on this library by default, however, it provides the users to change these settings.
When a TCP connection is being established, Winsock by default will try to establish a connection up to 3 times (controlled by registry key MaxConnectRetries). In other words, Winsock will send the SYN packet up to three times and wait for an SYN-ACK from the server. The first time Winsock will wait 3 seconds, and it will double on everyconsequentretry. It will send the first SYN at 0 seconds, it will retry at 3 seconds, then retry again at 9 seconds (+6 seconds from second SYN), and it will wait up to the 21st second before timing out (+12 seconds from third SYN).
In this case, when the target host replies with an ACK/RST packet, for connection to an un-used port, Winsock will not increase the timeout value. However, it will continue to retry the connection until the maximum retry value (default 3) is reached. In this case, it will send the first SYN at 0 seconds, it will retry at 3 seconds, then retry again at 6 seconds (+3 seconds from second SYN), and it will wait up to the 9th second before timing out (+3 seconds from third SYN).
Winsock relies on a slightly different logic to retransmit packets. It has a default retry of 5 times for the segments, and it is controlled by a registry key TcpMaxDataRetransmissions.
The retransmission timer is initialized to three seconds and adjusted on the fly to match the characteristics of the connection using Smoothed Round Trip Time (SRTT) calculations described in RFC793. The timer for a given segment is doubled after each transmission of that segment. This feature allows TCP to adjust in a normal delay of a connection. This means high-delay links will take a much longer time for TCP to timeout.
The default for the retransmission timer is 240 seconds but you can modify the retransmission time for faster connection through Microsoft's support page: http://support.microsoft.com/kb/170359