Troubleshoot port exhaustion issues

TCP and UDP protocols work based on port numbers used for establishing connection. Any application or a service that needs to establish a TCP/UDP connection will require a port on its side.

There are two types of ports:

• Ephemeral ports, which are usually dynamic ports, are the set of ports that every machine by default will have them to make an outbound connection.
• Well-known ports are the defined port for a particular application or service. For example, file server service is on port 445, HTTPS is 443, HTTP is 80, and RPC is 135. Custom application will also have their defined port numbers.

Clients when connecting to an application or service will make use of an ephemeral port from its machine to connect to a well-known port defined for that application or service. A browser on a client machine will use an ephemeral port to connect to https://www.microsoft.com on port 443.

In a scenario where the same browser is creating a lot of connections to multiple website, for any new connection that the browser is attempting, an ephemeral port is used. After some time, you will notice that the connections will start to fail and one high possibility for this would be because the browser has used all the available ports to make connections outside and any new attempt to establish a connection will fail as there are no more ports available. When all the ports are on a machine are used, we term it as port exhaustion.

Default dynamic port range for TCP/IP

To comply with Internet Assigned Numbers Authority (IANA) recommendations, Microsoft has increased the dynamic client port range for outgoing connections. The new default start port is 49152, and the new default end port is 65535. This is a change from the configuration of earlier versions of Windows that used a default port range of 1025 through 5000.

You can view the dynamic port range on a computer by using the following netsh commands:

• netsh int ipv4 show dynamicport tcp
• netsh int ipv4 show dynamicport udp
• netsh int ipv6 show dynamicport tcp
• netsh int ipv6 show dynamicport udp

The range is set separately for each transport (TCP or UDP). The port range is now a range that has a starting point and an ending point. Microsoft customers who deploy servers that are running Windows Server may have problems that affect RPC communication between servers if firewalls are used on the internal network. In these situations, we recommend that you reconfigure the firewalls to allow traffic between servers in the dynamic port range of 49152 through 65535. This range is in addition to well-known ports that are used by services and applications. Or, the port range that is used by the servers can be modified on each server. You adjust this range by using the netsh command, as follows. The above command sets the dynamic port range for TCP.

The start port is number, and the total number of ports is range. The following are sample commands:

• netsh int ipv4 set dynamicport tcp start=10000 num=1000
• netsh int ipv4 set dynamicport udp start=10000 num=1000
• netsh int ipv6 set dynamicport tcp start=10000 num=1000
• netsh int ipv6 set dynamicport udp start=10000 num=1000

These sample commands set the dynamic port range to start at port 10000 and to end at port 10999 (1000 ports). The minimum range of ports that can be set is 255. The minimum start port that can be set is 1025. The maximum end port (based on the range being configured) cannot exceed 65535. To duplicate the default behavior of Windows Server 2003, use 1025 as the start port, and then use 3976 as the range for both TCP and UDP. This results in a start port of 1025 and an end port of 5000.

Specifically, about outbound connections as incoming connections will not require an Ephemeral port for accepting connections.

Since outbound connections start to fail, you will see a lot of the below behaviors:

Unable to sign in to the machine with domain credentials, however sign-in with local account works. Domain sign-in will require you to contact the DC for authentication which is again an outbound connection. If you have cache credentials set, then domain sign-in might still work.

Group Policy update failures:

File shares are inaccessible:

RDP from the affected server fails:

Any other application running on the machine will start to give out errors

Reboot of the server will resolve the issue temporarily, but you would see all the symptoms come back after a period of time.

If you suspect that the machine is in a state of port exhaustion:

Try making an outbound connection. From the server/machine, access a remote share or try an RDP to another server or telnet to a server on a port. If the outbound connection fails for all of these, go to the next step.

Open event viewer and under the system logs, look for the events which clearly indicate the current state:

a. Event ID 4227

b. Event ID 4231

Collect a netstat -anob output from the server. The netstat output will show you a huge number of entries for TIME_WAIT state for a single PID.

After a graceful closure or an abrupt closure of a session, after a period of 4 minutes (default), the port used the process or application would be released back to the available pool. During this 4 minutes, the TCP connection state will be TIME_WAIT state. In a situation where you suspect port exhaustion, an application or process will not be able to release all the ports that it has consumed and will remain in the TIME_WAIT state.

You may also see CLOSE_WAIT state connections in the same output, however CLOSE_WAIT state is a state when one side of the TCP peer has no more data to send (FIN sent) but is able to receive data from the other end. This state does not necessarily indicate port exhaustion.

Having huge connections in TIME_WAIT state does not always indicate that the server is currently out of ports unless the first two points are verified. Having lot of TIME_WAIT connections does indicate that the process is creating lot of TCP connections and may eventually lead to port exhaustion.

Netstat has been updated in Windows 10 with the addition of the -Q switch to show ports that have transitioned out of time wait as in the BOUND state. An update for Windows 8.1 and Windows Server 2012 R2 has been released that contains this functionality. The PowerShell cmdlet Get-NetTCPConnection in Windows 10 also shows these BOUND ports.

Until 10/2016, netstat was inaccurate. Fixes for netstat, back-ported to 2012 R2, allowed Netstat.exe and Get-NetTcpConnection to correctly report TCP or UDP port usage in Windows Server 2012 R2. See Windows Server 2012 R2: Ephemeral ports hotfixes to learn more.

Open a command prompt in admin mode and run the below command

Open the server.etl file with Network Monitor and in the filter section, apply the filter Wscore_MicrosoftWindowsWinsockAFD.AFD_EVENT_BIND.Status.LENTStatus.Code == 0x209. You should see entries which say STATUS_TOO_MANY_ADDRESSES. If you do not find any entries, then the server is still not out of ports. If you find them, then you can confirm that the server is under port exhaustion.

Troubleshoot Port exhaustion

The key is to identify which process or application is using all the ports. Below are some of the tools that you can use to isolate to one single process

Method 1

Start by looking at the netstat output. If you are using Windows 10 or Windows Server 2016, then you can run the command netstat -anobq and check for the process ID which has maximum entries as BOUND. Alternately, you can also run the below Powershell command to identify the process:

Most port leaks are caused by user-mode processes not correctly closing the ports when an error was encountered. At the user-mode level ports (actually sockets) are handles. Both TaskManager and ProcessExplorer are able to display handle counts which allows you to identify which process is consuming all of the ports.

For Windows 7 and Windows Server 2008 R2, you can update your Powershell version to include the above cmdlet.

Method 2

If method 1 does not help you identify the process (prior to Windows 10 and Windows Server 2012 R2), then have a look at Task Manager:

Add a column called “handles” under details/processes.

Sort the column handles to identify the process with the highest number of handles. Usually the process with handles greater than 3000 could be the culprit except for processes like System, lsass.exe, store.exe, sqlsvr.exe.

If any other process than these has a higher number, stop that process and then try to login using domain credentials and see if it succeeds.

Method 3

If Task Manager did not help you identify the process, then use Process Explorer to investigate the issue.

Steps to use Process explorer:

Download Process Explorer and run it Elevated.

Alt + click the column header, select Choose Columns, and on the Process Performance tab, add Handle Count.

Select View \ Show Lower Pane.

Select View \ Lower Pane View \ Handles.

Click the Handles column to sort by that value.

Examine the processes with higher handle counts than the rest (will likely be over 10,000 if you can’t make outbound connections).

Click to highlight one of the processes with a high handle count.

In the lower pane, the handles listed as below are sockets. (Sockets are technically file handles).

Some are normal, but large numbers of them are not (hundreds to thousands). Close the process in question. If that restores outbound connectivity, then you have further proven that the app is the cause. Contact the vendor of that app.

Finally, if the above methods did not help you isolate the process, we suggest you collect a complete memory dump of the machine in the issue state. The dump will tell you which process has the maximum handles.

As a workaround, rebooting the computer will get the it back in normal state and would help you resolve the issue for the time being. However, when a reboot is impractical, you can also consider increasing the number of ports on the machine using the below commands:

This will set the dynamic port range to start at port 10000 and to end at port 10999 (1000 ports). The minimum range of ports that can be set is 255. The minimum start port that can be set is 1025. The maximum end port (based on the range being configured) cannot exceed 65535.

Note that increasing the dynamic port range is not a permanent solution but only temporary. You will need to track down which process/processors are consuming max number of ports and troubleshoot from that process standpoint as to why its consuming such high number of ports.

For Windows 7 and Windows Server 2008 R2, you can use the below script to collect the netstat output at defined frequency. From the outputs, you can see the port usage trend.

Useful links

Port Exhaustion and You! — this article gives a detail on netstat states and how you can use netstat output to determine the port status

Detecting ephemeral port exhaustion: this article has a script which will run in a loop to report the port status. (Applicable for Windows 2012 R2, Windows 8, Windows 10)

Windows Kill Process By PORT Number [duplicate]

I’m using embedded Tomcat server in Spring Tool Suite IDE. My problem is when I run my project there is an error as follows,

There are some similar questions but none of the answers not working for me.

2 Answers 2

Run command-line as an Administrator

Please Make sure that new port you are going to set for your Application doesn’t listen to any other process

Solution 3: Another way is to terminate the process (in IDE) and clean and rebuild project.

UPDATE:

For solution 2, Please Make sure that new port you are going to set for your Application doesn’t listen to any other process.

How to check Port Status?

Run resmon.exe and go to Network -> Listening Port (Also can be viewed on TaskManager)

PowerShell

cmd

(Add -n to stop it trying to resolve hostnames, which will make it a lot faster.)

-a Displays all connections and listening ports.

-b Displays the executable involved in creating each connection or listening port. In some cases, well-known executables host multiple independent components, and in these cases, the sequence of components involved in creating the connection or listening port is displayed. In this case, the executable name is in [] at the bottom, on top is the component it called, and so forth until TCP/IP was reached. Note that this option can be time-consuming and will fail unless you have sufficient permissions.

-n Displays addresses and port numbers in numerical form.

-o Displays the owning process ID associated with each connection.

How to see if Windows Firewall is blocking a port or program

• Wondering how you can check if a website is blocked by your Firewall? Check the firewall settings.
• The Windows Firewall settings will give you show you any port is blocked on your PC.
• Easily fix any problem caused by your Firewall blocking the Internet with the solutions below.
• The Windows Firewall also allows you to select which programs can run on your computer.

1. Download Restoro PC Repair Tool that comes with Patented Technologies (patent available here).
2. Click Start Scan to find Windows issues that could be causing PC problems.
3. Click Repair All to fix issues affecting your computer’s security and performance

• Restoro has been downloaded by 0 readers this month.

The Windows Firewall is a built-in security application that comes with Windows OS since the beginning. The software is built to filter network data transmission to and from your Windows system.

The Firewall will block any suspicious and harmful connection depending on the threat level.

The users can configure the Windows Firewall settings as per their need to block or open port in Windows 10 and other versions. However, at times the Firewall may block ports or programs accidentally by user’s or administrator’s misconfiguration.

Now, if you want to know if the Windows Firewall is blocking a port or program on your system, you need to check your settings. Here’s how to check firewall settings.

How do I check if Windows Firewall is blocking a port ?

1. Check your firewall settings

1. Press Windows Key + R to open Run.
2. Type control and press OK to open Control Panel.
3. Click on System and Security.
4. Scroll down and open Administrative Tools.
   
5. In the Administrative Tools window, open Windows Defender Firewall with Advanced Security.
6. Click on Actions and select Properties.
   
   
7. Now select your preferred Profile (Domain, Private, Publick).
8. In the Logging section, click on the Customize button.
   
9. Click on the drop-down menu for Log dropped packets: and select Yes.
10. Take note of the pfirewall.log path in the Name section.
11. Click OK to save the changes.
    
12. Open File Explorer and go to the path where the log file is saved.
    • It should look something like this: %systemroot%system32LogFilesFirewall
13. Click on the pfirewall.log file and check for any blocked ports.

2. Check for Blocked Port using the Command Prompt

1. Type cmd in the search bar.
2. Right-click on the Command Prompt and select Run as Administrator.
   
3. In the command prompt, type the following command and hit enter.
   • netsh firewall show state
4. This will display all the blocked and active port configured in the firewall.

How to check if Windows Firewall is blocking a program ?

1. Press Windows Key + R to open Run.
2. Type control and press OK to open Control Panel.
3. Click on System and Security.
4. Click on Windows Defender Firewall.
5. From the left pane Allow an app or feature through Windows Defender Firewall.
   
6. In the allowed app window, scroll through all the apps.
7. Locate the app you want to check and see if the app is checked.
   • If it is unchecked, the app is blocked on the Firewall.
8. If your program is blocked, simply uncheck the app and click OK.

In the Customize Settings window, click the circle(s) next to Turn off Windows Defender Firewall for public networks, private networks, or both.

That’s about it for this article. Make sure you follow the solutions thoroughly and see what works for you. Also, feel free to send us any feedback regarding the subject in the comment section below.