The main problem about DoS and 'try-and-guess' attacks cause is that they put a huge burden on the server's computational and networking resources. This is especially applicable to secure servers, as every instance of such a server, besides allocating networking resources, also needs to maintain a set of cryptographic parameters, making per-session computational and resource costs much higher.

The exact measures to choose for counteracting DoS attacks depend on the specifics and circumstances of every particular service and the environment it is supposed to work in (e.g., they will obviously differ for a service that runs behind an IPS and a service publicly visible from the internet and not protected by anything). Still, there are certain general rules that are applicable nearly everywhere and will increase the DoS tolerance of any secure service significantly.

The general approach here is to kick the attacker out as early down the protocol as possible.

The main points to consider are the following:

1. A newly allocated secure environment consumes several hundred KBs of memory, more than a freshly accepted TCP connection, so rejecting supposed attackers just after the TCP accept stage will save you a lot of memory.
2. SSH/SSL key exchange is a heavy routine involving asymmetric cryptographic operations, so getting rid of potential attackers before the protocol reaches this stage decreases the CPU burden drastically.
3. The SSH authentication phase may be used by the attacker to 'hold' the session (they may emulate the 'alive' state by issuing continuous login attempts). This may lead to the exhaustion of the server's network resources.

Taking the above points into account, we suggest you employ the following techniques:

1. Keep a log of all incoming connections and introduce a blacklist of IP addresses. If many consecutive fake connections are coming from the same IP address and/or subnet, add the address/subnet to that list and simply deny all subsequent connections originating from that range on the network layer (i.e., straight after the Socket's Accept call, without promoting the connection up to the SSH layer). The length of time to keep the IP in the blacklist can be chosen based on the specifics of your service and the 'molestation level' of the particular IP address and can range from 5 minutes to infinite presence.
2. Think about the geographic scope of IP addresses where 'legal' connections to your service can come from. Is there a chance that your customers will connect to your server from, e.g., China, the Philippines, or Russia? If not, it makes sense to reject connections originating from the corresponding geographic IP ranges even before looking for them in the black list.
3. Do not allow more than a reasonable amount of parallel unauthenticated connections (i.e., those still in the negotiation phase) from the same IP address. Limiting them to 2 would be fair enough.
4. Introduce a specific timeout for the negotiation phase. If the negotiation does not complete for 1 minute, there is a chance that a hacker is trying to consume the network resources of your server.
5. Do not allow more than 3 unsuccessful login attempts per session and terminate the session immediately after the last unsuccessful attempt.
6. Do not allow too many sessions authenticated with the same user credentials. A big number of sessions (in a majority of cases, more than 1) indicate that the credentials have probably been stolen and are now being used to attack your server.
7. Temporarily block credentials that are supposed to be stolen.
8. Control the number of active SSH sessions made to your server. Introduce emergency threshold values after which no further sessions are accepted and/or a notification is sent to the server administrator.

The above techniques are the universal ones. Other methods of controlling the load and deciding between legal and illegal connections that consider the specifics of particular environments can also be used.

In your particular case (where the client software is built and operated by you), you can also use a software name-based approach to distinguish legitimate clients from the illicit ones. Every SSH implementation starts the session with a so-called version line.

This typically looks like the following:

SSH-(version)-(software name)

e.g.,

SSH-2.0-OpenSSH4_1

SSHBlackbox provides access to the contents of the (software name) part of the version line via the components' SoftwareName, ServerSoftwareName, and ClientSoftwareName properties. The SoftwareName property can be used to adjust the name reported by the component. The others can be used to read the names provided by the remote party.

Still, it makes sense to analyze the contents of the version line even before creating the SSH components. This can be done by reading several of the first protocol bytes from the TCP connection straight after accepting it. If the software name is OK, you could go ahead with the connection and create the components.