Our Service or Tool is Being Blocked

Why this happens and what to do about it

At SITE DISTRICT, we have a WordPress firewall that protects sites against malicious and other undesired traffic and requests.

In some cases, requests from external third-party services or tools are blocked by the firewall. This page talks about why this happens, and what to do about it.

Overview

Our Firewall

The SITE DISTRICT protects sites hosted on our platform by analyzing requests sent to our servers and blocking those that are known to be malicious, as well as requests that are similar in some way to those made by bots which often cause problems with sites if they are not stopped. You can read more about our firewall here.

If you're here reading this, then there is a good chance a tool or service that are trying to use fits that second category (similar to bad bots) for some reason.

Identifying a Block

If you suspect the request is being blocked by our firewall, there are a few ways to confirm this. They include:

• HTTP 403 Status Code - Our firewall returns an HTTP status code of 403. The title of the page returned is "Access Denied."
• Access Denied page - If the tool that is being blocked attempts to render the page that is returned by our firewall, it will appear as an "Access Denied" page with a yellow background. A robot image and some additional details are displayed.
• Access Logs -> Blocked page - If you open up the Site Details view for your site in the SiteDistrict dashboard, navigate to the Access Logs page, and select the Blocked sub-page, you can view requests blocked by our firewall.

Access Denied Page

The Access Denied page will look something like this:

Access Logs

If you know the User-Agent or the originating IP address used by the service or tool or the URL being requested, enter this in the Filter box, and see if the requests from your tool show up here.

You may use the time and date range picker to adjust the time span if the most recent 1000 requests do not go back far enough.

The Problem

Why is it happening?

The simplest explanation for why this is happening is that the tool or service is doing something that doesn't match up with our expectations at SITEDISTRICT, for what a valid request should look like.

This could happen for two reasons:

1. The requests appear similar to requests made by troublesome bots
2. There is a bug or issue with our firewall code

How can it be fixed?

This naturally means there are also two primary ways to fix the issue:

1. You change the way your service or tool behaves, or request that the vendor do so
2. We make a change or update to the SiteDistrict firewall

Either one, or both, might be required. Which one is necessary or recommended depends on several factors, which we'll discuss below.

Common Issues

There are many reasons why the requests from your service or tool might look similar to the thousands of legitimately bad requests that we block at SITEDISTRICT.

Some of the common reasons include:

• Improper Identification - external tools and services that send requests to sites hosting on SiteDistrict should identify themselves propertly using the User-Agent HTTP request header. More information about User-Agent strings can be found here.
• Bad Reputation - Services that use server or network providers that also host or proxy a large amount of "bad" traffic are more likely to be blocked if the Autonomous System has also been responsible for a large amount of malicious or suspect traffic.
• Not following WordPress / Internet standards - In some cases, requests from services may be blocked if they don't behave as expected. One example is requests to /wp-admin/admin-ajax.php. As the name implies, these URLs are meant to be used by browsers to make AJAX requests in response to user interaction, or to make background requests. External tools services typically should use the WordPess REST API instead. The /wp-admin/admin-ajax.php URL should also not be used to display HTML, such is in a popup or <iframe>.

User-Agent Issues

The most common issue we see is due to Improper Identification, which typically means a "bad" User-Agent header was sent with the request, or one was not sent at all.

The User-Agent HTTP request header is used to identify the tool, service, or browser that is making the request to the server.

The most common issues:

• Blank User-Agent - This header should be present and should not be blank.
• Invalid User-Agent - This should not be something like just Mozilla/5.0, or a random string.
• Browser Impersonation - The User-Agent string sent should not be one sent by a browser, unless it actually is that browser sending the request.
• Bad Browser-Tool Hybrid - Similar to the previous issue, the User-Agent string should not be copied from a common or outdated browser, and then just have the name of your tool inserted or tacked onto the end.
• Fake Search Engine Bot - Tools & services should not "impersonate" Google's Googlebot, Microsoft's bingbot, etc.

See this page from Mozilla for more information regarding the User-Agent string.

Bad Reputation

Another, less common issue, is that a tool or service provider will use a cloud platform that has a bad reputation.

A service provider will have a bad reputation if much of the traffic to SiteDistrict from this particular cloud or server provider is malicious or undesired. This might include brute-force attack, spam comments & form submissions, probing for files or vulnerabilites, attempts to hack sites, or large-scale and rapid crawls of sites.

Why So Strict?

The main reason that the SiteDistrict firewall is strict about having services identify themselves is because a significant amount of "bad" traffic to sites that we host fails the User-Agent checks above. It has proven to be an extremely reliable and effective signal for classifying traffic and blocking attacks.

In fact, the reason these checks are in place is because sites hosted on SiteDistrict used to suffer unblocked DDoS and DoS attacks. The rules are in place because they solve a very real problem that existed before they were added.

The scale of such attacks can be quite large, peaking at hundreds of requests per second. There is no time to react and manually block such attacks, nor is such a strategy desired. If such attacks are not mitigated, both the site under attack as well as other sites on the same server can suffer or become unavailable.

To be "less strict" would reduce the security of our sites, and open the door for such unblocked attacks - as well as the associated performance issues - to resume.

How do I fix it?

If a tool or service that you are using is being blocked by our firewall, the best solution depends on factors such as:

• Why it was blocked in the first place
• The urgency of the issue
• The responsiveness and competency of the tool's author or the service provider

Fixing the User-Agent

If your service or tool is making requests to a site hosted on SiteDistrict, you should ensure that the User-Agent HTTP Request header that is sent properly identifies your service, and does not include or imitate strings used by browsers, unless actually based on those browsers.

The best and safest thing to do is to send something like this for the User-Agent:

MyService/1.0 (+https://example.com/my-service)

where MyService is the name of your tool or service, 1.0 represents the version of the service or tool, and https://example.com/my-service is a URL of a page that describes your tool or service and what kind of requests it makes.

If you do not have a page describing your service, a link to the main page of your website is a good alternative.

For more information on User-Agent strings, see the Mozilla Developer Network page.

If you are a SiteDistrict customer, and your have a tool or service that is not identifying itself properly, you may send them a link to this page.

Fixing AJAX usage

If the plugin, tool or service you are using is making external or improper requests to /wp-admin/admin-ajax.php, or is using this script to load or display HTML in a popup or <iframe>, it should be changed to follow WordPress and Internet conventions & best practices.

Switch Services / Tools

If the tool or service you are using is unable or unwilling to change their User-Agent string, one solution is to switch services.

There may be alternative tools or services available that can meet your needs, and which already identify themselves properly.

If you are paying or have paid for the tool or service, you may consider asking for a refund.

SiteDistrict Adjustment

In rare cases, the SiteDistrict firewall maybe incorrectly classify a service even if it is valid and is properly identifying itself via the User-Agent HTTP request header.

If you suspect that your tool or service is "doing everything right", but requests are still be blocked, and you can see them on the Blocked page of our Access Logs dashboard, then please do the following:

1. Filter the Blocked requests down to show just the requests from your service, which likely means filtering based on the name of the service found in the User-Agent string
2. Take a screenshot of the Access Logs page
3. Send both the URL of the filtering Access Logs page, and your screenshot, to support@sitedistrict.com

SiteDistrict Exception

In some cases, we may be able to add an exception to the SiteDistrict firewall. Please see the next section on Procedure & Policy for details on that.

Resolution Process

If a tool or service is being blocked by the SiteDistrict firewall, please follow the steps below:

1. Review reasons why a service or tool might be blocked above. See if you can determine which might apply.
2. If we told you to review this page, then continue to next steps.
3. Contact the tool or service provider. Have them review this page.
• If we told you specifically what we think they should do, pass that information along.
4. If they are unable to make the recommended adjustments, we will want to see
• Evidence that they understand what is going on
• A detailed explanation and justification as to why they are unable or unwilling to fix the problem on their side
5. Send the information from step 4 to us, and ask if an exception might be made.
• Please forward emails directly, or include screenshots of any chats with their support team.

Notes

We don't communicate directly with tool or service providers. As the user of the tool or service, you or your client are their customer, and thus you or your client should contact them for support.

If the provider could not create a fix, consider alternatives, including switching tools, and/or asking for a refund.

Timeline

Once the steps above have been completed, and we determine that an exception is reasonable, we will implement an exception in about the same amount of time, as from when you either first contacted us or the provider, until the completion of step 4.

In other words, the faster the tool or service provider addresses your support request, and states they cannot or will not provide a solution on their side, the faster we will work to provide an exception on our side.

Bad Excuses

As we said above, if your tool or service is being blocked by the SiteDistrict firewall, there is simply not a match in terms of expectations between the tool or service, and SiteDistrict. This usually means that one or both need to make some type of change.

If you have been instructed to view this page, it is most likely because we believe that the tool or service provider should make a change.

There are several reasons which are not considered "good reasons" for SiteDistrict to make a change. They include:

• Slow response time - Just because our support is faster & more responsive, does not mean SiteDistrict should be the one to make a change. Think about demanding more from your other providers.
• It works elsewhere - At SiteDistrict, we pride ourselves on performance, security, and reliability. Most sites on SiteDistrict do not run any WordPress security or firewall plugins, nor are they recommended. While your service might work with a different hosting provider, sites on that host may also be far more vulnerable to attacks, spam, or performance degradation. In fact, if you have had unexplained performance issues at other hosts, you might consider that some of those could have due to unmitigated attacks.
• They're a big company - Maybe the other company is "big", and you are a small customer, or just one of many customers. Maybe the other company is "bigger" than SiteDistrict. But any company can and must adapt its services for changing needs. Also consider that the vast majority of services - including large services, such as Zapier, PayPal, Google, and others - already "do it right", and are not blocked by the SiteDistrict firewall.

Doing the Right Thing

When it comes to tools and services not behaving properly, we are also reminded of this children's book: What if everyone did that?

Fixing "bad" behavior and following certain best practices makes the Internet a better place for everyone. Other servers that process requests from these tools & services will often be able to more easily identify & debug the requests as well.

Scalability

On a similar theme, one reason that we avoid adding exceptions for different services is because it is not scalable.

The rules employed by our firewall are carefully crafted to be simple and efficient. Adding exceptions makes it harder to maintain, and can reduce security while increasing the risk of other issues.

What might seem like a simple request to add an exception may have implications and repercussions that you are not aware of, some of which can cause additional unanticipated issues for you and other SiteDistrict customers.

Frequently Asked Questions (FAQ)

Question: How do I whitelist my service?

Answer: To "whitelist" something is jumping past the problem definition to a solution. The solution depends on a complete understanding of the problem, which includes a full understandin of how our firewall works. Therefore, it's our job. Instead, you should provide us with all the details necessary for us to diagnose & understand the problem on our end, so we can determine the solution, which may or may not be to "whitelist" something. Any updates that we make must also be made such that they minimize security risks and so that they do not create maintenance problems in the future.

Question: Why can't your firewall be smarter?

Answer: We are always making the SiteDistrict firewall smarter. In some cases, it makes sense to make an adjustment on our end but it depends on the issue. In some cases, being "smarter" means keeping things simple and requiring that others follow Internet best practices.

Summary

We are constantly working to create a better, safer Internet for all. Our firewall is a critical part of our infrastructure and protects sites by blocking requests and attacks that would otherwise cause performance issues or downtime. Ensuring that external tools and services behave properly allows us to continue to provide efficient and high-quality support.