Category Archives: Uncategorized

How blacklists work behind the scenes

Every now and then we get an email from a user who wants to know why our Blacklist tool shows them as being on a blacklist but when they use the check tool on the blacklist’s web page, it shows them as being clear or vice versa. A little bit of background on how DNSRBLs work will explain why this happens and I hope you find it helpful when trying to troubleshoot blacklist problems.

Blacklist Results

Blacklist operators generate lists of IP addresses or domain names that they would like to share with the world. DNS is a great way to publish IP addresses and hostnames in a very lightweight, fast, distributed way. The operator creates a domain zone and publishes records on their DNS server. So let’s say we create a blacklist called Example. We announce it to the world and let everybody know we are going to publish it at rbl.example.com. For every IP Address that we want to add to our list, we publish an A record in our zone. Mail servers would attempt to resolve the IP at our domain and if an A record is returned they would know that the IP in question is “on the blacklist”. Domain based lists work similarly.

Just like with all other DNS records, you do not need to always ask the DNS server that actually host the zone for an answer. In fact most DNS queries are made against nearby DNS servers. Most people first query their ISPs DNS servers. Many business networks are setup with a local DNS server for security as well as performance reasons. This way once one person gets an answer for the IP address google.com additional queries are returned very quickly without having to traverse the internet. How long these cached results are stored is determined by the time to live (TTL) settings that are configured by the owner of the zone. This means that in addition to determining who they want to put on their list, blacklist operators determine how long you should remain listed even after they remove you from the zone. They could do this for policy reasons or for performance of their DNS servers. But what it means is that every person who finds out that you are on the list will consider you “listed” until that TTL expires.

So I think you can see now how you could get a different answer from our tool than from the blacklists own check tool. Either we got a negative answer recently and are caching that and showing you as not listed when you in fact are, or we have a legitimate listing record on our server that hasn’t expired yet and we will show you listed even after you have been taken off at the source. It is important to realize that we report these cached results for the reason that this is what other email servers in the wild will see. If you get a positive result on our tool, once you request delisting you should check with the provider’s own check tool to see if you have been removed. Then you can see from our tool how long your TTL is before you will appear clean again to the email server’s of the world.

IPv6 Addresses added to MX record results

We continue to add support for IPv6 to our tools and this week we are going to start showing IPv6 addresses for Mail Exchange records that have AAAA records for their hostnames.

IPv6 in MX records

Our last blog post went over a lot of the basics of IPv6 for folks who would like some background. We are going to continue adding support for IPv6 in more of our tools over time as we strive to keep our tools as awesome as possible in the ever changing world of technology.

AAAA DNS Lookups are now available

world IPv6 launchWe have recently added the ability to perform AAAA record DNS lookups in order to resolve hostnames to IPv6 addresses. Here’s a bit of background on IPv6 and AAAA records.

DNS is the backbone of computer networking today. Every time you use a web browser or other internet connected technology that references a server by name, it uses DNS to turn that name into a numeric address. Since the early 1980′s that numeric address was an IPv4 IP address. These “IP Addresses” are 32-bit numbers that can be written as a decimal number from 0 to 4294967295. The IP for MxToolbox.com is currently written as a decimal number as 1075110789.  An easier to read “dotted quad” format is more popular, with 4 8-bit numbers from 0 to 255 separated by periods. In this notation the IP Address for MxToolbox.com is 64.20.227.133.

IPv4 has served well for over 30 years, but it has a limitation. It only has 4.29 billion addresses. In 1981 when specification was published, computers were large, shared, and not terribly common. Today many people have multiple computers and internet connected devices, each needing an address. The solution for this dilemma is IPv6, an update that brings with it an increase in address space. IPv6 allows for 340 billion, billion, billion, billion addresses, so we shouldn’t run out of room too soon.

Having so many addresses is critical, and the by far the largest benefit to IPv6, however trying to communicate such a large number is problematic. For example, as of the time of this blog post our tool reports that google.com resolves to 2607:f8b0:4000:804::1004. However that same IP Address can be written several different ways. Here’s that IP as 128 binary 1′s and 0′s - 100110000001111111100010110000010000000000000000001000000001000000000000000000000000000000000000000000000000000001000000000100. Another not very practical method is to use decimal numbers, in our case the address is written as 50552053919381933569817860797397733380. Here’s that number again with comma’s so you can get a grasp as to how large that number really is 50,552,053,919,381,933,569,817,860,797,397,733,380.

The most common way to write IPv6 is so use hexadecimal, which uses 0-9 and then a-f to represent 16 bits as a single character. Our IPv6 looks like this in hex 2607:f8b0:4000:0804:0000:0000:0000:1004. There are several methods for “compressing” this number to remove unneeded characters. You can turn any group of 0000 into just one 0, after all zero is zero. That gets you to 2607:f8b0:4000:0804:0:0:0:1004. However you can also replace any group of zeros with a single double colon so you get back to our optimal version which we return - 2607:f8b0:4000:0804::1004. You are only allowed to use the :: once per address.

IPv6 and it’s super large address pool is great, and some organizations are using it, but since it’s incompatible with IPv4 directly, there has been and will continue to be a long road as the internet transitions from the old version to the new one. Many people have IPv6 addresses and the AAAA DNS record is how those addresses are published. The path forward to implementing and converting networks to IPv6 is a much longer story and there are plenty of places to read about it online. But for now, if you want to lookup AAAA records and PTR records for IPv6, the MxToolbox tools are ready to help.

Create a Dashboard with the REST API

Viewing your monitors is fairly simple on MxToolBox’s website, but wouldn’t it be nice to have your monitor status on your website? This post shows off a sample dashboard using HTML, Javascript, and the REST API we talked about in our previous blog post and on our websiteYou can view the sample or download the files

Breaking Down the Sample – How It Works

Note that this sample is using the sample API Key, which can only query “example.com.”  You will need to use your own api key within sample.js.  Search for “add your api key here, ” use your api key, and you are running.

dashboard
The mechanics start with calling the Monitor API to get the status, then it’s mostly a matter of formatting.  This api will return each of your monitors’ status:

  {
    "MonitorUID": "e442d19c-5746-4816-b5af-65624757c297",
    "ActionString": "blacklist:example.com",
    "LastTransition": "2009-11-03T06:15:34.867",
    "LastChecked": "2013-12-02T15:23:35.78",
    "MxRep": "100",
    "Failing": [],
    "Warnings": []
  },

This javascript is composed of the following frameworks -

On DocumentLoad, we use the monitor details to determine the status of the monitor based on the problems listed in “Failing” and “Warnings.”  We then pass that jsRender to create a nicely formatted table.

Extend the Dashboard with Tags

Some customers segment their monitors by tagging them.  You can use the monitor API to query by tags by adding a ‘tag’ querystring parameter to the api.  The link below gets only the example.com monitors that have the “blacklist” tag:

http://api.mxtoolbox.com/api/v1/monitor/?authorization={your_api_key_goes_here}&tag=blacklist

One customer uses the tag filter to create dashboards for each of his regions, and another uses it to show only his blacklist monitors.

Learn more about the REST API

Blacklist Activity

Have you ever wondered which Blacklists are the most active?

At MxToolbox, we generate gigabytes of global blacklist activity data weekly from our public tool usage. This type of data is especially useful to determine when a blacklist goes does down or a dns service falls over and can relay that information to you. Like the rest of you, we use multiple monitors to let us know when an event occurs.

The most useful data we look at is the number of blacklists “Adds” (or listings) that occur.  A sharp/sudden spike or drop usually indicates a problem with a blacklist.

All Recent Blacklist Activity

Blacklist Activity

Top Activity By Blacklist

Blacklist Activity

You can see these charts on your free Dashboard and they are included in our free weekly Summary email.

As of 11/8/2013, these blacklists (below) have the most “adds” or listings by each respective blacklist operator. Each “add” or blacklisting is equivalent to one IP Address or one Domain being added to each list.

Spamhaus-ZEN

spamhaus-zen Activity

CBL

CBL Activity

Spamcop

Spamcop Activity

Barracuda

Barracuda Activity

Analysis

As of 11/8/2013, the four blacklists that recently have the most listings are Spamhaus-ZEN, CBL (Spamhaus), Spamcop, and Barracuda. On average, these blacklist have generated the highest volume of listings for the last several years, and we do not expect that to change soon. SORBS and Lashback blacklists are usually close behind. IP Addresses and domains that make up these listings can be added by these blacklists for a variety of reasons such as spam and virus/malware infections. You can run a quick Blacklist Check on your domain name or IP Address here to make sure your IP Address or Domain are free of blacklist issues.

 

Network Solutions DNS problems

We have seen a large number of alert emails go out from our Monitoring System and they appear to be caused by problems with Network Solutions DNS  servers being intermittently unavailable. No word yet as to when it might be resolved.

Network Solutions Facebook – https://www.facebook.com/networksolutions
Network Solutions Twitter – https://twitter.com/netsolcares

You can check your DNS with our tool - http://mxtoolbox.com/DNSLookup.aspx. Network solutions servers names end with .worldnic.com.

REST API Access Now Available

We recently released an often requested feature – API access to MxToolbox.  With the API, you can easily integrate MxToolBox directly in your own tools and website.

What is it?

The API is a RESTful Web Service.  Basically, you send a url to the site, and it will return the results in JSON format.  RESTFul services allow for simple testing.  For example, if you want to do a dns lookup on example.com, you go to

http://api.mxtoolbox.com/api/v1/lookup/dns/example.com/

with the following results:

    {
    "UID": "bca9f444-f28d-4d49-9368-2386d70e3034",
    "Command": "dns",
    "CommandArgument": "example.com",
    "TimeRecorded": "2013-10-11T10:40:21.7445287-05:00",
    "ReportingNameServer": "a.iana-servers.net",
    "TimeToComplete": "5101",
    "HasSubscriptions": false,
...
    ],
    "Passed": [
        {
            "ID": 305,
            "Name": "DNS Bad Glue Detected",
            "Info": "No Bad Glue Detected"
        },
    ]
...
}

You may notice that this information is almost exactly what you would get if you ran this in SuperTool:

http://mxtoolbox.com/supertool.aspx?action=dns:example.com

With the API, there is no longer a need to “scrape” the SuperTool results.

In all, the REST API has three functions/urls available

  • Lookup - as shown in the prior example, gives access to the SuperTool engine.
  • Monitor - access to the status of your current subscriptions, similar to the Monitors Tab in the Dashboard.
  • Usage - view the number of available API calls available to your subscription

The above links show example uses of each function.  In order to regulate usage, you are required to Request an API Key.  Each of the example links above include a “API Access Key” section with a button to request a key.  The api key is associated with your account.  Like all good things, we require a paid subscription to access the full power of the API.  The limits are defined in the  API Documentation.

To help with testing, we also allow all “example.com” lookups to be completed without an api key. This is why the above example works.  If, for example, you try navigating to http://api.mxtoolbox.com/api/v1/lookup/dns/mxtoolbox.com/ , you will get an “Invalid ApiKey” error because the argument is “mxtoolbox.com,” not “example.com.”  Please see the api documentation and examples for the proper way to use your ApiKey.

Use Cases

Customers have asked for this functionality for multiple reasons.  Usually it is to run a blacklist on demand from their own toolset.  The second most popular request is to get a “real time” version of their monitors.  We are excited to see what other uses customers have for the API.

The API is currently in BETA, so expect some slight changes as we determine the needs of our customer.  As always, we appreciate any feedback.

August’s New Feature – SSL Certificate Analysis

http_url

We are trying to add a new monitor or feature every month at mxtoolbox.com and for August we have added SSL certificate analysis onto our HTTPS tool and monitor.

https_details

The new version of the HTTPS tool will still go and fetch a page at whatever https:// url you configure. This will make sure that the web server is online and serving pages. Just as with our HTTP monitor, you can include an optional word (or regular expression) that must be present on the page in order to pass, so you can configure the test to confirm an additional bit of status as well.

In addition, we have added SSL Certificate Analysis. We will inspect each link of the security chain for information and errors. In addition to errors, we will also issue a warning if any certificate in the chain is due to expire in less than one month. So you can use the lookup tool to quickly check your cert, and by adding a monitor, you will receive an alert when you are due to begin the process of obtaining a new certificate with plenty of time to obtain and install the new cert.

You will receive an alert if

  • DNS Check – If we can’t resolve your domain
  • Connectivity Check – If we can’t connect on port 443
  • Keyword Check- If your keyword is missing
  • Performance Check – If your page doesn’t return within 15 seconds
  • Certificate Validity Check – If your certificate is invalid
  • Expiration Check – If your certificate is expired
  • Expiration Reminder – If your certificate expires within a month

New Feature – Notification Delay

We have just introduced a new monitoring feature called “Notification Delay,” which allows you to adjust how long a Monitor must be down before generating a problem alert.

Most of our monitors are high priority–we are notified immediately if there is a problem. However, we have a few that are less important like batch jobs and non-critical systems. Notification is only necessary if they have been down for a longer period of time (maybe 30 minutes or an hour). Another example is automatic maintenance jobs over the weekend that briefly shut down some services and notification is only necessary if something has been down longer than expected.

You can set the delay on the monitor details screen which you can get to from the Monitor Tree, your Dashboard, or your main Monitors list. For your convenience we have put a link directly to the details in all alert emails.
Nofitication-Delay Screenshot 1

The default value is to Send Immediately. You can choose values of 15 minutes, 30 minutes, 1, 2 or 4 hours of sensitivity.

Nofitication-Delay Screenshot 2

Please note, notification delay is only applicable to MxToolBox “transactional” monitors such as Mailflow, SMTP, TCP, HTTP, and HTTPS. There is not a notification delay setting on Blacklist monitors. To adjust blacklist notification frequencies, contact support to activate the Summary Alert Message (SAM) feature that generates alert reports containing all of your monitors that are in transition in one nicely formatted email message.

Notification Delay is available now to all of our paid users. If you’ve thought about upgrading your free account, now is the time!

Domain based blacklist monitoring added

Domain Blacklist

Thousand of companies from around the world use MxToolBox’s blacklist lookup and monitoring tools to keep their online reputation safe.  This service checks your IP Address against multiple blacklists to tell you if your server is blacklisted.  Today we are adding domain-based blacklist lookups to the tools.

These domain blacklists (RHBLs, or Right-handed black lists) are an increasingly popular way to attempt to cut down on spam and fraud.  They work like this.  Instead of a server’s IP being listed, an entire domain name could be listed.  For example, if your domain name is example.com, companies may look at the From: field and the body of an email message to see if they contain links to that domain name.

Keep in mind that MxToolBox does not run or endorse any of these lists, and users of our paid monitoring service can opt to “Ignore” these lists from their tool results and monitoring alerts.

All you need to do is run a blacklist check like normal on the website, but instead of entering an IP address, you just enter a domain name (example.com).  MxToolBox will detect that you have entered a domain name and run BOTH the “normal” IP based lists and the domain-based ones as well.

We have added a few new lists this week and expect to add more over the coming weeks.  Anyone can run a domain-based lookup on our web site for free, and users of our paid service can subscribe to alerts on these.

Your next blacklist summary email should reflect these new blacklists for any domain based monitors you have.

Main image courtesy of ivanpw