Tyler's Site

Abstract

The Domain Name System (DNS) serves as one of the foundational protocols and services that allow the Internet to operate as it does today. DNS serves as the translation between conventional domain names (such as foxide.xyz) and the corresponding IP address (172.245.181.191). This has the obvious effect of making servers on the Internet easier to find as names tend to be easier to remember than random numbers; however, it has the secondary effect of allowing for things like virtual hosting which contributed to the boom of websites on the early Internet. To think of the modern Internet without DNS is nearly unthinkable, to the point where many untrained people would confuse DNS not working with the Internet not working. This post is going to teach about some basic DNS concepts, troubleshooting, and setting up a DNS server with OpenBSD and Unboound.

How DNS Works

As stated above DNS is a foundational protocol that runs the modern day Internet, and the services that provide name resolution for the Internet have a huge amount of power over what their down stream users can and cannot see. Let’s take a look at how DNS works to better understand why this is.

The life of a DNS query is conceptually quite simple; it is just asking that query to a series of services until an answer is received.

For a quick oversimplification of the process:

  1. Local machine attempts to resolve by looking at the hosts file and any cache on the machine.
  2. The machine will ask the nearest DNS server.
  3. The nearest DNS server will send the query upstream.
  4. Repeat until answer is received.

In practice it operates as follows: First some software asks the operating system to resolve a name to an IP address (e.g. foxide.xyz). The operating system first checks the ‘hosts’ file (/etc/hosts on Unix systems and C:\Windows\System32\drivers\etc\hosts on Windows) to see if the system’s administrator has declared IP address that should correspond to a particular domain name. If the name is declared in the hosts file, then the query will use the corresponding IP address as the destination for the domain. An example of the hosts file might be:

127.0.0.1       localhost
::1         localhost
0.0.0.0         foxide.xyz

In the example above, we can see that localhost (more often referred to as the loopback address) was defined as 127.0.0.1 and ::1. This corresponds to the reserved loopback addresses for IPv4 and IPv6 respectively; reserved in this case means that nothing on the Internet will have this address, so it can safely be assigned for another purpose. In this case, it is to test that networking is working on the local machine in some capacity.

If the query cannot be resolved via the hosts file, then the operating system and/or the web browser will check the cache to see if it is able to resolve the query. It is very important to remember that modern web browsers are very complex and retain their own DNS cache. The purpose behind this is increasing performance when cache is needed, however, it can also lead to odd behavior such as a DNS record has changed, but the browser is only resolving to the previous address even though the operating system resolves to the new address.

Once the local machine has tried and failed to resolve the query, the second step in DNS resolution occurs. The query will then go and ask the nearest DNS server to the machine; this will often be the DNS server set by the DHCP server on the network, but could also be set manually. In many home networks, the DHCP server will be the ISPs router that will automatically set their preferred public DNS server for each client. In most enterprise and home lab settings though, there will be a local DNS server to resolve local domain names, then send queries upstream for external services. The local services will usually have a Top Level Domain (TLD) corresponding to something such as ‘.home’ or ‘.local’. It is worth noting that as of 2018 RFC 8375 declared ‘.home.arpa’ to be the de facto TLD intended for local network use with a unicast DNS server, and RFC 6762 declared ‘.local’ to be the de facto TLD for multicast DNS (mDNS) use. An example of this use would be something like Bonjour 50p). For more information, a list of “special use” domain names can be found here.

If the local DNS server does not know where the address is (either by deceleration or by caching), then it will ask the upstream DNS server. This will generally be a public DNS server and will be configured by the network administrator, many people use Google’s DNS (8.8.8.8), however, many other options exist:

These DNS services are what provides the Internet experience that most users are used to, and without it the Internet might as well be gone completely.

By this point, the query has most likely been answered, as the public DNS servers will not only have a record of many different domain locations, but they will also keep a cache of commonly used domain name locations. However, there are cases in which a domain may not be known to these public DNS servers. So, what happens from here?

Well, basically the same thing that has happened before. The public DNS server asks another server. However, this time, it will work in a bit of a different direction. It will start by trying to find the DNS server that handles the TLD of the domain (for example, the TLD for foxide.xyz would be .xyz) to ask where the domain should point to. If the DNS server cannot find the server that handles the TLD, it will then reach out to one of the 13 Root Servers. These are the authoritative DNS servers that ultimately answer for all of the TLDs on the Internet; because of their special place in DNS, their TLD is technically just a period. It can be thought of an implied trailing period on each DNS query that signifies the root server. The root server will then tell the DNS server making the query (remember the upstream public DNS server for our machine) where to find the authoritative name server for the TLD it is looking for. From there, the name server for the TLD will give up the IP address that corresponds with the query and that will get sent back up to the client that originally requested it. The amazing part of all of this, is it happens over the course of milliseconds all across the world.

The next part of understanding DNS is understanding the various record types that DNS has to offer!

DNS Record Types

While a DNS query may ask where a host’s location is, the record type will tell the system what the location is intended for. It can almost be though of like the zoning for a building; where the IP address would be the building’s address and the record type would be the building’s zoning. A brief overview of the most common record types are as follows:

The above list is far from comprehensive, but should give a general idea on the most basic and common record types. For a more complete list Wikipedia has you covered with lots of information on record types. And if that is not enough, you can also look at the Internet Assigned Number Authority (IANA)’s website.

Now that we have a basic understanding of DNS and roughly how it works, let’s get a server setup so we can resolve our own queries.

Lab Setup

For this project, I am going to be using OpenBSD because it is secure, stable, and simple. I am using a virtual machine for testing this, however, real hardware should not be much different; the VM is using UEFI to emulate a machine made within the past decade though OpenBSD is only mildly concerned with that fact.

Following the installer, and selecting the defaults except for using a GPT partition scheme and actually creating a user. Once the system is installed and rebooted, I enabled doas using the following command:

# Must be run as root, using su -
echo "permit persist :wheel" > /etc/doas.conf

and then installing some optional quality of life packages:

# Must be run as root, but doas will work now
pkg_add fish-4.0.2p0 neovim screen-5.0.0
# Sym Link nvim to vim for muscle memory
ln -s /usr/local/bin/nvim /usr/local/bin/vim
# Change shell from ksh to fish
# This command will be run as your unprivileged user
chsh -s /usr/local/bin/fish

These packages are quality of life packages that are completely optional and will not really affect anything in this blog post other than the above section. The focus of this post will be on the unbound DNS server included in OpenBSD base. Below is the default configuration file for OpenBSD’s unbound, located at /var/unbound/etc/unbound.conf:

$OpenBSD: unbound.conf,v 1.21 2020/10/28 11:35:58 sthen Exp $

server: interface: 127.0.0.1 #interface: 127.0.0.1@5353 # listen on alternative port interface: ::1 #do-ip6: no

# override the default "any" address to send queries; if multiple
# addresses are available, they are used randomly to counter spoofing
#outgoing-interface: 192.0.2.1
#outgoing-interface: 2001:db8::53

access-control: 0.0.0.0/0 refuse
access-control: 127.0.0.0/8 allow
access-control: ::0/0 refuse
access-control: ::1 allow

hide-identity: yes
hide-version: yes

# Perform DNSSEC validation.
#
auto-trust-anchor-file: "/var/unbound/db/root.key"
val-log-level: 2

# Synthesize NXDOMAINs from DNSSEC NSEC chains.
# https://tools.ietf.org/html/rfc8198
#
aggressive-nsec: yes

# Serve zones authoritatively from Unbound to resolver clients.
# Not for external service.
#
#local-zone: "local." static
#local-data: "mycomputer.local. IN A 192.0.2.51"
#local-zone: "2.0.192.in-addr.arpa." static
#local-data-ptr: "192.0.2.51 mycomputer.local"

# Use TCP for "forward-zone" requests. Useful if you are making
# DNS requests over an SSH port forwarding.
#
#tcp-upstream: yes

# CA Certificates used for forward-tls-upstream (RFC7858) hostname
# verification.  Since it's outside the chroot it is only loaded at
# startup and thus cannot be changed via a reload.
#tls-cert-bundle: "/etc/ssl/cert.pem"

remote-control: control-enable: yes control-interface: /var/run/unbound.sock

Use an upstream forwarder (recursive resolver) for some or all zones.

#forward-zone: # name: “.” # use for ALL queries # forward-addr: 192.0.2.53 # example address only # forward-first: yes # try direct if forwarder fails

Use an upstream DNS-over-TLS forwarder and do not fall back to cleartext

if that fails.

#forward-zone: # name: “.” # forward-tls-upstream: yes # use DNS-over-TLS forwarder # forward-first: no # do NOT send direct # # the hostname after “#” is not a comment, it is used for TLS checks: # forward-addr: 192.0.2.53@853#resolver.hostname.example


The nice thing about the default configuration file is how well commented it is;
while you may not understand the lines completely, you can understand the basics of what they are doing.
Then the `man` pages are also always available as a reference for the specifics of a configuration option.
For a more practical example, here is the config that I have setup for this lab with the comments removed for readability:

``` config
server:
    interface: 192.168.122.40

    outgoing-interface: 192.168.122.40

    access-control: 0.0.0.0/0 refuse
    access-control: 127.0.0.0/8 allow
    access-control: 192.168.122.0/24 allow
    access-control: ::0/0 refuse
    access-control: ::1 allow

    hide-identity: yes
    hide-version: yes

    auto-trust-anchor-file: "/var/unbound/db/root.key"
    val-log-level: 2

    aggressive-nsec: yes

    local-zone: "home.arpa." static
    local-zone: "122.168.192.in-addr.arpa." static

    local-data: "puffy.home.arpa. IN A 192.168.122.40"
    local-data: "fedora.home.arpa. IN A 192.168.122.126"

    local-data-ptr: "192.168.122.40 puffy.home.arpa"
    local-data-ptr: "192.168.122.126 fedora.home.arpa"

remote-control:
    control-enable: yes
    control-interface: /var/run/unbound.sock

forward-zone:
    name: "."               # Use for all queries
    forward-addr: 9.9.9.9           # For upstream queries
    forward-first: yes          # try direct if forwarder fails

I am not going to re-hash each line of the above config as the default config already does a pretty good job of explaining what everything is. Rather, I am going to just explain what the config file is setting up in general, then I will leave the cross referencing to the end user as a learning exercise.

The above config sets the IP address 192.168.122.40 as the incoming (listening) address as well as the address is sends queries back out on. It also allows traffic from the following networks:

Then will deny incoming queries from all other networks.

It will also create a local TLD (home.arpa) for local services to use on the 192.168.122.0/24 network. Currently, there are two A records in the config with reverse DNS records:

Finally, the last major part is setting the forwarding zone. This will control what happens to queries that the DNS server cannot resolve by itself and needs to ask for upstream help. It will set quad 9 (9.9.9.9) as the upstream DNS server.

There is quite a bit more that unbound can do and this is an extremely basic config, however, most people do not need much more than A records for local use. If you do, there is always the OpenBSD unbound.conf man page as a shining resource, as well as many other online resources.

Troubleshooting DNS

Because of how integral DNS is to the use of the Internet for the vast majority of people, DNS troubleshooting is often taught at the most fundamental levels of IT training. Many of the tools and techniques that would be used to troubleshoot DNS issues are tools and techniques that many people that have done troubleshooting for network problems have already done. In the event that someone is reading this and is not already familiar with these tools and techniques (did you find the rabbit?) I will go over the basics in this section. I will also try to cover some slightly more advanced stuff for the people already in the know though. The easiest way to explain is going to be using some common scenarios of problems and how one might go about dealing with them, so let’s do that.

Before actually getting into the exercises, it is worth noting that I am going to actually focus mostly on Windows tools for troubleshooting these issues. The reason for that is, practically speaking, in the vast majority of cases in the operations side of IT, you will be dealing with Windows. When a user calls because a computer isn’t doing the thing, it is most likely going to be a Windows machine. So, to work with the tools that are most likely to be seen in the field, I am choosing Windows for the troubleshooting parts of the exercise.

Scenario 1: “Can’t get to the Internet”

The first scenario is a common occurrence in the field, as well as a common interview question. A user calls and “cannot get to the Internet”. When this call comes in, you have no information on what the problem actually is. If you don’t believe me, then you should read through r/talesfromtechsupport and question who you should believe. The correct answer is no one, no even yourself; the best you can do is gather information and go through the common things.

  1. What does “can’t get to the Internet” actually mean? I have had users say something similar, but the issue was one website and was not the Internet as a whole. Make sure you understand what is broken and what the problem actually is before doing anything. Troubleshooting something the user didn’t care about is not only not helpful to anyone, but it is wasting the most valuable resource in existence…time.
  2. Assuming that the user actually cannot get to the Internet, we can now start doing actual troubleshooting; start with the basics. Is there a network cable plugged into the machine, or are they connected to a wireless network. If neither one of these is true, it will be fairly difficult to connect to the Internet.
  3. If a network connection should at least theoretically exist, then it is time to start actually asking the computer what might be wrong. Open a command prompt (does not have to be administrative command prompt) and start running some commands:
:: Remember, this is Windows command prompt, where '::' is a comment

:: First, ping an easy to remember public IP address. Something like 1.1.1.1, 8.8.8.8, or 9.9.9.9 work fine.
ping 1.1.1.1
Pinging 1.1.1.1 with 32 bytes of data:
Reply from 1.1.1.1: bytes=32 time=18ms TTL=56
Reply from 1.1.1.1: bytes=32 time=17ms TTL=56
Reply from 1.1.1.1: bytes=32 time=15ms TTL=56
Reply from 1.1.1.1: bytes=32 time=20ms TTL=56

Ping statistics for 1.1.1.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 15ms, Maximum = 20ms, Average = 17ms

:: Sending a ping to a domain name
ping example.com
Ping request could not find host example.com. Please check the name and try again.

From the above commands, it appears that the “Internet” is working just fine, but something appears to be wrong with resolving domain names. The next step would be using a tool like nslookup to see what DNS is doing with that query:

nslookup example.com
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.122.40

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.

From the above command we can see that it appears that the DNS server was unreachable, this can be confirmed with a simple ping to the DNS server’s IP.

ping 192.168.122.40
Pinging 192.168.122.40 with 32 bytes of data:
Reply from 192.168.122.220: Destination host unreachable.
Reply from 192.168.122.220: Destination host unreachable.
Reply from 192.168.122.220: Destination host unreachable.
Reply from 192.168.122.220: Destination host unreachable.

Ping statistics for 192.168.122.40:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Not only does that confirm that DNS is the culprit of the issue, but also gives us something else to look for. Why can this client not reach the DNS server? In this case it is because I purposely turned the DNS server off for the lab, but in the real world it will likely take a bit more investigation.

Scenario 2: Unable to resolve local services

Another fairly common problem is end users are not able to get to local services. These service might be something like an Intranet web app that the end user cannot access with the domain name that is pointed to it. While this issue sounds more complicated, it is actually much easier in ways. Step one is still the same, confirm what the problem is specifically. Step two, is just make sure the network is working elsewhere; can the user get to an external site of some sort (e.g. example.com or foxide.xyz). Assuming yes, then it is probably something happening locally. Making another assumption that this service is mission critical to a large group of people in the company, it is probably running otherwise you would have a lot more calls about the service. The first step would be to just send a ping to the service’s name to confirm that it does in fact fail to connect. We could also send a ping to the service’s IP address if it is something that we have memorized to confirm that the client machine can in fact get to the service. If the client can successfully ping the service’s IP, than it is a DNS issue, which means that the nslookup command can tell us what might be going on. This scenario would happen if the client’s DNS server somehow changed from the ‘local’ one that can resolve the local service to a public DNS server such as 8.8.8.8. That would allow external sites and services to work without a problem, but anything local would not work at all.

Closing Thoughts

DNS is a highly important part of the Internet, and as such anyone working in the IT field should have a grasp on the basics of DNS and how it works, bonus points if you can setup a DNS server. The more an individual understands DNS, the more they can break in to the wonderful world of doing things on the Internet, and hopefully even doing them correctly. Hosting services and having others be able to see the thing that you are doing is nearly impossible without some foundational knowledge of how to set up the address for the name of the thing. In general, it can be thought of as trying to send mail to somewhere without a street address; it’s not impossible, but you probably don’t want to do it that way.