Abstract

The dream of Artificial Intelligence is one that has existed for over a century; a machine that can learn and understand concepts in a similar way to a human building skills like problem solving, decision making, and general intuition. While we are closer than ever to that dream of a man-made intelligence that can operate on those levels, we are still years (at least) away from doing that. However, we are generally still living with and paying the investment to create newer and better AI systems whether we like it or not. This article is going to go over what the costs of AI are, and how we as a society can use AI in a good way to offset some of the massive cost that we have, and are currently still paying. So, what is AI anyway?

What is AI?

The definition of AI is rather muddled and the current landscape of the tech industry trying to integrate AI into everything is certainly not helping. Before we really begin talking about ‘AI’, we need to understand what it is that we are actually talking about. Wikipedia defines Artificial Intelligence as follows:

Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making.

This definition is as good of a starting point as any. It outlines the dream of what AI might become one day, but as many might catch, AI is not currently capable of any of the examples of things listed in the definition on its own. Rather it is able to take data and find patterns that emulate things like reasoning, problem solving, and perception in a manner that a human might express the same ideas. This is because the popular AI of today tend to be Large Language Models (LLMs). Great…so, what is an LLM? Again, referencing Wikipedia:

A large language model (LLM) is a language model trained with self-supervised machine learning on a vast amount of text, designed for natural language processing tasks, especially language generation. The largest and most capable LLMs are generative pre-trained transformers (GPTs) and provide the core capabilities of modern chatbots. LLMs can be fine-tuned for specific tasks or guided by prompt engineering. These models acquire predictive power regarding syntax, semantics, and ontologies inherent in human language corpora, but they also inherit inaccuracies and biases present in the data they are trained on.

This definition is a bit more complicated, but essentially says that LLMs are designed for understanding and creating text to emulate human writing. They can also be tuned to perform better by guiding the LLM through prompt engineering to tailor results to a specific task. The last key point is that they inherit biases and inaccuracies from the training data that was used.

The Cost of AI

During the past few years that AI has been rising the cost of those AI systems have grown, and not just monetarily. There is a massive amount of time, energy, and raw materials that are being used for AI and AI research, and the demand is still increasing under the guise of creating better AI. These costs are ones ordered by a small group of people, but everyone gets to pay. Below are three examples of non-monetary costs that society gets to pay to receive the “benefits” of modern and future AI systems.

Computational Cost

This one sounds a bit silly, of course AI uses computation, but does that cost anything? Yes, it has the obvious cost of the computational power required to actually create and run the AI in data centers, but also for the added resources required for serving the websites being scraped by AI systems and for consumer devices trying to use online resources. This seems like a silly cost to consider, what is the cost of visiting or serving a resource online? Well, that depends on what the resource is doing, how much processing needs to be done, and where that processing is occurring. On small static HTML websites, resources load quickly and they are not difficult to serve as it is just throwing text files between machines, However, something like a WordPress site is much more complex that requires much more in the way of processing and loading resources. This takes time, both for the clients visiting the website as well as for the server trying to serve it. AI scrapers have become so prevalent that it is actually starting to cause major problems for websites even as large as Wikipedia. As much fun as it would be for me to write down an explanation for what is occurring with this episode of 2.5 Admins will likely explain it better than I would, especially in written format. The AI scrapers are so egregious that they are even negatively impacting well established and large-scale websites like Wikipedia.

Environmental Cost

Another major problem with AI is the environmental cost of creating, training, and running the models; the US department of energy has released ]]> https://foxide.xyz/articles/AI.html https://foxide.xyz/articles/AI.html Sun, 04 Jan 2026 00:00:00 -0500

Abstract

Recently I have been working on DPorts in DragonFlyBSD; during that process I have really come to appreciate many of the command-line tools available to me during that process. In that spirit, I thought I might do a write up on some command-line tools that people newer to Linux/Unix systems may or may not be aware of, as well as sharing some tools that make life easier that even experienced Linux/Unix users may or may not be aware of.

Base System Tools

The following are command-line tools that are available in most base systems on Linux, BSD, and Unix systems. Learning these tools in one place will almost certainly tansfer (though not 100% due to flag differences) between systems.

• Package manager for OS: Most modern Unix-like operating systems have a command-line package manager. They might be more difficult to use than the graphical package manager, they aren’t usually too difficult and often are much more powerful.

• vi/nvi: The classic editor that is always there. Yes, it is a bit difficult to get used too, but once you do it will never let you down. When nano isn’t available vi or nvi will be as it is an integeral part of many base systems.

• which: which is a simple tool, however, it makes it very easy to find where executable files (either binaries or scripts) are located within the user’s $PATH.

• find: The find command is another classic command that will assist in finding files on your computer. it can search with or without case sensitivity and even allows for some REGEX.

• grep:

• df and du: While these tools are separate, they go together very nicely. df will show the disk usage on the various drives on the system, du will show how much disk space individual files or entire directories are taking up on the system. It is worth noting that df may be inaccurate with file systems like ZFS because of the way that ZFS handles free space. If ZFS and df disagree with the amount of space available on the system, ZFS is the one that should be trusted as it knows much more about disk usage and what the file system is doing.

• scp: Secure File Copy, or scp is the command that allows users to send one or multiple files from one machine to another via ssh.

• make: The `make’ tool is very commonly used for programming and compiling code. It is not something that everyone will use, however, it can do quite a bit more than just compiling code. The relevant excerpt from the GNU make man page:

  In fact, make is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change.

Third Party Tools

The follow tools are ones that are worth mentioning as they are generally useful, and some are tools that newer user may not be aware of.

• htop(plenty of alteratives as well)
• Good editor: In the world of modern Unix and Unix-like operating systems, this is very similar to telling most people how to apply thermal paste for CPU coolers. The two main camps are vim and/or neovim versus emacs, and while I could make an argument for either, it really doesn’t matter. Both are extremely powerful and far more capable editors than things like notepad++; I recently got one of my co-workers using vim and it has improved his workflow immensely. He said things that used to take him hours now takes him minutes. I emaacs is equally powerful, and can even emulate vim keybindings with evil mode (what I generally prefer using on Spacemacs).
• lf: A terminal file manager written in go. Truthfully, I find most TUI file mangers faster and more efficient than using cd/ls to get around in directories, but lf is my favorite that I have used up to this point. It is fast, feature rich, and very customizable. The two alternatives that I have found that are also good are nnn and ranger.
• rsync: This tool is very similar to scp, however, rather than transferring the whole file the same way that scp does, it has a delta-transfer algorithm that reduces the amount of data sent on the wire. It can do this to move files around on one machine, or it can send files to a remote daemon (yes, rsync must be installed on both ends for it to work between machines).
• pandoc: pandoc is a tool that is written to convert from one markup format to another. I find it particularly useful for converting markup into a PDF or HTML. However, it has support for other formats like: docx, rtf, and txt. It makes taking good quality notes very easy.
• LaTeX: LaTeX is a huge typesetting program originally that is often used in academia for its formatting capabilities. It is difficult to summarize its capabilities while also doing the program’s power justice. If interested, here is a source that I often reference when I write a LaTeX document.
• git: There are few people that have not at least heard of Git. However, some people may not think that it worth taking the time to learn as they are not developers. While Git is most widely known for software version control, it works very well with any plain text file, including config files, making it easy to back up and version control config files for various parts of a system.
• yt-dlp: This is a feature rich command-line audio/video downloader that works on thousands of sites. It is a fork of youtube-dl based around the interactive youtube-dlc.
• iperf3

Abstract

Currently there are a lot of great technologies to sandbox and separate applications across powerful machines. However at times it can be difficult to choose the best way to go about doing that. In general, there are two paths to choose from: containers or virtual machines (VMs). Both paths have different benefits, concerns, and technologies that accompany them.

Technical Differences

The main technical difference between containers and virtual machines are that virtual machines are ‘real’ computers in a sense. They have their own hardware, operating system, file system, and kernel; it is just virtual. A container, on the other hand, does not have its own resources. Rather, it shares a kernel with the host, and is granted resources via the host’s kernel. This is slightly different than a VM platform allocating resources for a VM, because the VM will manage its allocated resources by itself while a container will not.

Virtual Machine overview

The concept of a VM is usually pretty simple to explain for those that do not already know what it is. It is a computer running entirely in software; it has a virtual processor with virtual RAM and virtual storage. All of these resources will eventually lead back to the hardware of course, but These resources are segregated from the host computer similar to how having multiple computers on the same network would be segregated from each other. This method of providing a sandbox for applications is quite secure, however, it is resource heave as you are running a new instance of an operating system with each VM. Additionally some software may take a performance hit or not work properly when run inside of a VM, depending on hardware and virtualization platform. The main benefits of running VMs is that there is much more separation between the host and guest OSs as well as each VM can be managed as easily as any other machine once set up.

Container overview

One of the major benefits of containers is the density. One gigabyte of RAM is enough to run a couple dozen containers or more. Running one dozen VMs on one gigabyte of RAM is impossible, let along a few dozen. The difference is that the segregation between the host and the container is not as good. It is much easier for an attacker to break out of a container than a VM because containers share a kernel with the host. Containerizing applications is perfectly sufficient for workstations and is even how most Android apps run. Containers can also make packages run more consistently on various systems, evidence of this is systems like Docker or Flatpak that can make applications run flawlessly regardless of which Linux distro the host is running. The issue with these systems are that the container images do not always contain packages that are up to date to assist the software’s comparability. This may be acceptable to some people on their workstation, however, this is almost never acceptable for a server, especially one running in a production environment. Other container systems, such as Linux Containers (LXC/LXD) do not necessarily have this particular issue as they function much more like lightweight virtual machines.

When to use which?

Obviously putting everything inside of separate VMs is the more secure way to operate, however, the resources to do that can quickly add up and with machine(s) will quickly be unable to operate. Additionally, some applications are recommended to be installed as containers of one type or another. Unfortunately, this is a decision that will ultimately have to be made on a case-by-case basis. If the application or service needs extra sandboxing, or will be publicly accessible then it should likely go in a VM. If an application is just operating internal services it will likely be alright inside of a container of some sort.

Compromising

A good compromise to this thought process is having virtual machines for containers that will contain similar types of applications. For example, one virtual machine might contain various web applications, while another VM will run various storage containers. These VMs can be categorized in a way that makes sense with the services that need to be run. This thought process is the way that QubesOS operates and becomes a “reasonably secure operating system”. It will use categorized of VMs that have various uses to sandbox what type of work is being done, rather than sandboxing each running application.

Abstract

I have been watching some art restoration videos on YouTube recently, specifically Baumgartner Restoration, and it has made me think about how I work. Specifically some of the “best practices” that I try to employ for my job, and if there are any lessons that I can learn from other trades and crafts that could help with my trade and craft. While I am attempting to generalize some of these concepts; I do work in tech, and many of these observations will be centered around that thought process.

Make a plan

Having a plan in mind is always a good place to start with anything; the plan doesn’t have to be extremely specific or complicated, but it is good to get into the habit of at least outlining a plan in your head before starting something. Building this habit will help when taking on more complex tasks or projects as it will make you take a few minutes to think about what needs to be done and in what order.

Be flexible

While the planning stage of a project is very important, it also tends to not last until the end of the project. Plans for complex projects and tasks tend not to survive those projects and/or tasks. Flexibility is a skill that is just as important as planning is, because flexibility means that you can survive when the plan falls apart, and they will fall apart.

Work on destructively

Working “non-destructively” is a term that I learned in college referring to photo editing. It’s a fairly self-explanatory term and means do not make changes that damages the original. When working in photo editing software, it is common to make a copy layer of the original photo and edit the copied layer while hiding the original. It’s done that way so that if mistakes are made, the original photo is not lost and the project has to be completely restarted, or canceled altogether.

Most IT professionals likely follow a similar principle, but possibly by a different name. Take a snapshot before doing something that could break the system, or backup the original config before changing the running config on a switch or router. The idea isn’t a new one, and not even that novel, but is one that I find extremely helpful to keep in mind.

Standardization and categorization

Categorizing problems and standardizing solutions to those types of problems are useful because it speeds up the planning process. You will not have to actively think about a plan as much if there is already a standardized way to handle a particular situation. Sometimes this isn’t possible as the issue is unique or has not be encountered before, but many issues can be categorized.

Having clear naming conventions is also important, as it helps the person doing the work (you at a later date, or someone else) find what they are looking for. Having random, non-descriptive names makes it very difficult for people to know what the thing does or is for. Even if it makes sense to you right now, think about what you come back in six months and have forgotten everything you are doing right now, will it make sense then?

Abstract

Cryptography is a field that sits between multiple fields of study, most obviously mathematics and computer science. Historically, cryptography was largely used for purposes of military or espionage operations, however, in modern times much of the ways that we interact with the digital world has a fundamental reliance on strong cryptography. This cryptography allows for anything from private and secure communication between two or more people, as well as authenticating communications between entities.

This blog post is intended to be an extremely simplified view into the very, very basic concepts of cryptography, and is being written by someone who is not an expert in the field. That being said, I will do my best to give information that is accurate and to cite sources where necessary, but ultimately there is a decent chance that there will be something in this post that is not 100% accurate. So, if you are interested in this field, please go and do your own research. In the resources section of this post, there is an Introduction to Cryptography playlist given to university students that will be much more reliable and accurate than this. If there is one lesson to be learned from this, or any other resource on Cryptography for that matter, DO NOT MAKE YOUR OWN CRPTO! New crypto algorithms are extremely difficult to make securely, however, are extremely easy to create insecure algorithms while thinking it is a secure algorithm. When a new algorithm is proposed, it is done in such a way that asks people to attack and try to break it; they are proposed this way because the only way to ensure something is secure, is for people to continuously fail to break it. When this process is not followed, the security is not proven, then serious security vulnerabilities are found after the algorithm is in production, such as with the case of every version of SSL. So, I want to stress again, DO NOT MAKE YOUR OWN CRPTO, the chances that you get it wrong are extremely high, and that is even more true if you do not understand the field and are not doing the mathematical calculations for the algorithms in use currently. Leave the optimiations and tweaking of these algorithms to the experts in the field.

What is Cryptography?

In short, cryptography is the practice of securily communicating over unsecure channels; the general method of doing this is by algorythmically scrambling the message in such a way that it become unreadable without the key. This is a practice that has existed for thousands of years and has been used in a variety of settings. The algorithms that handle the scrabling of the original message are called ciphers and will take clear text (sometimes also called “Plain Text”) to turn it into encrypted text. One of the most basic and well known examples of this is the simple Ceaser cipher. Essentially the method of “encrypting” the message is to shift the letters by some number. For example, take the message “The less you say the more you hear” and shift the letters by five so the ‘T’ becomes a ‘Y’, the ‘h’ becomes an ‘m’ and so on, the encrypted message would be “Ymj qjxx dtz xfd ymj rtwj dtz mjfw”.

This is an achient cryptography method that does not even come close to modern cryptography standards, but it is an easy to understand primer for those unfamiliar with the topic. There are many systems that have existed before the age of modern computers that provided varying degrees of security for encoding information. While those are interesting on their own, they have little to do with the modern standards of cryptography and thus will largely be skipped over. Let’s instead look at the concepts that lay the foundation for modern crptography.

Fundamental Concepts

While the mathematics and calculations that make the various crptographic algorithms function in a secure manner are rather complicated, most of them involve a few basic concepts that are much easier to understand.

Prime Numbers

Prime numbers, for those that are unaware, are numbers that can only be divided by 1 and themselves, such as 13. Most if not all crptographic algorithms require large prime numbers (somewhere in the range of 2⁷⁶⁸ bits or larger) to function properly. The main reason for this is that the prime numbers are often used as a sort of key or key generator; these prime numbers are often processed in such a way that becomes practically impossible to find the original prime number with the number that is available to a threat actor. A simple example of this is finding the factors of a large semiprime number, however, that is certainly not the only way prime numbers are used in cryptography.

Modulus Math

Modulus math is something that most people are generally familiar with, even if they have not heard of it by that name. Essentially, it is division but the only answer we are concerned with is the remainder. For example, take the equation 6 ÷ 26, most people would be able to answer with “4 remainder of 2”. Well, in modulis math, the answer would just be the remainder, or ‘2’, these equations are represented as: 6 mod  24 = 2. It is notable that modulus math has non-unique answers, meaning that the equations 19 mod  3 and 37 mod  3 are equivalent (with the answer being 1). This property is useful to increase the performance of decryption as we do not actually have to divide by a huge number, but rather just find the modulo of that number compared to another. It is also worth noting that in the world of computers, everything is going to be set to modulo 2 (remember everything is either ‘0’ or ‘1’), so no matter what values are brains are dealing with, the computer only understands and performs mathematical operations on bits of 0 and 1.

Entropy

Entropy, also referred to as randomness, is an important part of the key generation process. Without a good amount of entropy in the system, it would be easy for attackers to systematically guess what the key is. There are two major categories of random number generators (RNGs):

• True Random Number Generators (TRNGs): These RGNs are often based around randomness from physical actions, such as a coin flip. These actions are nearly impossible to predict or re-create. This makes them really good for creating encryption that is tough to break, however, because it is impossible to reproduce it also means that the key is not practical because it cannot be reproduced for later or for transfer to someone else.
• Pseudo Random Number Generators (PRNGs): These RNGs are based around mathematics, which makes them not really random. The numbers can be re-created by following the same processes and using the same “seed” (or input) value, in short, they are deterministic. This means that the keys are great for being able to re-use or transfer to someone else, unfortunately it also means that they could be cracked more easily by an attacker.

The unpredictability of these random number generators is one of the key points to help cryptography stay secure. If the key is supposed to be a “random” prime number above so many bits, but the PRNG has been compromised to only give one of a handful of numbers, then there are many fewer keys to check against, thus reducing the security of the entire system dramatically. Having sufficient entropy in a system is highly important if the security of the system relies on randomness, such as with PID randomization.

Basics of Cryptography

Now that the foundational concepts that modern cryptography relies on, we can start discussing the general methodologies for encrypting information. As stated in earlier sections of this blog post, many historic methods would process the letters of the language’s alphabet in various ways to encrypt the information. However, modern computers do not understand human language, rather simply numbers that represent the characters drawn on our screen.

Symmetrical

A symmetric encryption algorithm is a system in which the key can be used to both encrypt and decrypt the message. An example of this might be the Ceaser cipher from our earlier example; the number ‘5’ was the key and could be used to encrypt and decrypt (just move the letters by five each time). In the modern world, the symmetric encryption algorithms are much more complex and robust as well as can be used as a stream cipher or a block cipher. A stream cipher is a cipher in which the bits are encrypted one by one as they come across the wire, alternatively a block cipher will encrypt a fixed length of bits at a time, even padding the data to meet the required length if necessary.

One of the biggest issues with symmetrical encryption is securely establishing a key, as both the sender and receiver must use the same key. There are several methods to securely establish a key, depending on what the situation calls for. Transferring the key can be done by using something like public/private key pairs, communicating the key via a secure channel (like meeting in real life and physically giving a key), or, more commonly, using something like Diffie-Hellman key exchange. The Diffie-Hellman is one of the most common methods to exchange keys between a client and server, and is often used in conjunction with AES encryption to secure web traffic. While I probably could explain the specifics of how Diffie-Hellman works, Computerphile has a video on it that explains it much better than I will probably do in this blog post.

Symmetrical encryption is one of the most widely used forms of encryption, as AES is the foundation of many of the protocols that we use today including HTTPS, WPA2, WPA3, and various other applications. Similar to Dillie-Hellman key exchange, I am not going to cover the specifics of how it works because computerphile did that one too, and I feel like they will do a much better job explaining how it works than I am going to be able to. The last thing of note is that while AES might be the most popular (at least symmetrical) encryption algorithm, there have been quite a few that have come before including, but not limited to:

• Blowfish
• Twofish
• DES
• 3DES

Asymmetrical

Asymmetrical encryption is pretty much the opposite of symmetrical encryption, where the key for encryption and the key for decryption are different. There are many different examples of algorithms that use this basic idea for the cipher including, but not limited to:

• RSA
• Elliptic-Curve Cryptography (ECC)
• Pretty Good Privacy (PGP)

Asymmetrical keys are often used to communicate symmetric keys more securely, however, asymmetrical keys tend to be more vulnerable being broken by quantum computers. At the time of writing though, quantum computers have not been able to break strong strong modern encryption (such as 4096 bit RSA or 768 bit ECC algorithms). Though, a quantum computer with sufficient qubits would be able to break modern RSA and ECC algorithms much more easily than classical computers (more resources on that later), nothing of that sort has happened yet, and I personal suspect we are at least 10 years from that becoming reality.

Another problem with some asymmetrical algorithms is the “web of trust”. This is specifically a problem with things like PGP keys and other asymmetrical cryptographic infrastructure such as certificate authorities. While the idea of creating a giant web of connections that allows every person to trust every other person via proxy connections is great, this idea falls apart when you start thinking about it in the real world. How many of your friend’s friends would you trust? If somehow that is the case, you either have an awesome friend or you are hopelessly naive. In the more likely case that you would not trust all of your friend’s friends, then you see the problem of the web of trust. Just because you trust one guy, doesn’t mean you trust the people he thinks is okay. If you cannot trust your friend’s friends, then the idea of the web of trust completely falls apart.

The two most commonly used asymmetric algorithms right now are RSA and ECC. These algorithms are not terribly difficult to understand the basics of, and often have good resources to explain it. That is what I am going to provide for learning these, as it will be much easier than poorly explaining it in this blog post. Eddie Woo has two videos (RSA Part1 and RSA Part 2) describing how RSA works, and computerphile has one on ECC.

Making Cryptography Secure

This is the section that will hopefully go more in-depth on why it is likely a bad idea to roll your own crypto for production use. I want to point out that attempting to create a crypto system for educational purposes is fine, but do not expect it to be secure. Cryptography something that is very easy to make something that looks secure, but is very difficult to make something that is secure. Thus, came about Kerckhoffs’s principle. The origins of this principle came from a list of six rules that military ciphers should follow published in the Journal of Military Science, which are:

1. The system must be practically, if not mathematically, indecipherable;
2. It should not require secrecy, and it should not be a problem if it falls into enemy hands;
3. It must be possible to communicate and remember the key without using written notes, and correspondents must be able to change or modify it at will;
4. It must be applicable to telegraphy communications;
5. It must be portable, and should not require several persons to handle or operate;
6. Lastly, given the circumstances in which it is to be used, the system must be easy to use and should not be stressful to use or require its users to know and comply with a long list of rules.

While some of these rules did not age very well, rule number two is still relevant, and is now referred to as Kerckhoff’s principle. This principle ensures security through testing the standards by battle rather than by hiding; this will allow researchers to attempt to break the proposed cipher to see how well it fares against common attacks against crypto systems. These attacks using things like pattern recognition in the output vs what is known about the input; additionally attacks using poorly implemented RNGs can be used to gain some influence or insight as to what the key might be for the computer.

A great method of seeing the results of what rolling crypto in house is like, look no further than the Secure Sockets Layer (SSL) vulnerabilities. Modern day Transport Layer Security is not perfect, but it is at least not horribly vulnerable to man in the middle attacks. Most crypto systems that have not be exposed to the general public for a vetting period, likely cannot withstand the abuse that they will see in the real world. Which is why the advise is to never roll your own crypto.

Quantum Resistant Cryptography

The last part is the part that we have not actually approached in history yet, however, advancements are working towards breaking several encryption algorithms that would be detramental to the foundations of the Intenret. Many people are aware of this fact, but misunderstand the reasoning behind it. While quantum computers do out perform classical computers in certain workloads, they are highly specific. The reason for that is rather complicated, but has to do with the way that information is processed in a quantum computer; where classical computers have bits (0’s and 1’s), quantum computers have qubits. Qubits are able to take advantage of several quantum mechanic concepts that classical computers are not able to access, one of the most notable of these mechanics is quantum super position, that will allow quantum computers to out perform classical ones in things like Shor’g algorithm.

Using shor’s algorithm, we are able to break things like RSA encryption in a fraction of the that classical computers take. This is the “privacy ending” vulnerability that quantum computers expose. I use quote there, because it will take a long time before anyone will be able to beat modern encryption with quantum computers for one target, let alone generally doing this for all the data across the whole Internet. However, this is a problem that does need to be solved quickly to give the Internet an opportunity to adopt the quantum resistant tech. At the time of writing the National Institute of Standards and Technology (NIST) has found that the following encryption algorithms are sufficiently secure against quantum computers:

• CRYSTALS-Kyber: is a general purpose asymmetrical algorithm based around Learning With Errors (LWE) and cyclotomic rings that was designed to be quantum resistant.
• CRYSTALS-Dilithium
• FALCON
• SPHINCS+

With the mention of the NIST, I would be doing a disservice if I failed to mention the controversies surrounding them as well. While the NIST holds the responsibility of promoting strong cryptography both in the public and governmental sectors within the United States, they do have a least one controversy in which the NIST allegedly allowed a standard (SP 800-90) to be published and promoted for cryptographic security, when there was a potential backdoor in the standard’s PRNG (some allegations say it was put there by the National Security Agency (NSA)). That standard was quickly revoked due to the backlash it was causing the NIST, however, that situation also weakened their trust for a lot of people. So, certainly do your own research on who to trust, and whether that situation is enough for you to distrust the NIST.

Resources

Cryptography is a huge field of study that I have barely done justice on getting a write-up published. As such, I wanted to make sure to leave some good primer material in this section. There are many other good places to start as well, but most people can follow along with YouTube videos easily enough and the creators do a good job of breaking it down in easy to understand pieces.

• MinutePhysics - How Quantum Computers Break Encryption | Shor’s Algorithm Explained: This is a good video (in standard Minute Physics style) that demonstrates the general idea of how Shor’s algorithm works, and how it can be used to significantly reduce the computing time needed to break things like RSA encryption.
• Christof Paar - Introduction to Cryptography: This is a semester long course on cryptography from Ruhr University in Bochum Germany. Though the course is from a German university, the course is entirely in English and is easy enough to follow to get the high spots on how crypto works in-depth.
• Chalk Talk - Lattice-based cryptography: The tricky math of dots: This is a basic primer to how lattic based crypto works, similar to the computerphile videos on AES, but for a potentially quantum resistant algorithm.
• 3Blue1Brown - But what is quantum computing (Grover’s Algorithm): 3Blue1Brown has a small series on quantum computing from a mathematical perspective that will give some more insight on how these algorithms are pieced together and how the math actually ends up doing something useful.
• Microsoft Research - Quantum Computing for Computer Scientists: This is a lecture on how quantum computers actually process information as given from a Microsoft engineer that works with quantum computers. This lecture can be difficult to follow as it is a bath heavy on math, but is extremely informative if you can stick through it.

Abstract

Recently a video showed up in my feed with the title “Why I Deleted My Second Brain: A Journey Back to Real Thinking.” Initially I was a bit skeptical of the concept, however, I gave it a watch and some valid concerns and critiques of personal knowledge management (PKMS). Before reading this, I recommend skimming through the Wikipedia page on PKMS, or reading the section below to get a general understanding and background of what is and is not being discussed. Then, I encourage giving the video a watch in its entirety; both to fully understand the context of this post, as well as to properly give credit where it is due to the original creator. Once the video has finished, you will hopefully have formed an opinion about it, and are ready to come back to this post.

What is a Personal Knowledge Management System?

A Personal Knowledge Management System is a way that one collects and organizes notes on various bits of information. This information could be anything at all; however, what differentiates many note-taking systems that are taught in schools and PKMS are the ability to reference, categorize, and link notes. For example, many of the notes that I took in school were organized for the lecture, but not very searchable as a whole. Information was difficult to find, and the notes from one lecture to another did not necessarily connect as fluidly as it did in my head at the time. This would not have been a problem, had the information stayed in my head. Like most people though, I ended up forgetting things that I knew that I knew at one point and could not find that information again. PKMS set out to solve that problem.

The specifics of how PKMS do that are dependent on the system; some of them relay on categorizing information, other systems focus on the relationships between information. Methods, such as the Zettelkasten, specifically encourage taking small notes that link together to create the structure of the system, rather than creating categories of notes that will never quite fit all of the topics that will eventually live in the PKMS. These systems also tend to be called “Second brains” because the notes are supposed to be relationally structured similar to how a human brain stores information.

Critique(s) of Knowledge Management Systems

The points being made in the video by Westenberg outlines some of the traps that are very easy to fall into with various knowledge management and productivity systems:

1. Shaping Attention
2. Replacing Thinking
3. Outsourcing reflection
4. Guilt for not being “Caught Up”

Above are the points that I personally took away from the video, and I do personally think that each of these points are a problematic trap that is very easy to fall into when using these systems; however, rather than just express what the problem is, I wanted to go through each of these points and try to explore what might cause these feelings, and come up with a plan on how to avoid them.

Shaping Attention

Attention can be thought of as the combination of time and focus that we put into a task. Many systems for note taking, as well as task management, encourage attention to be shaped in a particular way. This can be beneficial at times when trying to prioritize tasks, or when you are feeling overwhelmed. However, these same systems can also encourage bad habits with attention. For example, imagine needing to read a book on a topic being researched. If you are focused on taking notes on the book, you are not really reading it, but rather trying to find ideas in the book that can go in your knowledge management system. Generally, that is not how books are meant to be read.

Rather, the book should be read through naturally, then as interesting ideas come up, note them down. This may be a slightly more boring approach but it allows the book room to breath and expand on its ideas. When trying to hunt for specific ideas, often the larger picture gets missed that strings these ideas together forming a mindset that the book is trying to communicate. A common saying for this would be “missing the forest for the trees.”

Replacing Thinking

This leads into the next point of replacing thinking. Continuing with the example of reading a book for research, imagine sitting down to specifically take notes on a book rather than sitting down to read the book. When I imagine sitting down to take notes on a book, I imagine it to be more like an assignment rather than something I am trying to learn. Now, sitting down to read a book can certainly be an assignment as well, however, it at least encourages following the author’s train of thought on the ideas of the book. Whereas simply taking notes on the book are many semi-related ideas that happen to be in the same package of paper.

Knowledge does not come from simply taking notes on random thoughts from the same reading packet in the same reading session, but rather comes from being able to string multiple ideas together in a creative way to form an idea. Those ideas can then be notes down and related to other ideas; this is how one builds a knowledge-base. Gamifying note taking can, in some people, encourage taking down notes on every idea that may or may not be important for your purposes, not only building up unimportant notes, but also never allowing enough focus on the text itself to allow an idea to truly form into something meaningful.

Outsourcing Reflection

This brings us to the point of outsourcing reflection. Many people today want information transplanted into their brain as quickly as possible, however, that is just not how information works with the human brain. It takes time for even the most open minded people to truly internalize and understand new ideas, concepts, and viewpoints. Rather than poorly parrioting it to another person or your future self, take some time to process and understand the new information. Let the idea breath and really form in your mind to get a better grip of the purpose of each piece of the puzzle.

A counter argument to this is that information will be lost, no one can possibly remember an entire book with 100% accuracy. That argument is entirely true, it is impossible; however, is that something that is necessary? I would argue that generally, it is not. Picking out and noting down the truly most actionable and important points of a text should be done, however, the greater concept of the text is truly the most important piece to be able to actually wield the newfound knowledge the text is giving.

Guilt for not being “Caught Up”

This point is targeted more again productivity systems rather than knowledge management systems, though they can certainly be intertwined. What does being “caught up” mean? Can anyone really be “caught up” in this modern age where information literally at the speed of light? There is no possible way to properly read all of the books that you will ever want to read and live your life; it is the same with TV shows, projects, and many other endeavors that people want to subject themselves to, but then stress about being behind on. It is okay to be behind, especially on more frivolous endeavors. It is good to push yourself to accomplish things, but it is not worth beating yourself up over not meeting the arbitrary goals that you have set upon yourself. Simply do what you can. Don’t let perfection be the enemy of progress, and don’t push yourself so hard to complete the goal that the purpose of the activity gets lost. The point of reading a book is to understand the story or ideas, don’t speed read it to simply cross it off the list; that is just wasting your time as nothing was really gained from the experience.

Mindful Use of Systems and Tools

This last section is going to try to focus a bit on some points to try to build a healthier mindset with various systems. I do want to note though, most, if not all, of these concepts are nothing even remotely new, and could be found in one or two self-help books on any grocery store shelf.

What To Note Down

This is the age-old question that many people have been asking since grade school. “What should I write in my notes?” Everything seems like a good answer, but it is generally not reasonable. So, what should actually go in a note? I often put down small information that I specifically seek out. What is the command that does this task? Where does this registry key live? How do you compile this thing? All of the answers to those questions go in my notes after being found a couple of times, they then also get connected to related ideas and concepts to build something resembling a Zettelkasten.

What about notes on longer-form content? I will try to pick out ideas and concepts that I enjoy, then relate those concepts to specific ideas at a later time after I have been able to mentally process the initial concept. This allows for understanding of the general idea that the long-form content is trying to express. The point of a knowledge management system is to keep track of the details and sources, not to think and understand for you.

Take No Input Time

Having time to mentally process ideas is important as well; the best way to mentally process information and emotions is taking some time where you are not taking in input. This could be meditating, walking, running, yoga, or any number of other things; the truly important part is to do this without input. What does that mean? Mostly, it means no listening to music or podcasts while doing those activities. Don’t talk to a friend while doing those things. Allow your brain to just turn through thoughts as they come without adding to the noise; this puts your brain in the same space it goes when taking a shower, similar to having “shower thoughts”. Many people find clarity in those moments because they are the only moments that they are not being bombarded by new information, specifically positioning yourself to receive less input will only increase that clarity.

Don’t Get Lost in Implementation Details

Not every problem has to be solved before making a change or starting something. Sometimes details are really important and should be considered, but other times they are just an excuse to not try. There is a lecture given by Nickolas Means in regard to the original skunk works team. This is the same team that pushed the envelope of aviation many times and still holds records for some of the most impressive feats an airplane has accomplished. One of the more famous planes that this team built is the SR-71 Blackbird; the team faced many challenges during the development of the plane, such as not being able to find a sealant that would work both at ground level, and at the required altitude of the plane. The solution was simple, don’t fix it. Let the plane leak fuel at ground level, and refuel it at altitude when the sealant works. That solution seems crazy, but that is one of the most successful recon planes to have ever flown.

Now, the point of that story isn’t that the solution is always “don’t fix the problem.” Rather, it is more like, is this a big enough problem to prevent us from proceeding? Obviously in the case of the SR-71 Blackbird, it wasn’t. Maybe the blocker on your project is something that can be worked around just as easily.

GNU vs BSD licenses

Licensing in software has a very interesting history, because in the early days of computing software was shared around freely. This changed when companies decided that they should be making money off of the software development as well as the computer hardware. Changing the licenses and how the source code was distributed made the software propreitary, and made a lot of developers in the late ’70s very upset. One of these developers was Richard Stallman, the founder of the Free Software Foundation (FSF) and the creator of the GNU General Public License (GPL). This license had a feature that not only allowed, but encouraged people to copy, modify, and share the software’s source code. Around the same time, some students from Berkely were working on their own version of Unix. This version was dubbed the Berkely Software Distribution (BSD). The creators of BSD wanted to give the software away as freely as possible to anyone that wanted a copy of it to do what they like with it.

On the surface these two licenses sound very similar, but in the free software community there is a huge debate over which is the better license to use. Much of this debate is just voting for the camp that you side with. Many Linux users prefer the GPL, while BSD users would rather use a BSD license. These licenses do have some major differences in features that they provide.

GPL

The GPL’s main goal is to spread free software far and wide. It does this by allowing people to use, copy, modify, and redistribute modified versions of the software; the only catch is that subsequent versions of the software must be released under the GPL. This is a nobal goal, because this way free software will spread. However, it does have a downside; which is that all software released under it must be released under the GPL, which is a bit of a turn off for major companies. This feature can be described as non-permissive, and is generally something that companies do not want to have to follow, because then they might have to release some of their propeitary code.

BSD

The BSD licenses, on the other hand, are known as permissive licenses. This means that the only real checks are that credit for the original software is given (gross over-simplification). The source code does not have to remain under this license, but can still be used for any purpose, copied, modified, and redistributed.

Abstract

I am working on five years in the IT industry; and while I would still consider that early in my career, it did seem like a logical point to reflect on the journey so far. This post will likely be quite ramble-y and not really containing anything technical. Rather, it is going to focus on various parts of working in IT that I have come to learn. This will likely be a boring read for people that have worked in the industry as long or longer than I have, but hopefully will be at least somewhat helpful to people aspiring to work or just started working in the field.

My Start in the Field

My path in IT so far has been rather fortunate; when I was hired at my current company, I was floundering a bit out of college and didn’t have a lot going for me on my resume other than the variety of personal projects I had done. Thankfully though, I had shown enough interest and capability of learning while also appearing to be a human being that I was hired as a desktop support technician. Starting the new career was intimidating and I often questioned whether I was an impostor or not; over time that feeling faded a bit by being able to tackle bigger problems while also working well within my team. I learned how to research problems, and not only when to ask someone else, but also how to ask someone else for help. Asking probing questions to more senior people exposed me to a lot of different things that a desktop support position typically would not have been allowed to do. It also eventually built myself up in a way that when the system administrator position opened, I was asked to take the role. Taking the sys admin role also means that I have to assist in training up the desktop support position to eventually turn them into sys admin material.

Lessons

In the short time I have been working IT, I have picked up on a couple of lessons that I was not taught in school that can make life a bit easier when working in the field:

• Precedence: This is a term often used in law, but it applies with IT requests as well. Do we have previous examples of this type of individual asking for this type of request. For example, has a manager asked for a new distribution list to be made before? If yes, how did we handle that? If no, why? This can help break requests down into categories that can be processed more easily, especially with requests that would be considered odd. What is the most similar request we have gotten? how did we handle that, and how is the current request different. What does that change? This is a lesson I was taught from my previous sys admin, and is something I use on a regular basis when making decisions on how to approach a problem.
• Most people do not understand computers: This one seems obvious because of course they don’t, that is why they pay you. However, the level of not understanding computers is difficult to really describe without experiencing it. On a regular basis, I have users that will call their computer tower a “CPU” or restart their monitor and claim they have “restarted the computer.” Unfortunately, the lack of understanding most people have for magic thinking rocks (computers) goes much deeper than that. It is not helpful to blame them, they don’t speak the language of strange abbreviations and fancy words, and what is a ‘server’ anyway. The vast majority of end users don’t know and don’t need to know, what they need is an IT person to figure out what they are trying to say, then fix their issue.
• Owning mistakes: Making mistakes really sucks, especially when it affects a lot of people. One of the wonderful things about working in IT is that your mistakes can affect lots of people! It’s part of the job. Accidentally cutting an Ethernet cable in use, a department may no longer have a network. Set the wrong VLAN config on a switch, network is broken downstream of the switch. Unplug the wrong cable on a server, whelp, server is down for the whole company (true story too). Making mistakes can and absolutely will happen over the course of a long and prosperous career, learning from the mistakes is what makes us better, and builds junior people into senior ones. Own your mistakes, and do your best to fix them, or at least try to come up with a game plan for fixing them.

Struggles

As unfortunate as it is, struggles will exist in every aspect of life, nothing and no one can be perfect. This section is going to go over some of the things that I struggle a bit with, and I suspect a lot of people do as well. It’s worth noting that most of these things are not going to be technical, everyone knows that IT people hate printers.

• Impostor Syndrome: I do believe this is a fairly common occurrence in many professions, but Impostor syndrome is something I do struggle with fairly often. Constantly questioning whether you are good enough to be there and whether or not you know enough. The best advise I have to combat this is keeping track of your growth and accomplishments; that can take form of a blog, a notebook, or just revisiting where you started. It won’t prevent the Impostor syndrome from hitting, but it might help negate the negativity it generates.
• Effective Writing: Effective writing is an important skill for anyone working in 21st century or later. Most of civilization is founded upon the idea of reading and writing to communicate ideas and concepts very quickly. Reading and writing are also skills that many people do not practice either of those skills on a regular basis, especially writing. Even more importantly, tailoring your writings to a specific audience. Much of this blog is targeted towards at least slightly technical people, but I have had to do quite a few pieces of writing that was intended towards a non-technical end user. Being able to communicate with the written (or more often typed these days) word is key to being successful in the modern world, whether that be in the form of email or a written report.
• Expectation management: This skill is one that can be applied with every level of worker in which they are having to operate with other people. Making people understand what is realistic and what is not is something that can save a lot of pain on the back end by having an unpleasant conversation on the front end. People do not always understand what they are asking for; as the expert, it is your job to inform and make them understand (at a high level) what the implications of their ideas will be. The best approach to this is to be very blunt, very honest, and have a very good understanding of the problem space the problem exists in.

Unsolicited Advice

We have not reached the point in the writing in which I give advice that no one asked for. This advice will very heavily focus on the operations side of IT, but an analog could probably be found on the dev side as well:

• Work on personal Projects: This is even more true for the people that currently do not work in IT. Personal projects are the way to show what you know without that pesky experience thing that a lot of companies require. As it turns out, software doesn’t really care whether you are certified in the tech or not. If you can click the buttons to make the product do the thing, that is a talking point during the interview. The more you can talk about projects that relate to things the company does, the better your chances of getting hired are. In addition to that, being able to talk about personal projects show interest in the field; people that have that interest will always do better than people that don’t because they want to learn more. One question I asked when interviewing candidates is “tell me about the project you have done that you are the most proud of.” For someone that really enjoys the field, this question will be the easiest one to answer, but to someone that looks at it as a paycheck, it will be extremely difficult.
• Start a blog: I have done another post about this, but there is no reason to not have a blog of some sort. There are so many platforms that will allow anyone to host static sites for zero cost that it is quite sad that more people do not take advantage to up their web presence. Blogging is a great way to show off various projects that you have done, as well as practicing soft skills, as well as showing off those soft skills to the world. If the blog is successful, it is possible that it could be a resume supplement and help get a job in the future as well.
• Practice soft skills: Soft skills are the skills that are not generally directly taught, but will make you a lot easier to work with. These are things like verbal and written communication skills, explaining technical concepts to non-technical users, and documentation. Those are not usually skills that are taught directly, rather it is expected that you will figure it out at some point. You do not have to wait to start working in IT to do this though. Write about the cool stuff you do! talk to people about your projects! Even if it is just people you know personally, try to explain to them some of the things you have been working on. You can gauge how much better you have gotten at explaining things by benchmarking how quickly their eyes glaze over.
• Perfect is the enemy of good, and also the enemy of progress: It doesn’t matter how “good” you are now. You are not perfect, and that is okay. No one is expecting that of you, there is no point in expecting it of yourself. Don’t get caught up on doing everything perfectly because in the “real world” nothing is perfect, it is merely trash vs trash-lite. So, find the best solution you know how to, and just start working with it. If something better comes up, re-address what you are currently doing.
• Just start: The last thing, is just start. By that, I mean just start doing something you find interesting. It doesn’t really matter whether it is actually interesting or not, because that is a subjective opinion. Nothing is ever as intimidating as it seems once you get started; even if you find yourself not understanding something, that just means you need to go learn some other things before coming back to this topic. I’ve done that on numerous projects and blog posts. There is no shame in putting something down for a while to level up the relevant skills and kill it later. There is also no shame in picking something up that you had no business (only in a test environment, don’t break prod) touching and trying to get it to work.

Abstract

I was recently a part of a project that had a massive amount of difficulties that I believe could have been mitigated. This blog post is my attempt to write some of the thoughts and ideas down that might help mitigate similar failures in the future; while many of these are likely obvious, I feel they will be beneficial for me to keep in mind for the future on any projects that I am leading or a part of.

Pre-planning

Most projects are started because of something; whether that be something as trivial as a personal project that will never actually be used for anything important, or something as major as revamping a company’s infrastructure to meet modern standards. Both of these have slightly different approaches to actually dealing with the project, but can be approached in a similar basic way:

Goals

The first and most important step for any project is figuring out what problem or problems the project is attempting to solve. It is almost impossible for a project to actually be successful if the problem is not defined within the scope of the project. Ideally the problem or problems will be very clearly outlined or itemized for the team working on the project.

Research and ask questions

The next step is to research various ways to solve the problem or problems that the project will be attempting to solve. Try to figure out how big the project will be, and if it makes sense for the various goals to be lumped into one project; it may not make sense to try to fix the plumbing of your house while also working on the electrical system. Figure out what route to solve the problem makes the most sense for you and your team, as well as get an idea of what the scope of the project might be.

Take notes

Taking notes on the various ideas that come from research is also highly important; it may seem silly for someone that has a lot of knowledge in an area to take notes on something very trivial, it will help communicate ideas to your future self, as well as the team helping you with the project. The more complex a project is, the more important this step is, especially when working as a team.

Planning

The planning stage is arguably the most important stage of a project. This is where the team comes together to share their cumulative knowledge on the various aspects of the project, and figure out a plan that makes the most sense for the team. If you are a team of one, then this process can be slightly less formal, however, if you are truly working as a team of many people, it is a good idea to try to keep leadership qualities in mind. They are extremely effective whether you are actually leading the team, or simply running as a part of the team. Some good examples are:

• Everyone can have a good idea and contribute to the team
• Challenge ideas, not people
• If you don’t agree with the plan, speak up, and explain why is clear ways

Who, what, where, when, why, and how?

Asking these questions should be obvious to most people, but it is always worth keeping in the back of the mind:

• Who is the team that will be implementing the project?
• What are the expectations of the project for the team to work towards?
• Where is the project to going be taking place and being worked on?
• When is the project going to start, and how long is it expected to take?
• Why are we doing the project in the first place (this question should have been answered by the problems that the project is attempting to solve)?
• How is the project going to be implemented, what steps will be taken to make sure everything goes as smoothly as possible.

Checkpoints

Working checkpoints into the project is an idea that I did not have until after my most recent project that went less than wonderful. One of the big problems was the inability to stop once we realized the project was taking longer than expected. Being able to stop a project is not always something that is needed and sometimes is not even desired, however, if there is a hard timeline for the project, being able to work within small checkpoints that can be stopped and re-addressed at a later time is huge. It also tends to feel much less overwhelming to have a small manageable list of things to do, and seeing it dwindle down as tasks get completed helps keep you motivated to keep going.

Work the project

Assuming that notes are taken, a plan is enacted, and everyone knows what they are supposed to be doing ahead of time, the actual ‘work’ on the project should be relatively straightforward. It may not be ‘easy’ work, however, the most difficult parts shouldn’t be a surprise to anyone; and assuming that the difficulties were known ahead of time, there should be ample time to tackle those challenges.

Post project analysis

This step often only seems important when things go wrong, however, it is also important to acknowledge the things that go right. Complex projects can especially have certain things that worked really well and went even better than planned, but there will inevitably something that went wrong, or was more difficult than expected. Hopefully these are left to a minimum, but it is still good practice to see what could have been done, if anything, to mitigate those problems. It is also important to acknowledge (even if in silence) specific qualities of the team and the inter-workings of the team. Did this team work well together? Who was good as doing which tasks? were there any unforeseen conflicts between the teams, and could those have been avoided?

While the post project analysis is usually reserved for the leadership of the project, it can be a beneficial exercise for anyone within the team to do. Most importantly, is there anything that you (the reader) could have done to make the project better, did you make any mistakes that made the project worse? It is important to reflect on your weaknesses to make yourself better; if you cannot lead yourself to be better, how would you lead a team of people to do better?

Abstract

Using tools is an integeral part of accomplishing tasks, and an important part of using tools is selecting the right tool for the job. This article is going to go over the importance of selecting tools to use for particlar projects both when working alone and working as a team.

Have a wide variety of tools

The tool that you pick for completing a task can shape the direction and/or outcome of a particular project; which is why having a wide variety of tools for a wide variety of problems is essential.

If the only tool you have is a hammer, every problem looks like a nail.

This is a common saying, at least in the US, that outlines how easy it is to become single minded when only knowing one tool. While a hammer is a fine tool, it doesn’t work in all situations; which is why it is beneficial to be able to use other types of tools for other problems.

Be flexible with your tools

In addition to using other types of tools for other types of problems, it is also good to try to generalize the problems that tools are trying to solve, and be familiar with the various ways that tools attempt to solve that problem. For example, a hammer and nail try to solve the problem of holding two pieces of wood together by driving a nail through both of them. Another tool that can do this is a screw using similar principles.

This works extremely well in the tech world, because being knowledgeable about a problem and the generalized solution means that it will be easier to pick up an unfamiliar tool and solve that problem. Because of this, it will be easier to integrate into a team that uses tools that you may or may not have experience with. It also means that you can bring new ideas to a team because you are not stuck within the limitations of their tools.

Pick the best tools for your team

The last major thing is selecting a tool that works well for the team. This one is particularly difficult for people with less experience or extremely passionate people. Sometimes the tool that you like or works really well for you, may not be the right tool for the team. Even if it fits the prompt perfectly, it may not be the right tool for the team. This has to do more so with the team’s expertise, skill, time, and willingness to learn. For example, I like using org-roam and org-roam-ui for my personal knowledge-base; however, it would be a terrible tool to use for my team. This isn’t because either of those tools are bad, nor does it mean that my team is bad. It just isn’t the right fit. They don’t have any experience using Emacs, additionally, we would have to have a way to manage changes (Git probably), and they don’t have much experience with that either. A browser based tool that uses a more simplified markup language works much better for my team, because it fits our needs well enough without getting in our way.

Closing thoughts

This was just a short though on things to keep in mind when picking tools to use and use within a team. I find myself sometimes wanting to use different and ‘better’ tools, but then I remember that those tools don’t really make sense for the goals I am trying to accomplish, and they would tend to get in my team’s way more than they would save time/money/effort.

Abstract

Do you have a website or blog? You should. There are many reasons and benefits that I could list off to support this, and I will get to that, but in short, you should document your interests and share them with strangers online who share those interests. While there are many “platforms” that allow you to do this, they often have various restrictions to the specifics of how to go about doing that. Having a blog or a website proper removes a vast majority of those restrictions and gives you nearly complete creative freedom. I would argue it is also much easier to archieve your knowledge of past projects with a website than it is something like a series of posts on a social media site; this entire website (markdown source files as well as HTML and all the images) are 12 megabytes and can be transferred via email, flash drive, or git. Backing up posts on a propreitary platform is not generally so simple though.

A personal website can also be a suppliment to your resume in a professional setting. Going through my various projects shows not only what projects I have done, but how well (or poorly) I have done them, which is much stronger than just claiming “I’ve used Linux before”. Writing about projects also has the secondary benefit of showing (and improving) written communication skills, which is something most employers look for these days.

The last point, and the most important in my opinion, is that individuals running a blog or a website is what makes up the charm of the Internet. It’s a difficult skill for most people and sometimes the pages don’t look very good, but that is part of the charm from the Internet of old. These days many sites look too clean and steralized to the point where they do not actually serve the purpose of communication.

“What do I write about”?

Hopefully at this point, I have conviced you to at least consider the idea of a blog, and if you have come that far you should think about what you are going to write about. This is probably the most difficult part because the possibilities are nearly endless. It doesn’t have to be difficult though; ask yourself what topics are interesting (or at least tolerable) to write about? For me, most of the time it is related to technology, but occationally I like to sit on a soap box and preach about things like I am doing in this post. That may not be your style, and that’s fine. What you want to write about may not even be novel or interesting to other people, and that’s fine. This website should be about what interests you, because if you are passionate about it, a following is more likely to come (if you are interested in that).

Another tip is start writing early. Even if you don’t plan on making a blog for a few months or more, nothing is stopping you from writing your thoughts/ideas/projects down now. You can always publish them on your website later, however, you can’t go back in time and retroactively write them. There are various things I wish I had documented on this website, but now if I want to document it, I will have to re-do the project.

“How do I get started”?

Now that you have started writing and have decided to publish something, you should figure out how you want to host the blog/website and how you want to publish it. This blog started out on Codeberg Pages because it was free, easy, and had “backups” in the form of Git hosting. However, there are many other ways to go about it that have varying costs and complexity accossiated with them. Some other options include:

• neocities
• Various shared hosting providers
• cheap VPS hosting: This one is going to be for advanced users that know how to setup a web server.

And if you need/want help with this project, contact me and we might be able to work something out.

Writing Workflow

One of the more personal choices that will have to be made is the workflow for writing posts. What tool(s) to use? how to convert the text? Advertising? These questions are difficult to answer without having hosting setup first; if your blog is on a Wordpress website, than publishing tools are built in. Simply edit the page as desired and click “Publish”, however, other tools are available as well. This website is made using a static site generator, specifically, ssg5. Or, you can hand write standard HTML and CSS for each page. The options are endless, and the world is your oyester; it is your website after all.

Final thoughts

A lot of people give a lot of reasons that they don’t have a blog, either it’s “too difficult” to get started, they’re “bad at writing”, or they “aren’t interesting enough”. None of those things matter, just write something that YOU find interesting; think about all the people that post mind numbing videos on social media, did any of those issues stop them? Why should you be stopped from staking your own piece of the Internet to share with the world?

I am going to leave two links below that I fall back too when I get in a bit of a creative rut and find writing these posts a bit too difficult:

• Marc Rebillet: Another Idea
• The Cult of Done Manifesto

Using Org Mode to Stay Organized and Productive

As many emacs users will know, org-mode is an extremely useful tool within emacs. It has become an important pasrt of my workflow to be able to stay busy and organized, both at work and at home.

Notes in Org

Org is a markup language similar to Markdown with relatively simple syntax that is powerful at the same time. The other features that org provides uses the org syntax, however, the file itself is extremely portable and can be edited on any system with a text editor. This post will not get into the specifics of the syntax for org; however, if the reader would like to know more, information is available here.

Tasks and Todo lists in Org Agenda

Agenda views takes the syntax that Org uses, and gives the ability to create todo lists and schedule tasks. This is the feature that really gives Org its value for me. I use Org Agenda daily to schedule tasks and take notes on each task.

Orgzly

I use an app on my phone to help keep my personal life organized as well. That app is Orgzly. The main issue I have with the app is the ‘sync’ options that it provides. Currently it is limited to local storage sync or Dropbox. It also has limitied support for Git repositories, however, that feature is in beta and not technically a supported option.

Using Spacemacs in WSL

I recently switched back to using a Windows machine at work from using a Mac. While Windows does have some advantages that MacOS does not provide, it is much easier to use spacemacs and get my Org Agenda file working as intended.

WSL has made installing emacs and spacemacs easier than installing emacs natively on Windows, but it still work much better on MacOS than it does even in WSL. Another issue I ran into is syncing my Org Agenda file from my OneDrive so that it would be backup in case my computer exploded. For some reason I could not get the .org file that Agenda uses to store tasks to default to the one I have in OneDrive. I ended up using a sym link to resolve the issue because I did not have time to give the problem a ‘proper’ solution.

GNU Hurd

This week I am playing with GNU Hurd, specifically Debian with the Hurd kernel. While Hurd is not something that is going to be widely adopted any time soon, I thought it might be interesting to see what it can do.

Downloading and running GNU Hurd

The easiest way to start with GNU Hurd is to download the pre-installed VM image; instructions available here. Running Debian Hurd in Qemu was the only way I was able to get it up and running properly; I believe this is mostly due to driver issues and/or limitations, as Hurd currently has very limited hardware support, and cannot even really handle SATA drives.

Applications

The prebuilt VM image comes without a display manager, but running the startx command will take you to LXDE for a grpahical interface. There is not a lot of software preinstalled on the system, however, according to this page, at least 79% of software available in Debian repos should be available for Debian/Hurd.

I wanted to try a different graphical enviornment, so I decided to start with EXWM. It seemed to run well within Hurd, and nothing more to say without reviewing EXWM as a window manager.

Web Browsers

A basic task of any modern OS is to be able to browse the web. While I did expect some sites to be broken or not load at all, I was hoping that I could at least get a graphical browser. However, I could not get Firefox to install at all with the output of:

fac3@debian:~$ sudoa apt install firefox
Reading package lists... Done
Building dependency tree... Done
Reading state informaiton... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
  firefox : Depends: libevent-2.0.5 (>= 2.0.10-stable) but it is not installable
            Depends: libffi6 (>= 3.0.4) but it is not installable
            Depends: libhunspell-1.4.0 but it is not installable
            Depends: libvpx3 (>= 1.5.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

After looking around online for solutions to the Firefox issue I found this user guide that had a section on surfing the web which recommended using Lynx. I have used Lynx in the past, however, generally for text only web browsing these days, I prefer w3m as it has a nicer interface and works a bit better in the modern web than Lynx does.

Both Lynx and w3m worked to browse the web in a limited capacity, but I could not get any graphical browser to work with any success. Firefox would not install at all, nor would any fork. Chromium would not install either. I was able to get Midori installed, but it failed to go out to the Internet in any useful manner; I also tried surf, both the version in the repo, and compiling from source, and has similar results.

Thoughts

GNU Hurd is an interesting idea that I will likely keep around to play with more, however, for me it is not much more than a toy. I’m sure that some people do live within the limitations of GNU Hurd as I have tested it, however, I’m not sure that I would be able to without some significant difficulties in working. Hopefully the project continues to grow and improve; I may also do follow ups as updates are put out, or I solve issues that I ran into.

What is OSMC

Open Source Media Center (OSMC) is a Kodi based project that allows users to turn devices into full blown open source media center. More information can be found on their website.

Installing OSMC

For my setup, I used a Raspberry Pi 3B which is officially supported by OSMC. Installing the OS is fairly straightforward. Just download the img.gz file, unzip and dd onto the microSD card. After getting the image on the SD card, put it in the pi and connect the power supply. The initial boot process will take a few minutes, but will then take you to a set up screen.

Setup

The setup process is also fairly straightforward, however, I do recommend connecting a keyboard to the Pi for it. Select the language, timezone, and give the machine a hostname. Next, open the ‘my OSMC’ app and go to the network settings to get the wireless setup (this step can be skipped if using wired Internet). Turn on wi-fi, and follow prompts to connect to network.

Now the device is connected to the Internet and updates can be installed. The device is also accessable via SSH, which will be needed for third party plug-ins like Netflix.

Getting Fire remote to work

I had a firestick remote laying around since my firestick recently went out, so I decided to use that for the remote. This part of the process was actually much easier than I thought it would be. Go to the network section in the ‘my OSMC’ app, turn on bluetooth, and scan for devices. Press the home button on the remote for 10 seconds, or until the light is flashing. Pair the remote on OSMC (mine came up as ‘AR’). From there the remote can be used to control OSMC.

Installing Plugins

The termininology in OSMC kind of threw me off a bit here. An ‘app’ is something like FTP or SMB, whereas a plugin is something used to connect to a service. After figuring out the differences I started looking at the various plugins offered in the default repo; unfortunately Netflix was not one of them. However, a quick search on the OSMC forum lead me to a GitHub project that allows OSMC to connect to and stream Netflix.

Installing Netflix

Follwing the guide for the Netflix Addon was not terribly difficult, and only required a few minutes.

After logging into my Netflix account, attempting to watch a video prompts for the installation of Widevine (1.2 GB download for those on data). Downloading and extracting the package takes a bit as it is a lot of data, however, after it is done the selected media plays without issues.

Installing YouTube

OSMC does have a YouTube player in their repository of software, however, it requires API keys provided by the user. While this process does not look very difficult, it also requires signing in to Google services to do, which is not something I am particularly interested in doing. The guide is available here for those interested in doing so.

What is a knowledge base

A knowledge base will generally help people keep track of complex topics and how various bits of information relate to each other. While giving examples of knowledge bases is not too difficult, I could not think of a good definition for one. So I used Wikipedia instead.

A knowledge base (KB) is a set of sentences, each sentence given in a knowledge representation language, with interfaces to tell new sentences and to ask questions about what is known, where either of these interfaces might use inference. [1] It is a technology used to store complex structured data used by a computer system. The initial use of the term was in connection with expert systems, which were the first knowledge-based systems. - Wikipedia page on Knowledge base.

Many people have used a knowledge base at one point or another. Whether that is trying to find an answer for a product they just bought, or for performing a certain task at work, or even to look at something on Wikipedia. Creating a personal one gives the ability to easily keep track of knowledge on a topic that are complex and difficult to keep track of, but is very beneficial if done.

Org-roam: The knowledge base within emacs

Of course emacs has a knowledge base package that is available. I am not really sure why I doubt emacs can really do anything except be a good text editor… and maybe browse the web without issues. There actually seems to be two packages that function as a knowledge base within emacs. There is org-brain and org-roam. I chose org-roam because it has more features at the time of writing such as org-roam-ui.

Setting up org-roam in Spacemacs

According to the Org layer page in the spacemacs documentation, add the following to .Spacemacs.

;; Enabling org-roam and org-roam-ui in spacemacs
(org :variables
      org-enable-roam-support t
      org-enable-roam-ui t)
;; Skipping down to the 'dotspacemacs/user-config' function

;; This line forces org-roam to use v2, which is required
;; to be able to use org-roam-ui
(setq org-roam-v2-ack t)

;; Sets the directory for org-roam, or use the default ${HOME}/Documents/org-roam
(setq org-roam-directory "/pick/a/good/dir/for/you")

;; Automates the database syncing (yes, org-roam is a databsae)
(org-roam-db-autosync-mode 1)

Then restart Spacemacs. Theorectically at this point it should be working, however, I had quite a few issues with this. It kept giving me an error about a database not being avaiable. The resolution that worked for me was to use the ‘Update Packages’ button on the Spacemacs home screen. Following that process, assuming no catastrophic errors, it should being working after it is finished updating adn building all the packages.

Using Org-roam

After getting the packages installed and built we can start using org-roam. The Spacemacs keybinding to create a new ‘node’ (think of this as a note) is SPC a o r i. Emacs will ask you for a name to call the node, then open an org-mode buffer. Org-roam gives all the same power that you would get with standard org-mode, but you can also link nodes together to help simplify complex ideas. Each node can link to other nodes, and interlink between common ideas, building a complex web of information.

Org-roam-ui

Org-roam-ui gives a very nice web visualization of not only each node, but how they connect. Clicking on the node will pull up the information contained within the node, allowing for very easy reading. The web interface is hosted on the local computer, though, hosting it remotely would surely not be too difficult if that is of interest.

About HaikuOS

Haiku is an open-source operating system that specifically targets personal computing. Inspired by the BeOS, Haiku is fast, simple to use, easy to learn and yet very powerful.

Hardware Support

HaikuOS supports both 32-bit and 64-bit processors; I installed the 64-bit version and have not really run into any issues, however, from the FAQ on Haiku’s website regarding whether there is a 64-bit version of HaikuOS or not:

Yes. Please note that the 64-bit release does not support BeOS binaries, but it is still compatible with the powerful BeOS API (while offering modern features). The 32-bit Haiku release can run most BeOS applications without modification or recompiling. - General FAQ

I did not have any issues with any applications that I tried to run, but most of the things I ran were likely not developed specifically for BeOS.

Using HaikuOS

I installed HaikuOS on my Thinkpad T400 with 4GB of RAM and a core2duo. Even on a hard disk drive HaikuOS boots very quickly. The issue I have with the boot is that I could not figure out how to require a login before being able to have root access to the system. I also could not figure out how to bind a key to lock the screen, which may not be an issue for some people, but definately worth noting for some users.

Shortcut Keys

The shortcut keys are based off of the ALT key rather than CTRL, which takes a minute to get used to if you are used to Windows or Linux. However, it does not take long to fluently work with it; and it can also be changed to the CTRL based shortcut keys in the keyboard settings if it is too annoying to deal with.

User Interface

The UI of Haiku is reminiscent of a 90’s OS with some design choices that take a minute to get used to. However, Haiku includes very good documentation on the desktop that will give a very good walkthrough of most of the common tasks that users would be interested in doing. It will also show how to customize things like the deskbar, which users may want to put somewhere they are more familiar with such as the bottom left corner rather than the top right.

It also take a slightly different approach to window management differently than a traditional stacking window manager does. Each application has a tab that can be dragged to position the window on the screen. If you would like to group applications together, you simply hold the Windows key and drag the current tab over the tab of the application you would like it grouped with. This will then allow the user to drag and resize the entire group of windows rather than just one window at a time. I like finding different ways to manage windows and actually use the system like this (even if I wouldn’t necessarily want to actually use it every day).

File Systems

HaikuOS does not support many filesystems at the time of writing. According to the DriveSetup program on my Haiku install, it will only support FAT32, Btrfs, NTFS, and the Be File system. The FAT32 and NTFS support means that you could transfer data to and from HaikuOS on a flash drive without too much trouble, as most people likely do not use ext4 for external drives. Though, more filesystem support would be nice.

Package Management

HaikuOS uses pkgman as its package manager which works at least as well as FreeBSD’s pkg for most basic tasks like searching for or installing software. There is also a graphical package manager that is called the ‘HaikuDepo’ if users would rather have a GUI for installing packages rather than a command-line tool. The Haiku Depo also has recommendations for popular software in various categories that help show some of the popular options for certain types of software (quite helpful for looking at browsers).

Applications

One of the major issues with more niche operating systems is the availability of software. This is a problem that even more popular OSs like Linux or any of the BSDs have to this day. The following sections will cover some of the highlights of a handful of categories for applications.

Web Browsing

My experience in browsing the web on HaikuOS was probably the worst part of using the OS for me. Websites were slow to load, and too often just didn’t load at all. The browser that I had the best luck with was Gnome’s ‘Web’ (formerly known as Epiphany). It tended to have the best site compatability and also tended to load the quickest out of the browsers that I tried. The next best browser on the list was Falkon, but sadly I had quite a bit of trouble even looking for a wallpaper on DuckDuckGo with Falkon. Finding and downloading an image for a wallpaper was little trouble with Gnome’s Web.

HaikuOS does have a built in browser called ‘Web Positive’, but with my testing the site compatability for that browser was not very good at all. Many sites did not load from something that looked like a TLS error. It seems like the OS thought the time was wrong on my machine, even thogh it appeared to be correct to me. I could not figure out how to fix that issue with the default browser, but the issue didn’t come up again with any other browser.

Media Consumption

The built in media player for Haiku was basic, but it was able to play the music and videos that I tried out. Audio switching between the laptop’s speakers and headphones worked without issues. I installed Clementine for a music player, however, Clementine was not able to play webm audio for some reason (though I did little to try to resolve this).

Office Suite

Libre Office is available in Haiku’s package manager, which should get most people close enough in an office suite. There is also Caligra, however, I have not used it so I don’t know how good or bad it is.

Coding

I want to make it clear that I do not code a lot, so my opinion may not be 100% accurate, but Haiku seems to have most of the tooling needed for at least basic programming. There are a variety of editors, compilers, interpreters, and utilities that make coding in Haiku very possible. It also has packages for a variety of version control systems including: Git, Subversion, and Mercurial.

Gaming

While I do not see many people using Haiku as their main gaming machine, there are quite a few games and emulators in the repositories. Some of the games that I noted are: Minetest, GZdoom, Hexen II, and SuperTuxKart. The main issue I had with gaming is the wireless mouse I connected to the laptop didn’t work, but I’m sure a wired mouse would work just fine.

Abstract

I wanted to research MFA methods for SSH connections that did not require a password. Passwords, as many people know, kind of suck. However, authentication is needed for SSH to work properly. How can you authenticate for SSH? Easy an RSA key; and RSA keys are pretty secure, especially if they are encrypted with a password. What if you are wanting more though? You could use a password for the user account as well, but again, passwords suck and we already have one for our RSA key. So what about One Time Passwords (OTPs)? Specifically TOTPs like you would get by using an app like Authy or Google Authenticator. That would be great! This is a guide on how to do just that with FreeBSD 13.2 and OpenSSH 9.3.

Use Case

The use case for this is probably pretty niche. For most people RSA keys (especially encrypted ones) will suffice for security with SSH sessions. People that are in need of more authentication than that likely have standards bodies to comply with; and I do not believe they would consult this blog for advise on how to configure authentication to their machines. However, if MFA with an RSA key and TOTP is interesting to you, here is how I went about doing it.

OpenSSH Server

I have OpenSSH 9.3 installed on the server that has this configuration.Other OpenSSH versions will likely work, however, be mindful in the differences between OpenSSH 9.3 and the version the reader may be using.

Generating Keys

Many people using SSH will already have keys for logging into their servers, however, for the sake of completeness the ssh-keygen command is how one would go about generating key pairs.

$ ssh-keygen -t rsa
> Enter passphrase (empty for no passphrase):
> Enter same passphrase again:

The ssh-keygen command will generate an asymetrical key pair. The default location for these files are ${HOME}/.ssh/ and they will be called id_rsa and id_rsa.pub. The contents of the .pub file will need to go on the remote server the file ${HOME}/.ssh/authorized_keys. This fill is the default file OpenSSH will be looking at to see if a key will authenticate a user to allow a successful login. The id_rsa file will need to be kept safe as it will authenticate to anything that corresponds to the .pub key that was generated with it. Alternatively the id_rsa.pub key can be shared with anyone and everyone; this file contains nothing sensitive.

Please note that while it is possible to not have a password for a key pair, it is a bad idea. The password (or passphrase as ssh-keygen calls it) will encrypt the key so that it cannot be used without knowing the password. It is also worth mentioning that there are other key pairs that ssh-keygen will generate. For more documentation on the utility please reference the ssh-keygen man page.

OpenSSH Server Config

Below is my OpenSSH server config with some comments removed. All of the comments in the config below were added during the time of writing to help explain some options better.

# Forces v2 protocol
Protocol 2
PermitRootLogin no
MaxAuthTries 5
PubkeyAuthentication yes

# Forces public key authentication, then allows for keyboard authentication
# the keyboard authentication needs to be enabled to allow entry of the TOTP
AuthenticationMethods publickey,keyboard-interactive
ChallengeResponseAuthentication yes
KbdInteractiveAuthentication yes
=======
X11Forwarding no
ClientAliveInterval 300
Subsystem       sftp    /usr/libexec/sftp-server

It is always a good idea to keep a backup terminal open to the remote server just in case. If something is configured incorrectly and a backup session is not open then you will have a difficult time getting back into the remote server. Restart the sshd service for the changes to take effect.

$ sudo service sshd restart

and attempt to login to the remote server. Rather than asking for a password from the SSH session, it should prompt for a password for the rsa key pair (you did set a password for that right?) and let you in once the key is unlocked.

Google Authenticator LibPam

The next part is the part that I found a bit more tricky as I am not as good with PAM configuration. However, there are a lot of resources online about it. I used this page from the ]]> https://foxide.xyz/projects/2023-07-18.html https://foxide.xyz/projects/2023-07-18.html Tue, 18 Jul 2023 00:00:00 -0400

Abstract

I have been wanting to learn more about coding and DevOps recently. While I am not currently very good at either right now, I certianly have the tools to learn more and improve. This led me to wanting to make a project that assists in creating and managing virtual machines (VMs) and containers on my home server that runs FreeBSD. While it is easy enough to SSH into the box and do anything that I need to, it would be nice to have a web interface that I can login to and manage things that way. It would be even nicer to have one click app deployments similar to what a VPS provider like Linode can do. So, let’s try to build one.

Ideas

Obviously this is not going to be a project that can be done, at least not for me, in one blog post. There is quite a bit of learning that I will need to do, and a few different kinds of problems that I will need to solve. That is on top of all the design and architect decisions that need to be made before actually creating it. I do have a few things in mind that I would like to work towards:

• Runs on FreeBSD: This one probably seems obvious as we are targeting Bhyve (FreeBSD virtualization platform) and jails (FreeBSD container platform), but in my mind it is good to specify the target OS(s) in a project, as it wouldn’t make sense to target FreeBSD if I am writing a .NET app.
• As few dependencies as possible: While there is nothing wrong with depending on other packages and software (to some degree this is unavoidable), I also do not want to have Gigabytes of required packages to run this app. I know it is annoying for me when I am downloading new tools to try out, and I assume it is not any different for anyone else in the world.
• One Click Apps: I really like the idea of being able to press a button and deploy a database or a new Git server. I am not really sure how I am going to accomplish this goal, but I am thinking things like Ansible, Chef, or Salt Stack would probably be the way to go. It would also be cool to allow for custom one-click-apps.

These goals feel rather ambitious right now, but this is more of a learning project than anything else. If I fail, hopefully I learned something. If I don’t fail, maybe I will have made something really cool.

Setting Up Dev Environment

Before I really get started trying to do any coding, I need to get an environment set up to work in. Since I am targeting for FreeBSD, I will need a FreeBSD box to test in. So I will be setting up a FreeBSD 13.2 VM with nested virtualization enabled. Enabliing nested virtualization is pretty easy to get working, assuming your hardware supports it:

# For intel processors
# Unloads KVM kernel module
$ sudo modprobe -r kvm_intel
# and re-loads it with nested virtualization enabled
$ sudo modprobe kvm_intel nested=1
# For AMD processors
# Unloads KVM kernel module
$ sudo modprobe -r kvm_amd
# and re-loads it with nested virtualization enabled
$ sudo modprobe kvm_amd nested=1

This change can be made permanet by editing /etc/modeprobe.d/kvm.conf. Just add either options kvm_intel nested=1 on Intel systems, or options kvm_amd nested=1 on AMD systems.

From there, just install FreeBSD 13.2 as normal and start hacking away on the project. I will be giving updates on the project as enough interesting stuff to write about comes up.

Abstract

When starting to use FreeBSD, one might become interested in the hypervisor that FreeBSD uses, Bhyve. There are many guides on how to get started with it such as this ]]> https://foxide.xyz/projects/2023-08-22.html https://foxide.xyz/projects/2023-08-22.html Tue, 22 Aug 2023 00:00:00 -0400

• Abstract Using virtual machines for separating services within a network is great, however, that is not always practical. Running lots of virtual machines can require a lot of resources. Alternatively using containers provide much better density than full virtual machines can. FreeBSD has a great container system called ‘jails’. While using jails is not anything particularly difficult, I want to learn more about using the built in tools for jails rather than ezjail for managing jails.

Like many other great features in FreeBSD, jails are built into the base system. The FreeBSD handbook’s section on Jails is a great resource to get started with jails and is mostly what I referenced to get them set up. There are a few ways to get a jail installation, but I chose to do use the method from the Internet. Running the following command as root will download all of the files needed for a jail to a directory located at /usr/jails/example:

bsdinstall jail /usr/jails/example

The script will download all of the files needed and also ask for some configuration from the user for things like root password and users for the system. After the script runs the system is downloaded and ready to be configured in the /etc/jail.conf file.

It is also possible to install a jail from a FreeBSD ISO or from FreeBSD’s source, however, it is also worth noting that jails cannot run a newer version of FreeBSD than the host. For example, a FreeBSD 12.4 host can run a FreeBSD 11.4 jail, but not a FreeBSD 13.2 jail.

Configuration

Basic configurations for the jails are not difficult to read or write. The example included here is for a jail named “example” located at /usr/jails/example.

# Configuration for exaple jail
# Name of jail on top line. This is how the jail will be referenced
# from rc scripts
example {
# Hostname of jail, can be anything
host.hostname = example.home;
# loopback address of jail if desired
ip4.addr += "lo0|127.0.1.1";
# Network address
ip4.addr += "vtnet0|192.168.122.22";
# Path to jail directory
path = "/usr/jails/example";
# Mount devfs so things like SSH works inside of the jail
mount.devfs
# Startup scripts
exec.start = "/bin/sh /etc/rc";
# Shutdown scripts
exec.stop = "/bin/sh /etc/rc.shutdown";
}

Getting the networking for the jails to be able to access the Internet took a little research, however, the issue was that I did not assign a network Interface to the jails network address. I found that information from this article that goes into more depth about networking within jails.

Using the jails

From here we are ready to start the jail by running the following command as root:

# to start the jail
service jail onestart example
# to stop the jail
service jail onestop example

If you are wanting the jails to start automatically upon system boot, simply add jail_enable="YES" to /etc/rc.conf. From there the above command can be changed to:

# to start the jail
service jail start example
# to stop the jail
service jail stop example

The jls command will show the current running jails and their associated Jail ID (JID). Running a command within a jail is as easy as running jexec ${JID} ${PATH_TO_COMMAND} as root. For example, getting a shell on a jail with a JID of 3 would look like:

# Make sure to run as root
jexec 3 /bin/sh

You are then greeted to a very standard FreeBSD userland that can be used for testing or running services.

More resources

As alluded to in an earlier section of this article, FreeBSD jails are a widely covered topic within the FreeBSD community because of how useful they are. One person that has a lot of information about FreeBSD’s jails, and many other BSD related topics, is Michael W Lucas. He has written a very good book called FreeBSD Mastery: Jails He also gave a talk at BSDCan in 2019 about jails: Michael W Lucas: Twenty Years in Jail. Both of those resources would be really good places to start if you need more information than what I have provided here concerning FreeBSD’s jail system.

Abstract

This is a quick post about some interesting and useful Android apps that I’ve found on F-Droid. Some of these apps may help people fix something they didn’t realize was broken, and some of them are just to kill some time.

Life and Productivity

1. More Days

More days is an app that helps track habits and goals. You categorize various activities, then quantify how much of each category of activity you need on a daily basis. More Days will then help keep track of how “full” your day was based around how many of those types of activities were done. When the day is finished, you can journal about your day tracking your mood and giving yourself an affirmation for the day. A graph for your mood will be generated over time to show data about how you believe you have been feeling recently. Finally you can also measure things like sleep quality with the ‘scale’ feature. This is a simple rating system for measuring things like sleep quality or productivity throughout the day.

2. Orgzly

Orgzly is the best way I have found to use Org-agenda on Android. The app can work with an existing .org file, or it can also generate one itself. It supports using Dropbox and also has git syncing as an available option, though is not technically supported.

3. Saber

Saber is a notes taking app for hand written notes. It is great if you have a stylus to write on your screen with, and it is also available on other platforms such as Linux. Much of the app is configurable including the background for the notes so the notes can look like they are on blank paper, ruled paper, or grid paper.

Honorable mention: LifeHQ

LifeHQ is a journaling app taken to the next level. By default, it will send you a notification at 9:00 and 21:00 for planning your day and reviewing your day. It will have you put down how you are feeling at that point, and choose an affirmation that fits the mode. It also has a proper journal built in. I stopped using it because I don’t like journaling on my phone, and I don’t feel like the app was helpful for me personally, however, I could definitely see someone getting use from it.

UI and Keyboards

1. Olauncher Clutter Free

Olauncher is a clean but minimal launcher. It is a bit more interesting than a black screen with some apps on the home page and an app drawer, however, it does not do much else. For example, it does not support widgets. The home screen has a clock and a configurable amount of apps. It also has an app drawer and will support actions from different gestures being input on the home screen. I personally like the clean and minimal look, however, some people may find it cumbersome to use or may just lack features.

2. 8-Vim

8-vim is a re-invention of phone keyboards. For 8-vim, you don’t press a key to get a letter, but rather you perform a gesture based around a circle. The letter that you type is dependent on which direction and how far around the circle you slid your finger. It is a bit cumbersome to use sometimes, but other times it makes typing feel very easy. It does have some configuration options like how big the circle should be, and will even support custom layouts now. If you like trying different methods of IO this is definitely one to try.

3. Thumb-key

Thumb key is another interesting keyboard design. The letters all reside on nine keys that give different letters based around how the button was used. For example, the letter ‘a’ is on the right and will appear if the button on the right is just pressed; however, if that button is pressed then swiped to the left the letter ‘l’ will appear. Thumb-key makes for a very good one handed keyboard and supports multiple languages.

Honorable Mention: Posidon launcher

Posidon launcher is a more standard and feature rich launcher than Olauncher. It supports pages of apps, and widgets as well as having gesture support. It also had a clean design, however, it did get to a point where it had some stability issues and would crash. It might be different now, but that is one of the reasons I switched to a different launcher.

Games and fun

1. Lichess

Lichess.org is one of the most popular chess websites online, and their app is available on F-Droid. Lichess offers a wide variety of puzzles and lessons to help you improve your chess skills as well as a variety of chess formats.

2. Drinkable

Drinkable is an app that has recipes for alcoholic beverages. You can search for drinks with different ingredients or search for a drink that you like and find what you need for it. You can even add your own recipes to the app for drinks that you like making yourself. It is a good resource to have on hand for a party, or for going to the store if you are wanting to make drinks at home.

3. 0m

0m is a simple meditation timer that has either guided or unguided meditation. The guided meditation is there to help people that are not familiar with meditation get a good foundation for meditating. The unguided meditation, however, is essentially just a timer that begins and ends with three loud chimes. There isn’t much to this app, but it was really nice to have an app that can help you meditate.

Honorable Mention: Drum On!

Drum On! is a fun app for creating drum beats. It has a configurable tempo with several sounds to choose from to add to the track. The beats can even be exported and sent to another device. I don’t use it a lot, but I do think it is fun to create a simple trap or drum beat from time-to-time.

Abstract

“CryptPad is a collaborative office suite that is end-to-end encrypted and open-source.” - CryptPad’s website

I’ve been hosting a CryptPad instance for a few years, but have been using it a lot more recently for things like notes and calendaring as it is easier than keeping everything synced between devices. It operates a lot like Office 365 or Google Drive’s office suite, however, CryptPad is self-hosted, open source, and puts privacy first.

Setting up a CryptPad instance is not particularly difficult depending on what your security concerns are. For example, my original CryptPad instance was self-hosted on my LAN inside of a FreeBSD jail. The concern for security and was less than if my instance would have been public, so, I just got it working rather than configuring it to production quality. However, for a public instance, there is a lot more tweaking involved to make sure the server itself is secure and has as few holes as possible for attackers. In this post I will be going over my experience with both an “insecure” setup and a “production” setup. The insecure setup is not recommended, but might make the most sense for someone only concerned about having access to it on their LAN rather than across the Internet. For a public CryptPad instance available on the Internet, please do your research. CryptPad has a very good guide on setting up a public instance for CryptPad, before trying anything in this blog post, start there. This post is more so recounting some issues that I had (that were likely self-imposed).

Insecure CryptPad setup

My “insecure” CryptPad instance is running inside of a FreeBSD jail, so all of the commands will assume FreeBSD. However, this should not be terribly difficult to translate into Linux (s/pkg/apt/g and s//usr/local/${dir}/\/${dir}/g should suffice in most cases). When setting up any new server, making sure everything is up-to-date is highly important. After that, install Nginx, node20, npm-node20, and git.

# Run the following as root
pkg update && pkg upgrade
pkg install -y git nginx node20 npm-node20

Note: Nginx is the recommended web server for use with CryptPad. Other web servers will work, but the documentation will references Nginx configs which might make getting the setup complete easier

Then download CryptPad and install dependencies.

git clone https://github.com/cryptpad/cryptpad.git cryptpad && cd cryptpad
npm install
npm run install:components

From here, copy the example config file by running cp config/config.example.js config/config.js then run node server to start CryptPad. While CryptPad will technically run like this, it will not actually be useful until modifying config/config.js. Within the config file, there are some variables that must be changed for CryptPad to actually be accessible from other machines on the network. The first set of variables that must be changed are: httpUnsafeOrigin and httpSafeOrigin. What these variables do is not particularly important in this section, however, will become much more important during the “Production” section of this post. For now, just set the variable to whatever domain name that the instance will be used from (i.g. localdocs.internal). The next configuration item that must be changed is httpAddress. By default it only listens on localhost and should be changed to whatever the IP address of the server is going to be. Alternatively, it can listen on all ports, but that creates much more attack surface on the server than necessary. All of the other defaults should be fine to get CryptPad working; though it is worth reading through the different configuration options available.

Next we configure the web server to be able to proxy the correct ports. Nginx has some configuration examples in the default config file, but it is just as easy to add another server block to do the proxy. The following example assumes that the domain name being used for the CryptPad instance is ‘localdocs.internal’, the IP address of the server is 192.168.1.100, and that the ‘httpPort’ in CryptPad’s config file has remained unchanged.

server {
    listen: 80;
    server_name localdocs.internal;

    location / {
        proxy_pass http://192.168.1.100:3000;
    }

# If this is not set, then CryptPad will not load properly,
# and the application will be unusable
    location = /cryptpad_websocket {
        proxy_pass http://localhost:3000;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection upgrade;
    }

}

Then just make sure that Nginx is enabled by adding nginx_enable="YES" to /etc/rc.conf, then start by running service nginx start as root. From there, run node server in the cryptpad directory, then attempt to visit the page at localdocs.internal. Assuming it loads, the last thing to do is to make CryptPad run as a daemon. This can be done by using the daemon utility in FreeBSD, or by using the example rc.d file to create a service.

Production CryptPad setup

The production instance is similar to the insecure instance, however, you will also need to have a domain with an SSL certificate. I used certbot for this as it is quick and easy while also costing nothing (Go EFF!). Also for this, we must have two domains (at least subdomains) to help prevent XSS attacks on the CryptPad instance. This section of the blog post will be in more general terms as there is quite a bit of information that will vary from instance to instance, plus the example Nginx config does quite a good job of documenting what everything does. The first thing to do is to make sure that each of the domain names is registered so that they point to the server, and thus can get an SSL cert.

# All commands must be run as root

# Install certbot
pkg install py-certbot-nginx
# Then get the certificates for the domains
certbot --nginx
# Alternatively you can just get the certs,
# but it will be more work
certbot certonly --nginx

The next step is to copy the example Nginx config file into the current Nginx config file so it can be modified to fit the instance we are trying to set up. While there are likely 100 ways to do this in FreeBSD, the way that made sense to me was to fetch the file, then as root run cat example.nginx.conf >> /usr/local/etc/nginx/nginx.conf. This command will copy the example.nginx.conf file to the end of the current nginx config file. It can then be opened in ${EDITOR} and the server block for CryptPad can be moved to a more suitable location and the editing process can begin. Again, this file will vary from person to person a bit, and there will be a bit of trial and error to get the configuration correct. The ‘main_domain’ is the domain (or subdomain) that the site will be accessed from. The ‘sandbox_domain’ is a secondairy domain that is used strictly as a security measure against XSS attacks. There will also be some SSL items that will have to be commented out as the example file has some SSL configuration that should have been handled already if you installed your certs with it.

Once nginx is configured, enabled, and started, we can then start working with CryptPad itself to get it configured and working. First, as in the other section of this post, copy the config.example.js to config.js in the config folder in CryptPad. Then open config.js in ${EDITOR} and begin configuring it for your instance. Change httpUnsafeOrigin to the same as the main_domain in the Nginx config file, change httpSafeOrigin to the same as the sandbox_domain in the Nginx config file. Most everything else can be left as default, however, it is worth a look at the rest of the config file as there are some other useful configuration items in there. From there, run node server and attempt to access the page. If it loads and you are able to login and access the drive or another app, the configuration works. Now just use the rc.d script to make CryptPad a service and manage it as such.

More Documentation

CryptPad is a very well documented project with a lot of helpful resources to get started. Because of that, if you are interested in setting up your own public instance, I recommend starting there rather than this documentation. These pages are good for starting out with CryptPad and asking for help.

• Basic User Guide
• Installation Documentation
• CryptPad Support Forum

Abstract

Recently I heard about Super Dimension Fortress (SDF) on IRC and thought it was a very neat project, so I thought I would write about it. There are a lot of services offered by the group in addition to the public access Unix system such as Git, email, web and gopher hosting, and a radio station. A list of other projects that the group has is available on their website here.

What is SDF

Super Dimension Fortress (SDF) is a community of people in various roles in the Free Software world. From this page on sdf.org:

Our mission is to provide remotely accessible computing facilities for the advancement of public education, cultural enrichment, scientific research and recreation. Members can interact electronically with each other regardless of their location using passive or interactive forums. Further purposes include the recreational exchange of information concerning the Liberal and Fine Arts.

Uses for SDF

When I first heard about SDF, I was curious what the system could be used for barring education purposes. It is easier to teach someone how to use Unix with a Unix or Unix like system, and installing Linux or a BSD is sometimes difficult for people. Setting them up with an SSH connection to a Unix system seems much simpler to me. However, for people that already use a Unix-like OS, what would be the benefit? Well SDF actually already has an answer to that question; typing ‘what’ into the prompt after logging in will output the following:

The SDF is comprised of many computer systems that you may use remotely. These are not computers like your PC, but specialised systems with fast processors, sophisticated memory management and several mass storage disk arrays.

So what can you do here, especially with a prevalidated account?

YOU CAN DO A LOT HERE .. START READING –> create/browse ‘gopherspace’, browse ‘webspace’, store files locally with ZMODEM, send/receive email (pop3/webmail available too), use icq/aol IM, chat locally, read/post to the SDF bboard, run whois, ping, traceroute, nslookup, dig, geoip and other network utilities, multiuser games (LOTS OF GAMES), navigate the UNIX filesystem and much more!

Since SDF is funded by its members, you are rewarded with membership features at various levels for small, one time donations. To get an over view of the membership levels available, type ‘how’. You can interact with thousands of other users on our systems with ‘com’ and ‘bboard’. If you want to read answers to questions that have been asked, type ‘faq’. You’ll notice that when you type ‘help’ some commands bring up another menu of commands. Feel free to try all the commands as you explore SDF and the internet.

Explore & Enjoy!

There are a load of different things that SDF provides without any cost to a user. Additional utilities and software can be accessed and used by paying a one time $36 fee for ARPA membership. This will grant things such as webmail and developer tools on the system. Overall this project is one that I don’t feel I am doing justice to by writing about, however, I thought was a very cool project. Visit the website for more information at sdf.org.

Abstract

Recently at work I was asked about a project called tiny11, which is a minimalist Windows 11 ISO. While it is a very cool project, I did not think it was a good idea to use third party ISOs in production as we could not verify the integrity of the ISOs and make sure they were not malicious. However, we could certainly see about making our own ISOs and attempting to match what tiny11 does, while making modifications that fit our needs a bit better than tiny11 does.

About Tiny11

For anyone that is not familiar with the project, tiny11 is an attempt to remove as much of the bloat and unnecessary software that is included in Windows 11 such as Spotify, Facebook Messinger, and Instagram. A basic overview of the project can be found on tomshardware.com. However, tiny11 goes even farther by removing things like the built-in email client, calendar, and Edge. This amount of software removal is great for someone wanting a super minimalist Windows 11 installation, however, I was asked to look at this for deploying workstations at my job and our team does not actually want to remove things like Edge or OneDrive. In addition to that, I did not think it was a good idea to use a third party ISO. Thankfully, the creator of tiny11 posted a nice script on GitHub that automates and documents the process so it is easy to make your own custom ISOs of Windows.

Creating Trimmed Windows ISOs with tiny11builder

Starting the process of creating a custom Windows image is not particularly difficult and only requires two things: A Windows ISO (Windows 11 in this case) and a working Windows installation. Thankfully, the official Windows 11 ISO is available on Microsoft’s Website, from there you can create a Windows 11 VM, or install Windows 11 on a physical computer to continue with the process.

Note: If you are running Windows you will likely have to change your user agent in your browser to be given an option to download a Windows ISO. Microsoft prefers users using their tool to create Windows installation media.

Then simply go to GitHub and download the scripts, that can be done either by downloading the zip file, or running a git clone on this repo. Taking a look at the tiny11creator.bat script shows that the DISM tool is the star of the show. Not only removing a lot of the bloat from Windows, but also simple things like just mounting the image to be modified. Unfortunately the script will likely not work as is though; the package names that DISM uses to remove software change with each Windows release. So, if the Windows ISO downloaded does not match the version used in this script, it will not actually remove the software from the image. There is a solution to this though; simply mount the image and ask DISM for the software installed on the image. That can be done by doing the following:

1. Mounting the downloaded Windows ISO in File Explorer
2. Open an Administrative command-prompt to run the following lines

:: Directory to copy Windows image filess/folders to
md C:\WindowsImage

:: Copies Windows image located at %DriveLetter% to C:\WindowsImage
xcopy.exe /E /I /H /R /Y /J %DriveLetter% c:\WindowsImage

:: Shows the different versions of Windows that can be worked with
:: (home, pro, enterprise, etc) We will use Pro which is 6 by default
dism /Get-WimInfo /wimfile:C:\WindowsImage\sources\install.wim

:: Creates a scratch directory to mount the image in
:: Note, we are mounting the copy to not mess up the original ISO
dism /mount-image /imagefile:C:\WindowsImage\sources\install.wim /index:6 /mountdir:C:\scratchdir

:: From here we use DISM to ask about package information on the image
:: It is a lot of data, and would probably be worth copying/pasting into a text file
:: for later use

:: Retrives ProisionedAppxPackages
:: Copy the output into a text file to keep for later
dism /image:C:\scratchdir /Get-ProvisionedAppxPackages

:: Retrives System packages
:: Copy the output into a text file to keep for later
dism /image:C:\scratchdir /Get-Packages

:: Unmounts the image
dism /unmount-image /mountdir:C:\scratchdir /commit

This process will not only give the packages that are come installed in the ISO, but more importantly give us the package names so that we can use DISM to remove them. Make a copy of the tiny11 creator.bat script in the repo and open it in your favorite text editor. This is the most tedious part of the process, and where a choice of text editor will potentially save a lot of time, replace the package name in the script with the package name you retrieved from the Windows ISO you downloaded. If the package name in the script does not match what is in the Windows image exactly, DISM will not remove it and the package will remain on your computer. Once that is done, do the same thing for the regular packages in the script. There may be some packages that you would like to keep, for those just comment the line out by putting a :: at the beginning of the line; Some of the package names are not very clear, so using the echo above the DISM command will help give an idea of what is being removed. You can also comment out anything else that you may not want to modify, for example, our team does not actually want to remove the hardware requirements from Windows 11, and we do not need to enable the ability to use a local account as it is relatively easy to bypass the online account without the registry modifications. You can either type user and the username and password for the online account, or you can create a provisioning package that creates a local user for you. Lastly you may want to modify the ISO creation line as when I ran it, Windows did not understand %~dp0 at the beginning of the line, so I replaced it with .\ and changed the location of the ISO to C:\temp\tin11.iso. Now that we have our script that matches with the Windows ISO we downloaded, we can follow the instructions in the readme.md file in the repo, and run our script instead of the tiny11 builder.bat, following the instructions in the script and waiting for a while, we will get an ISO image for Windows 11. After several iterations of testing, I have gotten Windows 11 to run happily on 4GB of RAM and with 2 virtual cores. While it’s not quite as impressive as the original author of the script, it fits my companies needs much better than his version of tiny11 does.

Securing and Hardening SSH servers

SSH is the tool that many users use to manage and interact with their Unix systems. It is a vital protocol that must be protected, as SSH being compromised can be detrimental. Thankfully there are many tools and guides to help people secure their SSH servers. The tool that we will be focusing on today is ssh-audit. It is a tool that can analyze both SSH clients and servers to show insecurities and places that SSH could be stronger. For publicly accessable servers, they even have a tool on their website that will grade your SSH server security.

Most of the knowledge in this post comes from Özgür Konstantin Kazanççı’s website, and really is just re-stating it for my own memory and use.

FreeBSD

Start with making sure the system is updated (and please upgrade to FreeBSD 14.0 if you have not already done so. Here is a guide if one is needed).

pkg update
pkg upgrade
pkg install ssh-audit

Running ssh-audit localhost will likely make you feel bad about deploying a server in such a state. That’s okay, that is what we are currently in the process of trying to fix. The first step is to regenerate SSH host identification keys for higher security keys:

ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""

Then we need to generate our own moduli. The ones that come with most operating systems are often too short, and are vulnerable to attack. In some cases, they are just re-used and are even vulnerable to a pre-computation attack, so regenerating them is vital. Keep in mind, this process can take a very long time depending on the hardware that is generating the moduli (it took over an our on the test VM for the next section).

ssh-keygen -M generate -O bits=3072 moduli
ssh-keygen -M screen -f moduli moduli-final
mv moduli-final /etc/ssh/

Next we need to set some configuration options by putting the following lines into our sshd_config file (located at either /etc/ssh/sshd_config or /usr/local/etc/ssh/sshd_config depending on whether you are using the built in SSH daemon or downloaded another one from pkg or ports).

HostKeyAlgorithms rsa-sha2-512,rsa-sha2-256,ssh-ed25519

KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com

These options restrict which kind of key exchange alrogithms and encryption algorithms can be used for our SSH host.

Finally, restart the SSH server for the changes to take effect, however, if you are managing the machine from an SSH session, do not close it until you verify that the new settings work correctly by attampting to connect from another SSH session.

service sshd restart

Void Linux

The process for hardening Void Linux’s SSH server is very similar to the previous section, however, it is worth documenting the process in my opinion.

We start by making sure our operating system and all of our binary packages are up-to-date, as well as downloading ssh-audit to see how well or how poorly SSH is currently configured.

xbps-install -Syuv
xbps-install -Sy ssh-audit

Now we run ssh-audit to see whether our SSH server is configured well or not. Note that the IP address of the SSH server I used for this was 192.168.122.27; also the -n flag was to turn off color output to paste into this article better, and will not necessarily be needed for all cases.

ssh-audit -n 192.168.122.27
# general
(gen) banner: SSH-2.0-OpenSSH_9.6
(gen) software: OpenSSH 9.6
(gen) compatibility: OpenSSH 8.5+, Dropbear SSH 2018.76+
(gen) compression: enabled (zlib@openssh.com)

# key exchange algorithms
(kex) sntrup761x25519-sha512@openssh.com    -- [info] available since OpenSSH 8.5
(kex) curve25519-sha256                     -- [info] available since OpenSSH 7.4, Dropbear SSH 2018.76
                                            `- [info] default key exchange since OpenSSH 6.4
(kex) curve25519-sha256@libssh.org          -- [info] available since OpenSSH 6.4, Dropbear SSH 2013.62
                                            `- [info] default key exchange since OpenSSH 6.4
(kex) ecdh-sha2-nistp256                    -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp384                    -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp521                    -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) diffie-hellman-group-exchange-sha256 (3072-bit) -- [info] available since OpenSSH 4.4
                                                      `- [info] OpenSSH's GEX fallback mechanism was triggered during testing. Very old SSH clients will still be able to create connections using a 2048-bit modulus, though modern clients will use 3072. This can only be disabled by recompiling the code (see https://github.com/openssh/openssh-portable/blob/V_9_4/dh.c#L477).
(kex) diffie-hellman-group16-sha512         -- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) diffie-hellman-group18-sha512         -- [info] available since OpenSSH 7.3
(kex) diffie-hellman-group14-sha256         -- [warn] 2048-bit modulus only provides 112-bits of symmetric strength
                                            `- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) ext-info-s                            -- [info] pseudo-algorithm that denotes the peer supports RFC8308 extensions
(kex) kex-strict-s-v00@openssh.com          -- [info] pseudo-algorithm that denotes the peer supports a stricter key exchange method as a counter-measure to the Terrapin attack (CVE-2023-48795)

# host-key algorithms
(key) rsa-sha2-512 (3072-bit)               -- [info] available since OpenSSH 7.2
(key) rsa-sha2-256 (3072-bit)               -- [info] available since OpenSSH 7.2
(key) ecdsa-sha2-nistp256                   -- [fail] using elliptic curves that are suspected as being backdoored by the U.S. National Security Agency
                                            `- [warn] using weak random number generator could reveal the key
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(key) ssh-ed25519                           -- [info] available since OpenSSH 6.5

# encryption algorithms (ciphers)
(enc) chacha20-poly1305@openssh.com         -- [info] available since OpenSSH 6.5
                                            `- [info] default cipher since OpenSSH 6.9
(enc) aes128-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes192-ctr                            -- [info] available since OpenSSH 3.7
(enc) aes256-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes128-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes256-gcm@openssh.com                -- [info] available since OpenSSH 6.2

# message authentication code algorithms
(mac) umac-64-etm@openssh.com               -- [warn] using small 64-bit tag size
                                            `- [info] available since OpenSSH 6.2
(mac) umac-128-etm@openssh.com              -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-256-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-512-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha1-etm@openssh.com             -- [fail] using broken SHA-1 hash algorithm
                                            `- [info] available since OpenSSH 6.2
(mac) umac-64@openssh.com                   -- [warn] using encrypt-and-MAC mode
                                            `- [warn] using small 64-bit tag size
                                            `- [info] available since OpenSSH 4.7
(mac) umac-128@openssh.com                  -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 6.2
(mac) hmac-sha2-256                         -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 5.9, Dropbear SSH 2013.56
(mac) hmac-sha2-512                         -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 5.9, Dropbear SSH 2013.56
(mac) hmac-sha1                             -- [fail] using broken SHA-1 hash algorithm
                                            `- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 2.1.0, Dropbear SSH 0.28

# fingerprints
(fin) ssh-ed25519: SHA256:0etkEayA+4dGscclMee05Z18EDlB4TH3lyRZ8zD1ORs
(fin) ssh-rsa: SHA256:brHIQBb1RqGrM//hj7PKVjzMlCV/eoCHbloyyx5srMs

# algorithm recommendations (for OpenSSH 9.6)
(rec) -diffie-hellman-group14-sha256        -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp256                   -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp384                   -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp521                   -- kex algorithm to remove 
(rec) -ecdsa-sha2-nistp256                  -- key algorithm to remove 
(rec) -hmac-sha1                            -- mac algorithm to remove 
(rec) -hmac-sha1-etm@openssh.com            -- mac algorithm to remove 
(rec) -hmac-sha2-256                        -- mac algorithm to remove 
(rec) -hmac-sha2-512                        -- mac algorithm to remove 
(rec) -umac-128@openssh.com                 -- mac algorithm to remove 
(rec) -umac-64-etm@openssh.com              -- mac algorithm to remove 
(rec) -umac-64@openssh.com                  -- mac algorithm to remove 

# additional info
(nfo) For hardening guides on common OSes, please see: <https://www.ssh-audit.com/hardening_guides.html>
(nfo) Be aware that, while this target properly supports the strict key exchange method (via the kex-strict-?-v00@openssh.com marker) needed to protect against the Terrapin vulnerability (CVE-2023-48795), all peers must also support this feature as well, otherwise the vulnerability will still be present.  The following algorithms would allow an unpatched peer to create vulnerable SSH channels with this target: chacha20-poly1305@openssh.com.  If any CBC ciphers are in this list, you may remove them while leaving the *-etm@openssh.com MACs in place; these MACs are fine while paired with non-CBC cipher types.

Much like FreeBSD, there is a lot of red and yellow in the default configuration. That was to be expected, however, can be fixed by running a few commands.

First we need to change the host identification key. The following commands will change the SSH server’s remote identification, and SSH will give you an error saying “IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!” This error is occuring because we are removing the weak host identification key in favor of generating a stronger one.

cd /etc/ssh/
# Removes weak host keys
rm -f ssh_host_*
# Generates stronger RSA host key
ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N ""
# Generates stronger ed25519 key
ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N ""

Next, we will generate our own SSH moduli; this is a critical SSH hardening step as many operating systems ship pre-computed moduli that is too short and potentially vulnerable to attacks. A minimum of 3072-bit modulus is needed for providing 128 bits of security.

This process will potentially take a while depending on the specs of the computer it is running on.

ssh-keygen -M generate -O bits=3072 moduli
ssh-keygen -M screen -f moduli moduli-final
mv moduli-final /etc/ssh

Finally add the following lines to /etc/ssh/sshd_config

HostKeyAlgorithms rsa-sha2-512,rsa-sha2-256,ssh-ed25519

KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com

and restart the sshd service by running:

sv restart sshd

Now, re-running ssh-audit:

ssh-audit -n 192.168.122.27
# general
(gen) banner: SSH-2.0-OpenSSH_9.6
(gen) software: OpenSSH 9.6
(gen) compatibility: OpenSSH 7.4+, Dropbear SSH 2018.76+
(gen) compression: enabled (zlib@openssh.com)

# key exchange algorithms
(kex) curve25519-sha256                     -- [info] available since OpenSSH 7.4, Dropbear SSH 2018.76
                                            `- [info] default key exchange since OpenSSH 6.4
(kex) curve25519-sha256@libssh.org          -- [info] available since OpenSSH 6.4, Dropbear SSH 2013.62
                                            `- [info] default key exchange since OpenSSH 6.4
(kex) diffie-hellman-group16-sha512         -- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) diffie-hellman-group18-sha512         -- [info] available since OpenSSH 7.3
(kex) diffie-hellman-group-exchange-sha256  -- [info] available since OpenSSH 4.4
(kex) ext-info-s                            -- [info] pseudo-algorithm that denotes the peer supports RFC8308 extensions
(kex) kex-strict-s-v00@openssh.com          -- [info] pseudo-algorithm that denotes the peer supports a stricter key exchange method as a counter-measure to the Terrapin attack (CVE-2023-48795)

# host-key algorithms
(key) rsa-sha2-512 (4096-bit)               -- [info] available since OpenSSH 7.2
(key) rsa-sha2-256 (4096-bit)               -- [info] available since OpenSSH 7.2
(key) ssh-ed25519                           -- [info] available since OpenSSH 6.5

# encryption algorithms (ciphers)
(enc) chacha20-poly1305@openssh.com         -- [info] available since OpenSSH 6.5
                                            `- [info] default cipher since OpenSSH 6.9
(enc) aes256-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes128-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes256-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes192-ctr                            -- [info] available since OpenSSH 3.7
(enc) aes128-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52

# message authentication code algorithms
(mac) hmac-sha2-256-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-512-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) umac-128-etm@openssh.com              -- [info] available since OpenSSH 6.2

# fingerprints
(fin) ssh-ed25519: SHA256:Q2qtOTp16c67r2gUCwQoft08Ze2c/hWZU3lBedWK3pY
(fin) ssh-rsa: SHA256:eAQoG/aPar0Eyy5II0/YrPx89bYGJL9hQouZ5JA7KP8

# algorithm recommendations (for OpenSSH 9.6)
(rec) +sntrup761x25519-sha512@openssh.com   -- kex algorithm to append 

# additional info
(nfo) Be aware that, while this target properly supports the strict key exchange method (via the kex-strict-?-v00@openssh.com marker) needed to protect against the Terrapin vulnerability (CVE-2023-48795), all peers must also support this feature as well, otherwise the vulnerability will still be present.  The following algorithms would allow an unpatched peer to create vulnerable SSH channels with this target: chacha20-poly1305@openssh.com.  If any CBC ciphers are in this list, you may remove them while leaving the *-etm@openssh.com MACs in place; these MACs are fine while paired with non-CBC cipher types.

Now if running with color enabled, we see a lot more green and no red with only one yellow entry that is just reminding us that both the SSH client and server must be secured to fully protect our servers.

Hardening SSH clients

Unfortunately, hardening the SSH server is not enough to mitigate all attack vectors. Some attacks only need the SSH server or SSH client to be improperly configured to be vulnerable to some attack. Thankfully hardening your SSH client for your current user can be done with the following command:

printf "\nHost *\n  Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr\n  KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256\n  MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com\n  HostKeyAlgorithms ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com\n" > ~/.ssh/config

This command works both on FreeBSD and Void Linux; I imagine it would work on many other Unix-like operating systems as well.

Abstract

Git is an extremely valuable tool for devs and admins to be able to manage changes to their configs and code. There are many options for Git hosting, however, many options that people hear about are proprietary with little to no self-hosting options. While there isn’t anything wrong with that per se, self-hosting is a better option in my opinion. So, in this post we are going to look at using Forgejo (a fork of Gitea made by the folks at codeberg.org) to self-host Git services with many of the same features offered by something like GitHub or GitLab. Gitea is another product that would work, however, there have been some disagreements between some of the former Gitea maintainers and the company behind the Gitea project, Gitea Ltd. Additionally, setting up Gitea is well documented on their website, here.

The host for these notes is FreeBSD 14.0-RELEASE-p3 and has an IP address of 192.168.122.11.

Hosting with Forgejo

As of the time of writing, Forgejo is not available via pkg in FreeBSD, but it did not seem particularly difficult to compile myself. Really the only things that are needed to be installed are: Git, Go (version 1.21 or newer), and npm 20. So, here is what the process for installing the dependencies and compiling Forgejo look like:

# Always a good idea to make sure packages are up-to-date
sudo pkg update
sudo pkg upgrade
# You must use go121, as the standard go package is not a new enough version
sudo pkg install npm-node20 go121 git
# Symlinking go121 to go is required to pass the go check and make
# Forgejo recognize that go is installed
sudo ln -s /usr/local/bin/go121 /usr/local/bin/go
# Clone the Forgejo repo
git clone https://codeberg.org/forgejo/forgejo && cd forgejo
TAGS="bindata sqlite sqlite_unlock_notify" make build
./gitea

From here, open a web browser to complete the setup. Navigate to the IP address of the server with the port 3000 (e.g. 192.168.122.11:3000). We can use the daemon utility to run the gitea binary in the background using the following command:

daemon -f ./gitea

Alternatively, we can make a FreeBSD service to be able to manage the application better. This is a very basic service file that is based on the service file from installing gitea on a FreeBSD machine. To create the service file simply run touch /usr/local/etc/rc.d/gitea && chmod +x /usr/local/etc/rc.d/gitea as root, then add the following into the file:

#!/bin/sh
# PROVIDE: gitea
# REQUIRE: NETWORKING SYSLOG
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf to enable gitea:
#
#gitea_enable="YES"

. /etc/rc.subr

name="gitea"
rcvar="gitea_enable"

load_rc_config $name

: ${gitea_user:="git"}
: ${gitea_enable:="NO"}
: ${gitea_facility:="daemon"}
: ${gitea_priority:="debug"}
: ${gitea_shared:="/usr/local/share/${name}"}

command="/usr/local/bin/gitea"
pidfile="/var/run/${name}.pid"
start_cmd="${name}_start"

gitea_start() {
  for d in /var/db/gitea /var/log/gitea; do
    if [ ! -e "$d" ]; then
      mkdir "$d"
      chown ${gitea_user} "$d"
    fi
  done

  daemon -T ${name} \
  -u ${gitea_user} -p ${pidfile} \
  $command --work-path ${gitea_shared}
}

run_rc_command "$1"

From there, add gitea_enable="YES" into /etc/rc.conf, then start the service with service gitea start. The setup will have to be re-done unless the service was pointed to the directory of the initial setup or the files were moved to the new directory. Moving forward with this setup, it will just be important to remember to update the software from time-to-time by running git pull origin master and recompiling from time-to-time (probably at least monthly). If and when there is a ports version or a binary, it would be easier to switch to that so that the software updates with the rest of the OS, but until then, this solution will work.

In the next article I will be going over making a more minimalist Git server using Git daemon.

Abstract

This is a second part to my other blog post concerning Git services. On the last post, I showed how to host a Git server with Forgejo; this time, I will be using Git daemon to setup a minimalist Git service.

The host for these notes is FreeBSD 14.0-RELEASE-p3 and has an IP address of 192.168.122.11.

Hosting Git with Git Daemon

The most minimal Git server possible will be just using git-daemon. It is included with Git and will give all of the basic features of a Git repository, however, there are not many features to Git daemon other than being able to sync Git repositories.

  doas pkg install git
  cd ~
  git init --bare repo1.git
  git daemon --reuseaddr --export-all --enable=receive-pack

The repo can then be cloned by running git clone git://192.168.122.11/home/${USER}/repo1.git. At this point, we have an extremely basic and simple Git server. There is no thrills, frills, or even authentication due to the --enable=recieve-pack flag. However, without this flag, at least without more setup, we would not be able to push changes to the repos. That being said, this is what the git-daemon man page says about the receive-pack service:

receive-pack This serves git send-pack clients, allowing anonymous push. It is disabled by default, as there is no authentication in the protocol (in other words, anybody can push anything into the repository, including removal of refs). This is solely meant for closed LAN setting where everybody is friendly. This service can be enabled by setting daemon.receivepack configuration item to true.

So, while this is a way to accomplish the goal of having a remote Git repo, it is not really a good way to do it unless you are very sure no one will be inside your network doing anything nefarious. Git daemon itself does not really have anything authentication built in, however, we can interact with the Git server via SSH (which is how most Git providers work). Doing this is as simple as changing the clone command: git clone ssh://${USER}@192.168.122.11/home/${USER}/repo1.git, or if the repo has already been cloned, changing the remote source works also:

git remote remove origin
git remote add origin ssh://${USER}@192.168.122.11/home/${USER}/repo1.git

    doas pkg install git
    doas mkdir -p /usr/local/git
    git daemon --reuseaddr --base-path=/usr/local/git --export-all --user-path

DragonFly BSD

DragonFly BSD is the often forgotten BSD within the family tree. While it does have some really interesting technologies behind it, such as Hammer2, it just does not have the same popularity as the other BSDs. DragonFly BSD’s main appeal are the performance metrics that it hits compared to any of the other BSDs; and while I do not feel confident enough to attempt to explain why that is, there is a page about performance on their website. It does honestly feel faster too. My installation of DragonFly BSD is running on a 2.5’’ spinning rust drive on a Thinkpad T430; and while it does feel slow, it is almost certainly because of the hard drive. Even with the spinning disk though, it does not feel that slow. I can tell it is not as snappy as any of my machines as an SSD, but that’s not really a fair comparison. It does certainly run better than my FreeBSD install on the same laptop with a similar 2.5’’ spinning rust drive though.

Installation

I had trouble booting into the install media on my T430, and after doing some digging online I tried using the USB 3.0 ports for the install media rather than the USB 2.0 which allowed it to boot. While it’s a strange issue that I wasn’t really expecting, it allowed me to get into the installation of DragonFly. The installation process is a rather straightforward one, especially for users with any experience in Linux or any of the BSDs. When partitioning the disk, I chose the Hammer2 file system as it is one of the big features touted by DragonFlyBSD’s website, however, I did run into an issue while trying to use encrypted partitions for the file system. This was likely an issue on my part as I did not try very hard to correct the issue, and looking back, attempting to encrypt the efi partition likely would have caused issues that DragonFlyBSD may have detected and prevented. I decided to just go with an un-encrypted file system, as I was planning on using the machine to learn more about DragonFly and DragonFly development rather than use it as a primary computer. The rest of the installation process went rather smoothly without any hiccups.

System Configuration

In a lot of ways DragonFlyBSD is similar to FreeBSD and many of the same concepts of running a FreeBSD machine will apply to DragonFly BSD. Initially, the biggest issue was lack of WiFi, which required downloading and enabling wpa_supplicant, then adding the following lines to /boot/loader.conf to load the correct wireless NIC drivers:

if_iwm_load="YES"
iwm3160fw_load="YES"
iwm3168fw_load="YES"
iwm7260fw_load="YES"
iwm7265fw_load="YES"
iwm7265Dfw_load="YES"
iwm8000Cfw_load="YES"
iwm8265fw_load="YES"
iwm9000fw_load="YES"
iwm9260fw_load="YES"

From here, you can configure the NIC in /etc/rc.conf:

wlans_iwn0="wlan0"
ifconfig_wlan0="WPA DHCP"

The next steps for me were installing some standard tool that I like using:

# including intel video driver as it is necessary for X to work
pkg install doas neovim xorg xf86-video-intel
# configure doas
echo "permit persist :wheel" > /usr/local/etc/doas.conf
# I also downloaded my suckless tools as my 'normal' user at this point
mkdir -p ~/.config/suckless && cd ~/.config/suckless
git clone https://codeberg.org/fac3plant/st.git
git clone https://codeberg.org/fac3plant/slstatus.git
git clone https://codeberg.org/fac3plant/dwm.git
git clone https://codeberg.org/fac3plant/dmenu.git
git clone https://codeberg.org/fac3plant/slock.git

I started out only compiling dwm and st as those are the two most important ones that are difficult to work without. Neither of them were particularly difficult after getting the correct libraries installed and making sure the config.mk was set up properly. Finally put exec dwm in ~/.xinitrc.

Then I was able to run xinit normally (xinit was used over startx as the startx command wouldn’t work for some reason), and… the keyboard does not work while X is running. The trackpoint appears to work, and it can click, however, I cannot use the keyboard. I also cannot get out of my window manager as it is keyboard driven, so the only option is a hard reset.

After messing with Xorg configs and installing several keyboard drivers, this blog post gave me the secret sauce for a working keyboard:

pkg install xf86-input-evdev
# I'm not sure why, but the keyboard only works after putting this value in /etc/sysctl.conf
echo kern.evdev.rcpt_mask=3 >> /etc/sysctl.conf

From here, the keyboard had no issues when using X.

More Suckless Utils

Most of the other Suckless utilities compiled without much issues after modifying the config.mk file, which is normally something that has to be done on FreeBSD anyway as many of the library files are located at /usr/local/lib or /usr/local/include rather than /usr/X11R6/lib or /usr/X11R6/include. However, slstatus was rather difficult to get working properly; specifically the components/cpu.c and components/battery.c files were particularly annoying. The trick that made them compile correctly was to modify both files at the line:

// Original line
#elif defined(__OpenBSD__)
// Modified line
#elif defined(__DragonFly__)

From here, the programs seemed to compile properly, and even run. Huzzah! Now we can install the rest of the applications that I like using:

pkg install -y firefox-esr qutebrowser emacs feh mpv groff mupdf

and proceed to configure the applications more or less the same as any other Linux distro or BSD.

Power saving

Battery life is an important aspect to any mobile device for me, so naturally getting the most life out of the battery was appealing. Thankfully there is a section concerning battery life on this blog post that shows how to get some more CPU efficiency. Simply add the following lines to /etc/sysctl.conf:

# CPU States
hw.acpi.cpu.cx.lowest=C3 # This will make the computer run slower, but save more power (fine for my uses)
# hw.acpi.cpu.cx_lowest=C2 # This will make the computer run faster, but consume more power (may be better for some users)

Additionally, adding powerd_enable="YES" to /etc/rc.conf will help with battery life some. Unfortunately, I could not get suspend and resume to work at time of writing.

Building and Installing Custom Kernel

While compiling a custom kernel is completely unnecessary to using the OS, it is something that I have been interested in learning more about. It also has some more practical benefits, such as speeding up the boot process and memory efficiency. Building from source is also not quite as intimidating as it sounds initially.

# Download the source tree
cd /usr/
make src-create
cd /usr/src

# Building world is required first to get the appropriate toolchain
make -j4 buildworld # replace the '4' with the number of threads you have in your CPU
make installworld   # using more than one core is not necessary nor recommended for the install step

# Now build the kernel, just to make sure it will build properly
make -j4 buildkernel # again, replace the '4' to match your machine's thread count
make install kernel  # again, only one core is necessary

# Copy the current kernel config to a new file
cp sys/config/X86_64_GENERIC sys/config/MYKERNEL
# Then make the desired changes
${EDITOR} sys/config/MYKERNEL

# Now build the kernel
make -j4 buildkernel kernconf=MYKERNEL
make installkernel kernconf=MYKERNEL

# You can also use faster build methods
make -j4 quickkernel kernconf=MYKERNEL
make installkernel kernconf=MYKERNEL

# View the Makefile for all of the different build options,
# as it is outlined there in more detail

There is a lot of information in the kernel config and it can be a bit overwhelming, especially for someone new to building kernels from source. However, DragonFly BSD has a page to help explain what some of the options are and what they do. It will still require some thought and knowledge of the hardware that you are using, but it does make it a bit easier to understand the plethora of configuration items in that file.

Further TODOs

The big thing that I would like to get working is suspend and resume. I know that is something that tends not to work well even on Linux, however, I have a FreeBSD install on the same laptop and suspend and resume works quite well. It would be nice to see if I can get it working to some degree in DraognFly to save on some battery power when the lid is closed.

I also could not get the volume up and down buttons to work correctly, which is not really a big deal for me as I have volume up and down bound to a key combination in DWM that I like better anyway, but it would still be worth exploring why that did not work.

Resources

These are some of the resources that I used to solve many of my issues I had with this installation. Most of the issues were straightforward enough to solve, and others took a bit of thinking and playing around with (at least for me). I will likely continue to play around with DragonFly BSD as it is an interesting OS and does appear to perform very well considering the hardware it is installed on.

• DrgonFly on Laptops
• DragonFly BSD on a Thinkpad T480s: This blog post was a good place to get started with configuring my T430 as some of the sticking points were similar.
• DragonFly BSD documentation: Project documentation is always a good resource to keep in mind when working on resolving issues.

Abstract

Some people may be familiar with groff as a typesetting format for man pages, however, groff can also produce PDF documents. These produced documents give a very similar level of control and consistency as something like LaTeX can provide. In this blog post, I will be going over some basics of using groff to produce PDFs though don’t necessarily have a lot of justification as to why one should use it over something like LaTeX or Markdown with pandoc.

Brief History

While groff is the modern program to work with nroff macros, the family tree for groff and related software starts with RUNOFF, which released in 1964 and was one of the earliest text formatting programs. RUNOFF was originally written for the Compatible Time-Sharing System, then was re-written a few times in a few different languages. Fast forward to the 1970’s when Ken Thompson wanted to upgrade the current PDP-7 to a PDP-11 which would be a very expensive upgrade at the time. The justification given to upper management for needing this new system was for word processing, which is where roff came from. It was originally “transliterated” from MAD-assembler to PDP-7 assembler, then transliterated again once they acquired the PDP-11. Some time later, a more flexible language was needed so, nroff or “newer roff” was created and will provide the basis for all future versions of the programs. Another version called Troff was later created to support multiple fonts and proportional spacing. Groff was written in 1990 for the GNU system as a free software implementation of troff, which is still in use today.

Getting started and basics

Working with groff, much like many other typesetting systems, is as easy as opening a text editor and knowing how to run the command to actually convert the text into a document. This article will be focusing on GNU roff (groff) as that is the implementation that I have installed on all of my machines.

This is a very basic document with Groff just to get started and make sure things compile properly. Save the following text into a file called test.ms.

.TL
Test Document
.AU
Tyler Clark
.PP
Hello World!

Then open a terminal and type the following in the same directory as the test.ms file.

groff -ms -Tpdf test.ms > test.pdf

From there, a PDF document should be generated looking as follows:

Now that we have compiled a basic document to make sure that everything is working properly, let’s get into the macros that handle the formatting for the text. groff has a very large variety of macros and even different sets of macros; this post will focus exclusively on the ‘ms’ macros.

.TL
This would be a title

.AU
Author's Name Here

.AB
The abstract section is initiated with the '.AB' and ended with the '.AE'
.AE

.PP
A new paragraph with a serif font is initiated by the '.PP' macro.

.NH 1
Heading
.PP
Headings are initiated by the '.NH' macro with a number following for the level of heading desired.

.NH 2
Subheading
.PP
This heading was followed by a '2' which means that it will be a subheading.
The higher the number following the macro, the more of a subheading it is.

.B
Text can also be bold with '.B'.

.I
Text can be italicized with '.I'.

.BI
The two an be combined for bold italics.

.CW
For monospace fonts, the '.CW' macro can be used.

.PP

Table of contents

.TL
Document with table of contents
.AU
Tyler Clark

.AB
This document will show how to do a table of contents.
.AE

.NH 1
Heading
.XS
Heading
.XE

.PP
This is going to be the first heading for our document with the table of contents.
The heading must be followed by a set of tags '.XS' and '.XE' respectively with the heading in between;
these macros will add the text between them to the table of contents.

.NH 1
Another heading
.XS
Another Heading
.XE

.PP
This will be another heading for the table of contents.
The '.TC' macro provides the table of contents for us.

.TC

What is noteworthy here is that the table of contents macro .TC comes at the end of the document. This is because if it comes at the beginning, it will not find any macros telling it about the headings as they have not occurred in the document yet. However, if we use the same compilation command, the table of contents will be at the end of the document. How do we fix that? Easy, we change the compilation command. There is another utility called pdfroff that is effectively a wrapper around the original groff command that does some nice things, such as moving the table of contents from the end of the document to the beginning. The command looks as follows:

pdfroff test.ms -mspdf > test.pdf

Closing thoughts and more resources

Groff is a very large program that has a lot of other components to add more functionality to it for things like images and math equations. If I were to try to cover all of these topics, this blog post would be massive. So rather, I included a bit to get started, then am also including some more resources to hopefully aide further exploration of groff and its related programs.

Resources for learning and getting better with groff

• Gavin Freeborn: Troff/Groff Tutorials
• Luke Smith: groff/troff for Minimalist Document Complication
• r/groff
• groff_man

Abstract

I typically cover more open source technology on this blog, however, I recently upgraded our high availability ESXi hosts for work and thought it would make a good blog post because of the configuration and process that was taken to accomplish the goal.

Pre-upgrade tasks

Before performing the upgrade, there are some things that should be done ahead of time to make the process as smooth as possible. This, in no particular order, is what my list for prepping for this upgrade was.

• Blank flash drives
• Download ESXi installation ISOs and burn to flash drives for install media
• Download vCenter upgrade ISO and have Windows machine on the same network as server(s)
• Backup configuration of each of the ESXi servers
• Confirm credentials for vSphere and ESXi servers
• Make sure to have correct licensing

Confirming and practicing recovery

While upgrading ESXi should go smoothly, there is not really a way to guarantee there will not be any problems, which is way it is always good practice to create a backup and practice the restoration before everything blows up. Thankfully, there is documentation on how to create a backup and restore from the backup available here. Then to make ]]> https://foxide.xyz/projects/2024-04-12.html https://foxide.xyz/projects/2024-04-12.html Fri, 12 Apr 2024 00:00:00 -0400

Abstract

In the not too recent past, playing video games on Linux (GNU Linux or otherwise) was an extremely difficult and often fruitless venture; however, with the improvements of tools like Wine, and contributions from other projects such as Valve’s Proton, there has been a massive amount of progress in the realm of playing games on Linux. This is likely nothing new to those who use Linux as a desktop operating system.

Open Source Games

While they are not extremely popular, it is worth mentioning the fair bit of fun open source games in a wide variety of genres. There are too many to really put on one comprehensive list, and I am certainly not going to try in this blog post, but there are some lists that have quite a lot of games, and will show other lists. Some decent lists I found were:

• Open Source Games: bobeff - GitHub
• List of open source video games - Wikipedia

It is absolutely worth checking some of those games out as they have taken a lot of effort and in many cases the games are better than some of the proprietary rivals.

Running Games in Steam

For many games on Steam, this just works without any issues, for evidence of this check ProtonDB. Many of the games just work out of the box without any issues, and games are being added and improved every day. The games that are the most likely to give problems are the ones with kernel level anti-cheat. Some examples of games that have this “feature” are:

• Destiny 2
• Tom Clancy’s Rainbox Six Siege
• Call of Duty

The games will either not launch, or will immediately kick/ban you when trying to play the game. While it is unfortunate, it is the current reality of playing games on Linux. However, that is just games available on Steam; what about games that aren’t on Steam?

Non-Steam Gaming

With all of the work that Valve has put into playing games on Linux, it’s not surprising that the experience is very good, but what about playing games outside of Steam? Well there are quite a few options… The first, and possibly the simplest is to use Steam’s “Add a Game” feature to run the game using Valve’s Proton; however, there is also an argument to be made that doing this is not necessary and could eventually cause issues with the game. This has never been my experience, however, your mileage may vary. There are some other tools that will be able to run Windows games as well as other Windows software on Linux.

Wine

Wine is the original software to make Windows games and applications work on Linux, and like the name implies, has only gotten better with age. The website manages a database of games and software that are known to be working with standard Wine. In many cases, Wine will be enough to get the game to run, however, there may or may not be some performance issues in some cases. Many games work well, but many give low frame rates without tweaks to use better graphics APIs such as DXVK rather than OpenGL.

Lutris

Lutris is a gaming platform designed to make playing games on Linux easier. From their website:

Lutris is an open gaming platform for Linux. Lutris helps you install and play video games from all eras and from most gaming systems. By leveraging and combining existing emulators, engine re-implementations and compatibility layers, it gives you a central interface to launch all your games. The client can connect with existing services like Humble Bundle, GOG and Steam to make your game libraries easily available. Game downloads and installations are automated and can be modified through user made scripts. Before installing Lutris (or at least before running any games), make sure your computer is fully setup to run games. This includes making sure your graphics drivers are installed and up to date, that Vulkan is installed and that the 32bit libraries for OpenGL / Vulkan are present. If any of those components are missing Lutris will give a warning. We also very highly recommend installing Wine from your package manager. This can be any version of Wine (stable, development or staging) as long as it pulls both 32bit and 64bit variants. Lutris will usually not make use of it directly but the peer dependencies installed from the Wine package are necessary to run our Wine version properly.

Lutris allows players to download configuration profiles for various games and game clients to allow running the software on Linux. This can be done by going to Lutris.net and search for a game that you would like to install (pro tip, DuckDuckGo has a bang for Lutris, !lutris). Then once that game is found, then download and open it with Lutris; Lutris will take care of the rest and get the game working. It will handle not only PC games, but emulator games as well.

Bottles

Bottles is another game/software manager for Linux that is installed via Flatpak. This software is similar to Lutris in the sense of it will manage different versions of Wine and its forks for the various games and software it will have to run. It acts almost like a virtual machine manager, but for Wine configurations. Bottles allows users to use different versions of Wine with different options set for Wine, as well as different versions of things like DXVK to allow for better graphics support depending on the game and/or software that the bottle is running.

More In-depth topics

There are quite a few more topics that could be covered in this post; however, it would get very length very quickly. So, rather than continuing with the technical bits, I will give some more things to look into that might be interesting for some one that is wanting to learn more about it.

• DXVK: Vulkan based translation layer for DirectX 9, 10, and 11. This software allows for running 3D applications using Wine (or related software).
• Winetricks: Helper script to download and install libraries in Wine that are needed to make some Windows software work properly.
• Protontricks: Like Winetricks, but for Proton.
• ProtonGE: This is a build of Proton that uses the most bleeding edge code, as well as usually shipping the newest DXVK version. There are many games that do not run, or do not run well on Valve’s version of Proton, that run very well with ProtonGE. It is worth downloading and trying the latest version to run a game that is giving problems.

Abstract

Running a Windows server to have a domain controller with things like Active Directory are great, however, it requires dealing with Windows and actually activating a Windows server in one method or another. A better solution would be to run something that is free not only as in beer, but as in speech as well. This blog post is going to explore the trials and tribulations of running a Samba 4 domain controller.

Setting up and getting started

I decided to use Parabola GNU/Linux for this project. There wasn’t really a particular reason for that, other than I just wanted to get a bit more familiar with Parabola; the main concepts in this blog post should stay the same, just replace some of the distro specific commands with the equivalent of your distro (for example pacman -Sy samba on an Arch based distro would be apt install samba on a Debian based distro, also doas is interchangeable with sudo in this case).

I am not going to cover the installation as it is fairly straightforward from just following the installation guide and getting to the installed system. From there, I just setup a few basic quality of life things (neovim, doas, sshd, etc).

$ doas pacman -Sy python-markdown samba
$ doas samba-tool domain provision --server-role=dc -use-rfc2307 --dns-backend=SAMBA_INTERNAL --realm=GALAXY.FOXIDE.LOCAL --domain=GALAXY --adminpass=S3curePW

Note: the python-markdown package is required (at least when I tried it) to actually get the domain to finish provisioning. If the package is not installed, the provisioning will fail saying that it is missing the required markdown packages.

The next step is getting authentication to work as expected. To do that, simply copy the krb5.conf example file included with Samba (located at /usr/share/samba/setup/krb5.conf on my system, but mileage may vary) to /etc/krb5.conf:

$ doas cp /usr/share/samba/setup/krb5.conf /etc/krb5.conf
$ doas nvim /etc/krb5.conf

Then edit the /etc/krb5.conf to match the provisioned domain settings.

[libdefaults]
    default_realm = GALAXY.FOXIDE.LOCAL
    dns_lookup_realm = false
    dns_lookup_kdc = true

[realms]
GALAXY = {
    default_domain = FOXIDE.LOCAL
}

[domain_realm]
    DC01 = FOXIDE.LOCAL

Then, we need to enable and start Samba as a service, then finally check that the authentication is working properly:

$ doas systemctl enable samba
$ doas systemctl start samba
$ kinit Administrator
Password for Administraotr@GALAXY.FOXIDE.LOCAL:
S3curePW
Warning: Your password will expire....

If you received the warning about the password expiring for the account, than the authentication has passed and is working on the domain. The kinit command can also be used to check authentication for other users as well.

Adding users

While it is not strictly necessary, it is usually a good idea to create a separate admin account that is only used for privilege escalation. The idea behind that is that stolen user credentials are less dangerous than stolen admin credentials; though ideally credentials do not get stolen in the first place.

$ doas samba-tool user create tyler
$ doas samba-tool user create tyler.admin
$ doas samba-tool group addmembers "Domain Admins" tyler.admin
$ doas samba-tool group addmembers "Administrators"

I added my admin account to both the Administrators and the Domain Admins group to give myself both global admin access, as well as the ability to mange computers on the domain. If your admin account is not part of the Domain Admins group, than it will not be able to escalate past the User Account Control (UAC) prompt in Winodws.

There are some other options that have to be set for users on Unix machines that is documented here, however, this article is mostly focused on Windows clients for the domain as that is what I am most familiar with currently. I will likely make another post specifically about Unix users and workstations on a Samba domain as it looks slightly more complex (or possibly just unfamiliar) than a Window enviornment.

Joining Windows Computer to Domain

I am only going to be covering how to do this in Windows 11, as that is what I am testing with; and really what you should be using at this point. If you would like a less bloated version of Windows or have problems with the hardware requirements, check out tiny11builder. I also did a previous post about making a smaller, more trimmed down version of Windows 11, if you would like to read that explaining the process.

Settings -> Accounts -> Access work or school -> Connect -> Join this device to a Local Active Directory domain Then enter the domain realm, and provide administrative credentials for the domain to join the device. Here is an article with more information and diagrams if you are having trouble.

Managing Domain with Windows

The Windows Remote Server Administration Tools (RSAT) are the standard tools used to manage an Active Directory domain from a Winodws workstation. There is also the other option of using Winodws Admin Center, however, this post is not about all the differnet methods of managing a domain, so I am not going to cover that here.

I installed the RSAT tools in PowerShell as installing them under the Windows feature menu didn’t work for me. To do that, open PowerShell as an administrator and run the following command:

Get-WindowsCapability -Name RSAT* -Online | Add-WindowsCapability –Online

You can do something a bit more cleaver than installing all of them, but that took a bit more effort than I thought it was worth; however, if you are only wanting specific tools you can follow this guide.

This will give some familiar tools such as Active Directory Users and Group and Group Policy Management that will allow for managing different aspects of the domain. From here, many Windows system administrators should see some semblance of familiarity with managing groups, users, and computers. As far as setting up shared printers, Samba can do that as well, however, I am not covering that in this particular blog post; if that is a use-case that is interesting to you, there is this page that documents the process.

Resources

These are some of the resources that I used to get started with this project. The Samba documentation was great, but Matei Cezar’s blog post really helped filling in the gaps on using the samba-tool utility. Alternatively, just typing in samba-tool on the command line will bring up the various things that can be done with it, and some documentation will be provided from typing in samba-tool followed by the specific area of interest. For example, if I wanted to learn more about the DNS section of samba-tool, I could type samba-tool dns and it would spit out the available commands.

https://www.tecmint.com/manage-samba4-active-directory-linux-command-line/ https://wiki.samba.org/index.php/Adding_users_with_samba_tool https://wiki.samba.org/index.php/Group_Policy

Abstract

Recently, I saw a lifetime deal for a DirectAdmin panel from Silicom Network for only $10.00 USD (Thanks LowEndBox). It sounded interesting and it provided some benefits that I thought might be worth $10.00 (namely getting email setup, and the potential for cheap and easy web hosting). After purchasing it, I ran into some issues as I had never used or really seen any domain management panel like it before purchasing; thankfully it those panels are designed to be relatively straightforward and easy to understand.

Recently I got my own email and calendar service running on it, complete with web access to both services, and remote access via a client like Thunderbird. This blog post documents the process and some of the growing pains that I didn’t understand about DirectAdmin and Silicom’s offer beforehand.

Prerequisites

The major thing I didn’t understand about Silicom’s offer was that the claimed that you would receive one domain. I initially took that as the offer comes with a domain name, however, that is incorrect. Really the offer is for the ability to manage one domain. That was fine with me as I already had a domain at that point that I could just change the information for DirectAdmin to make applicable for my domain. For this guide, the following things will be required (at least if the services will need to interact with the Internet rather than just the LAN).

• Need a domain name
• Access to second email address to test sending/receiving messages (Google/Outlook/Yahoo are best, if they are received on those email services, they will likely be received everywhere)

Email

Creating email accounts with a DirectAdmin panel is very straightforward, click on ‘E-mail Accounts’, then the ‘CREATE ACCOUNT’ button, and type in the information for the email account to be created. Once the account is made, it will show the server information to put into an email client to start sending and receiving emails. However, that does not mean that emails sent from this account will be receivable from other email providers; to do that, the records must be setup appropriately with the registrar that handles the domain name.

Email Authentication (SPF, DKIM, and DMARC)

Email is much more complex than it used to be, and there is a lot more security and authentication that has to be setup before an email will actually be delivered through the various email gateways and spam filters that are so commonly used these days. Beyond publishing the standard MX records for mail, it is also important to publish records for DomainKeys Identified Mail (DKIM), Domain-based Message Authentication Reporting and Conformance (DMARC), and Sender Policy Framework (SPF). These are three text records that should be published with the registrar of choice to help determine that the emails coming from your domain are authenticate emails and not spam or impersonation. Without these three methods of email authentication, it is very likely that mail coming from your domain will be blocked, especially by major email providers (Google, Yahoo, and Exchange). As stated above, DirectAdmin makes this process easy enough to setup as you can just create new TXT records and copy the information from the DNS section in the DirectAdmin panel. After putting the information into the registrar records and giving an appropriate amount of time for the records to update (mine updated in ~10 minutes, however, they can take much longer so patience is key), we are ready to setup an email client and test sending an email.

Setting up mail clients (Roundcube Webmail and Thunderbird)

Setting up email clients is not a task that is particularly difficult, especially in the case of Thunderbird. All I had to do to add my account to Thunderbird was type in the email address and password and it automatically found the correct settings.

As for the webmail, it isn’t strictly required, but I thought it might be nice to have. I went with Roundcube for this because I liked the interface the best. I also set the webmail address on a mail subdomain of my domain. Doing this was easy enough with the Softaculous software center, just make sure to set the IMAP and SMTP settings correctly (specifically the port numbers, I did it incorrectly and could not get the webmail to work until I fixed the problem). Once there, you can sign into the webmail client by going to the domain (i.e. mail.example.domain) and logging in with the email account credentials.

Calendar

Setting up a calendar was slightly more tricky for me; I wanted a calendar that I could sync to multiple devices, while keeping the ability of having a web based client. The options on the Softaculous software center for calendars are slightly limited with the following 4 being available:

• WebCalendar
• Booked
• LuxCal
• SuperCali

I couldn’t get the demo to load for SuperCali and it was rated rather poorly, so I did not try that one; Booked seemed like it was a good solution for a reservation scheduler, which is not quite what I wanted. I initially tried LuxCal, and liked the interface, but could not figure out how to get syncing setup with either iCal or CalDav, so, I went with WebCalendar. While the interface on WeebCalendar isn’t wonderful, it is serviceable to get started with and do quick checks on scheduling, it also has iCal syncing to be able to sync to clients that support iCal. Setting it up was as easy as entering the proper information into the Softaculous software center, then logging in with the admin credentials. Though, I would recommend creating another user and disabling the default admin account, as the name couldn’t be changed upon install.

Calendar client in Thunderbird

Again, I went with Thunderbird as it is one of the most popular open source email clients, as well as it is fairly feature rich. Adding the calendar was as simple as clicking the ‘New Calendar’ button in Thunderbird, then ‘On the Network’ and entering the username for the user as well as the iCal syncing URL (located at Settings->Preferences->Subscribe/Publish and enabling ‘Allow remote publishing’, then copying the URL that follows). From there, the WebCalendar events will sync with Thunderbird’s calendar.

Things to improve on

While I do really like this setup, and it made setting up an email service much easier, there are some paper cuts that I would like to improve on:

• Enable SSO for Calendar and Email
• Find a good iCal client for Android that does syncing, not just subscribing
• Find TUI iCal client with vim keybindings
• Update the CKEditor in WebCalendar as apparently v4 is outdated, this shouldn’t be super difficult, but all documentation is command-line based, and I didn’t pay for SSH access to this service

These are things that can be improved on later, and will likely just take more time to research and figure out.

Abstract

ZFS is an extremely powerful tool, and one that I do not know enough about. To remedy that, I thought I might do a series of blog posts on ZFS to help me not only learn about it, but make it easy (for me at least) to reference in the future. For those that are unaware of what ZFS is:

ZFS (previously Zettabyte File System) is a file system with volume maangemtn capabilities. It began as part of the Sun Amicrosystems Solaris operating system in 2001. Large parts of Solaris, including ZFS, were published under an open source license as OpenSolaris for around 5 years from 2005 before being placed under a closed source license when Oracle Coporation acquired SUn in 2009-2010. During 2005 to 2010, the open source version of ZFS was ported to Linux, Mac OS X (continued as MacZFS) and FreeBSD. In 2010, the illumos project forked a recent version of OpenSolaris, including ZFS, to continue its development as an open source project. In 2013, OpenZFS was founded to coordinate the development of open source ZFS. OpenZFS maintains and manages the core ZFS code, while organizations using ZFS maintain the specific code and validation processes required for ZFS to integrate within their systems. OpenZFS is widely used in Unix-like systems.

• Wikipedia Entry on ZFS

Learning Environment

There are a variety of ways to setup a lab to learn about ZFS and ZFS management. For these labs, I am going to be using FreeBSD as ZFS is a tier 1 filesystem which makes the installation process quicker and easier, though the ZFS concepts should largely be the same, but some of the surrounding command (like to list disks detected by the OS) may differ.

Installing ZFS on root with FreeBSD

For anyone that has installed FreeBSD before, ZFS on root is a rather trivial. During the partitioning phase of the installation, there are options for:

• Auto (ZFS): Guided Root-on-ZFS
• Auto (UFS): Guided UFS Disk Setup
• Manual: Manual Disk Setup (experts)
• Shell: Open a shell and partition by hand

The option that we are going with for this post is the Auto (ZFS) option. We are then given an option to change the name for the ZFS pool (zroot is the default). After selecting the ZFS pool name, come the important decisions for the ZFS configuration. Many of these options should be self-explanatory, but the one that we are going to focus on right now is the ‘Pool Type/Disks’ option. Six options are given in the ‘Select Virtual Device type’ menu:

• stripe: No Redundancy
• mirror - n-Way Mirroring (Where ‘n’ is the number of disks added to the pool)
• raid10 - n x 2-Way Mirrors (Mirrored stripes)
• raidz1 - Single Redundant RAID
• raidz2 - Double Redundant RAID
• raidz3 - Triple Redundant RAID

For this post, we are just going to be dealing with stripes and mirrors as those are generally the easiest to grasp; later posts will cover the various types of RAID schemes with ZFS.

Once the virtual device type is selected, the installer will ask which disks will be apart of the pool. Select the desired disks by pressing the spacebar when they are highlighted, changing the empty [ ] next to the disk name to [X]. For a device that only has one drive, using a ‘stripe’ and selecting the one available drive for the pool will suffice. Then continue through the FreeBSD installation as normal.

Replacing failed disk in ZFS mirror

The first lab that I did was to replace a “failed” disk in a ZFS mirror; this lab assumes that the drive is completely dead and unable to be recovered or used in any capacity.

First, find and detach the disk from the storage pool (all commands are run as root!):

# look at the status of the ZFS pools on the system
zpool status
# Remove the dead drive (ada1 in this case) from the pool (zroot) in this case
zpool detach zroot ada1

Once the drive is removed from the storage pool, shutdown the machine and replace the disk (shutting down is not necessary if the storage is hot swappable on the machine). After the dead drive is replaced, reboot the machine (again, all commands are run as root):

# List the disks that are detected in FreeBSD, similar to the `lsblk` command in Linux
geom disk list
# Show the partition scheme of the good disk in the pool for reference
# though, the partition scheme does not necessarily have to stay the same
gpart show
# Create a partition scheme on the drive
# the drive will be `ada1` in this case, and the partition scheme will be `gpt` in this case
gpart create -s gpt ada1
# This example is just copying the same partition scheme of the disk that is still in the pool
# however, that does not have to be the case, and the disk can be partitioned differently
gpart add -t freebsd-boot -s 512k ada1
# partition for swap space
gpart add -t freebsd-swap -s 2G ada1
# replace `disk1` with a disk name that might be appropriate in your environment/situation
# a good place to reference for current names would be under /dev/gpt in this particular case
gpart add -t freebsd-zfs -l disk1 ada1
# attach the new storage disk to the pool
# make sure to put the existing storage disk before the new storage disk
zpool attach zroot ada0 ada1
# This step is to add the required boot code to the disk so it can boot FreeBSD properly
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
# verify the work done above
zpool list

It is also possible to add the disk to the pool before removing the failing/failed disk as ZFS will allow for having 2^64 (18,446,744,073,709,551,616) devices in a zpool. There may be practical reasons for doing one way over the other, but both ways are entirely possible and valid.

Migrating to bigger drive

Another common task with storage drives is upgrading to a larger drive. ZFS also makes this easy by allowing us to attach the larger drive to the smaller one in a mirror, then after the data migration has occurred between the two removing the smaller drive. This process is highly similar to the one above; the main difference is making sure that zpool autoexpand is set to on. The example below is assuming only one drive, but modifying the steps for multiple drives should be quite easy to figure out.

# Make sure to enable `autoexpand` for the pool.
# This will allow ZFS to automatically utilize more drive space once the smaller drives are detached from the pool
zpool autoexpand=on zroot # the example pool is `zpool` here, change appropriately
geom disk list
gpart show # again only referring to the current partitioning scheme
# ada1 will again be the new disk and ada0 will be the original disk in this example
gpart create -s gpt ada1
gpart add -t freebsd-boot -s 512k ada1
gpart add -t freebsd-swap -s 2G ada1
gpart add -t freebsd-zfs -l disk1 ada1
# attach the drive to the zpool, remembering to put the original drive first and the new one second
zpool attach zroot ada ada1
# after the resilvering is done, don't forget to add the boot code otherwise the system will not boot
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
# verify the work with
zpool list

To do this with a ZFS mirror, just replace one disk as if it failed. After the new drive is in and has replaced the first disk, repeat the process with the second.

Other Resources for ZFS

I am still a bit of a novice with ZFS and its extremely powerful tools and ways of handing storage issues. Because of that I am using a lot of resources to attempt to teach myself the basics to begin to feel comfortable enough to play with it and come up with something novel. In the meantime, I thought I would share some sources of knowledge that might be helpful to another ZFS novice.

• ZFS 101 - Understanding ZFS storage and performance: This is an Ars Technica article written by Jim Salter, that goes through the basics of ZFS.
• Practical ZFS: This is a community that was created by one of the former r/ZFS mods on Reddit. This community was stood up after the API change occurred on Reddit.
• Klara Systems: ZFS: Klara Systems is a company that will support and interface open source software and communities to make life easier for companies. It has a lot of helpful resources on various topics, but the ZFS resources are particularly good.

Abstract

The last post of this series worked through replacing failed disks with ZFS. This post will be on working with ZFS snapshots and data recovery using ZFS; for the purposes of this post, I am going to be using a ZFS on root FreeBSD machine and only working with the one zroot pool that is the default in FreeBSD when installing ZFS on root.

Taking snapshots with ZFS

By default non-root users have very little privilege when it comes to ZFS; this is a good thing, but makes permissions more annoying and is overall bad practice. So, the first thing to do is to grant another user necessary permissions within ZFS.

# The following command must be run as root
# substitue ${USER} with the username that will need ZFS permission
zfs allow -u ${USER} canmount,compression,create,destroy,hold,mount,mountpoint,receive,send,snapshot zroot/ROOT

For more information on what these permissions are/do, read the zfs-allow man page

Now the user specified in the command above should be able to create a snapshot, as well as use some other features of ZFS. Taking a snapshot of the zroot/ROOT pool is as simple as running:

# This would take a snapshot of zroot/ROOT with the tag of bk01
zfs snapshot zroot/ROOT@bk01
# This would recursively take a snapshot of zroot/ROOT with the tag of bk02
zfs snapshot -r zroot/ROOT@bk02

While that is great, that doesn’t actually back up the entire system. Running a zfs list command on a standard zfs on root install of FreeBSD will turn up something like the following:

zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                959M  25.7G    66K  /zroot
zroot/ROOT           956M  25.7G    66K  none
zroot/ROOT/default   956M  25.7G   951M  /
zroot/home           400K  25.7G    67K  /home
zroot/home/${USER}   333K  25.7G   179K  /home/${USER}
zroot/tmp            134K  25.7G    73K  /tmp
zroot/usr            198K  25.7G    66K  /usr
zroot/usr/ports       66K  25.7G    66K  /usr/ports
zroot/usr/src         66K  25.7G    66K  /usr/src
zroot/var            741K  25.7G    66K  /var
zroot/var/audit       68K  25.7G    68K  /var/audit
zroot/var/crash     66.5K  25.7G  66.5K  /var/crash
zroot/var/log        340K  25.7G   212K  /var/log
zroot/var/mail       134K  25.7G    77K  /var/mail
zroot/var/tmp         67K  25.7G    67K  /var/tmp

The zpool/ROOT pool only contains the basic installation files of the system; backing up from zpool/ROOT@bk02 will be missing the /home, /tmp, /usr, and /var directories from the original system. Getting a snapshot of those can be done by running the following:

# This will create a full system snapshot with the tag full-bk01
zfs snapshot -r zroot@full-bk01

To get a list of available snapshots on the system, run zfs list -t snapshot. This is some test output from the VM I am using for testing that had a snapshot taken of the entire system called bk01:

zfs list -t snapshot
NAME                      USED  AVAIL  REFER  MOUNTPOINT
zroot@bk01                  0B      -    66K  -
zroot/ROOT@bk01             0B      -    66K  -
zroot/ROOT/default@bk01  4.99M      -   948M  -
zroot/home@bk01             0B      -    67K  -
zroot/home/${USER}@bk01   154K      -   180K  -
zroot/tmp@bk01             61K      -    73K  -
zroot/usr@bk01              0B      -    66K  -
zroot/usr/ports@bk01        0B      -    66K  -
zroot/usr/src@bk01          0B      -    66K  -
zroot/var@bk01              0B      -    66K  -
zroot/var/audit@bk01        0B      -    68K  -
zroot/var/crash@bk01        0B      -  66.5K  -
zroot/var/log@bk01        128K      -   177K  -
zroot/var/mail@bk01      56.5K      -  72.5K  -
zroot/var/tmp@bk01          0B      -    67K  -

Next, let’s see how to send those snapshots off to some backup storage in case the system fails.

Sending snapshots to a backup server

Having a backup file of a system is great, but it needs to be accessible from another location in case the system fails so the original machine can be restored to a functioning state. Thankfully doing that with ZFS is decently straight-forward. The tool to do this is zfs send; these examples only go over sending the snapshots as files, however, they can also be sent as entire datasets to another system using ZFS. I had issues doing this with a root on ZFS snapshot, but that might have been a skill issue that could have been overcome by someone that is more knowledgeable. To send the snapshot to a file, run the following:

# This will create a file called `full-backup.zfs` that will contain the snapshot `zpool@full-bk01`
# this snapshot can be sent with standard tools like scp, rsync, or stored on a fileshare.
zfs send -R zpool@full-bk01 > full-backup.zfs
# This example will send the snapshot `zpool@full-bk01` to be compressed via gzip, then to a remote machine called `backup.server`
# The compressed file will then be available on that server
zfs send -R zpool@full-bk01 | gzip | ssh ${USER}@backup.server "cat > /backup/directory/location/full-bk01.zfs.gz"

Now that the snapshot has been moved to another system as a proper backup; let’s go over how to actually use the snapshot to recover lost/destroyed data.

Restoring from snapshots

Now that we have a proper backup, and it is located elsewhere besides the machine, we need to figure out how to actually utilize it like a backup. There are three main cases that I am going to cover in this particular blog post that should cover most situations.

Restoring files, a la carte

This case assumes that some (probably user) error happened and the system itself is okay, but some files were deleted or modified in a negative way and need to be restored/recovered. Assuming we have a snapshot that contains the necessary files (having regular snapshots with a good naming convention is going to save you here), this can be done very easily. ZFS has a hidden directory .zfs throughout the file system for all the directories that a snapshot exists for. Within that .zfs directory it will show the different snapshots by their tag name (for example if the snapshots are: zroot@bk01 and zroot@bk02 there will be bk01 and bk02 directories), and within those directories will be the files that were there at the time of the snapshot. Accessing and restoring the files on the machine can be done by:

# assuming we are trying to recover `file.md` that was contained in snapshot `zroot@full-bk01`
# cd into the snapshot dir
cd ~/.zfs/snapshot/full-bk01
# copy the file back to the file regular file system
cp -r file.md ~/

The simplicity of this file recovery method is amazing, it is odd that the .zfs directory doesn’t show up when running a ls -a, but running a cd .zfs will work. However, recovering files feels like just a normal and mundane file operation that any Linux or BSD user should be comfortable with.

Rolling back the file system

This example assumes that something has gone quite wrong, and the entire system needs to be taken back to a previous snapshot state (this could also be done with specific datasets, i.e. zpool/home). For this we use the zfs rollback command. It is also worth noting that I did not set those permissions up for a non-root user in this blog post, so unless you see rollback somewhere when running zfs allow on your pool and/or dataset, then you will have to run it as root as well. I think I prefer rollbacks being segregated to root users as it could potentially be damaging, and people should think about commands more carefully when running as root anyway, but to roll the system back run:

zfs rollback -r zroot@full-bk01

This will bring the entire system to the state that it was in during the time of taking the snapshot zroot@full-bk01. To rollback a specific dataset:

# assuming rolling back the zroot/tmp dataset at snapshot full-bk01
zfs rollback -r zroot/tmp@full-bk01

Entire root file system

The final example assumes that the entire system has been corrupted/deleted/caught fire/nuked/etc and has to be fully recovered from backup onto a brand new hard drive. Make sure the new drive is connected to the system, and boot a live FreeBSD environment. I used mfsBSD (found thanks to this blog post) as it allowed me to mount a zpool to /mnt where as the standard live FreeBSD system would not allow that as it was read-only.

# assuming the drive is `ada0`
# also assuming a basic FreeBSD installation partition scheme
gpart create -s gpt ada0
gpart add -t freebsd-boot -s 512k ada0
gpart add -t freebsd-swap -s 2G ada0
gpart add -t freebsd-zfs -l disk0 ada0
# don't forget this otherwise system will not boot
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

From here, we are ready to actually get the backup onto the system and restore it. It’s worth noting that there are a lot of different ways to get the snapshot file from the backup machine onto the machine we are trying to recover (recovery machine). We can do a pull, where we are logged into the recovery machine and transfer the file from the backup server to the recovery machine by “pulling” it off. We can do a push, where we log into the backup machine, and push a copy of the file onto the recovery machine. We could also do something like transfer the file via USB flash drive.

zpool create -d -o altroot=/mnt zroot ada0p3
# Example of a "pull", pulling snapshot off of the backup server using the recovery machine
ssh ${USER}@backup.server "cat /path/to/backup/full-bk01.zfs.gz | gunzip | zfs receive -vF zroot"
# Example of a push, pushing the snapshot from the backup server onto the recovery machine
cat /path/to/backup/full-bk01.zfs.gz | gunzip | ssh ${USER}@recovery.machine "zfs receive -vF zroot"
# also do not forget this otherwise system will not boot
zpool set bootfs=zroot/ROOT/default zroot

Closing thoughts

ZFS is a lot easier to learn than I thought it would be, however, that does not make it simple. It is definitely a great choice for a sys admin trying to make life easier once the sys admin knows how to use it.

Researching resources

Some of the resources that I found helpful while working on this blog post.

• zfs and zpool man pages on FreeBSD system
• OpenZFS man pages - Good for general reference regarding zfs commands
• Restoring FreeBSD from a ZFS snapshot - This blog explained recovering an entire ZFS system from a snapshot.
• zfs send | zfs receive as non-root - Blog that helped explain some zfs permissions, and showed different send/receive examples. ## Other mentions
• mfsBSD - Small BSD distro that is useful in helping recover from ZFS snapshots; allows mounting a zpool to /mnt while the live FreeBSD installer ISO will not allow it.

Abstract

I didn’t feel like writing a post on ZFS this time, so I decided I would try out Slackware. Slackware is the oldest still maintained Linux distro out there, and it is not known for being user friendly, there is a moniker: “if you need to know something about Linux, ask a Slackware user**. While that is funny, I wanted to learn more about it, and try out a distro that might be different and unique compared to most modern distros.

Impressions before installing

I thought it might be a good idea to jot down some thoughts on what I think Slackware will be like before installing the distro; it might be nice compare the perception of the distro to the reality of using it. So, my initial thoughts are:

Potential issues

• Dependency Hell: I have heard a lot about Slackware’s package manager not resolving dependencies for you. While I don’t think it will be detrimental, it will be extremely annoying having to go through and find all the required C libraries to get even basic applications to work.
• Lack of documentation: Documentation can make or break a distro, and I have heard that Slackware is a distro in which you either know, or you don’t. I am confident in my skills to figure things out, but I do need a jumping off point; if the documentation cannot provide that, I might have a very difficult time using it.

Things that might be cool

• Linux History: Because Slackware is one of the oldest distros, it will be cool to see Linux as it may have been in the 90’s.
• Learning experience: I am decently confident in my Linux skills, however, I am sure that going through this experience will teach me something about using Linux, which is always nice.

Slackware!

Slackware is the oldest still maintained Linux distribution (since April 1993). The core philosophy of Slackware is to provide the more “Unix-like” Linux distribution available; another pillar of the distribution is simplicity. The distro is also designed to be full featured for both users and developers alike, offering various desktop environments, web tools, editors, and even developer libraries out of the box.

Getting Started

Getting started with Slackware is easy enough, just download the ISO and write it to a flash drive:

# can alternatively use `fetch` command in FreeBSD
wget https://mirrors.slackware.com/slackware/slackware-iso/slackware64-15.0-iso/slackware64-15.0-install-dvd.iso
# be very careful of the disk that you write to, DD will not hesitate to nuke your drives
# let's assume the drive path here is /dev/sdb
dd if=slackware64-15.0-install-dvd.iso of=/dev/sdb bs=4M status=progress

Then restart the computer and boot off of the live media.

Installation

Booting the flash drive and getting a shell in Slackware is easy enough; there is also instructions on what to do after logging in as root on the live media. Partition the disk with your tool of choice. I am going to use cfdisk, but fdisk is also available. Then run setup, once the partitions are made. I partitioned my drive as follows:

• 500M for UEFI
• 109G for Storage
• 9.8G for swap

Then run setup; I did not read the HELP menu, and did not need to change my keybaord layout, so I immediately started working on formatting filesystems on the partitions. Select the installation source, in my case a USB drive; impressively, there are a lot of package options that can be installed from the live disk:

• Base Linux system
• Various Applications that do not need X
• Program Development (C, C++, Lisp, Perl, etc)
• GNU Emacs
• FAQ lists. HOWTO documentation
• Linux kernel source
• The KDE Plasma Desktop
• System Libraries (some needed by both KDE and XFCE)
• Networking (TCP/IP, UUCP, Mail, News)
• TeX typesetting software
• Tcl/Tk script languages
• X Window System
• X Applications
• The XFCE Desktop Enviornment for X
• Games

I thoroughly enjoying the option to install basic and useful applications during the installation process.

The next step it to select a “prompting mode”. I assume that means prompting for the build options for each application; I could try to do something more advanced and fancy, but I am going to go with the “full” prompting mode because I don’t feel like messing with config options that much right now. This prompting mode wastes a bit more space (installer says its about 15G in total), but that is fine for the purposes of me just trying out Slackware. Now, wait for the installation process…

The installer recommends making a bootable USB device to boot into the root filesystem, additionally LILO is not capable of booting UEFI systems, so ELILO is required.

Configure mouse, networking, ConsoleFonts, and some other options. Unfortunately at the end of it, the system was not bootable from just the hard drive for some reason. Not a big deal, I used the install media to boot the OS and was able to get in.

Using Slackware

Now that I am in a real Slackware system, let’s figure out how to connect to the network, install some packages, and create a user. Connecting to a network is fairly straightforward; during the install process, I enabled NetworkManager, which handles networking for me (mostly). I was unable to figure out connecting to WiFi easily, however, I can connect an Ethernet cable and work with that until I can either figure out WiFi, or get a tool that makes it easier. Creating a user can be done the same as any other Linux distro, here is the command I generally use:

useradd -m -s /bin/bash -U -G wheel,video,audio,users ${USERNAME}
passwd ${USERNAME}
# Also have to enable the wheel group for sudo
# I am going to replace this with doas later
visudo

Now I can download, build, and install my suckless applications that I use. Surprisingly, they built without trouble or having to download any other dependencies. With other distros, that is generally a bit more annoying to have to go through, but it just werks™ on Slackware.

Next is package management. This was more complicated than I was initially thinking it would be as Slackware handles packages much differently than other distros that I have used in the past. Slackware doesn’t build and provide much in the way of the repos outside of the base system software. Installing a remote package provided by Slackware can be done with slackpkg (Guide on Linode); alternatively, there is installpkg for installing packages that have already been downloaded to the machine. Finally, there is sbopkg, which functions similar to a traditional repo. After realizing this, I can start getting the rest of my normal utilities installed. Installing the package is as simple as:

# Download sbopkg
wget https://github.com/sbopkg/sbopkg/releases/download/0.38.2/sbopkg-0.38.2-noarch-1_wsr.tgz
# Install
installpkg sbopkg-0.38.2-noarch-1_wsr.tgz
# First sync of repo
sbopkg -r

Now we can use sbopkg to install software like any other distritbuion. An example of installing software with sbopkg might look like:

# searching for package in the repo
sbopkg -s imlib2
# build and install said package
sbopkg -i imlib2

However, do note that it does not seem that you can install multiple packages at a time with sbopkg like you might be able to in Arch or Ubuntu (ext pacman -Sy doas imlib2 lf).

WiFi

I was lazy and connected to wifi via nmcli; there are plenty of guides on doing this as well as the man page.

Upgrading to Slackware current

While using Slackware, specifically when trying to install Spacemacs, I noticed that the packages were a tad outdated. That might not be a horrible problem, but the version of Emacs installed on the system was too old for Spacemacs, so I wanted to upgrade it to the current version; I thought while I was at it, I might as well upgrade everything to current as the kernel is also 5.15 by default, and I was hoping for at least 6.1. The process of upgrading was simple enough by opening up /etc/slackpkg/mirrors in a text editor and un-commenting the repo URL below the section that says “Slackware64-current”. That URL is http://mirrors.slackware.com/slackware/slackware64-current/ for me. Then running slackpkg upgrade-all as root. The process took a while, but once it finally finished I rebooted.

Minor issues

The first reboot after installing Slackware was where I started to get the hint that the distro was a bit more complex that I initially thought. During the slackpkg upgrade-all, many of the system libraries were uninstalled; this would not have been a big deal, however that also broke the networking. After doing another reboot and looking at the available boot options, I realized my error was assuming (like most other distros) the first boot option was the one that I wanted. Reading the boot options more carefully, I booted into the 6.10.5-generic boot option, in which networking was working on. X was still broken, but reinstalling the library that the error message was complaining about fixed that issue.

After researching this a bit, I found this page explaining the recommended update/upgrade procedure. Here is the cliff notes for this, though I recommend reading the actual page if you are an aspiring Slackware user (or also asking in IRC might be good too).

# Update repo
slackpkg update
# Upgrade slackpkg
slackpkg upgrade slackpkg

# Upgrade packages marked in ChangeLog.txt, will not install packages that are currently not on the system
slackpkg install-new
# Compare Slackware packages with currently installed packages and install the ones that are out of date
slackpkg upgrade-all
# Show a list of removed packages
slackpkg clean-system

Closing thoughts

Slackware is an interesting distro that I think I am going to keep around for a bit. I am not sure that I will become a long time Slackware user, but it does seem like an interesting system that I still have a lot to learn about. It can definitely teach new users how Linux works, and challenge Linux users that have not used it before.

Abstract

Lately I have been wanting to learn more about development and the process behind pushing code. For many entities, a big part of the process is the CI/CD pipelines that automate various aspects of testing and releasing code. It sounds like an interesting and very useful idea and wanted to learn more about it. So, I decided to pick up and play with Buildbot.

From Buildbot’s docs

Buildbot is a framework to automate the compile and test cycle that is used to validate code changes in most software projects.

Test Environment and General Goals

I didn’t have a lot of goals for this project other than get something to compile stuff (very scientific I know). However, I also know with my lack of development skills and how much I am actually writing code, a CI pipeline is probably not going to be super useful. So, for now the main part of the project is to set up two machines, a master and a worker, that can pull from a git repo and build software. From there, I will work with it further to find more useful things Buildbot can do.

Host 1

Host 1 is going to be a FreeBSD machine acting as the Buildbot master as well as a Forgejo host for the code we will be using in Buildbot. This post is not going to go over setting Forgejo up or installing FreeBSD; however, if you are interested I already made a post on that exact topic. I will mostly be focusing on getting Buildbot setup on a FreeBSD host:

# Install Buildbot with a web interface front-end
pkg install -y buildbot-www
# enable the service (many different ways to do this on FreeBSD)
service buildbot enable
# Go to the directory for Buildbot install (/var/db/buildbot by default)
cd /var/db/buildbot
# edit master.cfg to fit needs
vim master.cfg

It took me a bit to figure out what I was doing with this file; in an attempt to help anyone reading this in the future, I am posting my master.cfg as well as the diff compared with master.cfg.sample.

My master.cfg after editing to make it build stuff.

# -*- python -*-
# ex: set filetype=python:

from buildbot.plugins import *

# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### WORKERS

# The 'workers' list defines the set of recognized workers. Each element is
# a Worker object, specifying a unique worker name and password.  The same
# worker name and password must be configured on the worker.
c['workers'] = [worker.Worker("example-worker", "secret_password"), worker.Worker("buildbot", "secret_password")]

# 'protocols' contains information about protocols which master will use for
# communicating with workers. You must define at least 'port' option that workers
# could connect to your master with this protocol.
# 'port' must match the value configured into the workers (with their
# --master option)
c['protocols'] = {'pb': {'port': 9989}}

####### CHANGESOURCES

# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes.  Here we point to the buildbot version of a python hello-world project.

c['change_source'] = []
c['change_source'].append(changes.GitPoller(
        'https://github.com/buildbot/hello-world.git',
        workdir='gitpoller-workdir', branch='master',
        pollInterval=300))

####### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.  In this
# case, just kick off a 'runtests' build

c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
                            name="all",
                            change_filter=util.ChangeFilter(branch='master'),
                            treeStableTimer=None,
                            builderNames=["duskOS"]))
c['schedulers'].append(schedulers.ForceScheduler(
                            name="force",
                            builderNames=["duskOS"]))

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which workers can execute them.  Note that any particular build will
# only take place on one worker.

factory = util.BuildFactory()
# check out the source
factory.addStep(steps.Git(repourl='http://192.168.122.128:3000/fac3/duskOS/', mode='incremental'))
# run the tests (note that this will require that 'trial' is installed)
factory.addStep(steps.ShellCommand(command=["make"],
                                   env={"PYTHONPATH": "."}))

c['builders'] = []
c['builders'].append(
    util.BuilderConfig(name="duskOS",
      workernames=["buildbot"],
      factory=factory))

####### BUILDBOT SERVICES

# 'services' is a list of BuildbotService items like reporter targets. The
# status of each build will be pushed to these targets. buildbot/reporters/*.py
# has a variety to choose from, like IRC bots.

c['services'] = []

####### PROJECT IDENTITY

# the 'title' string will appear at the top of this buildbot installation's
# home pages (linked to the 'titleURL').

c['title'] = "Hello World CI"
#c['titleURL'] = "https://buildbot.github.io/hello-world/"
c['titleURL'] = "http://192.168.122.128:3000/fac3/duskOS/"

# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server is visible. This typically uses the port number set in
# the 'www' entry below, but with an externally-visible host name which the
# buildbot cannot figure out without some help.

c['buildbotURL'] = "http://192.168.122.128:8010/"

# minimalistic config to activate new web UI
c['www'] = dict(port=8010,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))

####### DB URL

c['db'] = {
    # This specifies what database buildbot uses to store its state.
    # It's easy to start with sqlite, but it's recommended to switch to a dedicated
    # database, such as PostgreSQL or MySQL, for use in production environments.
    # http://docs.buildbot.net/current/manual/configuration/global.html#database-specification
    'db_url' : "sqlite:///state.sqlite",
}

The diff comparing master.cfg.sample to my edited one.

--- master.cfg.sample   2024-09-05 22:27:32.771221000 -0400
+++ master.cfg  2024-09-11 22:59:26.934922000 -0400
@@ -15,7 +15,7 @@
 # The 'workers' list defines the set of recognized workers. Each element is
 # a Worker object, specifying a unique worker name and password.  The same
 # worker name and password must be configured on the worker.
-c['workers'] = [worker.Worker("example-worker", "pass")]
+c['workers'] = [worker.Worker("example-worker", "pass"), worker.Worker("buildbot", "buildbot")]
 
 # 'protocols' contains information about protocols which master will use for
 # communicating with workers. You must define at least 'port' option that workers
@@ -45,10 +45,10 @@
                             name="all",
                             change_filter=util.ChangeFilter(branch='master'),
                             treeStableTimer=None,
-                            builderNames=["runtests"]))
+                            builderNames=["duskOS"]))
 c['schedulers'].append(schedulers.ForceScheduler(
                             name="force",
-                            builderNames=["runtests"]))
+                            builderNames=["duskOS"]))
 
 ####### BUILDERS
 
@@ -58,15 +58,15 @@
 
 factory = util.BuildFactory()
 # check out the source
-factory.addStep(steps.Git(repourl='https://github.com/buildbot/hello-world.git', mode='incremental'))
+factory.addStep(steps.Git(repourl='http://192.168.122.128:3000/fac3/duskOS/', mode='incremental'))
 # run the tests (note that this will require that 'trial' is installed)
-factory.addStep(steps.ShellCommand(command=["trial", "hello"],
+factory.addStep(steps.ShellCommand(command=["make"],
                                    env={"PYTHONPATH": "."}))
 
 c['builders'] = []
 c['builders'].append(
-    util.BuilderConfig(name="runtests",
-      workernames=["example-worker"],
+    util.BuilderConfig(name="duskOS",
+      workernames=["buildbot"],
       factory=factory))
 
 ####### BUILDBOT SERVICES
@@ -83,14 +83,15 @@
 # home pages (linked to the 'titleURL').
 
 c['title'] = "Hello World CI"
-c['titleURL'] = "https://buildbot.github.io/hello-world/"
+#c['titleURL'] = "https://buildbot.github.io/hello-world/"
+c['titleURL'] = "http://192.168.122.128:3000/fac3/duskOS/"
 
 # the 'buildbotURL' string should point to the location where the buildbot's
 # internal web server is visible. This typically uses the port number set in
 # the 'www' entry below, but with an externally-visible host name which the
 # buildbot cannot figure out without some help.
 
-c['buildbotURL'] = "http://localhost:8010/"
+c['buildbotURL'] = "http://192.168.122.128:8010/"
 
 # minimalistic config to activate new web UI
 c['www'] = dict(port=8010,

The important parts to get the master and the worker to communicate is to modify the ‘workers’ list to add the worker credentials to be used in the buildbot.tac file on the work (will be explained below). From there, you can modidy the steps in the factory.addStep section to add the repository of code to be downloaded as well as the corresponding command to build the code. To finish the process and get it to show up on the web interface, modify the util.BuilderConfig and workernames to match the project repo and the worker to be used respectively. Run the following command to reload the config:

buildbot reconfig

Oncethe worker is setup appropriately, the project should be available and ready to run.

Host 2

Host 2 is going to be the buildbot worker running a very standard FreeBSD install, so as on the first host I will not go into detail about that. Installing the worker is very straightforward:

# Installing the worker
pkg install buildbot-worker
# Enabling the service
echo 'buildbot_worker_enable="YES"' > /etc/rc.conf
# Adding the base directory for the worker service
echo 'buildbot_worker_basedir="/var/db/buildbot/workers"' > /etc/rc.conf
# Generate the `buildbot.tac` file for the worker
# Make sure the ${BUILDBOT_USER} and ${SECRET_PASSWORD} match
# the username and password given to buildbot master in the `master.cfg` file
# the directory location should match what went into `/etc/rc.conf`
su -m buildbot -c 'buildbot-worker create-worker /var/db/buildbot/workers 192.168.122.128:9989 ${BUILDBOT_USER} ${SECRET_PASSWORD}'
# Start the service
service buildbot-worker start

Below is roughly what buildbot.tac looks like on my buildbot-worker system:

import os

from buildbot_worker.bot import Worker
from twisted.application import service

basedir = '/var/db/buildbot/workers'
rotateLength = 10000000
maxRotatedFiles = 10

# if this is a relocatable tac file, get the directory containing the TAC
if basedir == '.':
    import os.path
    basedir = os.path.abspath(os.path.dirname(__file__))

# note: this line is matched against to check that this is a worker
# directory; do not edit it.
application = service.Application('buildbot-worker')

from twisted.python.logfile import LogFile
from twisted.python.log import ILogObserver, FileLogObserver
logfile = LogFile.fromFullPath(
    os.path.join(basedir, "twistd.log"), rotateLength=rotateLength,
    maxRotatedFiles=maxRotatedFiles)
application.setComponent(ILogObserver, FileLogObserver(logfile).emit)

buildmaster_host = '192.168.122.128'
port = 9989
connection_string = None

workername = 'buildbot'
passwd = 'secret_password'
keepalive = 600
umask = None
maxdelay = 300
numcpus = None
allow_shutdown = None
maxretries = None
use_tls = 0
delete_leftover_dirs = 0
proxy_connection_string = None
protocol = 'pb'

s = Worker(buildmaster_host, port, workername, passwd, basedir,
           keepalive, umask=umask, maxdelay=maxdelay,
           numcpus=numcpus, allow_shutdown=allow_shutdown,
           maxRetries=maxretries, protocol=protocol, useTls=use_tls,
           delete_leftover_dirs=delete_leftover_dirs,
           connection_string=connection_string,
           proxy_connection_string=proxy_connection_string)
s.setServiceParent(application)

Actually trying stuff out

Now that the workers and the jobs are setup in Buildbot, let’s try it out to see what happens. First, to confirm our worker go to TODO{insert path here} and the example worker should show up as well as the new worker. In my case, the new worker is called “buildbot”; to confirm that the worker has been added properly go to Builds -> Workers in the sidebar.

Next let’s run a job. Currently the only job I have is building duskOS. To see the build jobs, go to Builds -> Builders

To actually get something to run, click the force button at the top right corner of the screen, then the Start Build button on the bottom button on the window that pops up.

After clicking the Start Build button a new screen will appear showing the STDOUT of the build job.

Closing Thoughts

I am going to close out this blog post, but will continue working with Buildbot and writing about it once I have learned more about it, and CD/CI systems in general.

Abstract

FreeBSD’s Boot Environments feature is a tool that takes some of the fear and uncertainty out of things like upgrades by taking a snapshot of the root file system; in the event that the root file system needs to be rolled back to a known good version, simply changing an option in the boot menu will allow for that without affecting user files. This feature is a massive benefit of using FreeBSD with ZFS-on-root. I recently ran into an issue with upgrading my FreeBSD laptop and had to rely on a previous boot environment, which inspired me to do a small write-up on the topic.

Getting Started with Boot Environments

Getting started with boot environments is very easy these days; all that is needed is a FreeBSD install with ZFS-on-root. Similar systems are available on Linux, however, I do not know of distributions that have an out-of-the-box equivalent that works as well as boot environments. All currently supported FreeBSD systems (13.x, 14.x, as well as 15-CURRENT) will enable boot environments by default when using ZFS for the root file system, no further configuration required. During major and minor version upgrades a boot environment of the previous (the boot environment that was used before upgrading) is created. This will allow the system administrator to rollback the upgrade if something did not work as intended, or if the upgrade left the system completely broken.

Creating and Managing Boot Environments

The automatic backup of a boot environment is great for upgrading a system, however, there are more reasons that you might want to create a backup of the current boot environment. The simplest reason is to make sure that the system is bootable after something (probably an “experiment”) really stupid happens. Working with boot environments is fairly straightforward:

# This will create a boot environment called 'test01'
bectl create test01
# Then the environment can be listed with:
bectl list
# And to activate the boot environment:
bectl activate test01

While this process technically works, it is not an ideal process for several reasons. The first reason is the manageability of the boot environments; theoretically, the name shouldn’t really matter, but it is foreseeable to have half a dozen or more boot environments. No one is going to remember what ‘test01’ is or when it was created. Secondly, what if there is an issue with the boot environment, and you cannot get it boot back? That isn’t a problem if is a local machine, but for remote machines, you’d like it to fail into a known good state, right? Let’s see if we can improve this process a bit.

The first thing is the naming convention, it is silly, but many experienced IT staff will say that creating a following a naming convention is a key point of good IT practices. Names should mean something and be able to communicate important information quickly, so, keeping that in mind for our boot environments a good naming convention might be:

DATE−{ACTION}, for example 20240123-UPGRADE. The example name would communicate that the boot environment was created on January 23, 2024 and was created before upgrading the system. Next, let’s see about fixing the failing boot issue:

# Create the boot environment
bectl create 20240927-test
# Set the boot environment to be the snapshot that was just taken (current environment)
bectl activate 20240927-test
# Then set the boot environment to boot the "upgraded" system just once (that is set by the '-t' flag)
bectl activate -t default

Assuming the upgrade worked as intended, the boot environment can be permanently set by running bectl activate default (note the lack of a ‘-t’).

Actually Booting into the Environment

In the event that this boot environment should be re-used, simply reboot the machine and in the FreeBSD bootloader hit the ‘e’ key on the keyboard. This will open another menu in which an alternative boot environment can be selected by simply pressing ‘a’. Pressing ‘a’ will change the environment name from zfs:zroot/ROOT/default, to zfs:zroot/ROOT/${BOOT_ENV_NAME}. From there, press backspace to return to the main section of the bootloader, and press ‘Enter’ to continue booting the computer. Once the computer is booted, the root file system should return to the state in was in before performing the upgrade. It is worth noting that zroot/HOME is not affected by changing boot environments, meaning that any files in /home will remain unaffected.

A Creative Use for Boot Environments

Using boot environments as a method to safely restore a broken system is great, however, there are more creative things that it can be used for. The main one that comes to mind is a method for atomic in-place upgrades that Alan Jude came up with. I first heard the idea on an episode of 2.5 Admins, and it sounded complex, but genius for managing a lot of remote machines. When writing about boot environments I wanted to take a look at the method to actually make this happen, and realized it is slightly more difficult than I originally thought. Because of that, implementing this idea is going to take a bit more time than I have allotted to getting this post out, however, I do want to make it happen in the future. When I do, I am going to come back to this post and describe the method that I used to make it happen. For now, I have some resources in the below section that will give more information on not only this idea, but boot environments in general.

Resources

• FreeBSD Wiki entry on Boot Environments
• Boot Environments at Scale: Alan Jude
• ZFS Powered Magic Upgrades Using boot environments for atomic in-place upgrades

Abstract

In modern times, the main method of posting to the Internet is using the World Wide Web, or more often known as the “Web” (for those that know the difference between the web and the Internet). However, in the 90s, there were other protocol options for having a presence online; the main alternative to web protocols was (and still is in many ways), gopher. This article will go over setting up a modern day gopher server with gophernicus.

Setting up a Gopher Server with Gophernicus

Gophernicus is a modern and full featured gopher server that is packaged in a lot of distributions, but is also very easy to build from source. For the purposes of this article, I am going to build from source.

Download and build Gophernicus from source

Building gophernicus from source is slightly more than just git clone ${thing} and make. There is a configuration step that needs to be done before-hand. See below for an example:

git clone https://github.com/gophernicus/gophernicus
cd gophernicus
./configure --listener=inetd --gopherroot=/usr/gopher
make
doas make install

The --listener=inetd and --gopherroot=/usr/gopher are customization flags that override some of the default behaviors of gophernicus. Only the --listener flag is required for the configuration script to run, but there are other options that can be defined. Reading the configure script itself will show what options are available and what the default values are. The listener option that we want for FreeBSD (host I am using for this project) is inetd. Enable inetd and start it if it is not already running:

echo 'enable_inetd="YES"' >> /etc/rc.conf
service start inetd

gophernicus itself does not actually need a service to run automatically, the inetd service will take care of that.

To test whether it is working, an application that can understand gopher will be required. There are several applications out there, such as lynx, cgo, gopher, or even Firefox with an extension called OverbiteWX. For example, using lynx:

lynx gopher://192.168.122.94

and you should get a page that looks similar to the following:

Welcome to Gophernicus!
        _______               __                      __
       |     __|.-----.-----.|  |--.-----.----.-----.|__|.----.--.--.-----
       |    |  ||  _  |  _  ||     |  -__|   _|     ||  ||  __|  |  |__ --
       |_______||_____|   __||__|__|_____|__| |__|__||__||____|_____|_____
                      |__|
       If you can see this, it means that the installation of Gophernicus
       on this system was successful. You may now add content to this
       directory and replace this page.

       Generic information:
           your ip address: 192.168.122.1
           server time....: Wed Oct  9 22:43:18 EDT 2024
           server uptime..: 51 mins
           server version.: Gophernicus/3.1.1 "Dungeon Edition"
           server platform: FreeBSD/amd64 15.0
           description....:

       Server configuration:
           config file....: /etc/inetd.conf
           server hostname: fbsd-build
           root directory.: /usr/gopher/site1
           running as user: nobody
           output charset.: UTF-8
           output width...: 67 characters

       ___________________________________________________________________
                      Gophered by Gophernicus/3.1.1 on FreeBSD/amd64 15.0

From here, you can modify the default gophermap file, as well as add your own.

Writing Gophermaps and Gopher Pages

Writing a gopher page is slightly different than writing a page for a website. The protocol is a bit more rigid than HTML, which does not allow for as much customization of the page itself. Gopher is also structured slightly differently than HTML. At least for the purposes of this post, gophernicus would only recognize gophermap files as files to display gopher properly. Because of this, I had to structure my test site as a group of folders containing a gophermap file as well as anything else that should be available on the page. I have an example below:

!Tyler's Page!

1Home   /   192.168.122.94:70

1Projects   /projects   192.168.122.94:70

1Code   /code   192.168.122.94:70

Welcome to Tyler's gopherhole

This example will render as follows using lynx:

       Tyler's Page!

 (DIR) Home

 (DIR) Projects

 (DIR) Code

       Welcome to Tyler's gopherhole
       ___________________________________________________________________
                      Gophered by Gophernicus/3.1.1 on FreeBSD/amd64 15.0

The (DIR)s are links that can be navigated through to get to other parts of the page. I found getting the gophermap file to work as I expected a bit tricky. There is a guide on the for writing gophermap files available on gophernicus’ Github, however, I still had quite a bit of trouble understanding it well enough to get the structure I wanted in my Gopher Site. As an attempt to explain this, take the following line from an example gophermap file:

1Projects   /projects   192.168.122.94:70

Each line starts with a code, which will tell the gopher protocol how to handle that particular file. For navigating to another page in the same gopher site, the only way I could get it to work and still render the gopher as gopher, is to create a directory projects in the example above, and place a new gophermap file in that directory. Then for actually creating the link, use the number code 1 (for a directory), then state the directory, /projects (yes the ‘/’ must be included as relative site references are not usually a good idea), as well as giving the IP of the server and the port number of the service.

Resources

These are some resources that I found helpful when working on this post:

• How to Gopher
• Running a Gopher Server in 2018
• Gophernicus Github Page

Abstract

I have been using Spacemacs for roughly a year or so now, and thought it might be time to try out EXWM. DWM is a very good window manager, and I don’t have any issue that is making me want to move away from DWM, but I also like trying new things. So, here is my experience in working with EXWM on Spacemacs.

Getting Started

Getting started with EXWM on Spacemacs is as simple as enabling the layer, docs can be found here. For myself, I added the following to my .spacemacs file:

;; In the `dotspacemacs-configuration-layers` section
(exwm :variables exwm-systemtray-mode t
    exwm-terminal-command "st"
    exwm-locking-command "slock"
    exwm-hide-tiling-modeline t
    exwm-workspace-switch-wrap t
    exwm-randr-mode t)

;; In the `dotspacemacs/user-config` section

;; This puts workspaces 2 and 3 on my second monitor
(setq exwm-randr-workspace-monitor-plist '(2 "HDMI-A-0" 3 "HDMI-A-0"))
;; This auto refreshes randr so the workspaces go to the appropraite
;; monitor automatically
(exwm-randr-refresh)
;; Turns on transparency
(spacemacs/toggle-transparency)

Then simply modify your .xinitrc file to run emacs rather than your window manager or desktop environment, for example my .xinitrc looks as follows:

# This swaps the functionality of the escape and caps lock key
exec setxkbmap -option caps:swapescape &
# Sets the wallpaper to ~/.config/wall.png
exec feh --bg-fill ~/.config/wall.png &
# Turns on transparency settings I like
exec xcompmgr -slt &
# Network Manager applet for FreeBSD
exec networkmgr &
# Starts emacs and EXWM
exec emacs

From there, just exit X and run startx again; the process will be slightly different for people that use display managers, I might recommend using the Arch Wiki for guides on adding EXWM to the one you are using. Once emacs is running, Spacemacs will download and install the relevant layer packages for emacs, and after a few minutes you will be using EXWM.

That all seems good, but checking the Spacemacs documentation for the keybindings, some (many) of them don’t work as documented. For example, s-RET (super/meta/windows + return I assume) should launch the terminal that was configured in the .spacemacs file, but hitting the meta and return keys just shows an error. After digging through the files and messing around a bit, I found that the file: ~/.emacs.d/layers/+window-management/exwm/packages.el is where the keybindings are configured. From there, it was as simple as changing some lines, though I am still playing around with them. The diff of what I have done so far is as follow:

- (require 'exwm-config)
- (unless exwm-workpsace-number)
-   (custom-set-variables '(exwm-workspace-number (/ (length exwm--randr-displays) 2)))
+ (setq exwm-workspace-number 4)

- (exwm-input-set-key (kbd "s-TAB") #'exwm/jump-to-last-exwm)
+ (exwm-input-set-key (kbd "s-W") #'exwm-workspace-move-window)
+ (exwm-input-set-key (kbd "s-<tab>") #'exwm/jump-to-last-exwm)
- (exwm-input-set-key (kbd "s-RET") #'exwm/terminal-command)
+ (exwm-input-set-key (kbd "s-<return>") #'exwm/terminal-launcher)

There are still some paper cuts that I am trying to figure out a more convenient way to handle such as:

• Launching lf (I know emacs has dired, but I don’t think it works as well for all types of files as a proper file manager)
• Launching neomutt (emacs can do email, so might replace it with something like GNUs or Mu4e)
• Launching moc (I have used an emacs music player before, and it is alright so I might switch, but am still undecided)

Usability

Getting EXWM setup is one thing, but how pleasant is it to use? The answer to that question, in my opinion, really comes down to how comfortable you are in emacs. EXWM is designed so that you truly never leave emacs, thus every aspect of window management will be based around emacs concepts. Coming from other window managers such as DWM and i3, EXWM takes a lot of getting used to. The is even more true because I only started using emacs about a year ago, that being said, I also feel quite a bit more focused using EXWM then I did in DWM. For example, I can do an org capture on the fly without having to leave the window that I am on; simply pressing s-i to put the workspace into line mode will allow me to use Spacemacs keyboard shortcuts to add something to my org-agenda or any other capture that I want to do. This makes taking notes or scheduling very easy and minimally disruptive to what I am currently doing.

Gaming?

Now that I have spoken about how EXWM makes me feel more productive, how does it handle gaming? Games that automatically go fullscreen seem to work without issue; I was able to play Tomb Raider without much of a problem just like normal. Other games, such as Payday 2, worked as expected, however, I could not figure out how to make the game fullscreen, nor could I figure out how to center the game onto the screen. The EXWM keybindings were not working from within the game window, and I could not figure out how to manipulate the game window otherwise. There is probably a way to fix this, but I could not figure out what it was at the time of this blog post.

Final Thoughts

EXWM is a pretty good window manager, especially when considering productivity, for people that are already comfortable in emacs. However, if you are new to emacs, expect to struggle with it for a while. Again, I have been using Spacemacs for a bit more than a year and I am still struggling with it consistently. I am not sure that it will be replacing DWM in the long term, but it is to play around with something different.

Abstract

Modern Unix-like operating systems have a variety of ways of getting software. The most common method is probably the package manager of the operating system you are using; however, some other options are:

• Containers, such as Snaps, Flatpaks, or AppImages
• Build recipes, such as ports for the BSDs or the AUR for Arch Linux
• Downloading the binaries directly, such as with Chrome

In addition to those above options, there is the option to build from source (at least with software the source is available). While doing this may not be extremely useful for many desktop Linux users, many jobs that involve working with Linux will want some experience in building (and maybe patching) software to run on the system. This article is a quick introduction into some common ways to get , compile, and patch source code. It also aims to cover the logic behind each of these steps for the case that they WILL fail, as building software from source is generally not a something that can be copy and pasted from one application’s source to another. As an example for this write up, I am going to be using a fairly simple project that is relatively easy to build. The project is a redshifting program called SCT, and uses a bit of C to change the color temperature of your screen.

I also want to go ahead and thank my co-worker, William, for helping proof-read this post ahead of time as he pointed out some technical issues that I did not initially catch; specifically, that I should mention the required dependencies for SCT to compile and take some guesswork out, as well as some vague instructions towards the beginning of the post.

Obtaining Source Code

The first hurdle in dealing with source code is actually obtaining it. As far as the ‘how’, it really depends on what you are trying to build. That being said, many projects have either a read-only or an unofficial mirror on sites like GitHub.

For this example, there are two ways to go about getting the source code. Option 1: Click on the ‘Code’ button towards the top right of the GitHub page, then click ‘Download ZIP’ at the bottom of the small dialog window, or Option 2: use git to download the software. Both options work, however, using git will allow you to keep track of software changes (future updates, or custom patches added by you) easier than the ZIP file will.

# One option for opening the ZIP archieve if that was the chosen method
7z x sct-master.zip

# Alternatively, the command if git was used
git clone https://github.com/faf0/sct

From there, simply cd into the sct directory and we can begin the next step.

Building the software

Before building the software, we need to make sure we have all the required dependencies; I am not going to list the command to install the required dependencies for all Linux distros ever, but I will provide what you would need for Debian/Ubuntu/Mint. If you do not use one of those, you can search for similar package names in your package manager of choice and you should find it. Debian also does not, at least by default, include a compiler or make, so that has to be installed as well.

apt install -y libx11-dev libxrandr-dev make gcc

Now that we have the required libraries, we have to compile (or build) the software. This step is where things can vary heavily depending on a lot of different factors such as build system, language used, dependencies, target operating system, etc. Most projects will have instructions in their README file on what the dependencies are and roughly how to build it. However, it is important to read it very carefully as minor details can end up being the difference between successfully compiled software and hours of incredibly frustrating troubleshooting as to what is causing it to SEGFAULT. SCT happens to be a fairly easy software to build and can simply be built with make in the project directory. Once the software is built, it can be installed by running make install (again in the project directory). Wonderful, but what did we learn? Really nothing… So, let’s backup, what is make and why does it install stuff on my system?

The wonderful world of Makefiles

Make is a command line tool that is often used to automate building software, although it is not limited to just building software. Going over the ins-and-outs of all the things make can do is well outside the scope of this one blog post, however, if the reader is interested in learning more about the topic the GNU Make man page is a great place to start. This post will go into some general ideas to get started with reading a basic Makefile (configuration file for what make will do when invoked).

Getting back to the actually getting code compiled, let’s begin by looking at what the manual compilation command looks like, as that is what make will end up doing:

gcc -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include xsct.c -o xsct -L /usr/X11R6/lib -lX11 -lXrandr -lm -s

That looks super annoying to type each time you want to compile something; thankfully make can automate that. Let’s look at the Makefile to see what is going on in there:

CC ?= gcc
CFLAGS ?= -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include
LDFLAGS ?= -L /usr/X11R6/lib -s
PREFIX ?= /usr
BIN ?= $(PREFIX)/bin
MAN ?= $(PREFIX)/share/man/man1
INSTALL ?= install

PROG = xsct
SRCS = src/xsct.c

LIBS = -lX11 -lXrandr -lm

$(PROG): $(SRCS)
    $(CC) $(CFLAGS) $^ -o $@ $(LDFLAGS) $(LIBS)

install: $(PROG) $(PROG).1
    $(INSTALL) -d $(DESTDIR)$(BIN)
    $(INSTALL) -m 0755 $(PROG) $(DESTDIR)$(BIN)
    $(INSTALL) -d $(DESTDIR)$(MAN)
    $(INSTALL) -m 0644 $(PROG).1 $(DESTDIR)$(MAN)

uninstall:
    rm -f $(BIN)/$(PROG)
    rm -f $(MAN)/$(PROG).1

clean:
    rm -f $(PROG)

Anyone familiar with shell scripting may be able read and kind of see what the file is doing. However, not everyone is already used to shell scripting, so let’s add some comments to make it a bit more clear for those that are not as familiar:

# This section is just setting variables that will be used later
# Variables set with just '?=' are conditional variables
# that only matter if the variable is not already set by the environment.
# variables set with '=' are ones that will be used verbatim

# This variable is just setting the compiler, gcc in this case
CC ?= gcc

# This variable is setting some C compilation flags
CFLAGS ?= -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include
LDFLAGS ?= -L /usr/X11R6/lib -s

# This variable is setting the prefix for where things should be installed
# This doesn't /really/ matter, but on BSD systems, would probably
# be changed to `/usr/local`
PREFIX ?= /usr
# The $(PREFIX) is how the variable can be recalled
# In this case, it is just being called while setting another
# variable, which is quite common in Makefiles
BIN ?= $(PREFIX)/bin
MAN ?= $(PREFIX)/share/man/man1
# Install is a program in Linux and many other Unix-like systems
# You can check whether install is a program on your machine
# by running `which install`, my computer shows it is installed
# at `/usr/local/bin/install`
INSTALL ?= install

PROG = xsct
SRCS = src/xsct.c

# This variable is setting the libraries that must be invoked
# during compilation, in this case:
# X11 libraries, Xrandr libraries, and math libraries
LIBS = -lX11 -lXrandr -lm

# This part of the Makefile is where the compilation happens
# I will go through the specifics of this in a minute
$(PROG): $(SRCS)
    $(CC) $(CFLAGS) $^ -o $@ $(LDFLAGS) $(LIBS)

# This section is describing what `make` should do if
# `make install` is called
install: $(PROG) $(PROG).1
    $(INSTALL) -d $(DESTDIR)$(BIN)
    $(INSTALL) -m 0755 $(PROG) $(DESTDIR)$(BIN)
    $(INSTALL) -d $(DESTDIR)$(MAN)
    $(INSTALL) -m 0644 $(PROG).1 $(DESTDIR)$(MAN)

# This section is describing what `make` should do if
# `make uninstall` is called
uninstall:
    rm -f $(BIN)/$(PROG)
    rm -f $(MAN)/$(PROG).1

clean:
    rm -f $(PROG)

Often times, a large majority of a Makefile is just getting variables set properly, this example is no different. Let’s substitute the variables in the section that builds the software so it is a bit more straightforward to read:

xsct: src/xsct.c
    gcc -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include src/xsct.c -o xsct -L /usr/X11R6/lib -lX11 -lXrandr -lm

After all of the variables are replaced with what they will evaluate to (I know I glossed over the $^ and $@, I will come back to those), the build command in the Makefile looks almost identical to the manual build command.

Applying and Creating Patches for Software

When working with source code, you may eventually come across something that you want changed, modified, or removed. With proprietary applications this is next to impossible, however, with open source projects it is allowed and even encouraged. It is worth pointing out that while patch management is not required if the changes are only for you; utilizing version control and patching tools will make this process much easier even if you do not plan on contributing your changes back to the project (though you should if they are relevant).

Using Git

There are a lot of ways to go about version control and managing patches, however, I am only going to go over two. The first one will be some basic use of Git, and the second one will be more traditional Unix tools. I am going to start with Git as it is the industry standard for version control and managing patches by far. So, going back to the SCT project, let’s download it using git if we have not already done so (may need to install git on your machine). After that, we are going to make a new branch that we can make changes to.

# 'clone' or download the repository (and history) using git
git clone https://github.com/faf0/sct
cd sct
# Add new branch 'new-feature'
git branch new-feature
# Switch to branch 'new-feature'
git checkout new-feature
# Alternatively, you can create and switch to the new branch 'new-feature'
# in one command by running
git checkout -b new-feature

Now that we have a new branch, we can begin making modifications to the code. Just to make a simple example of this, I will be adding a line to the README.md file. To get the diff of the changes I have made versus what the master branch has, simply run:

git diff master

The results of this, at least for the change I made, are as follows:

diff --git a/README.md b/README.md
index afc027f..00b24df 100644
--- a/README.md
+++ b/README.md
@@ -21,6 +21,7 @@ Minor modifications were made in order to get sct to:
 - return `EXIT_SUCCESS`

 # Installation
+This is a new line to the README file.

 ## Make-based

To make a proper patch of the changes that were made, we need to commit the changes to the new branch, then make a patch file. A ‘commit’ is the way of telling git that you are happy with the changes and you would like them to be submitted into the tree. Essentially, this is the way to ‘save’ the file and be able to track those specific changes in the version history. Making a commit and a patch file can be done by running:

# This command adds the new or modified files to be commited
git add README.md
# This command makes the commit, the argument '-m' will make
# the commit message inline. Without this argument
# git will open your $EDITOR for the commit message to be
# typed out
git commit -m "Some helpful message"
# This command will format the patch file and compare
# the current commit branch to the master branch
git format-patch origin/master

The output of the last command will be a file that contains the diff as well as the commit message for the commits ahead of the master branch. Essentially, just describing the differences between the current branch and the master branch. From there, that patch file can be stowed away, emailed, or otherwise shared. Now let’s see how to apply that patch to the code base:

# Switches back to the master branch to 'undo' our changes
git checkout master
# Creating a test branch, it is bad practice to modify the master branch directly
git checkout -b apply-test
# Then apply the patch, it is worth noting that the .patch file
# may have a different name that what I have here
git apply 0001-Test-change.patch

This is the absolute basics of dealing with patches in git; there is a lot about git that I did not cover, because it would make this blog post a medium length novel. So, rather than boring you, I will just provide places to get more information on working with Git and open source code:

• Git documentation
• How to contribute to open source
• Many projects have a ‘how to contribute’ section in their documentation

Manual way of dealing with patches

In the interest of history, I am also going to give a quick overview of how to deal with patches without using git. This is not something I recommend doing without a reason as it is quite a bit more painful, at least in my opinion. It also does not transfer to many open source projects the same way that learning git does; however, it is important to keep the old ways in mind.

So, let’s go back before any changes were made to any files for the application. Before we actually work on any changes, we need to copy the file(s) that we are going to work on to another file; generally, the naming convention is to just append the filename with ‘.orig’, but anything will work as long as you keep up with it. After making the desired changes (will be same changes as the previous section) you can run the following command:

# The '-u' flag is to make the output of the diff
# look more similar to the git output, I find it easier
# read than the default output that diff uses
diff -u README.md.orig README.md
# To save the output into a file, simply use the Unix file redirection
diff -u README.md.orig README.md > README.md.patch

The patch file can then be sent and shared around similar to the patch file using git. Now how do we apply these patches to a code base? With a utility called patch, that will likely not be on the machine by default, thus will need to be installed. Once installed, patches can be applied as follows:

# Notice the '<' rather than the '>', if the '>' is used, the program will just hang
patch < README.md.patch README.md

Closing thoughts

Working with source code can be intimidating at times, especially if you are not used to doing it. Hopefully, this post helps alleviate some of the fears and concerns with working with source code, and with making patches to software.

Abstract

Learning to run and maintain an email system can be a daunting task, especially for a young IT professional that potentially did not use email before starting their career (me). That are a lot of prerequisite skills that are required to get the service to run at all, then there are more pieces of knowledge that must be sprinkled in to make the service actually useful (by getting mail delivered and blocking spam). I did a write-up on getting email setup on a previous blog post, however, that post did not really explain any of the logic that went into setting up an email server. This article intends to provide that basic overview of how the various pieces of email fit together to make a working system. It is also worth noting that this post will not go in-depth on configurations for anything as this is supposed to be a general overview; many of the configuration items that are required can be found online for the platform you are using. Additionally, I will have more resources that go more in-depth on email at the end of this post.

Prerequisite Skills

As eluded to in the intro paragraph, email is built on various other skills that are critical to being able to run a successful mail service. These skills are:

• System Administration for relevant operating system(s)
• Domain Name System (DNS)
• Networking
• Understanding SSL/TLS (and which one you should be using) is a plus

Basic Requirements for Email Server

In earlier days of the Internet, the list of requirements for running an email server was pretty short. You basically just needed a server, a domain name, and some software to send and receive email (SMTP and POP3/IMAP server). Once that is setup, then you can be on your way sending all the email that you want. Technically that is still the case; however in practice, the requirements are a bit higher today for emails to actually be received by major providers such as Gmail, Yahoo, and Outlook/Hotmail.

Hard Requirements

This list is a minimal list of requirements to have the technical capability to send and receive email.

• Domain Name: The server that is receiving the email should have a domain name that is available on the hosts that will be sending email to the server. The domain is probably not a “hard” requirement if email is only being send internally, but then you are forced to use IP addresses as the domain, which is terrible to work with. This is even less practical on the open Internet, as not only would people have to type your public IP to send anything to you, you also couldn’t do any of the soft requirements in the next section. Without those, email will likely be blocked by a large group of people you may want to send messages to. Specifically, you want an MX record for your domain.
• IMAP Server: An Internet Message Access Protocol (IMAP) server. This is what will actually be receiving the mail that is sent to your server; without it, the email server will effectively be deaf, and not able to actually receive any messages. There are quite a few IMAP servers out there, but one of the most popular IMAP servers for Linux/*BSD is Dovecot. In Microsoft world, the IMAP and SMTP server will almost certainly be Exchange, which can be hosted locally (though they are attempting to move everyone into the cloud).
• SMTP: Simple Mail Transfer Protocol (SMTP). This is what is responsible for sending email. Traditionally Sendmail was the tool of choice for Unix admins, however, much like IMAP, there are a variety of options to choose from. And again, like IMAP, in the land of Microsoft the only choice is Exchange.

Soft Requirements

While these items are not technically required to have mail sent to another email server, for the mail to actually be accepted and received by other email servers on the Internet, these requirements must be met.

• TLS: Transport Layer Security (TLS), specifically TLS 1.2 or higher as other versions are deprecated because of weak cryptography. Thanks to Let’s Encrypt, there is absolutely no reason to not have TLS on your website, your email, and anything else that might benefit from it.
• SPF: Sender Policy Framework (SPF) is a DNS authentication protocol that declares a list of valid senders for a particular email domain. SPF can also give hints to remote servers what should be done if the email domain in question does not pass SPF. Some explanation on creating an SPF record with some examples can be found here.
• DKIM: DomainKeys Identified Mail (DKIM) is another DNS authentication protocol that cryptographically authenticates the domain’s emails via public/private key pairs. Some explanation on creating DKIM records with examples can be found here.
• DMARC: Domain-based Message Authentication Reporting and Conformance (DMARC) is an email authentication protocol that works in conjunction with DKIM and SPF to help prevent email domain spoofing. It does this by instructing remote mail servers what to do if the received email does not pass DKIM or SPF, as well as what to do if there is a misalignment of either. A bit more explanation on how DMARC works and how it can be configured for your domain can be found here.

Checking your records

Once you think your records are setup for your email domain, it is then time to check it using online tools. This Email health tool from MX Toolbox is a very solid one that will show a lot of information about your email domain and point out issues that should be addressed. Example output from my mail server:

It is highly recommend to make sure the checks on this health check tool pass before changing SPF to hard-fail and DMARC to reject or quarantine as improper configuration of DNS records can affect mail deliverability. If something is not setup correctly on the DNS records, MX Toolbox will very helpfully offer some additionally info to clear up what the problem might be.

Handling Spam

Most people hate email spam; and the reasons for this are two fold. The obvious reason is because you have to deal with the spam in your email box. However, the second reason is that email spam has been a huge driving force to make email the headache that it is today. The idea was that by having email providers declare where email will come from and showing some validation that it was really sent from their email server (SPF and DKIM) it would cut down on spoofing and spam. The ironic thing, is that spammers tend to have the best SPF, DKIM, and DMARC records, and will generally allow anyone to send email on their behalf, and will actively tell other email server to ignore DKIM.

Spam filters

Since SPF, DKIM, and DMARC do little to prevent spam, the option is an email spam filter. While this isn’t required per se, in reality this is something you will likely want sooner or later, especially if your email domain is associated with a business. Spam filters come in software form or a hardware appliance; each of the different options for a spam filter will work slightly differently, however, in general spam filters will analyze incoming email and give it a spam score. This score will determine whether the email will be considered spam by the filter; in addition to this, actions can be setup on a tier system, so an email with a lower score could be handled differently than an email with a higher score.

To determine the spam score of the incoming emails, various parts of the email will be analyzed including:

• SPF records, missing records or failing SPF will negatively impact score
• Email headers (will explain more in next section)
• The language of subject and body of the message
• Some filters also aggregate data to determine bad actors en mass for lots of customers

The last point is the one that can be really difficult to get around if you run your own mail server. Specifically, entire IP ranges can get blocked because too much spam was being sent; this will affect the entire IP block’s ability to send email. Microsoft did this to Linode, and was not very helpful in resolving the issue until much, much later. And, even when they did fix it, it was only silently.

Dissecting Email Headers

Many spam filters look at email headers for information about the emails to attempt to determine whether they are spam or not. Unfortunately, email headers are roughly as readable as compiler output and can look like a wall of information if you haven’t worked with them before. Here is an example email header that we will be working with:

From 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com Fri Dec  6 07:30:08 2024
Delivered-To: person@email.com
Received: by 2002:aa6:c561:0:b0:2ac:443d:529e with SMTP id z1csp807162lkp;
        Fri, 6 Dec 2024 04:30:09 -0800 (PST)
X-Google-Smtp-Source: AGHT+IGGzFj9rG8DAig1r5R5L5v4J0hhtWtnpjSTxXCTgWwNr2/+2Z/kUANsVLg/AwSqXtLz4Ksi
X-Received: by 2002:a05:6214:5086:b0:6d8:9abb:3c28 with SMTP id 6a1803df08f44-6d8e71ad0abmr48222746d6.29.1733488208169;
        Fri, 06 Dec 2024 04:30:08 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1733488208; cv=none;
        d=google.com; s=arc-20240605;
        b=I1tzm9YUP3Ow+AtYga97dUCaUscPCFS4+Ghqgd0MPVNl/cUU8bWx5kWVtv1gLTNYot
         5NLiqCaNvxoXOwbYMGBtO/4nkoZ1Wml4Mtm1PyA3LsnjIUnCkdikjaevpXlSHzzr70/+
         Wf2EnzBykF0oGUyWGZKTZdvZgDHnh1aY9PZG5qEiIDT5YFmO86szXO5MGPfZSqHFDZxe
         TjKvhopvfhFH+Q6rEbXXemHjEYl1YG7IINGJ40DNVZpLwZzc7zIU9w8dbGZEbu8A7aRc
         VhIze0e0eYugkCHYBEO0R7x2mHtc1EHe9M18VaUP7HXV8ePLLB1/gJ9IUEA8WL0LauyT
         RWkQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605;
        h=feedback-id:date:message-id:list-unsubscribe-post:list-unsubscribe
         :reply-to:subject:to:from:mime-version:dkim-signature:dkim-signature;
        bh=UbDnVmK+d3a5KWc8tc7jzr3yiQ1VTsl2vRyjB3oX5YE=;
        fh=cYhEvxUJVCrPuHhuhnYjrlZs1arjc/bcOjCL+MocS9Y=;
        b=U1t7XlEIb2HYPoHqDlfMkM5oynMZDQJKj56UN52F+ymAN5+GShWXIn3kyXdzgHhT4v
         GV/nlt075V17N+5/i+lOdUEnkjdQfGKWS11qtqLQhAYLFVWo5PApv5LdHolbPHuciM9w
         9bCxduC5qFsRBtmnR7JRg912F2yU9iml4vQxg5goUMPbJqtcv492ZHclERm+wWAM36f8
         39MqrjWuOjrSyu0+JhrJZeJio0t1f6PKppslQz7NM88LJUFtaha8QKyyjSZk1q0/KJr2
         dsrlqNkeaqVN/s6ri9V2QZU2upjaIj7MfGwtlVuCKm2D/eiufH5wYuku8aTxexLjsVKV
         N6lw==;
        dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@duolingo.com header.s=nrqmk37y4yzwsm3xc6vb7hxfdn5ugmwd header.b=n3+GQSgE;
       dkim=pass header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=Hwmiz7tQ;
       spf=pass (google.com: domain of 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com designates 54.240.10.133 as permitted sender) smtp.mailfrom=010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com;
       dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=duolingo.com
Return-Path: <010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com>
Received: from a10-133.smtp-out.amazonses.com (a10-133.smtp-out.amazonses.com. [54.240.10.133])
        by mx.google.com with ESMTPS id 6a1803df08f44-6d8dac0949esi42739666d6.346.2024.12.06.04.30.07
        for <person@email.com>
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 06 Dec 2024 04:30:08 -0800 (PST)
Received-SPF: pass (google.com: domain of 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com designates 54.240.10.133 as permitted sender) client-ip=54.240.10.133;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@duolingo.com header.s=nrqmk37y4yzwsm3xc6vb7hxfdn5ugmwd header.b=n3+GQSgE;
       dkim=pass header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=Hwmiz7tQ;
       spf=pass (google.com: domain of 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com designates 54.240.10.133 as permitted sender) smtp.mailfrom=010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com;
       dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=duolingo.com
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
    s=nrqmk37y4yzwsm3xc6vb7hxfdn5ugmwd; d=duolingo.com; t=1733488207;
    h=Content-Type:MIME-Version:From:To:Subject:Reply-to:List-Unsubscribe:List-Unsubscribe-Post:Message-ID:Date;
    bh=UbDnVmK+d3a5KWc8tc7jzr3yiQ1VTsl2vRyjB3oX5YE=;
    b=n3+GQSgE3I9a8CN9baF2Qic3u+4XWRMKMtXumZpBi2x6E5W5LQbrz8JazcbP2Pk8
    stEVgA9hym6Uoj0PeVdjFKSXCxqLVw8R6dTyE3ab1Gvbx6A5j+IYUq1bkwLyG57YngQ
    oFX5uSARMoT7w5aKlM+0NcsF3tXXePz0b0L2TVRY=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
    s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1733488207;
    h=Content-Type:MIME-Version:From:To:Subject:Reply-to:List-Unsubscribe:List-Unsubscribe-Post:Message-ID:Date:Feedback-ID;
    bh=UbDnVmK+d3a5KWc8tc7jzr3yiQ1VTsl2vRyjB3oX5YE=;
    b=Hwmiz7tQUR/YIbVr4rhu2ecJ/d1KJdvvSqospSf7JElz8cRQAJSh+09Jyg38RJYO
    /e8yIMWvnLI+zsWD5wHZ+NGTKGWrOHyg4qr4Og06KAZyEgOeksu3nCahR6SfSBI2l0f
    PuxaO7Pu0EuR97IPfsLQQWb3Y9E5MZGxavfDA7Zw=
Content-Type: multipart/alternative; boundary="===============0697970980562405415=="
MIME-Version: 1.0
From: Duolingo <no-reply@duolingo.com>
To: person@email.com
Subject: =?utf-8?q?=F0=9F=98=B2_Your_Year_in_Review_is_here=2E?=
Reply-to: Duolingo <no-reply@duolingo.com>
List-Unsubscribe: <https://blast.duolingo.com/web-redirect/200567?from_email=16715badf50216a7b0f662868d27190658681a0eImZhY2VwbGFudDM2QGdtYWlsLmNvbSI=&user_id=cdcde602db5914665fd0d6917a97c8faa53d4f23NTMxMTU5OQ==&email=16715badf50216a7b0f662868d27190658681a0eImZhY2VwbGFudDM2QGdtYWlsLmNvbSI=&sample_id=8603&target=aHR0cHM6Ly93d3cuZHVvbGluZ28uY29tL3Vuc3Vic2NyaWJlP3R5cGU9bm90aWZ5X2Fubm91bmNlbWVudCZlbWFpbD1bRW5jb2RlZEVtYWlsXSZ1dG1>
List-Unsubscribe-Post: List-Unsubscribe=One-Click
Message-ID: <010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@email.amazonses.com>
Date: Fri, 6 Dec 2024 12:30:07 +0000
Feedback-ID: ::1.us-east-1.RpxhJRmOpL41XzJPFX+GBBQj4+ioASSIVb8HK9KAN9A=:AmazonSES
X-SES-Outgoing: 2024.12.06-54.240.10.133

Again, massive wall of barely parsable (by a human) text. So, let’s look for our basic information and go from there.

From 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com Fri Dec  6 07:30:08 2024
Delivered-To: person@email.com
Received: by 2002:aa6:c561:0:b0:2ac:443d:529e with SMTP id z1csp807162lkp;
        Fri, 6 Dec 2024 04:30:09 -0800 (PST)
Received: from a10-133.smtp-out.amazonses.com (a10-133.smtp-out.amazonses.com. [54.240.10.133])
        by mx.google.com with ESMTPS id 6a1803df08f44-6d8dac0949esi42739666d6.346.2024.12.06.04.30.07
        for <person@email.com>
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 06 Dec 2024 04:30:08 -0800 (PST)
Received-SPF: pass (google.com: domain of 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com designates 54.240.10.133 as permitted sender) client-ip=54.240.10.133;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@duolingo.com header.s=nrqmk37y4yzwsm3xc6vb7hxfdn5ugmwd header.b=n3+GQSgE;
       dkim=pass header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=Hwmiz7tQ;
       spf=pass (google.com: domain of 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com designates 54.240.10.133 as permitted sender) smtp.mailfrom=010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com;
       dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=duolingo.com
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
    s=nrqmk37y4yzwsm3xc6vb7hxfdn5ugmwd; d=duolingo.com; t=1733488207;
    h=Content-Type:MIME-Version:From:To:Subject:Reply-to:List-Unsubscribe:List-Unsubscribe-Post:Message-ID:Date;
    bh=UbDnVmK+d3a5KWc8tc7jzr3yiQ1VTsl2vRyjB3oX5YE=;
    b=n3+GQSgE3I9a8CN9baF2Qic3u+4XWRMKMtXumZpBi2x6E5W5LQbrz8JazcbP2Pk8
    stEVgA9hym6Uoj0PeVdjFKSXCxqLVw8R6dTyE3ab1Gvbx6A5j+IYUq1bkwLyG57YngQ
    oFX5uSARMoT7w5aKlM+0NcsF3tXXePz0b0L2TVRY=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
    s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1733488207;
    h=Content-Type:MIME-Version:From:To:Subject:Reply-to:List-Unsubscribe:List-Unsubscribe-Post:Message-ID:Date:Feedback-ID;
    bh=UbDnVmK+d3a5KWc8tc7jzr3yiQ1VTsl2vRyjB3oX5YE=;
    b=Hwmiz7tQUR/YIbVr4rhu2ecJ/d1KJdvvSqospSf7JElz8cRQAJSh+09Jyg38RJYO
    /e8yIMWvnLI+zsWD5wHZ+NGTKGWrOHyg4qr4Og06KAZyEgOeksu3nCahR6SfSBI2l0f
    PuxaO7Pu0EuR97IPfsLQQWb3Y9E5MZGxavfDA7Zw=
From: Duolingo <no-reply@duolingo.com>
To: person@email.com
Subject: =?utf-8?q?=F0=9F=98=B2_Your_Year_in_Review_is_here=2E?=
Reply-to: Duolingo <no-reply@duolingo.com>
Message-ID: <010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@email.amazonses.com>
Date: Fri, 6 Dec 2024 12:30:07 +0000

There is still a lot of information there, but it should look a bit less intimidating and some of the fields may even look obvious. For example, Delivered-To, is the email address that the email was delivered too. The subject, is the subject of the email, etc. Some of the things that might look slightly less straightforward are going to be things like the Received-SPF and DKIM-Signature. In the case of the Received-SPF:

Received-SPF: pass (google.com: domain of 010001939bf2e76b-c528bc30-1748-4eba-9ca2-91507276373b-000000@bounces.duolingo.com designates 54.240.10.133 as permitted sender) client-ip=54.240.10.133;

We can tell that the message passed SPF because of the pass toward the beginning of the line. That is auto echoed in the section for Authentication-Results. That section also shows that DKIM was passed as well, Then at the bottom of the Authentication-Results we see the line dmarc=pass then some more information on what the email is supposed to do if DMARC is not passed. This means that this email has passed the three main email authentication requirements. Past that, we also get a specific time stamp the email was received, along with a Message-ID to better track down the email if we need to look for it.

Great, but what was the other stuff that got removed? Some of it was junk that doesn’t really affect anything. That is pretty much any line that starts with X-. Those headers are not standardized, and thus can kind of be made up by the email composer. The other interesting section is the ARC-Authentication-Results. Authenticated Received Chain (ARC) is a method of validating email that could be changed by a spam filter. Some spam filters will modify an email’s headers, and this modification can cause the email to fail DKIM when it gets through to the user. ARC fixes this by allowing intermediate servers to sign off on the original validation results. This helps prevent authentication issues, and increases deliverability.

Resources

Email is a topic that has a lot happening all at once, and is not really something for people just getting started with IT. It has a lot of concepts that you really should know before jumping in, and there are a lot of things that must be done correctly for it to actually work. Unfortunately, it is also a moving target. What works today, may not work tomorrow if the big mail providers change their mind on something. This blog post is not intended to be an in-depth tutorial on anything, rather, just a primer to get familiar with some of the different aspects of email and a general idea of how they fit together. If you are interested in running your own mail server, I have included some resources that I highly recommend checking out before doing so:

• Solene: Setting Up Open BSD as an Email Server
• Michael W Lucas: The State of Email NYC*BUG
• Michael W Lucas: Run Your Own Mail Server
• MailerCheck: ]]> https://foxide.xyz/projects/2024-12-06.html https://foxide.xyz/projects/2024-12-06.html Fri, 06 Dec 2024 00:00:00 -0500

  Abstract

  In many cases, it can be difficult to gather information about problems with resources using the same tools that an end user would use to access those resources. For example, it may be difficult to understand an email authentication issue by just using an email client to attempt to authenticate. Sometimes the client will provide helpful error codes that can be searched, but other times it might give you a very generic error that is difficult to track down. In such situations, it might be nice to be able to interact with the application a bit more directly to see what might be going on. This is where tools like Telnet and OpenSSL can help out.

  Telnet is deprecated and insecure, why use it?

  Telnet was originally used as a network protocol to interact with systems remotely, however, because it is a clear text protocol it is highly insecure and is extremely susceptible to various network attacks. That being said, Telnet does still have modern uses. As stated from the Wikipedia article on Telnet:

  The Telent client may be used in debugging network services such as SMTP, IRC, HTTP, FTP or POP3, to issue commands to a server and examine the responses. For example, Telnet client applications can establish an interactive TCP session to a port other than the Telnet server port. However, communication with such ports does not involve the Telnet protocol, because these service merely use a transparent 8-bit TCP connection, because most elements of the telnet protocol were designed around the idea of accessing a command line interface and non of these options or mechanisms is employed in most other internet service connections.

  Essentially, telnet can be used to make a TCP connection on an arbitrary port and the admin can make command line calls to the service to see what the responses are. As an example of this from this StackOverflow post:

  telnet stackoverflow.com 80
  Trying 172.64.155.249...
  Connected to stackoverflow.com.
  Connection closed by foreign host.
  Escape character is '^]'.
  GET /questions HTTP/1.0
  Host: stackoverflow.com
  
  HTTP/1.1 403 Forbidden
  Date: Tue, 17 Dec 2024 02:14:16 GMT
  ...

  Running those commands will spit out the application headers (HTTP headers in this case) as well as what the response would be from a given client. This example is for HTTP. Using similar commands for this website gives the following response:

  Trying 172.245.181.191...
  Connected to foxide.xyz.
  Escape character is '^]'.
  GET /index.html HTTP/1.1
  Host: foxide.xyz
  
  HTTP/1.1 301 Moved Permanently
  Server: nginx/1.26.2
  Date: Tue, 17 Dec 2024 02:20:57 GMT
  Content-Type: text/html
  Content-Length: 169
  Connection: keep-alive
  Location: https://foxide.xyz/index.html
  
  <html>
  <head><title>301 Moved Permanently</title></head>
  <body>
  <center><h1>301 Moved Permanently</h1></center>
  <hr><center>nginx/1.26.2</center>
  </body>
  </html>
  ^CConnection closed by foreign host.

  Nginx returns a 301 error, because I automatically re-direct HTTP traffic on port 80 to HTTPS traffic on port 443. Since telnet does not support encryption, how would one go about doing this for HTTPS traffic?

  OpenSSL for Modern Command Line Troubleshooting

  The tool to troubleshoot modern applications that are behind (most likely) Transport Layer Security TLS, and thus telnet will not be able to understand the encrypted traffic; this is where a tool like OpenSSL comes into play. Specifically, OpenSSL’s s_client function; from the OpenSSL man`` page ons_client**:

  This implements a generic SSL/TLS client which can establish a transparent connection to a remote server speaking SSL/TLS. It’s intended for testing purposes only and provides only rudimentary interface functionality but internally uses mostly all functionality of the OpenSSL ssl library.

  Here is the analog of the telnet example in the previous section, but using OpenSSL’s s_client to perform the same task.

  openssl s_client -connect foxide.xyz:443
  Connecting to 172.245.181.191
  CONNECTED(00000003)
  depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
  verify return:1
  depth=1 C=US, O=Let's Encrypt, CN=E6
  verify return:1
  depth=0 CN=foxide.xyz
  verify return:1
  ---
  Certificate chain
   0 s:CN=foxide.xyz
     i:C=US, O=Let's Encrypt, CN=E6
     a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
     v:NotBefore: Oct 21 03:09:36 2024 GMT; NotAfter: Jan 19 03:09:35 2025 GMT
   1 s:C=US, O=Let's Encrypt, CN=E6
     i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
     a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
     v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
  ---
  Server certificate
  -----BEGIN CERTIFICATE-----
  MIIDmjCCAx+gAwIBAgISBO5wd7WEOdqbgg2ZxG8R9J3ZMAoGCCqGSM49BAMDMDIx
  CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF
  NjAeFw0yNDEwMjEwMzA5MzZaFw0yNTAxMTkwMzA5MzVaMBUxEzARBgNVBAMTCmZv
  eGlkZS54eXowWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASB3sT482vbUr8zsGWd
  GGvRVctZVGcXPschnLuTUoxFTT/sgsB6CnyfD3gqIKqGppb4L58t1wdNxMmmCZ0A
  Ptrco4ICMDCCAiwwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMB
  BggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQtds7pKOO9N/h5hM1N
  ovja00JAwzAfBgNVHSMEGDAWgBSTJ0aYA6lRaI6Y1sRCSNsjv1iU0jBVBggrBgEF
  BQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9lNi5vLmxlbmNyLm9yZzAiBggr
  BgEFBQcwAoYWaHR0cDovL2U2LmkubGVuY3Iub3JnLzA6BgNVHREEMzAxghJjb250
  YWluLmZveGlkZS54eXqCD2RvY3MuZm94aWRlLnh5eoIKZm94aWRlLnh5ejATBgNV
  HSAEDDAKMAgGBmeBDAECATCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AKLjCuRF
  772tm3447Udnd1PXgluElNcrXhssxLlQpEfnAAABkq1CxWcAAAQDAEYwRAIgFY8h
  E3vu1bTL2UmWBypEhfbN5Fw/cgYjtK0nbwf7YAICIGePDHPQIOpKrkRbGgXx+jxh
  qvwyJOCssJaGs/medWCBAHYAE0rfGrWYQgl4DG/vTHqRpBa3I0nOWFdq367ap8Kr
  4CIAAAGSrULG2AAABAMARzBFAiATEsvrayeCsWpcm1sGzmuBFP8sazeCDe6i8bfY
  LrjV2wIhAPVshF00u1Chtu5C8LEhXhLX3vVNOSClQfAlm/6ZoEsIMAoGCCqGSM49
  BAMDA2kAMGYCMQCGExlw5KTniWzQqLRodBDWzGT6RB/TM5Kaux/ARKiX0N7OQnV6
  SsJ6xyX+74uyUAQCMQCel3ISqjn2Y0oMBiR5d2p/j8p71KwUTPEnmCkzBBWDDKUE
  yF9vARxsOe3ltetRRno=
  -----END CERTIFICATE-----
  subject=CN=foxide.xyz
  issuer=C=US, O=Let's Encrypt, CN=E6
  ---
  No client certificate CA names sent
  Peer signing digest: SHA256
  Peer signature type: ECDSA
  Server Temp Key: X25519, 253 bits
  ---
  SSL handshake has read 2421 bytes and written 398 bytes
  Verification: OK
  ---
  New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
  Server public key is 256 bit
  This TLS version forbids renegotiation.
  Compression: NONE
  Expansion: NONE
  No ALPN negotiated
  Early data was not sent
  Verify return code: 0 (ok)
  ---
  ---
  Post-Handshake New Session Ticket arrived:
  SSL-Session:
      Protocol  : TLSv1.3
      Cipher    : TLS_AES_256_GCM_SHA384
      Session-ID: DECD4B808369B03981BD1DFD7CB9510D851C9BBBD77F38AB0C5A4E8916803BC7
      Session-ID-ctx: 
      Resumption PSK: 5AE0AC37D01DC46CECFE733C43D6DDCEECEE801780232EA5B2EE24ED38FFC77B7D0A57A27270B75F1A07F39ACAEB9A29
      PSK identity: None
      PSK identity hint: None
      SRP username: None
      TLS session ticket lifetime hint: 86400 (seconds)
      TLS session ticket:
      0000 - 8b f7 43 30 03 60 3b 21-10 ae 15 9b bd 17 ac d8   ..C0.`;!........
      0010 - f5 32 77 0e 39 af 1a 90-ee f0 1c 6d fe f8 8b 95   .2w.9......m....
  
      Start Time: 1734663386
      Timeout   : 7200 (sec)
      Verify return code: 0 (ok)
      Extended master secret: no
      Max Early Data: 0
  ---
  read R BLOCK
  ---
  Post-Handshake New Session Ticket arrived:
  SSL-Session:
      Protocol  : TLSv1.3
      Cipher    : TLS_AES_256_GCM_SHA384
      Session-ID: 90EACD532A3149074067947556B18A029C09188C70D94D06E48737504FE3B6EF
      Session-ID-ctx: 
      Resumption PSK: 2B3A17104B7E4978AFEFE581A53DAEC95C0AC1C409FDB8EE0D5CF5CF5FA8D91F4653875A3670E656D21F3F5F72F69BC9
      PSK identity: None
      PSK identity hint: None
      SRP username: None
      TLS session ticket lifetime hint: 86400 (seconds)
      TLS session ticket:
      0000 - f3 da a3 44 ad b1 8d 65-ad 8c a3 09 95 f2 7d 87   ...D...e......}.
      0010 - c6 c6 f6 98 38 f8 7b 69-c4 c9 af 6c f0 35 ee 35   ....8.{i...l.5.5
  
      Start Time: 1734663386
      Timeout   : 7200 (sec)
      Verify return code: 0 (ok)
      Extended master secret: no
      Max Early Data: 0
  ---
  read R BLOCK
  GET /index.html HTTP/1.1
  Host: foxide.xyz
  
  HTTP/1.1 200 OK
  Server: nginx/1.26.2
  Date: Fri, 20 Dec 2024 02:56:46 GMT
  Content-Type: text/html
  Content-Length: 1157
  Last-Modified: Tue, 01 Oct 2024 02:28:32 GMT
  Connection: keep-alive
  ETag: "66fb5e50-485"
  Accept-Ranges: bytes
  
  <!DOCTYPE html>
  <html lang="en">
      <head>
          <meta charset="utf-8" />
          <meta name="viewport" content="width=device-width, initial-scale=1" />
          <link rel="icon" type="image/png" href="favicon.png" />
          <link rel="stylesheet" type="text/css" href="style.css" />
          <link rel="stylesheet" type="text/css" href="../style.css" />
          <title>Tyler's website</title>
      </head>
  
      <body>
          <header>
              <h1>Tyler's Site</h1>
          </header>
          <nav>
              <a href="https://foxide.xyz/">home</a>&emsp;
              <a href="https://foxide.xyz/articles.html">articles</a>&emsp;
  			]]> 
  		 
  		https://foxide.xyz/projects/2024-12-20.html
  		https://foxide.xyz/projects/2024-12-20.html
  		Fri, 20 Dec 2024 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  The Logical
  Volume Manager (LVM) is a framework that provides logical volume
  management.
  
  
  
  
  Though the description for logical volume management is on another
  Wikipedia page.
  
  In computer storage, logical volume management or LVM provides a
  method of allocating space on mass-storgae devices that is more flexible
  than conventional partitioning schemes to store volumes. In particular,
  a volume manager can concatenate, stripe together or otherwise combine
  partitions (or block devices in gneeral) into larger virtual partitions
  that administrators can re-size or move, potentially without
  interrupting system use.
  
  In simpiler terms, logical volume management allows software to
  create a virtualized partitioning scheme made from one or many
  partitions, disks, or other block devices. Doing this allows the virtual
  file system to comine them in many different any interesting ways such
  as RAID-1 (mirroring), RAID-0 (striping), and other RAID-like schemes.
  Linux’s LVM also allows for adding, removing, or resizing partitions on
  a live system, though the file system itself will have to be resized. In
  general, LVM offers many of the same features that next generation copy
  on write (CoW) file systems, such as ZFS, btrfs, and bcachefs offer. The
  main downsides to LVM over something like ZFS or btrfs is that LVM
  management can be more complex the equivalent configurations on other
  CoW file systems, as well as LVM is generally not going to perform as
  well as CoW file systems. This is because LVM is a layer that sits on
  top of the file system, rather than the file system providing the
  management directly. With those points in mind, let’s setup an
  environment to start working with LVM.
  Lab Setup
  For this lab, I am using a KVM virtual machine running Void Linux, though distro shouldn’t
  matter too much. The VM’s root drive is 30GB, then virtual disks will be
  added or removed depending on what is needed for the particular section.
  I am not going to go over the specifics of the Void Linux installation,
  if you are wanting to follow along using Void Linux and need help, you
  can check out their installation tutorials:
  
  Basic
  Installation Guide
  Installation
  via chroot
  
  It is also important to load (at least on Void Linux) the proper
  kernel modules; the ones I have had to make sure are loaded for things
  to work properly are:
  
  raid1
  dm-raid
  
  This
  page in the Void Linux handbook explains how to automatically load
  kernel modules on Void Linux.
  Setting up LVM in a
  RAID-like configuration
  LVM gives the ability to setup software RAID on multiple partitions
  or disks. For this example, I am going to be using disks
  /dev/sda and /dev/sdb as the disks on the test
  machine; I am also going to setup a mirror, however, I will also explain
  how to setup a stripe or other kinds of RAID.
  The first step of the process is to partition the disk(s), and while
  this isn’t a hard requirement, it does make managing the pool
  easier later on. If we were to add the raw disks to the pool without
  partitioning them, other systems may not pick up on the LVM pool and
  could overwrite the data on the disk(s). So, first we will create one
  partition on each disk that takes the entirety of the space available on
  the disk. There are many different tools that can do this, I find the
  simplest one to be cfdisk, but use what you are comfortable
  with.
  # Run the following commands as root
  
  # Opens /dev/sda in cfdisk allowing for partitioning
  cfdisk /dev/sda
  # Opens /dev/sdb in cfdisk allowing for partitioning
  cfdisk /dev/sdb
  Then create a Physical Volume (PV); this step allows
  for creating a Volume Group (VG) using the devices, as well as writing
  some meta data to the group that labels it as an LVM group. Creating a
  PV and a VG is very straightforward and will only take two commands. The
  first one will initialize the PV using partitions /dev/sda1
  and /dev/sdb1, then we will create a volume group named
  STORAGE with the same partitions. That process looks as follows:
  # Run as root
  pvcreate /dev/sda1 /dev/sdb1
  vgcreate STORAGE /dev/sda1 /dev/sdb1
  Now that the volume group is initialized, we can get onto creating a
  usable file system that can be mounted and store data. The command that
  does this is lvcreate. This command is very powerful and
  can cover a lot of different things that an admin might want to do with
  their storage pool, however, this blog post will only cover very basic
  use of this command to get setup quickly, however, more options are
  available by looking at the man
  page.
  # Run as root
  
  # The following command will create a stripe of the two disks in the VG 'STORAGE',
  # and uses the '-n' flag to name it 'storage_volume'. The '-l' flag is neat as it
  # specifies what percentage of the disk should be used rather than how many gigabytes.
  # This example will use 100% of the avialable disk space (should be around 80G).
  lvcreate -l 100%FREE -n storage_volume STORAGE
  
  # This command will create a mirror (RAID-1) of the two disks in the same VG
  # and with the same name as the command above. However, because this is a RAID-1
  # The available disk space will be 40G rather than 80G because of the mirror.
  lvcreate -l 100%FREE --type raid1 -n storage_volume STORAGE
  
  # Then create a file system in the storage volume so that we can mount it on our system
  mkfs.ext4 /dev/mapper/STORAGE-storage_volume
  There are a few things worth noting at this point. The first thing
  worth mentioning is the difference between the -l and the
  -L flags. The uppercase (-L) allows
  administrators to specify the size of the pool by setting its size in
  gigabytes; while the lowercase (-l) allows setting the size
  of the pool based on a percentage of the disk. Take the following
  commands:
  # This command will set the available storage capacity to 35 Gigabytes.
  # Attempting to allocate more space than is available will cause lvcreate
  # to fail with error 5.
  lvcreate -L 35G -n --type raid1 -n storage_volume STORAGE
  
  # This command will allocate 90% of the available space in the storage pool
  # Which is about 36G in our 40G disks.
  lvcreate -l 90%FREE -n --type raid1 -n storage_volume STORAGE
  Another point worth noting is the various types of RAID that are
  available using lvcreate. According to the LVMRAID
  man page, LVM supports the following RAID types when using the
  --type flag:
  
  RAID0: Also known as a stripe. This RAID type will combine the
  devices into one large pool, but offers no redundancy. This type
  can be used to increase performance, but is generally not
  recommended for stand alone use.
  RAID1: Also known as mirroring. This RAID type will span the data
  across multiple devices so that if a device is lost, the data is not.
  The minimum amount of devices required for this type is two.
  RAID4: This is another type of striping that uses a device as a
  parity device, which will provide a level of redundancy. In this RAID
  type, if one device is lost data can be recovered. Minimum devices
  required for this RAID type is three
  RAID5: This RAID type is similar to RAID4 in the sense that it is a
  form of striping that uses a parity device, however, the difference
  between RAID4 and RAID5 is the way the parity is handled. With a RAID4
  the parity blocks are always stored on the same device, while in RAID5
  the parity blocks are placed on difference devices in a “round-robin
  sequence”. This RAID type requires at least three devices, but typically
  offers better performance than RAID4
  RAID6: This RAID type is similar to RAID4 and RAID5 before it, but
  will use two devices for parity blocks rather than one. Allowing for up
  to two devices to fail without data loss. The main caveat to this RAID
  type is that it requires five devices to use.
  RAID10: This RAID type is a combination of RAID1 and RAID0, striping
  data across a set of mirrors. This offers the speed of RAID0 with the
  redundancy of RAID1, but at the cost of needing more devices. The
  minimum amount of devices required for this RAID type is four.
  
  Finally, we can check the status of the volume groups using the lvs
  command. Simply running lvs will give a small amount of
  information about the volume group, but to get more details on the
  entire group and its devices run the following:
  # Run as root
  lvs -a -o +device
  LV                        VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
    storage_volume            STORAGE rwi-a-r--- <36.00g                                    15.17            storage_volume_rimage_0(0),storage_volume_rimage_1(0)
    [storage_volume_rimage_0] STORAGE iwi-aor--- <36.00g                                                     /dev/sda1(1)
    [storage_volume_rimage_1] STORAGE Iwi-aor--- <36.00g                                                     /dev/sdb1(1)
    [storage_volume_rmeta_0]  STORAGE ewi-aor---   4.00m                                                     /dev/sda1(0)
    [storage_volume_rmeta_1]  STORAGE ewi-aor---   4.00m                                                     /dev/sdb1(0)
  
  This will show 1. all logical volumes (-a) and show each
  device (-o +device).
  Replacing Disks and
  Resizing Partitions
  It’s great that the logical volumes are setup now, but what happens
  if we need to replace a disk? Either because the previous disk failed,
  or is being upgraded to a larger disk. Thankfully, LVM allows for that;
  in this scenario, let’s imagine that one of our 40G disks died, and is
  being replaced with a 50G disk. My current lvs output:
  # Run as root
  lvs -a -o +device
  WARNING: Couldn't find device with uuid HLdFpC-sWym-TnKZ-TiyQ-E9A4-e5P7-3iNpO0.
    WARNING: VG MIRROR is missing PV HLdFpC-sWym-TnKZ-TiyQ-E9A4-e5P7-3iNpO0 (last written to /dev/sdb1).
    LV                        VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
    storage_volume            STORAGE rwi-aor-p- <38.00g                                    100.00           raid_rimage_0(0),raid_rimage_1(0)
    [storage_volume_rimage_0] STORAGE iwi-aor--- <38.00g                                                     /dev/sda1(1)
    [storage_volume_rimage_1] STORAGE Iwi-aor-p- <38.00g                                                     [unknown](1)
    [storage_volume_rmeta_0]  STORAGE ewi-aor---   4.00m                                                     /dev/sda1(0)
    [storage_volume_rmeta_1]  STORAGE ewi-aor-p-   4.00m                                                     [unknown](0)
  As we can see, one of the devices on the mirror is missing.
  Thankfully, I put in a 50G drive that can replace it, so let’s get that
  process started. First partition the drive using your favorite tool
  (cfdisk for me) to create one large partition on the disk.
  Then run:
  # Run as root
  vgreduce --removemissing STORAGE --force
  This will remove the device from the volume group, but will leave the
  group in state in which it needs to be refreshed. We will do this after
  adding the new device:
  # Run as root
  
  # First create a physical volume for the new device
  pvcreate /dev/sdc1
  # Then add the new device to the previous volume group
  vgextend STORAGE /dev/sdc1
  # Finally repair the volume using the lvconvert command
  lvconvert --repair /dev/mapper/STORAGE-storage_volume
  # Allow a few minutes for LVM to copy all of the data over to the new disk
  # before replacing the old one. The status of the copy can be seen by running:
  lvs -a
   LV                        VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    storage_volume            STORAGE rwi-aor--- <36.00g                                    4.83
    [storage_volume_rimage_0] STORAGE iwi-aor--- <36.00g
    [storage_volume_rimage_1] STORAGE Iwi-aor--- <36.00g
    [storage_volume_rmeta_0]  STORAGE ewi-aor---   4.00m
    [storage_volume_rmeta_1]  STORAGE ewi-aor---   4.00m
  
  # The Cpy%Sync section describes how much data has been synced between the two disks
  # once that is at 100, the first disk is safe to remove
  Now the larger disk should be a part of the mirror just as the
  previous 40G disk was. Let’s repeat the process for the other 40G to
  have the volume group on two 50G disks rather than two 40G ones.
  After the 40G disks have been swapped out with 50G ones, let’s resize
  the volume group to utilize more space. This is a fairly straightforward
  step; additionally the lvextend utility even includes a
  flag to resize the file system itself. To do this run:
  # Run as root
  
  # Note the '+' between the -l and the 90
  lvextend -l+90%FREE -r STORAGE
  NOTE: In the above command, the 90%FREE will not
  allocate 90% of the total disk size, but rather will add 90% of the
  available disk space to the current volume.
  At the end of it, my lvs output looks as follows:
  # Run as root
  lvs -a -o +devices
   LV                        VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
    storage_volume            STORAGE rwi-aor--- 48.59g                                    100.00           storage_volume_rimage_0(0),storage_volume_rimage_1(0)
    [storage_volume_rimage_0] STORAGE iwi-aor--- 48.59g                                                     /dev/sdb1(1)
    [storage_volume_rimage_1] STORAGE iwi-aor--- 48.59g                                                     /dev/sdc1(1)
    [storage_volume_rmeta_0]  STORAGE ewi-aor---  4.00m                                                     /dev/sdb1(0)
    [storage_volume_rmeta_1]  STORAGE ewi-aor---  4.00m                                                     /dev/sdc1(0)
  
  Resources
  
  Wikipedia:
  Logical Volume Manager (Linux): General background on Linux’s
  LVM
  ArchWiki: LVM:
  More background on LVM with some more specifics and technical
  details
  LVM
  Man Page: Good reference document for LVM as well as other included
  utilities such as lvcreate or lvs
  GoLinuxCloud:
  Create Mirrored Logical Volumes in Linux: Main guide I referenced to
  get started
  
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-01-03.html
  		https://foxide.xyz/projects/2025-01-03.html
  		Fri, 03 Jan 2025 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  Another post on LVM,
  this time going over snapshots and backups. This is a territory that
  many next generation file systems, especially ZFS excel in. There is an
  argument to be made for using ZFS rather than using LVM, however, that
  is not a debate that will be made in this post. This one is simply going
  over how to use LVM for those things.
  Lab Setup
  This lab will be run on a Void Linux VM similarly to the lab in my
  last post. The root drive for the VM is 30G, however, this time the LVM
  drive is only 5G, as we just need enough to prove the concept.
  Additionally, before beginning make sure to load the
  dm-snapshot kernel module. This can be done by running
  modprobe dm-snapshot as root. It can also be done
  automatically by putting the following line in the file:
  /etc/modules-load.d/dm-snapshot.conf
  dm-snapshot
  According to the Void
  Linux Handbook; putting that same file in
  /etc/modprobe.d should have the same effect, but it did not
  work in my testing.
  Basic LVM Setup
  Be sure to partition the drive(s) first, this can be done either
  using cfdisk or fdisk with cfdisk
  being the easier one.
  # Run as root
  
  # Create physical volume on the drive
  pvcreate /dev/sda1
  # Create volume group with the drive
  vgcreate DRIVE /dev/sda1
  # Allocate 75% of the free space in volume group 'DRIVE'
  lvcreate 75%FREE -n D01 DRIVE
  # Create file system on volume group allocation
  mkfs.ext4 /dev/mapper/DRIVE-D01
  Now that we have the volume group made, let’s see how snapshots and
  backups work within LVM. As a general note before getting into the
  commands, it is generally much easier to prove that these commands have
  worked if there is some data on the volume group. That way when we
  delete the volume group and restore the snapshot on a new volume group,
  we can verify that the data was restored as it was on the original
  volume.
  Creating the snapshot
  LVM has two kinds of snapshots, a Copy on Write (CoW) snapshot, and a
  thin snapshot. The thinkk Rather, this post will be going over the CoW
  snapshots. The CoW snapshots are created when a size for the snapshot is
  specified in the snapshot creation command. From the lvcreate
  man page:
  
  COW (Copy On Write) snapshots are create when a size is specified.
  The size is allocated from space in the (volume group), and is the
  amount of space that can be used for saving COW blocks as writes occur
  to the origin or snapshot. The size chosen should depend upon the amount
  of writes that are expected; often 20% of the origin (logical volume) is
  enough. If COW space runs low, it can be extended with lvextend
  (shrinking is also allowed with lvreduce.) A small amount of the COW
  snapshot LV size is used to track COW block locations, so the full size
  is not available for COW data blocks. Use lvs to check how much space is
  used, and see –monitor to to
  automatically extend the size to avoid running out of space.
  
  The following command will create a 500 Megabyte copy on write
  snapshot of the previously created logical volume.
  # Run as root
  lvcreate --snapshot --size 500M --name backup01 DRIVE/D01
  We can see the snapshot was taken by running lvs -a as
  root:
   LV   VG    Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    BK01 DRIVE swi-a-s--- 500.00m      D01    0.01
    D01  DRIVE owi-aos---  <3.75g
  The size specified in the snapshot is not relative to the size of the
  storage pool, but rather describes the amount of changed data the
  snapshot can store. So, in this example, 500 megabytes of data can be
  modified on the storage pool, and once the snapshot storage is full, it
  will cease to be able to keep up with modified data on the main pool.
  The recommended snapshot size is 20% of the storage pool’s available
  space; I modified the above snapshot command to change to that
  value:
  # Run as root
  lvcreate --snapshot -l 20%ORIGIN --name backup01 DRIVE/D01
  Restoring to a previous
  snapshot
  The following commands are how one might go about restoring a logical
  volume back to the state of the snapshot backup01:
  # Run as root
  lvconvert --merge /dev/mapper/DRIVE-backup01
  # Deactive Volume
  vgchagne -an
  # Re-activate Volume
  vgchange -ay
  The biggest annoyance with LVM is the requirement to reload the
  volume group. If the volume group is not the root device for a running
  operating system, then it can be reloaded using the commands shown
  above, but if the root file system has to be reverted to a previous
  snapshot, then it requires that the machine be restarted as you cannot
  unload a currently running block device.
  Backing up the snapshot with
  tar
  Another point that should be pointed out is that the Arch Wiki page on
  LVM specifically points out that a copy on write snapshot is not a
  backup.
  
  A [Copy on Write] snapshot is not a backup, because
  it does not make a second copy of the original data. For example, a
  damanged disk secotor that affects original data also affects the
  snapshots. That said, a snapshot can be helpful while using other tools
  to make backups, as outlined below.
  
  The way that this issue is generally addressed, appears to be to
  mount the snapshot and copy it using something like tar
  or dd. I have included an example below of creating a
  backup of the snapshot using tar:
  # Run as root
  mkdir /backup/mount
  mount /dev/mapper/DRIVE-backup01 /backup/mount
  tar -cf backup01.snapshot /backup/mount/*
  From there, the snapshot file can be moved to another storage
  location, and it can act as the backup of the pool.
  Restoring a file from backup
  Restoring individual files from the backup is fairly straightforward,
  just extract the snapshot file that was made earlier and grab any files
  that are required.
  Restoring entire backup
  Restoring the entire backed up snapshot is a bit of a combination of
  restoring individual files and restoring to a previous snapshot.
  Essentially, we create a new snapshot and load the new snapshot with the
  previous snapshot’s data, then recover it. The process is shown
  below:
  # Run as root
  
  # Creating location to mount the snapshot
  mkdir -p /backup/mount
  # Creating restoration snapshot, adjust size appropriately
  lvcreate --snapshot --size 500M -n restore01 DRIVE/02
  # Mount snapshot
  mount /dev/mapper/DRIVE-restore /backup/mount
  # Extract tar archieve
  tar -xf backup01.snapshot
  # Restore snapshot
  lvconvert --merge /dev/mapper/DRIVE-restore01
  # Deactive Volume
  vgchagne -an
  # Re-activate Volume
  vgchange -ay
  Deleting snapshot
  Deleting a snapshot is as easy as deleting a logical volume, just
  make sure to get the name correct and not delete the main LV. The names
  can be checked using lvs -a.
  # Run as root
  lvremove DRIVE/backup01 DRIVE/D01
  Resources
  
  lvcreate
  man page
  ArchWiki
  page, specifically the snapshot section
  Linux
  Config page on LVM snapshots
  
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-01-17.html
  		https://foxide.xyz/projects/2025-01-17.html
  		Fri, 17 Jan 2025 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  Applying good disk encryption is a great way to protect data at rest.
  This is important in a variety of contexts; whether it be a journalist
  protecting a source’s story, a company protecting company assets, or
  just a person protecting their right to privacy. Good disk encryption
  can provide a layer of protection in all of these situations, but how
  does one go about setting up disk encryption on Linux? The standardized
  method of doing this is by using LUKS
  and cryptsetup
  What is LUKS and Cryptsetup?
  Some Notes Before Getting
  Started
  Cryptography is a very difficult field that requires a lot of study
  and mathematics. Cryptsetup allows users to handle their own encryption
  via the dm-crypt kernel module, however, this is
  specifically pointed out in the cryptsetup man page, as
  well as in the FAQ:
  
  Unless you understand the cryptographic background well, use LUKS.
  With plain dm-crypt there are a number of possible user errors that
  massively decrease security. While LUKS cannot fix them all, it can
  lessen the impact for many of them.
  
  Effectively, if you aren’t sure that you understand cryptography,
  don’t use plain dm-crypt and expect it to be secure. You will most
  likely do it wrong and leave your disks vulnerable to avoidable
  attacks.
  This blog post will only cover using LUKS as I do not have enough
  background in cryptography to even begin to explain how to properly use
  plain dm-crypt.
  Examples
  The commands for encrypting a disk (or really any block device) are
  not particularly difficult.
  Encrypting a disk
  The following commands show how to encrypt a disk located at
  /dev/sda, then create a file system and mount it at
  /mnt:
  # Run as root
  
  # Format the entire drive with LUKS1 encryption
  cryptsetup luksFormat --type luks1 /dev/sda
  # Open the encrypted block device
  cryptsetup luksOpen /dev/sda drive
  # Create a file system on the now decrypted device
  mkfs.ext4 /dev/mapper/drive
  # Mount the device
  mount /dev/mapper/drive /mnt
  It is worth noting, the drive that follows the
  /dev/sda is just a label and can be anything as long as
  there are not duplicate files in /dev/mapper (where
  cryptsetup maps decrypted devices). drive will be used for
  the remainder of this post, but know that it could be any other
  label.
  Using a Key File
  Additionally, a keyfile can be created to unlock the encrypted volume
  without having to type a password. NOTE, this command
  uses dd, which can and will happily overwrite anything that
  you point it at with no warnings. Before running the command, please
  make sure the syntax is correct, specifically the of
  location, as that is the location where the data will be written. Pro
  tip, if stands for ‘input file’ and of stands
  for ‘output file’.
  # Run as root
  
  # Create the keyfile with pseudo random data
  dd bs=1 count=64 if=/dev/urandom of=${HOME}/keyfile
  # Add the key to the current encrypted volume
  cryptsetup luksAddKey /dev/sda ${HOME}/keyfile
  # Then use this command to unlock the encrypted volume with the key
  cryptsetup luksOpen /dev/sda drive --key-file ${HOME}/keyfile
  Additionally, another passphrase can be added rather than a key by
  just removing the location of the keyfile:
  # Run as root
  cryptsetup luksAddKey /dev/sda
  Changing the passphrase
  At some point, changing or removing a key (either passphrase or a
  keyfile) may be required for one reason or another. This can be done by
  running the following command:
  # Run as root
  
  # This command is for changing a passphrase
  cryptsetup luksChangeKey /dev/sda
  # This command will replace an existing passphrase with a keyfile
  cryptsetup luksChangeKey /dev/sda ${HOME}/keyfile
  Cryptsetup will then ask the user to enter the passphrase that needs
  to be changed, then a new passphrase to replace it with. However,
  changing a keyfile, or replacing a keyfile with a passphrase is slightly
  less straightforward. The process would actually look more like removing
  the old keyfile and adding a new keyfile or passphrase to the encrypted
  volume:
  # Run as root
  
  # Removing old keyfile
  cryptsetup luksRemoveKey /dev/sda ${HOME}/keyfile
  Then add the new keyfile or passphrase as shown previously.
  Backups
  As most everyone should know in 2025, backups are important, and this
  is true even with encrypted data; but how does one backup encrypted
  data? That’s a good question, that has two basic answers.
  
  Backup the entire encrypted disk: Basically, just dd
  the entire disk to another storage pool as the backup, if restoration is
  needed, just dd it back onto a disk and carry on. This has
  the benefit of the files staying encrypted, however, it is going to take
  a lot more space and doing differential backups is not really possible
  (at least to my knowledge).
  Backup just the files on the encrypted disk. This has the benefit of
  keeping storage space for the backups to a minimum, but the encryption
  will have to be handled separately from the original files.
  
  Once the preferred type of backup is chosen, the specifics are really
  just which tools are going to be used. There are many different backup
  tools for Linux and other systems that use cryptsetup, so I am not going
  to go in-depth on that.
  Guides for Encrypted Root
  An Linux install with full disk encryption is not very difficult to
  do. Quite a few distros with proper installers have the option for it,
  but some distros don’t. I am leaving two guides that should be useful in
  getting started. The basic idea is just to use cryptsetup
  to encrypt the disk, create your partitions and file systems, then
  install onto those file systems. Both the Arch wiki and the Void Linux
  docs page have guides on how to do this that are probably better than
  what I would produce.
  
  Arch
  Wiki page on full disk encryption
  Void
  Linux page on full disk encryption
  
  Simple, right?
  Sure, the commands are simple enough to follow, unfortunately the
  tricky bit with cryptography is understanding the reasons behind what
  you are doing. Why use LUKS1 over LUKS2? Which algorithms are secure?
  For those answers, I lean on people smarter than myself on the topic…
  the team that made cryptsetup. They have an FAQ
  page that answers most any question that you could dream up for the
  tool. Additionally, if you dream up a question that is not on that list,
  there is a mailing list
  that questions are welcomed at.
  Resources
  
  Arch
  wiki page for cryptsetup
  Cryptsetup
  FAQ
  LUKS1
  White paper
  LUKS2
  White paper Draft
  
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-02-01.html
  		https://foxide.xyz/projects/2025-02-01.html
  		Sat, 01 Feb 2025 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  In the olden days of IT, managing computers was a very manual task
  that took hours and hours; however, these days, we have tools that allow
  for automating some of those tasks making them suck less. The issue with
  that is that they will eventually become a requirement as the amount of
  work that needs to be done will increase until it is impossible to do
  the task manually. This tipping point was many years ago; now automation
  a task is a must, especially the really mundane ones. Anisble is a tool
  that allows for exactly that. It will allow for auto deployment of
  things like websites, services, and apps, and best of all, it can be run
  in the background while listening to something else that is more
  interesting.
  Operating Systems
  For working on this project, I wanted to test out and see how well
  Anisble held up with mixed integrations. While I understand that most
  companies would not have a scheme quite this chaotic. Since this is a
  lab, I can make it as much or as little chaotic as I decide. Here are
  the operating systems I decided on.
  
  FreeBSD
  OpenSUSE
  Tumbleweed
  Debian
  
  Lab Setup
  To get started, the first thing is to setup a controller to actually
  run Anible playbooks from. This means installing an operating system of
  some sort (if you somehow don’t already have one installed somewhere). I
  am setting up a new OpenSUSE Tumbleweed virtual machine to service this
  purpose; going through the installer is extremely straight forward, if
  you have installed a Linux distro before, you should be able to manage
  this installer.
  Once the system is installed, configure it to your liking and also
  install Ansible. OpenSUSE uses the zypper
  package manager; the command I used to install the packages I wanted
  looks like:
  sudo zypper install -y neovim emacs git ansible
  # You can search for packages by running:
  zypper search ${PACKAGE}
  Additionally, you will want to generate SSH keys to allow for running
  playbooks automatically. This can be done by running the following
  command:
  # The default /should/ be ed25519, but to make sure
  # you are using a good key type:
  ssh-keygen -t ed25519
  Next to setup the “fleet” of machines that we will be managing with
  Ansible. For this lab, I am going to be setting up one SUSE Tumbleweed
  machine, one FreeBSD machine, and one Debian machine. There will not be
  anything special about these machines other than I will be installing
  Ansible, enabling SSH, and installing the public key we generated a
  minute ago.
  Setting up Ansible on the
  Controller
  Ansible operates on various configuration files, making it easy to
  version control. This is something that you should absolutely do if you
  are working with Ansible in any appreciable way. The most common version
  control system is Git, however,
  there are others such as Subversion and Mercurial. The
  important point is that version control is being done, as managing
  playbooks (especially with a team) and being able to roll back changes
  in critical. This post isn’t going to focus on how to version control
  the playbooks, but I do want to emphasize the importance of doing this,
  especially if working in a team of people that are also going to be
  working with playbooks. For a primer on Git, there are some point in another blog post
  of mine, but you can also search online for a Git tutorial.
  Let’s get started. First, I will create a directory called
  ‘AnsibleLab’ that our Ansible files will live in. Then I am going to
  create two Git repositories, one for the inventory file(s) and one for
  the playbook(s). Realistically, this could probably live in one
  repository, however, it felt slightly more organized to separate
  them.
  # Create and cd into the 'AnsibleLab' directory
  mkdir AnisbleLab && cd AnisbleLab
  # Create git repo for the invenotry files
  git init inventory
  # Create git repo for the playbook files
  git init playbooks
  As for committing changes, I approach it the same way you would with
  source code. Commit the smallest amount of measurable ‘work’ for the
  repository. Updating the hosts, make a commit. Creating a new playbook,
  commit, etc. Use your best judgment on what your
  process is, but that is how I approach it.
  Setting up the Inventory
  Before we can run any playbooks, we need to build an inventory of
  hosts to run commands against. So we need to create an file
  (inventory.ini in this case) that will include the various
  hosts that we will be managing. Just to make sure things are working,
  the file will look fairly simple right now:
  [hosts]
  192.168.122.49
  192.168.122.98
  192.168.122.237
  Domain names can also be used over IP addresses as long as there is a
  DNS entry for each host, and for production environments, the IPs of the
  hosts should at least be static, if not have domain names attached to
  them. Once the inventory file is built, we can test to make sure that
  Ansible can connect to the hosts by running:
  # The first argument to the 'ansible' command should be the hosts
  # that we are targeting. In this example, it is just 'hosts' from the inventory
  ansible hosts -m ping -i inventory.ini
  As an alternative to ini files, YAML format can be used to specify
  the hosts inventory, and allows for much more granular definition of the
  hosts such as defining the SSH user and operating system so that it does
  not have to be passed when running the Ansible command. So, let’s see
  what our previous playbook might look like as YAML:
  prod:
    hosts:
      dbs:
        ansible_host: 192.168.122.237
      web:
        ansible_host: 192.168.122.49
      dev:
        ansible_host: 192.168.122.98
  Then running our ping command, but changing ‘hosts’ to ‘prod’ should
  successfully ping our Ansible machines. In addition to that, we can also
  create multiple groups and assign hosts to those multiple groups. For
  example, we have the group ‘prod’ in our current example, but we could
  also add a group to separate the hosts by operating system as
  follows:
  prod:
    hosts:
      dbs:
        ansible_host: 192.168.122.237
      web:
        ansible_host: 192.168.122.49
      dev:
        ansible_host: 192.168.122.98
  
  os-group:
    hosts:
      freebsd:
        ansible_host: 192.168.122.237
      debian:
        ansible_host: 192.168.122.49
      SUSE:
        ansible_host: 192.168.122.98
  In our small inventory of three machines, this is a bit silly.
  However, for a fleet of many, many machines serving different roles and
  running on different operating systems, this kind of granularity is
  extremely appreciated. That will bring us to another problem though… The
  inventory file will eventually become a massive file that is extremely
  difficult to work with and parse through. Thankfully, Ansible can handle
  splitting the inventory files into multiple files within a directory to
  make it not only easier for admins to manage grouping hosts in multiple
  ways, but also for different teams to manage hosts in ways that matter
  to them without harming other teams.
  To do this, we are simply going to split the two sections into
  different files, one file ‘prod.yml’ containing the ‘prod’ section in
  the example YAML, and an ‘os.yml’ containing the ‘os-group’ section. We
  will then put both of these files in a directory called ‘hosts’ and
  create a new YAML file called ‘inv.yml’ with the contents:
  # Comments are something else that is useful.
  # To make a comment, simply type a '#' character, and everything after will be commented out
  hosts/
    prod.yml   # Groups the inventory hosts by role within production
    os.yml     # Groups the inventory by Operating System
  Playbooks
  Playbooks are the files to tell Ansible what work needs to be done on
  which host(s). To get started, I am going to steal the example
  playbook from Ansible’s documentation. We create a file,
  playbook.yml, with the following contents:
  # Note, that the 'hosts' line will look for items with the same name
  # in your inventory files to figure out which hosts the commands need
  # to run on
  - name: My first play
    hosts: prod
    tasks:
     - name: Ping my hosts
       ansible.builtin.ping:
  
     - name: Print message
       ansible.builtin.debug:
         msg: Hello world
  Then we run that playbook with the following command:
  ansible-playbook -i inventory/hosts/prod.yml playbooks/playbook.yml
  Useful Things
  Now let’s see about doing something useful with Ansible. A common
  task for system administrators is to install software; doing this can be
  tedious across a dozen machines, and highly impractical over a few
  dozen. Ansible will cut the time down on installing software on dozens
  of machines down from hours or days, to minutes. This playbook will
  install neovim and git onto all of the hosts in the ‘prod’ group.
  # This playbook with install the following software onto the target machines:
  #
  # - Neovim
  
  - name: Install required software
    hosts: prod
    tasks:
      - name: Install Neovim
        ansible.builtin.package:
          name: neovim
          state: present
      - name: Install Git
        ansible.builtin.package:
          name: git
          state: present
  Changing which machines will receive this playbook is as simple as
  changing the ‘Host’ parameter. Let’s look at another playbook that
  installs and starts nginx, as well as removes a package
  doas for only hosts in the ‘web’ group.
  # This playbook with install the following software onto the target machines:
  #
  # - nginx
  #
  - name: Install required software for production machines
    hosts: web
    tasks:
      - name: Install Nginx
        ansible.builtin.package:
          name: nginx
          state: present
      - name: Start nginx service
        ansible.builtin.service:
          name: nginx
          state: started
      - name: Remove doas if present
        ansible.builtin.package:
          name: doas
          state: absent
  At this point, it is worth noting that Ansible playbooks run from top
  to bottom, so in the playbook above the order of operations would
  be:
  
  Install nginx
  Start nginx service
  Remove doas
  
  If I were to change the playbook to look as follows:
  # This playbook with install the following software onto the target machines:
  #
  # - nginx
  #
  - name: Install required software for production machines
    hosts: web
    tasks:
      - name: Start nginx service
        ansible.builtin.service:
          name: nginx
          state: started
      - name: Install Nginx
        ansible.builtin.package:
          name: nginx
          state: present
      - name: Remove doas if present
        ansible.builtin.package:
          name: doas
          state: absent
  The playbook would fail to run because the first task could not be
  completed (assuming nginx was not already installed). So, be mindful of
  the order of operations for the playbooks, ensure that any users that
  will be used in the tasks are created first, and services get installed
  before attempting to start them.
  While the previous playbook did do things that were actually useful,
  Ansible can actually go even further and do full deployments for
  applications. For example, this playbook would install and deploy a CryptPad instance on a machine (tested
  with Debian).
  # This playbook will set a machine up for a CryptPad instance from the
  # ground up. In doing this, it will also install the following software:
  #
  # - Nginx
  # - Node
  # - npm-node
  # - Git
  
  - name: Deploy CryptPad
    hosts: web
    tasks:
      - name: Install nginx
        ansible.builtin.package:
          name: nginx
          state: present
      - name: Install Node
        ansible.builtin.package:
          name: nodejs
          state: present
      - name: Install npm
        ansible.builtin.package:
          name: npm
          state: present
      - name: Install Git
        ansible.builtin.package:
          name: git
          state: present
      - name: create service group
        ansible.builtin.group:
          name: cryptpad
          state: present
          gid: 5000
      - name: create service user
        ansible.builtin.user:
          name: cryptpad
          comment: CryptPad service account
          shell: /bin/bash
          create_home: yes
          uid: 5000
          group: cryptpad
      - name: Create working directory
        ansible.builtin.file:
          path: /srv
          owner: cryptpad
          group: cryptpad
          state: directory
      - name: Download CryptPad
        ansible.builtin.git:
          repo: 'https://github.com/cryptpad/cryptpad'
          dest: /srv/cryptpad
      - name: Change ownership of cryptpad directory
        ansible.builtin.file:
          path: /srv/cryptpad
          recurse: true
          owner: cryptpad
          group: cryptpad
          mode: '0755'
      - name: Deploy Config
        ansible.builtin.copy:
          src: /home/fac3/Documents/configs/CryptPad-config.js
          dest: /srv/cryptpad/config/config.js
          owner: cryptpad
          group: cryptpad
          mode: '664'
      - name: Deploy Service
        ansible.builtin.copy:
          src: /home/fac3/Documents/configs/cryptpad.service
          dest: /etc/systemd/system/cryptpad.service
          owner: root
          group: root
          mode: '755'
      - name: Install Cryptpad
        command:
          cmd: "npm ci && npm run install:components && node server &disown"
          chdir: /srv/cryptpad
        become: true
        become_user: cryptpad
      - name: Start Cryptpad service
        ansible.builtin.service:
          name: cryptpad
          state: started
  As you can see, there are a few modules that can do useful things
  like adding users and handling file permissions. Combining these modules
  give more or less the same control that one might have one a system with
  a standard shell, but this is automated and reproducible. Because it is
  automated and reproducible, it makes administration much easier.
  Configuring Ansible
  The last part that I will be covering in this post is how to begin to
  configure Ansible. Ansible has a lot of configuration options, but most
  of them have sane defaults that do not necessarily need to be changed.
  To get a ‘default’ configuration for Ansible, you can simply run
  ansible-config init. To put that into a file, simply modify
  the previous command to:
  ansible-config init > ansible.cfg.
  As stated above, many of these options are good in most cases,
  however, changing some of them may make sense. For example, for working
  on this project i modified the default as follows:
  --- default.cfg 2025-03-01 00:15:39.155023749 -0500
  +++ ansible.cfg 2025-02-23 16:19:10.305749807 -0500
  @@ -1,112 +1,112 @@
   [defaults]
   # (boolean) By default, Ansible will issue a warning when received from a task action (module or action plugin).
   # These warnings can be silenced by adjusting this setting to False.
  -action_warnings=True
  +;action_warnings=True
  
   # (list) Accept a list of cowsay templates that are 'safe' to use, set to an empty list if you want to enable all installed templates.
  -cowsay_enabled_stencils=bud-frogs, bunny, cheese, daemon, default, dragon, elephant-in-snake, elephant, eyes, hellokitty, kitty, luke-koala, meow, milk, moofasa, moose, ren, sheep, small, stegosaurus, stimpy, supermilker, three-eyes, turkey, turtle, tux, udder, vader-koala, vader, www
  +;cowsay_enabled_stencils=bud-frogs, bunny, cheese, daemon, default, dragon, elephant-in-snake, elephant, eyes, hellokitty, kitty, luke-koala, meow, milk, moofasa, moose, ren, sheep, small, stegosaurus, stimpy, supermilker, three-eyes, turkey, turtle, tux, udder, vader-koala, vader, www
  
   # (string) Specify a custom cowsay path or swap in your cowsay implementation of choice.
  -cowpath=
  +;cowpath=
  
   # (string) This allows you to choose a specific cowsay stencil for the banners or use 'random' to cycle through them.
  -cow_selection=default
  +;cow_selection=default
  
   # (boolean) This option forces color mode even when running without a TTY or the "nocolor" setting is True.
  -force_color=False
  +;force_color=False
  
   # (path) The default root path for Ansible config files on the controller.
  -home=~/.ansible
  +;home=~/.ansible
  
   # (boolean) This setting allows suppressing colorizing output, which is used to give a better indication of failure and status information.
  -nocolor=False
  +;nocolor=False
  
   # (boolean) If you have cowsay installed but want to avoid the 'cows' (why????), use this.
  -nocows=False
  +;nocows=False
  
   # (boolean) Sets the default value for the any_errors_fatal keyword, if True, Task failures will be considered fatal errors.
  -any_errors_fatal=False
  +;any_errors_fatal=False
  
   # (path) The password file to use for the become plugin. ``--become-password-file``.
   # If executable, it will be run and the resulting stdout will be used as the password.
  -become_password_file=
  +become_password_file=/home/fac3/Documents/AnsibleLab/pw.txt
  
   # (pathspec) Colon-separated paths in which Ansible will search for Become Plugins.
  -become_plugins=/home/fac3/.ansible/plugins/become:/usr/share/ansible/plugins/become
  +;become_plugins=/home/fac3/.ansible/plugins/become:/usr/share/ansible/plugins/become
  
   # (string) Chooses which cache plugin to use, the default 'memory' is ephemeral.
  -fact_caching=memory
  +;fact_caching=memory
  
   # (string) Defines connection or path information for the cache plugin.
  -fact_caching_connection=
  +;fact_caching_connection=
  
   # (string) Prefix to use for cache plugin files/tables.
  -fact_caching_prefix=ansible_facts
  +;fact_caching_prefix=ansible_facts
  
   # (integer) Expiration timeout for the cache plugin data.
  -fact_caching_timeout=86400
  +;fact_caching_timeout=86400
   
   # (list) List of enabled callbacks, not all callbacks need enabling, but many of those shipped with Ansible do as we don't want them activated by default.
  -callbacks_enabled=
  +;callbacks_enabled=
   
   # (string) When a collection is loaded that does not support the running Ansible version (with the collection metadata key `requires_ansible`).
  -collections_on_ansible_version_mismatch=warning
  +;collections_on_ansible_version_mismatch=warning
   
   # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
   
  -collections_path=/home/fac3/.ansible/collections:/usr/share/ansible/collections
  +;collections_path=/home/fac3/.ansible/collections:/usr/share/ansible/collections
   
   # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
  -collections_scan_sys_path=True
  +;collections_scan_sys_path=True
   
   # (path) The password file to use for the connection plugin. ``--connection-password-file``.
  -connection_password_file=
  +;connection_password_file=
   
   # (pathspec) Colon-separated paths in which Ansible will search for Action Plugins.
  -action_plugins=/home/fac3/.ansible/plugins/action:/usr/share/ansible/plugins/action
  +;action_plugins=/home/fac3/.ansible/plugins/action:/usr/share/ansible/plugins/action
   
   # (boolean) When enabled, this option allows lookup plugins (whether used in variables as ``{{lookup('foo')}}`` or as a loop as with_foo) to return data that is not marked 'unsafe'.
   # By default, such data is marked as unsafe to prevent the templating engine from evaluating any jinja2 templating language, as this could represent a security risk. This option is provided to allow for backward compatibility, however, users should first consider adding allow_unsafe=True to any lookups that may be expected to contain data that may be run through the templating engine late.
  -allow_unsafe_lookups=False
  +
   
   # (boolean) This controls whether an Ansible playbook should prompt for a login password. If using SSH keys for authentication, you probably do not need to change this setting.
  -ask_pass=False
  +;ask_pass=False
   
   # (boolean) This controls whether an Ansible playbook should prompt for a vault password.
  -ask_vault_pass=False
  +;ask_vault_pass=False
   
   # (pathspec) Colon-separated paths in which Ansible will search for Cache Plugins.
  -cache_plugins=/home/fac3/.ansible/plugins/cache:/usr/share/ansible/plugins/cache
  +;cache_plugins=/home/fac3/.ansible/plugins/cache:/usr/share/ansible/plugins/cache
   
   # (pathspec) Colon-separated paths in which Ansible will search for Callback Plugins.
  -callback_plugins=/home/fac3/.ansible/plugins/callback:/usr/share/ansible/plugins/callback
  +;callback_plugins=/home/fac3/.ansible/plugins/callback:/usr/share/ansible/plugins/callback
   
   # (pathspec) Colon-separated paths in which Ansible will search for Cliconf Plugins.
  -cliconf_plugins=/home/fac3/.ansible/plugins/cliconf:/usr/share/ansible/plugins/cliconf
  +;cliconf_plugins=/home/fac3/.ansible/plugins/cliconf:/usr/share/ansible/plugins/cliconf
   
   # (pathspec) Colon-separated paths in which Ansible will search for Connection Plugins.
  -connection_plugins=/home/fac3/.ansible/plugins/connection:/usr/share/ansible/plugins/connection
  +;connection_plugins=/home/fac3/.ansible/plugins/connection:/usr/share/ansible/plugins/connection
   
   # (boolean) Toggles debug output in Ansible. This is *very* verbose and can hinder multiprocessing. Debug output can also include secret information despite no_log settings being enabled, which means debug mode should not be used in production.
  -debug=False
  +;debug=False
   
   # (string) This indicates the command to use to spawn a shell under, which is required for Ansible's execution needs on a target. Users may need to change this in rare instances when shell usage is constrained, but in most cases, it may be left as is.
  -executable=/bin/sh
  +;executable=/bin/sh
   
   # (pathspec) Colon-separated paths in which Ansible will search for Jinja2 Filter Plugins.
  -filter_plugins=/home/fac3/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
  +;filter_plugins=/home/fac3/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
   
   # (boolean) This option controls if notified handlers run on a host even if a failure occurs on that host.
   # When false, the handlers will not run if a failure has occurred on a host.
   # This can also be set per play or on the command line. See Handlers and Failure for more details.
  -force_handlers=False
  +;force_handlers=False
   
   # (integer) Maximum number of forks Ansible will use to execute tasks on target hosts.
   forks=5
   
   # (string) This setting controls the default policy of fact gathering (facts discovered about remote systems).
   # This option can be useful for those wishing to save fact gathering time. Both 'smart' and 'explicit' will use the cache plugin.
  -gathering=implicit
  +;gathering=implicit
   
   # (string) This setting controls how duplicate definitions of dictionary variables (aka hash, map, associative array) are handled in Ansible.
   # This does not affect variables whose values are scalars (integers, strings) or arrays.
  @@ -117,311 +117,312 @@
   # Changing the setting to ``merge`` applies across variable sources, but many sources will internally still overwrite the variables. For example ``include_vars`` will dedupe variables internally before updating Ansible, with 'last defined' overwriting previous definitions in same file.
   # The Ansible project recommends you **avoid ``merge`` for new projects.**
   # It is the intention of the Ansible developers to eventually deprecate and remove this setting, but it is being kept as some users do heavily rely on it. New projects should **avoid 'merge'**.
  -hash_behaviour=replace
  +;hash_behaviour=replace
   
   # (pathlist) Comma-separated list of Ansible inventory sources
  -inventory=/etc/ansible/hosts
  +#inventory=/etc/ansible/hosts
  +inventory=/home/fac3/Documents/AnsibleLab/inventory
   
   # (pathspec) Colon-separated paths in which Ansible will search for HttpApi Plugins.
  -httpapi_plugins=/home/fac3/.ansible/plugins/httpapi:/usr/share/ansible/plugins/httpapi
  +;httpapi_plugins=/home/fac3/.ansible/plugins/httpapi:/usr/share/ansible/plugins/httpapi
   
   # (float) This sets the interval (in seconds) of Ansible internal processes polling each other. Lower values improve performance with large playbooks at the expense of extra CPU load. Higher values are more suitable for Ansible usage in automation scenarios when UI responsiveness is not required but CPU usage might be a concern.
   # The default corresponds to the value hardcoded in Ansible <= 2.1
  -internal_poll_interval=0.001
  +;internal_poll_interval=0.001
   
   # (pathspec) Colon-separated paths in which Ansible will search for Inventory Plugins.
  -inventory_plugins=/home/fac3/.ansible/plugins/inventory:/usr/share/ansible/plugins/inventory
  +;inventory_plugins=/home/fac3/.ansible/plugins/inventory:/usr/share/ansible/plugins/inventory
   
   # (string) This is a developer-specific feature that allows enabling additional Jinja2 extensions.
   # See the Jinja2 documentation for details. If you do not know what these do, you probably don't need to change this setting :)
  -jinja2_extensions=[]
  +;jinja2_extensions=[]
   
   # (boolean) This option preserves variable types during template operations.
  -jinja2_native=False
  +;jinja2_native=False
   
   # (boolean) Enables/disables the cleaning up of the temporary files Ansible used to execute the tasks on the remote.
   # If this option is enabled it will disable ``ANSIBLE_PIPELINING``.
  -keep_remote_files=False
  +;keep_remote_files=False
   
   # (boolean) Controls whether callback plugins are loaded when running /usr/bin/ansible. This may be used to log activity from the command line, send notifications, and so on. Callback plugins are always loaded for ``ansible-playbook``.
  -bin_ansible_callbacks=False
  +;bin_ansible_callbacks=False
   
   # (tmppath) Temporary directory for Ansible to use on the controller.
  -local_tmp=/home/fac3/.ansible/tmp
  +;local_tmp=/home/fac3/.ansible/tmp
   
   # (list) List of logger names to filter out of the log file.
  -log_filter=
  +;log_filter=
   
   # (path) File to which Ansible will log on the controller.
   # When not set the logging is disabled.
  -log_path=
  +;log_path=
   
   # (pathspec) Colon-separated paths in which Ansible will search for Lookup Plugins.
  -lookup_plugins=/home/fac3/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup
  +;lookup_plugins=/home/fac3/.ansible/plugins/lookup:/usr/share/ansible/plugins/lookup
   
   # (string) Sets the macro for the 'ansible_managed' variable available for :ref:`ansible_collections.ansible.builtin.template_module` and :ref:`ansible_collections.ansible.windows.win_template_module`.  This is only relevant to those two modules.
  -ansible_managed=Ansible managed
  +;ansible_managed=Ansible managed
   
   # (string) This sets the default arguments to pass to the ``ansible`` adhoc binary if no ``-a`` is specified.
  -module_args=
  +;module_args=
   
   # (string) Compression scheme to use when transferring Python modules to the target.
  -module_compression=ZIP_DEFLATED
  +;module_compression=ZIP_DEFLATED
   
   # (string) Module to use with the ``ansible`` AdHoc command, if none is specified via ``-m``.
  -module_name=command
  +;module_name=command
   
   # (pathspec) Colon-separated paths in which Ansible will search for Modules.
  -library=/home/fac3/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
  +;library=/home/fac3/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
   
   # (pathspec) Colon-separated paths in which Ansible will search for Module utils files, which are shared by modules.
  -module_utils=/home/fac3/.ansible/plugins/module_utils:/usr/share/ansible/plugins/module_utils
  +;module_utils=/home/fac3/.ansible/plugins/module_utils:/usr/share/ansible/plugins/module_utils
   
   # (pathspec) Colon-separated paths in which Ansible will search for Netconf Plugins.
  -netconf_plugins=/home/fac3/.ansible/plugins/netconf:/usr/share/ansible/plugins/netconf
  +;netconf_plugins=/home/fac3/.ansible/plugins/netconf:/usr/share/ansible/plugins/netconf
   
   # (boolean) Toggle Ansible's display and logging of task details, mainly used to avoid security disclosures.
  -no_log=False
  +;no_log=False
   
   # (boolean) Toggle Ansible logging to syslog on the target when it executes tasks. On Windows hosts, this will disable a newer style PowerShell modules from writing to the event log.
  -no_target_syslog=False
  +;no_target_syslog=False
   
   # (raw) What templating should return as a 'null' value. When not set it will let Jinja2 decide.
  -null_representation=
  +;null_representation=
   
   # (integer) For asynchronous tasks in Ansible (covered in Asynchronous Actions and Polling), this is how often to check back on the status of those tasks when an explicit poll interval is not supplied. The default is a reasonably moderate 15 seconds which is a tradeoff between checking in frequently and providing a quick turnaround when something may have completed.
  -poll_interval=15
  +;poll_interval=15
   
   # (path) Option for connections using a certificate or key file to authenticate, rather than an agent or passwords, you can set the default value here to avoid re-specifying ``--private-key`` with every invocation.
  -private_key_file=
  +;private_key_file=
   
   # (boolean) By default, imported roles publish their variables to the play and other roles, this setting can avoid that.
   # This was introduced as a way to reset role variables to default values if a role is used more than once in a playbook.
   # Starting in version '2.17' M(ansible.builtin.include_roles) and M(ansible.builtin.import_roles) can individually override this via the C(public) parameter.
   # Included roles only make their variables public at execution, unlike imported roles which happen at playbook compile time.
  -private_role_vars=False
  +;private_role_vars=False
   
   # (integer) Port to use in remote connections, when blank it will use the connection plugin default.
  -remote_port=
  +;remote_port=
   
   # (string) Sets the login user for the target machines
   # When blank it uses the connection plugin's default, normally the user currently executing Ansible.
  -remote_user=
  +;remote_user=
   
   # (pathspec) Colon-separated paths in which Ansible will search for Roles.
  -roles_path=/home/fac3/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
  +;roles_path=/home/fac3/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
   
   # (string) Set the main callback used to display Ansible output. You can only have one at a time.
   # You can have many other callbacks, but just one can be in charge of stdout.
   # See :ref:`callback_plugins` for a list of available options.
  -stdout_callback=default
  +;stdout_callback=default
   
   # (string) Set the default strategy used for plays.
  -strategy=linear
  +;strategy=linear
   
   # (pathspec) Colon-separated paths in which Ansible will search for Strategy Plugins.
  -strategy_plugins=/home/fac3/.ansible/plugins/strategy:/usr/share/ansible/plugins/strategy
  +;strategy_plugins=/home/fac3/.ansible/plugins/strategy:/usr/share/ansible/plugins/strategy
   
   # (boolean) Toggle the use of "su" for tasks.
  -su=False
  +;su=False
   
   # (string) Syslog facility to use when Ansible logs to the remote target.
  -syslog_facility=LOG_USER
  +;syslog_facility=LOG_USER
   
   # (pathspec) Colon-separated paths in which Ansible will search for Terminal Plugins.
  -terminal_plugins=/home/fac3/.ansible/plugins/terminal:/usr/share/ansible/plugins/terminal
  +;terminal_plugins=/home/fac3/.ansible/plugins/terminal:/usr/share/ansible/plugins/terminal
   
   # (pathspec) Colon-separated paths in which Ansible will search for Jinja2 Test Plugins.
  -test_plugins=/home/fac3/.ansible/plugins/test:/usr/share/ansible/plugins/test
  +;test_plugins=/home/fac3/.ansible/plugins/test:/usr/share/ansible/plugins/test
   
   # (integer) This is the default timeout for connection plugins to use.
  -timeout=10
  +;timeout=10
   
   # (string) Can be any connection plugin available to your ansible installation.
   # There is also a (DEPRECATED) special 'smart' option, that will toggle between 'ssh' and 'paramiko' depending on controller OS and ssh versions.
  -transport=ssh
  +;transport=ssh
   
   # (boolean) When True, this causes ansible templating to fail steps that reference variable names that are likely typoed.
   # Otherwise, any '{{ template_expression }}' that contains undefined variables will be rendered in a template or ansible action line exactly as written.
  -error_on_undefined_vars=True
  +;error_on_undefined_vars=True
   
   # (pathspec) Colon-separated paths in which Ansible will search for Vars Plugins.
  -vars_plugins=/home/fac3/.ansible/plugins/vars:/usr/share/ansible/plugins/vars
  +;vars_plugins=/home/fac3/.ansible/plugins/vars:/usr/share/ansible/plugins/vars
   
   # (string) The vault_id to use for encrypting by default. If multiple vault_ids are provided, this specifies which to use for encryption. The ``--encrypt-vault-id`` CLI option overrides the configured value.
  -vault_encrypt_identity=
  +;vault_encrypt_identity=
   
   # (string) The label to use for the default vault id label in cases where a vault id label is not provided.
  -vault_identity=default
  +;vault_identity=default
   
   # (list) A list of vault-ids to use by default. Equivalent to multiple ``--vault-id`` args. Vault-ids are tried in order.
  -vault_identity_list=
  +;vault_identity_list=
   
   # (string) If true, decrypting vaults with a vault id will only try the password from the matching vault-id.
  -vault_id_match=False
  +;vault_id_match=False
   
   # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
   # If executable, it will be run and the resulting stdout will be used as the password.
  -vault_password_file=
  +;vault_password_file=
   
   # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
  -verbosity=0
  +;verbosity=0
   
   # (boolean) Toggle to control the showing of deprecation warnings
  -deprecation_warnings=True
  +;deprecation_warnings=True
   
   # (boolean) Toggle to control showing warnings related to running devel.
  -devel_warning=True
  +;devel_warning=True
   
   # (boolean) Normally ``ansible-playbook`` will print a header for each task that is run. These headers will contain the name: field from the task if you specified one. If you didn't then ``ansible-playbook`` uses the task's action to help you tell which task is presently running. Sometimes you run many of the same action and so you want more information about the task to differentiate it from others of the same action. If you set this variable to True in the config then ``ansible-playbook`` will also include the task's arguments in the header.
   # This setting defaults to False because there is a chance that you have sensitive values in your parameters and you do not want those to be printed.
   # If you set this to True you should be sure that you have secured your environment's stdout (no one can shoulder surf your screen and you aren't saving stdout to an insecure file) or made sure that all of your playbooks explicitly added the ``no_log: True`` parameter to tasks that have sensitive values :ref:`keep_secret_data` for more information.
  -display_args_to_stdout=False
  +;display_args_to_stdout=False
   
   # (boolean) Toggle to control displaying skipped task/host entries in a task in the default callback.
  -display_skipped_hosts=True
  +;display_skipped_hosts=True
   
   # (string) Root docsite URL used to generate docs URLs in warning/error text; must be an absolute URL with a valid scheme and trailing slash.
  -docsite_root_url=https://docs.ansible.com/ansible-core/
  +;docsite_root_url=https://docs.ansible.com/ansible-core/
   
   # (pathspec) Colon-separated paths in which Ansible will search for Documentation Fragments Plugins.
  -doc_fragment_plugins=/home/fac3/.ansible/plugins/doc_fragments:/usr/share/ansible/plugins/doc_fragments
  +;doc_fragment_plugins=/home/fac3/.ansible/plugins/doc_fragments:/usr/share/ansible/plugins/doc_fragments
   
   # (string) By default, Ansible will issue a warning when a duplicate dict key is encountered in YAML.
   # These warnings can be silenced by adjusting this setting to False.
  -duplicate_dict_key=warn
  +;duplicate_dict_key=warn
   
   # (string) for the cases in which Ansible needs to return a file within an editor, this chooses the application to use.
  -editor=vi
  +;editor=vi
   
   # (boolean) Whether or not to enable the task debugger, this previously was done as a strategy plugin.
   # Now all strategy plugins can inherit this behavior. The debugger defaults to activating when
   # a task is failed on unreachable. Use the debugger keyword for more flexibility.
  -enable_task_debugger=False
  +;enable_task_debugger=False
   
   # (boolean) Toggle to allow missing handlers to become a warning instead of an error when notifying.
  -error_on_missing_handler=True
  +;error_on_missing_handler=True
   
   # (list) Which modules to run during a play's fact gathering stage, using the default of 'smart' will try to figure it out based on connection type.
   # If adding your own modules but you still want to use the default Ansible facts, you will want to include 'setup' or corresponding network module to the list (if you add 'smart', Ansible will also figure it out).
   # This does not affect explicit calls to the 'setup' module, but does always affect the 'gather_facts' action (implicit or explicit).
  -facts_modules=smart
  +;facts_modules=smart
   
   # (boolean) Set this to "False" if you want to avoid host key checking by the underlying connection plugin Ansible uses to connect to the host.
   # Please read the documentation of the specific connection plugin used for details.
  -host_key_checking=True
  +;host_key_checking=True
   
   # (boolean) Facts are available inside the `ansible_facts` variable, this setting also pushes them as their own vars in the main namespace.
   # Unlike inside the `ansible_facts` dictionary where the prefix `ansible_` is removed from fact names, these will have the exact names that are returned by the module.
  -inject_facts_as_vars=True
  +;inject_facts_as_vars=True
   
   # (string) Path to the Python interpreter to be used for module execution on remote targets, or an automatic discovery mode. Supported discovery modes are ``auto`` (the default), ``auto_silent``, ``auto_legacy``, and ``auto_legacy_silent``. All discovery modes employ a lookup table to use the included system Python (on distributions known to include one), falling back to a fixed ordered list of well-known Python interpreter locations if a platform-specific default is not available. The fallback behavior will issue a warning that the interpreter should be set explicitly (since interpreters installed later may change which one is used). This warning behavior can be disabled by setting ``auto_silent`` or ``auto_legacy_silent``. The value of ``auto_legacy`` provides all the same behavior, but for backward-compatibility with older Ansible releases that always defaulted to ``/usr/bin/python``, will use that interpreter if present.
  -interpreter_python=auto
  +;interpreter_python=auto
   
   # (boolean) If 'false', invalid attributes for a task will result in warnings instead of errors.
  -invalid_task_attribute_failed=True
  +;invalid_task_attribute_failed=True
   
   # (boolean) By default, Ansible will issue a warning when there are no hosts in the inventory.
   # These warnings can be silenced by adjusting this setting to False.
  -localhost_warning=True
  +;localhost_warning=True
   
   # (int) This will set log verbosity if higher than the normal display verbosity, otherwise it will match that.
  -log_verbosity=
  +;log_verbosity=
   
   # (int) Maximum size of files to be considered for diff display.
  -max_diff_size=104448
  +;max_diff_size=104448
   
   # (list) List of extensions to ignore when looking for modules to load.
   # This is for rejecting script and binary module fallback extensions.
  -module_ignore_exts=.pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, .rst, .yaml, .yml, .ini
  +;module_ignore_exts=.pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, .rst, .yaml, .yml, .ini
   
   # (bool) Enables whether module responses are evaluated for containing non-UTF-8 data.
   # Disabling this may result in unexpected behavior.
   # Only ansible-core should evaluate this configuration.
  -module_strict_utf8_response=True
  +;module_strict_utf8_response=True
   
   # (list) TODO: write it
  -network_group_modules=eos, nxos, ios, iosxr, junos, enos, ce, vyos, sros, dellos9, dellos10, dellos6, asa, aruba, aireos, bigip, ironware, onyx, netconf, exos, voss, slxos
  +;network_group_modules=eos, nxos, ios, iosxr, junos, enos, ce, vyos, sros, dellos9, dellos10, dellos6, asa, aruba, aireos, bigip, ironware, onyx, netconf, exos, voss, slxos
   
   # (boolean) Previously Ansible would only clear some of the plugin loading caches when loading new roles, this led to some behaviors in which a plugin loaded in previous plays would be unexpectedly 'sticky'. This setting allows the user to return to that behavior.
  -old_plugin_cache_clear=False
  +;old_plugin_cache_clear=False
   
   # (string) for the cases in which Ansible needs to return output in a pageable fashion, this chooses the application to use.
  -pager=less
  +;pager=less
   
   # (path) A number of non-playbook CLIs have a ``--playbook-dir`` argument; this sets the default value for it.
  -playbook_dir=
  +;playbook_dir=
   
   # (string) This sets which playbook dirs will be used as a root to process vars plugins, which includes finding host_vars/group_vars.
  -playbook_vars_root=top
  +;playbook_vars_root=top
   
   # (path) A path to configuration for filtering which plugins installed on the system are allowed to be used.
   # See :ref:`plugin_filtering_config` for details of the filter file's format.
   #  The default is /etc/ansible/plugin_filters.yml
  -plugin_filters_cfg=
  +;plugin_filters_cfg=
   
   # (string) Attempts to set RLIMIT_NOFILE soft limit to the specified value when executing Python modules (can speed up subprocess usage on Python 2.x. See https://bugs.python.org/issue11284). The value will be limited by the existing hard limit. Default value of 0 does not attempt to adjust existing system-defined limits.
  -python_module_rlimit_nofile=0
  +;python_module_rlimit_nofile=0
   
   # (bool) This controls whether a failed Ansible playbook should create a .retry file.
  -retry_files_enabled=False
  +;retry_files_enabled=False
   
   # (path) This sets the path in which Ansible will save .retry files when a playbook fails and retry files are enabled.
   # This file will be overwritten after each run with the list of failed hosts from all plays.
  -retry_files_save_path=
  +;retry_files_save_path=
   
   # (str) This setting can be used to optimize vars_plugin usage depending on the user's inventory size and play selection.
  -run_vars_plugins=demand
  +;run_vars_plugins=demand
   
   # (bool) This adds the custom stats set via the set_stats plugin to the default output.
  -show_custom_stats=False
  +;show_custom_stats=False
   
   # (string) Action to take when a module parameter value is converted to a string (this does not affect variables). For string parameters, values such as '1.00', "['a', 'b',]", and 'yes', 'y', etc. will be converted by the YAML parser unless fully quoted.
   # Valid options are 'error', 'warn', and 'ignore'.
   # Since 2.8, this option defaults to 'warn' but will change to 'error' in 2.12.
  -string_conversion_action=warn
  +;string_conversion_action=warn
   
   # (boolean) Allows disabling of warnings related to potential issues on the system running Ansible itself (not on the managed hosts).
   # These may include warnings about third-party packages or other conditions that should be resolved if possible.
  -system_warnings=True
  +;system_warnings=True
   
   # (string) A string to insert into target logging for tracking purposes
  -target_log_info=
  +;target_log_info=
   
   # (boolean) This option defines whether the task debugger will be invoked on a failed task when ignore_errors=True is specified.
   # True specifies that the debugger will honor ignore_errors, and False will not honor ignore_errors.
  -task_debugger_ignore_errors=True
  +;task_debugger_ignore_errors=True
   
   # (integer) Set the maximum time (in seconds) for a task action to execute in.
   # Timeout runs independently from templating or looping. It applies per each attempt of executing the task's action and remains unchanged by the total time spent on a task.
   # When the action execution exceeds the timeout, Ansible interrupts the process. This is registered as a failure due to outside circumstances, not a task failure, to receive appropriate response and recovery process.
   # If set to 0 (the default) there is no timeout.
  -task_timeout=0
  +;task_timeout=0
   
   # (string) Make ansible transform invalid characters in group names supplied by inventory sources.
  -force_valid_group_names=never
  +;force_valid_group_names=never
   
   # (boolean) Toggles the use of persistence for connections.
  -use_persistent_connections=False
  +;use_persistent_connections=False
   
   # (bool) A toggle to disable validating a collection's 'metadata' entry for a module_defaults action group. Metadata containing unexpected fields or value types will produce a warning when this is True.
  -validate_action_group_metadata=True
  +;validate_action_group_metadata=True
   
   # (list) Accept list for variable plugins that require it.
  -vars_plugins_enabled=host_group_vars
  +;vars_plugins_enabled=host_group_vars
   
   # (list) Allows to change the group variable precedence merge order.
  -precedence=all_inventory, groups_inventory, all_plugins_inventory, all_plugins_play, groups_plugins_inventory, groups_plugins_play
  +;precedence=all_inventory, groups_inventory, all_plugins_inventory, all_plugins_play, groups_plugins_inventory, groups_plugins_play
   
   # (string) The salt to use for the vault encryption. If it is not provided, a random salt will be used.
  -vault_encrypt_salt=
  +;vault_encrypt_salt=
   
   # (bool) Force 'verbose' option to use stderr instead of stdout
  -verbose_to_stderr=False
  +;verbose_to_stderr=False
   
   # (integer) For asynchronous tasks in Ansible (covered in Asynchronous Actions and Polling), this is how long, in seconds, to wait for the task spawned by Ansible to connect back to the named pipe used on Windows systems. The default is 5 seconds. This can be too low on slower systems, or systems under heavy load.
   # This is not the total time an async command can run for, but is a separate timeout to wait for an async command to start. The task will only start to be timed against its async_timeout once it has connected to the pipe, so the overall maximum duration the task can take will be extended by the amount specified here.
  -win_async_startup_timeout=5
  +;win_async_startup_timeout=5
   
   # (list) Check all of these extensions when looking for 'variable' files which should be YAML or JSON or vaulted versions of these.
   # This affects vars_files, include_vars, inventory and vars plugins among others.
  @@ -430,47 +431,47 @@
   
   [privilege_escalation]
   # (boolean) Display an agnostic become prompt instead of displaying a prompt containing the command line supplied become method.
  -agnostic_become_prompt=True
  +;agnostic_become_prompt=True
   
   # (boolean) When ``False``(default), Ansible will skip using become if the remote user is the same as the become user, as this is normally a redundant operation. In other words root sudo to root.
   # If ``True``, this forces Ansible to use the become plugin anyways as there are cases in which this is needed.
  -become_allow_same_user=False
  +;become_allow_same_user=False
   
   # (boolean) Toggles the use of privilege escalation, allowing you to 'become' another user after login.
  -become=False
  +become=True
   
   # (boolean) Toggle to prompt for privilege escalation password.
  -become_ask_pass=False
  +become_ask_pass=True
   
   # (string) executable to use for privilege escalation, otherwise Ansible will depend on PATH.
  -become_exe=
  +;become_exe=
   
   # (string) Flags to pass to the privilege escalation executable.
  -become_flags=
  +;become_flags=
   
   # (string) Privilege escalation method to use when `become` is enabled.
   become_method=sudo
   
   # (string) The user your login/remote user 'becomes' when using privilege escalation, most systems will use 'root' when no user is specified.
  -become_user=root
  +;become_user=root
   
   
   [persistent_connection]
   # (path) Specify where to look for the ansible-connection script. This location will be checked before searching $PATH.
   # If null, ansible will start with the same directory as the ansible script.
  -ansible_connection_path=
  +;ansible_connection_path=
   
   # (int) This controls the amount of time to wait for a response from a remote device before timing out a persistent connection.
  -command_timeout=30
  +;command_timeout=30
   
   # (integer) This controls the retry timeout for persistent connection to connect to the local domain socket.
  -connect_retry_timeout=15
  +;connect_retry_timeout=15
   
   # (integer) This controls how long the persistent connection will remain idle before it is destroyed.
  -connect_timeout=30
  +;connect_timeout=30
   
   # (path) Path to the socket to be used by the connection persistence system.
  -control_path_dir=/home/fac3/.ansible/pc
  +;control_path_dir=/home/fac3/.ansible/pc
   
   
   [connection]
  @@ -479,234 +480,234 @@
   # It can result in a very significant performance improvement when enabled.
   # However this conflicts with privilege escalation (become). For example, when using 'sudo:' operations you must first disable 'requiretty' in /etc/sudoers on all managed hosts, which is why it is disabled by default.
   # This setting will be disabled if ``ANSIBLE_KEEP_REMOTE_FILES`` is enabled.
  -pipelining=False
  +;pipelining=False
   
   
   [colors]
   # (string) Defines the color to use on 'Changed' task status.
  -changed=yellow
  +;changed=yellow
   
   # (string) Defines the default color to use for ansible-console.
  -console_prompt=white
  +;console_prompt=white
   
   # (string) Defines the color to use when emitting debug messages.
  -debug=dark gray
  +;debug=dark gray
   
   # (string) Defines the color to use when emitting deprecation messages.
  -deprecate=purple
  +;deprecate=purple
   
   # (string) Defines the color to use when showing added lines in diffs.
  -diff_add=green
  +;diff_add=green
   
   # (string) Defines the color to use when showing diffs.
  -diff_lines=cyan
  +;diff_lines=cyan
   
   # (string) Defines the color to use when showing removed lines in diffs.
  -diff_remove=red
  +;diff_remove=red
   
   # (string) Defines the color to use when emitting a constant in the ansible-doc output.
  -doc_constant=dark gray
  +;doc_constant=dark gray
   
   # (string) Defines the color to use when emitting a deprecated value in the ansible-doc output.
  -doc_deprecated=magenta
  +;doc_deprecated=magenta
   
   # (string) Defines the color to use when emitting a link in the ansible-doc output.
  -doc_link=cyan
  +;doc_link=cyan
   
   # (string) Defines the color to use when emitting a module name in the ansible-doc output.
  -doc_module=yellow
  +;doc_module=yellow
   
   # (string) Defines the color to use when emitting a plugin name in the ansible-doc output.
  -doc_plugin=yellow
  +;doc_plugin=yellow
   
   # (string) Defines the color to use when emitting cross-reference in the ansible-doc output.
  -doc_reference=magenta
  +;doc_reference=magenta
   
   # (string) Defines the color to use when emitting error messages.
  -error=red
  +;error=red
   
   # (string) Defines the color to use for highlighting.
  -highlight=white
  +;highlight=white
   
   # (string) Defines the color to use when showing 'Included' task status.
  -included=cyan
  +;included=cyan
   
   # (string) Defines the color to use when showing 'OK' task status.
  -ok=green
  +;ok=green
   
   # (string) Defines the color to use when showing 'Skipped' task status.
  -skip=cyan
  +;skip=cyan
   
   # (string) Defines the color to use on 'Unreachable' status.
  -unreachable=bright red
  +;unreachable=bright red
   
   # (string) Defines the color to use when emitting verbose messages. In other words, those that show with '-v's.
  -verbose=blue
  +;verbose=blue
   
   # (string) Defines the color to use when emitting warning messages.
  -warn=bright purple
  +;warn=bright purple
   
   
   [selinux]
   # (boolean) This setting causes libvirt to connect to LXC containers by passing ``--noseclabel`` parameter to ``virsh`` command. This is necessary when running on systems which do not have SELinux.
  -libvirt_lxc_noseclabel=False
  +;libvirt_lxc_noseclabel=False
   
   # (list) Some filesystems do not support safe operations and/or return inconsistent errors, this setting makes Ansible 'tolerate' those in the list without causing fatal errors.
   # Data corruption may occur and writes are not always verified when a filesystem is in the list.
  -special_context_filesystems=fuse, nfs, vboxsf, ramfs, 9p, vfat
  +;special_context_filesystems=fuse, nfs, vboxsf, ramfs, 9p, vfat
  
  
   [diff]
   # (bool) Configuration toggle to tell modules to show differences when in 'changed' status, equivalent to ``--diff``.
  -always=False
  +;always=False
   
   # (integer) Number of lines of context to show when displaying the differences between files.
  -context=3
  +;context=3
   
   
   [galaxy]
   # (path) The directory that stores cached responses from a Galaxy server.
   # This is only used by the ``ansible-galaxy collection install`` and ``download`` commands.
   # Cache files inside this dir will be ignored if they are world writable.
  -cache_dir=/home/fac3/.ansible/galaxy_cache
  +;cache_dir=/home/fac3/.ansible/galaxy_cache
   
   # (bool) whether ``ansible-galaxy collection install`` should warn about ``--collections-path`` missing from configured :ref:`collections_paths`.
  -collections_path_warning=True
  +;collections_path_warning=True
   
   # (path) Collection skeleton directory to use as a template for the ``init`` action in ``ansible-galaxy collection``, same as ``--collection-skeleton``.
  -collection_skeleton=
  +;collection_skeleton=
   
   # (list) patterns of files to ignore inside a Galaxy collection skeleton directory.
  -collection_skeleton_ignore=^.git$, ^.*/.git_keep$
  +;collection_skeleton_ignore=^.git$, ^.*/.git_keep$
   
   # (bool) Disable GPG signature verification during collection installation.
  -disable_gpg_verify=False
  +;disable_gpg_verify=False
   
   # (bool) Some steps in ``ansible-galaxy`` display a progress wheel which can cause issues on certain displays or when outputting the stdout to a file.
   # This config option controls whether the display wheel is shown or not.
   # The default is to show the display wheel if stdout has a tty.
  -display_progress=
  +;display_progress=
   
   # (path) Configure the keyring used for GPG signature verification during collection installation and verification.
  -gpg_keyring=
  +;gpg_keyring=
   
   # (boolean) If set to yes, ansible-galaxy will not validate TLS certificates. This can be useful for testing against a server with a self-signed certificate.
  -ignore_certs=
  +;ignore_certs=
   
   # (list) A list of GPG status codes to ignore during GPG signature verification. See L(https://github.com/gpg/gnupg/blob/master/doc/DETAILS#general-status-codes) for status code descriptions.
   # If fewer signatures successfully verify the collection than `GALAXY_REQUIRED_VALID_SIGNATURE_COUNT`, signature verification will fail even if all error codes are ignored.
  -ignore_signature_status_codes=
  +;ignore_signature_status_codes=
   
   # (str) The number of signatures that must be successful during GPG signature verification while installing or verifying collections.
   # This should be a positive integer or all to indicate all signatures must successfully validate the collection.
   # Prepend + to the value to fail if no valid signatures are found for the collection.
  -required_valid_signature_count=1
  +;required_valid_signature_count=1
   
   # (path) Role skeleton directory to use as a template for the ``init`` action in ``ansible-galaxy``/``ansible-galaxy role``, same as ``--role-skeleton``.
  -role_skeleton=
  +;role_skeleton=
   
   # (list) patterns of files to ignore inside a Galaxy role or collection skeleton directory.
  -role_skeleton_ignore=^.git$, ^.*/.git_keep$
  +;role_skeleton_ignore=^.git$, ^.*/.git_keep$
   
   # (string) URL to prepend when roles don't specify the full URI, assume they are referencing this server as the source.
  -server=https://galaxy.ansible.com
  +;server=https://galaxy.ansible.com
   
   # (list) A list of Galaxy servers to use when installing a collection.
   # The value corresponds to the config ini header ``[galaxy_server.{{item}}]`` which defines the server details.
   # See :ref:`galaxy_server_config` for more details on how to define a Galaxy server.
   # The order of servers in this list is used as the order in which a collection is resolved.
   # Setting this config option will ignore the :ref:`galaxy_server` config option.
  -server_list=
  +;server_list=
   
   # (int) The default timeout for Galaxy API calls. Galaxy servers that don't configure a specific timeout will fall back to this value.
  -server_timeout=60
  +;server_timeout=60
   
   # (path) Local path to galaxy access token file
  -token_path=/home/fac3/.ansible/galaxy_token
  +;token_path=/home/fac3/.ansible/galaxy_token
   
   
   [inventory]
   # (string) This setting changes the behaviour of mismatched host patterns, it allows you to force a fatal error, a warning or just ignore it.
  -host_pattern_mismatch=warning
  +;host_pattern_mismatch=warning
   
   # (boolean) If 'true', it is a fatal error when any given inventory source cannot be successfully parsed by any available inventory plugin; otherwise, this situation only attracts a warning.
   
  -any_unparsed_is_failed=False
  +;any_unparsed_is_failed=False
   
   # (bool) Toggle to turn on inventory caching.
   # This setting has been moved to the individual inventory plugins as a plugin option :ref:`inventory_plugins`.
   # The existing configuration settings are still accepted with the inventory plugin adding additional options from inventory configuration.
   # This message will be removed in 2.16.
  -cache=False
  +;cache=False
   
   # (string) The plugin for caching inventory.
   # This setting has been moved to the individual inventory plugins as a plugin option :ref:`inventory_plugins`.
   # The existing configuration settings are still accepted with the inventory plugin adding additional options from inventory and fact cache configuration.
   # This message will be removed in 2.16.
  -cache_plugin=
  +;cache_plugin=
   
   # (string) The inventory cache connection.
   # This setting has been moved to the individual inventory plugins as a plugin option :ref:`inventory_plugins`.
   # The existing configuration settings are still accepted with the inventory plugin adding additional options from inventory and fact cache configuration.
   # This message will be removed in 2.16.
  -cache_connection=
  +;cache_connection=
   
   # (string) The table prefix for the cache plugin.
   # This setting has been moved to the individual inventory plugins as a plugin option :ref:`inventory_plugins`.
   # The existing configuration settings are still accepted with the inventory plugin adding additional options from inventory and fact cache configuration.
   # This message will be removed in 2.16.
  -cache_prefix=ansible_inventory_
  +;cache_prefix=ansible_inventory_
   
   # (string) Expiration timeout for the inventory cache plugin data.
   # This setting has been moved to the individual inventory plugins as a plugin option :ref:`inventory_plugins`.
   # The existing configuration settings are still accepted with the inventory plugin adding additional options from inventory and fact cache configuration.
   # This message will be removed in 2.16.
  -cache_timeout=3600
  +;cache_timeout=3600
   
   # (list) List of enabled inventory plugins, it also determines the order in which they are used.
  -enable_plugins=host_list, script, auto, yaml, ini, toml
  +;enable_plugins=host_list, script, auto, yaml, ini, toml
   
   # (bool) Controls if ansible-inventory will accurately reflect Ansible's view into inventory or its optimized for exporting.
  -export=False
  +;export=False
   
   # (list) List of extensions to ignore when using a directory as an inventory source.
  -ignore_extensions=.pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, .rst, .orig, .ini, .cfg, .retry
  +;ignore_extensions=.pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, .rst, .orig, .ini, .cfg, .retry
   
   # (list) List of patterns to ignore when using a directory as an inventory source.
  -ignore_patterns=
  +;ignore_patterns=
   
   # (bool) If 'true' it is a fatal error if every single potential inventory source fails to parse, otherwise, this situation will only attract a warning.
   
  -unparsed_is_failed=False
  +;unparsed_is_failed=False
   
   # (boolean) By default, Ansible will issue a warning when no inventory was loaded and notes that it will use an implicit localhost-only inventory.
   # These warnings can be silenced by adjusting this setting to False.
  -inventory_unparsed_warning=True
  +;inventory_unparsed_warning=True
   
   
   [netconf_connection]
   # (string) This variable is used to enable bastion/jump host with netconf connection. If set to True the bastion/jump host ssh settings should be present in ~/.ssh/config file, alternatively it can be set to custom ssh configuration file path to read the bastion/jump host settings.
  -ssh_config=
  +;ssh_config=
   
   
   [paramiko_connection]
   # (boolean) TODO: write it
  -host_key_auto_add=False
  +;host_key_auto_add=False
   
   # (boolean) TODO: write it
  -look_for_keys=True
  +;look_for_keys=True
   
   
   [jinja2]
   # (list) This list of filters avoids 'type conversion' when templating variables.
   # Useful when you want to avoid conversion into lists or dictionaries for JSON strings, for example.
  -dont_type_filters=string, to_json, to_nice_json, to_yaml, to_nice_yaml, ppretty, json
  +;dont_type_filters=string, to_json, to_nice_json, to_yaml, to_nice_yaml, ppretty, json
   
   
   [tags]
   # (list) default list of tags to run in your plays, Skip Tags has precedence.
  -run=
  +;run=
   
   # (list) default list of tags to skip in your plays, has precedence over Run Tags
  -skip=
  +;skip=
   
  For understanding what these options do, and which options may help
  out with something specific, refer to the Ansible
  Documentation.
  The next question is where should this configuration file go? Well
  that depends. According to the Ansible-Config
  the options are:
  
  /etc/ansible/ansible.cfg - General system-wide configuration that
  will be used if present
  ~/.anisble.cfg - User config file, overrides the system
  configuration file if it is present
  ./ansible.cfg - Local config file, assumed to be project specific
  and will override previous options if present
  ANSIBLE_CONFIG - If this environment variable is set, it will
  override all other configuration options
  
  Choose the option that makes the most sense; for me on this project,
  I used the local directory because it was easy to change. It is also
  easy to put in Git (you are version controlling these configs
  right?).
  More Resources
  Ansible has plenty of resources online; many Stack Overflow questions
  on it, and various blog posts as well as the official documentation. This post
  was mostly a showing of what I was able to put together using the
  official documentation, some testing, and a bit of creativity. Keep in
  mind, that this blog post is intended to be a primer, and there are many
  other cool topics that I did not even touch on with Ansible such as the
  vault and Ansible’s roles.
  Both of which can be extremely useful for dealing with Ansible in a
  large production environment. As a quick run down, the vault is an
  encrypted file for storing secret and roles allow for re-using various
  Ansible artifacts (vars, files, tasks, etc) for the fleet. This not only
  makes understanding what is going on slightly easier, but also prevents
  unnecessary work being re-done.
  I do hope to touch back on the topic of Ansible and other automation
  software suites that allow for extremely rapid deployment and management
  that is not practical or reasonable for IT management to expect their
  team to manage by hand.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-02-28.html
  		https://foxide.xyz/projects/2025-02-28.html
  		Fri, 28 Feb 2025 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  Setting up a basic web server is one of the first projects that many
  people getting started in system administration and Linux take on. There
  are many good guides on the topic with some of the best ones I’ve seen
  coming from Vultr;
  however, not many of them include setting up a web server with HTTP/2
  and HTTP/3 support, as well as other security headers and optimizations.
  So, I am going to throw my guide in the hat as a resource for getting
  started.
  Requirements
  Before actually setting up a public web server, there are some things
  that are needed.
  
  Domain: As many people know, a domain is the name gets associated
  with IP addresses to make finding resources on the Internet easier. For
  example, my domain is foxide.xyz, which contains various kinds of
  records for different protocols.
  Hosting: The software and pages that will be served have to live
  somewhere, and generally this would be publicly accessible if the
  website is public. There are a few different options for this, but for
  most people a VPS is going to be
  the simplest. Though, it is also possible to host the website from home,
  however, most ISPs aren’t very happy with customers that chose to do so
  (unless you have a business plan).
  Operating System: This one could probably be lumped into hosting,
  however, picking an operating system is extremely important and there
  are some notes worth pointing out. The most important one being, use
  something that you are comfortable with and understand how to use.
  FreeBSD is a wonderful operating system and runs at least a quarter of
  the Internet’s traffic (via Netflix), however, I would have a difficult
  time recommending it to someone that has no experience in a Unix-like
  system. If you are extremely knowledgeable in an OS, even if it is not
  generally a server OS like Arch Linux, I would say that is a better
  choice for YOU than something else. Because when there is an issue, you
  will be more likely to know how to fix it rather than having to stumble
  around an unfamiliar environment.
  
  For this blog post, I am going to be using example.com as the domain
  for our website, and I am going to be going through the process both on
  Debian and FreeBSD. Nginx will be the web server of choice as it is a
  fairly popular one, and has support for all of the protocols that we
  need. I am not going to get into the process of installing the operating
  system, but if needed this guide
  will show how to install Debian, and this
  one will show how to install FreeBSD. The only real recommendation
  that would potentially go against default install options would be to
  make sure that the file system is ZFS, or is using LVM in the case of
  Debian (unless you installed ZFS as the file-system for Debian). Doing
  one of these will make backups much easier in the future.
  Setup
  After installing, configuring, and updating your new web server, we
  need to install anything that we will need for being a web server,
  namely Nginx. We will, however, need some other tools such as certbot and a good
  editor.
  Installing packages for Debian:
  apt install vim nginx certbot
  FreeBSD:
  pkg install nginx vim-tiny py311-certbot
  I will be using vim as the editor, but any other editor can be used;
  simply replace vim with any other editor and it should be
  fine. The next step is to enable nginx so that it can
  deliver web pages. For Debian and other SystemD based distros that looks
  like:
  # Commands should be run as root
  systemctl enable nginx
  systemctl start nginx
  and for FreeBSD:
  # Commands should be run as root
  sysrc nginx_enable
  service nginx start
  Then traveling to the IP (or domain name if that is already setup)
  will show a default nginx page that should look as follows:
  
  
  
  
  Technically, this is a functioning web server. HTML could be put into
  the relevant locations, and web pages would become available; however,
  this is well below the bare minimum for modern websites. We can do
  better.
  DNS Settings
  Next, let’s go ahead and setup the domain name to route to the
  correct IP address; many of the following steps will hinge on this being
  setup appropriately, so it would be good to have it done now (or
  sooner).
  To set the DNS records, login to the panel for your registrar and
  edit the domain settings. This is an exercise left to the reader, as
  each registrar’s website is going to look completely different from each
  other. For the web page to work we at least need to add an ‘A’ record or
  a ‘AAAA’ record; adding both record types is ideal as that will cover
  both IPv4 and IPv6 addresses and thus being reachable by nearly any
  device that can reach the Internet. The first record type is the ‘A’
  record which is for IPv4
  addresses The value for the A record should be the public IP address
  of the web server, to find out what that value is on Debian:
  ip addr
  
  1: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether 70:85:c2:f4:22:7d brd ff:ff:ff:ff:ff:ff
      inet 172.245.181.191 brd 172.245.181.255 scope global dynamic noprefixroute enp9s0
         valid_lft 567sec preferred_lft 567sec
  and on FreeBSD
  ifconfig
  vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
          options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
          ether 00:16:3c:d3:f3:4e
          inet 172.245.181.191 netmask 0xffffff80 broadcast 172.245.181.255
          media: Ethernet autoselect (10Gbase-T <full-duplex>)
          status: active
          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  The IPv4 address is the number next to ‘inet’ (172.245.181.191 in the
  example cases), that address needs to be added as an ‘A’ record in your
  registrar. Once that record propagates throughout the Internet (can take
  up to 48 hours) then the domain name should point to that IP address.
  This can be tested by using a simple ping command, or by
  using nslookup:
  # Ping simply checks that the connection can be established
  ping example.com
  PING example.com (96.7.128.175): 56 data bytes
  64 bytes from 96.7.128.175: icmp_seq=0 ttl=51 time=78.648 ms
  64 bytes from 96.7.128.175: icmp_seq=1 ttl=51 time=71.742 ms
  64 bytes from 96.7.128.175: icmp_seq=2 ttl=51 time=74.054 ms
  64 bytes from 96.7.128.175: icmp_seq=3 ttl=51 time=71.280 ms
  ...
  
  --- example.com ping statistics ---
  4 packets transmitted, 4 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 71.280/73.931/78.648/2.919 ms
  
  # nslookup will show the DNS information used to find the domain
  nslookup example.com
  
  Server:     172.16.0.10
  Address:    172.16.0.10#53
  
  Non-authoritative answer:
  Name:   example.com
  Address: 23.215.0.138
  Name:   example.com
  Address: 2600:1406:bc00:53::b81e:94ce
  My VPS does not have IPv6, however, it would appear as under the
  ‘inet’ entry with the entry name as ‘inet6’. Similar to the ‘A’ record
  for the IPv4 address, put that address as a record in your registrar,
  but the record type should be ‘AAAA’ rather than ‘A’. Similar commands
  can be used to check the work, nslookup will show both IPv4
  and IPv6 information, while ping will need to add the -6
  flag to force IPv6.
  Server Blocks
  Nginx operates with server blocks for each domain (or subdomain) that
  is being hosted on that instance of nginx. Certbot will use DNS to see
  if the domain is pointing to the correct domain, then will also check
  the nginx server blocks to make sure that nginx is prepared to serve web
  traffic for that domain. Server blocks will usually reside in the HTTP
  block of the nginx config file, or they will be in separate files to be
  included in the main config file. By default Debian (as well as many
  other distros) will encourage creating separate files for each domain or
  subdomain; the main advantages of this are better separation of files
  for making changes as well as being able to more quickly find where the
  relevant configuration will be. The method that Debian uses to do this
  is to have two directories in the /etc/nginx directory:
  sites-available and sites-enabled. The
  sites-available directory is for domains that could be
  enabled, but are not currently active. On the other hand, the
  sites-enabled directory are for the currently active sites.
  To make an available site active, simply create a symbolic link
  (symlink) to the sites-enabled directory:
  # must be run as root
  ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com
  On FreeBSD, this is not setup by default and server blocks could just
  go within the HTTP block in the main nginx.conf file. However, if you
  wanted to re-create this setup on FreeBSD (or another system that did
  not already have it) simply put the following line in your HTTP block in
  the nginx.conf file:
  # File path for most BSD systems
  # for Linux systems, remove the /usr/local
  include /usr/local/etc/nginx/sites-enabled/*;
  Then create the sites-available and
  sites-enabled directories manually.
  Now let’s create a server block for our domain! Below is a very
  simple example of a server block for the domain ‘example.com’:
  server {
      listen 80;
      listen [::]:80;
      server_name example.com;
  
      location / {
          root /srv/www/example.com
          index index.html index.htm
      }
  }
  
  The above example should go into a either go in a file in
  sites-available then symlinked, or in the HTTP block of
  nginx.conf. This server block will listen on port 80 (HTTP)
  for a request for example.com. If an HTTP request is made for that
  domain on that port, it will serve either index.html or index.htm
  located in /srv/www/example.com. The config can be tested
  by running nginx -t; assuming that no errors are returned
  than nginx can be restarted.
  # These commands must have root privileges
  # On Debian:
  systemctl restart nginx
  # On FreeBSD:
  service nginx restart
  Then typing http://example.com in the URL bar of a web browser should
  now render one of the index files located in
  /srv/www/example.com. The next step is setting up our TLS
  certificates.
  Transport Layer Security
  (TLS)
  After the DNS records and the server blocks in nginx have been
  configured, then we can setup TLS,
  still often referred to as SSL. The tool for doing
  this (at least for the average person) is Certot, which is a tool
  that provides free TLS certificates via LetsEncrypt. Certbot will interface
  with LetEncrypt to acquire a certificate for a domain, but it will also
  write the relevant configurations for the domain in the web server
  (nginx in our case) to enable it. Install the tool if it is not already
  installed (within the context of this guide, it was installed alongside
  nginx). Then we simply need to run the following command:
  # Commands must be run as root
  certbot --nginx
  Certbot will look in the nginx configuration file for the
  server_name deceleration to find which domains to add to
  the certificate. Alternatively, the domains and subdomains can be
  manually declared using the -d flag as follows:
  # Commands must be run as root
  certbot -d example.com,git.example.com,docs.example.com --nginx
  # Multiple -d flags can be used instead of commas
  certbot -d example.com -d git.example.com -d docs.example.com --nginx
  After running one of the two commands, certbot will acquire a TLS
  certificate, and edit the nginx config to enable it for the relevant
  domains. The simplest way to confirm that the certificate is installed
  and valid, simply visit the URL in a browser and look for the lock icon
  in the URL bar. However, for a command line way to check use the
  following command:
  echo | openssl s_client -connect example.org:443 2>/dev/null | openssl x509 -noout -dates
  Firewall Rules
  Explaining firewall configurations are well outside of the scope of
  this already long post. That being said, it is very important to set up
  some sort of a firewall on the server since it is public facing. In
  general, the rules should cover at least the following:
  
  Allow TCP traffic on port 80 (HTTP) and port 443 (https)
  Allow UDP traffic on port 80 and 443 (this is for QUIC as it is a
  UDP protocol)
  Allow for a means of managing the server. This would generally be
  SSH (TCP traffic on port 22)
  Any other traffic that should be allowed (do not allow just anything
  through the firewall)
  Default deny anything else
  
  There are many different firewall softwares to choose from, however,
  one of the more popular ones for Linux systems is the Uncomplicated
  Firewall (UFW) and on FreeBSD PF
  is the most common firewall.
  Nginx Configuration
  Improvements
  At this point, you should have a pretty solid web server setup.
  However, there are still things that can be done to improve the setup a
  bit more. The big ticket items are adding support for HTTP/2 and HTTP/3
  (and QUIC) as well as
  enabling some other security features for the domain. The main benefit
  of HTTP/2 over HTTP/1 is the ability to multiplex requests, meaning that
  rather than having to request one asset at a time, multiple assets can
  be requested at a time reducing load times. It also offers things like
  header compression, prioritization, and binary framing. HTTP/3 and QUIC
  further improve the protocol by building in TLS (TLS 1.3 specifically)
  by default, thus reducing connection requests. It also improves
  congestion control and adds support for connection migration. One of the
  biggest changes to HTTP/3 over HTTP/2 is that it uses UDP over TCP, and
  the connection verification as well as packet loss handling are built
  into QUIC. This is why web servers that want to support QUIC must enable
  HTTP and HTTPS traffic on UDP as well as TCP.
  We are going to start with an example server block that we will build
  off of in the following sections:
  listen example.com:443 ssl;
  server_name example.com;
  
  # Other server config stuff below
  ...
  All of the features that we will be working with are added toward the
  beginning of the server block near the listening and server name
  sections, so, I am only including that in the example.
  HTTP/2 Support
  Adding HTTP/2 support is fairly simple, just add
  http2 on; in the server block as follows:
  listen example.com:443 ssl;
  server_name example.com;
  
  # Enable HTTP/2 Support
  http2 on;
  
  # Other server config stuff below
  ...
  Then restart the nginx service and confirm. The easiest way to
  confirm is to find an online service that will check for you such as keycdn.com. However, if
  you want to verify yourself, you can do so by using Firefox’s developer
  tools. Open the developer tools by pressing CTRL+Shift+I, then reload
  the page. Click on the line that has the HTML file for the root of the
  domain, and look for the HTTP version in the box on the right. If you
  see ‘HTTP/2’, then HTTP/2 support is enabled and working (be mindful
  that browsers like to cache things, and Firefox may be using a cached
  version of the website).
  HTTP/3 and QUIC Support
  Adding support for HTTP/2 is very simple and straightforward,
  however, HTTP/3 and QUIC are not quite that simple.
  # Adding listen address to specificly listen for quic traffic
  listen example.com:443 quic reuseport;
  listen example.com:443 ssl;
  server_name example.com;
  
  # Enable HTTP/3
  http3 on;
  quic_retry on;
  add_header Alt-Svc 'h3=":$server_port"; ma=86400, h3-29=":$server_port"; ma=86400';
  
  # Enable HTTP/2
  http2 on;
  
  # Other server config stuff below
  ...
  Extra Security Features
  The last items for this post is enabling some extra security headers
  in Nginx for the site. Namely, HTTP
  Strict Transport Security (HSTS), Content Security Policy
  (CSP), and secure headers. These items work together to further
  reduce threat actors from performing various types of attacks both to
  the server itself, and to clients that are interacting with the server
  to boost the overall security of the website. The following configs
  should be enough to get started for basic sites (such as static sites),
  but definitely do research as to how these options should be configured
  for your site. Using things like Firefox’s built-in developer tools are
  super helpful here.
  
  # Resources
      # Security Header Stuff
          add_header X-XSS-Protection "1; mode=block";
          add_header X-Content-Type-Options nosniff;
          add_header Access-Control-Allow-Origin "${allowed_origins}";
          add_header Access-Control-Allow-Credentials true;
          add_header X-Frame-Options "SAMEORIGIN";
       
          # Opt out of Google's FLoC Network
          add_header Permissions-Policy interest-cohort=();
       
          # Enable SharedArrayBuffer in Firefox (for .xlsx export)
          add_header Cross-Origin-Resource-Policy cross-origin;
          add_header Cross-Origin-Embedder-Policy require-corp;
  
         # HSTS (ngx_http_headers_module is required) (63072000 seconds)
         add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
  
         # OCSP stapling
         ssl_stapling on;
         ssl_stapling_verify on;
  
         # 0-RTT Stuff
         ssl_early_data on;
  
         # Content Security Policy
         add_header Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline';" always;
  
      # verify chain of trust of OCSP response using Root CA and Intermediate certs
      ssl_trusted_certificate /usr/local/share/certs/ca-root-nss.crt;
  
      # replace with the IP address of your resolver
      resolver 9.9.9.9 149.112.112.112 208.67.222.222 208.67.220.220 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1;
  Final Thoughts
  This post was quite big, but much of it is stuff that is easily found
  online in one place or another, the idea of this post was putting most
  everything one would need to get an HTTP/2 and HTTP/3 enabled web server
  up and running in one place as I have not found that online (at least
  not yet).
  Using online tools to assist in confirming that things are working
  properly is, in my opinion, the way to go especially for the security
  headers. The tool I have found and like the most is this one, but
  your mileage and opinions may vary. I also wanted to note that I did not
  cover the 0-RTT header because it is not enabled on my web server.
  Reason for that is that I used OpenSSL, which does not (at the time of
  writing) support that particular header. If you want that to be enabled,
  the recommendation in this
  article is to use boringSSL. This
  might be worth doing in some cases, but I did not feel it was worth it
  in mine.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-03-30.html
  		https://foxide.xyz/projects/2025-03-30.html
  		Sun, 30 Mar 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  The BSDs are known to have some wonderful built-in tools to the base
  operating system, and the tools I am going to cover today are no
  different. IPFW is one of the three firewalls built into the FreeBSD
  base system, and while it is a very solid firewall, it also has a
  network testing/shaping utility built in as well in the form of dummynet.
  IPFW
  Before enabling and trying to use IPFW, it is recommended to make
  sure we do not lock ourselves out of our systems. This is less important
  if you are locally working on a machine, but if you are accessing the
  machine via a network protocol (like SSH) firewall rules are a great way
  to lock yourself out. Generally, the recommended way to allow all
  traffic rather than blocking all traffic. The way I have had the most
  luck with is the running the following command before starting IPFW:
  # Must be run as root
  kenv net.inet.ip.fw.default_to_accept=1
  Then enable IPFW by running either of the following commands as
  root:
  # Must run as root
  service ipfw onestart
  # or running
  kldload ipfw
  Now that we have the basic firewall setup, let’s add some rules. I
  will now present one of the most secure, if not the most secure,
  firewall configurations possible. This is also the default configuration
  for IPFW:
  ipfw -q add 65535 deny ip from any to any
  In this configuration all network traffic on the machine is blocked,
  thus making it extremely secure. The biggest problem with this
  configuration is that it is also quite unhelpful. In the vast majority
  of cases in 2025 computer use will require a network at some point which
  the above rule will block. If this is the rule you are really wanting,
  just unplug the Ethernet cable and remove the wireless card from the
  machine. Assuming that some amount of networking is desired on the
  machine, let’s look at some rules that might be more useful. The
  following firewall rules will disable all traffic except for TCP traffic
  coming in on port 22:
  ipfw -a add 00100 allow tcp from any to me 22 in via vtnet0 setup limit src-add 2
  ipfw -a add 00200 deny ip from any to any
  With this rule set, we are able to SSH into the machine, but are not
  able to do much of anything else. While that is not super useful, it
  does give remote access to the machine. More importantly, it shows an
  example of how firewall rules are processed in IPFW (and many other
  firewalls). The rules are processed by going down the rule numbers and
  when the packet comes across a rule that matches, the packet is then
  processed according to that rule. So, given the following rules:
  00100 allow tcp from 192.168.122.1 to me 22 via vtnet0 setup limit src-addr 2 :default
  00500 deny tcp from any to me via vtnet
  65535 deny ip from any to any
  We would see that rule ‘00100’ and rule ‘00500’ conflict (as well as
  the default deny all). However, I can still SSH into the machine because
  the packet is coming from 192.168.122.1 on port 22 to the correct
  interface. Once IPFW sees that the SSH packet matches that rule, it
  ceases to process the packet any further. It is very important to keep
  this in mind as rules may not be effective, or be too effective if not
  ordered appropriately.
  Now that we have a very basic understanding of IPFW, let’s see about
  making a basic firewall configuration for a public facing server. During
  the process of developing your firewall rules, it is highly important to
  not lock yourself out of the system. As stated above, the common method
  of doing this is changing the default ‘block all’ to ‘allow all’,
  however, it is possible that you will lock yourself out at some point
  during the creation of these rules (especially once you turn the deny
  all back on). So, as another way to prevent this, it might be worth
  setting up an automated timed restart as a backup method of accessing
  the system. The shutdown command can do this, but after
  running it no more user logins are allowed; an alternative that does not
  run into this problem is using sleep:
   # Must run as root
  sleep 15m && reboot &
  This command will reboot the system after fifteen minutes, which will
  then revert IPFW to the previously known state (off if the service isn’t
  enabled). Adjust the time to whatever seems reasonable for you (or don’t
  use it at all at your own risk). To prevent the reboot, simply kill the
  sleep command:
  # Must run as root
  ps -aux | grep sleep
  root  4139  0.0  0.2 14076 2116  0  SC   22:27      0:00.00 sleep 15m
  root  4141  0.0  0.0   432  264  0  R+   22:27      0:00.00 grep sleep
  kill -9 4139
  Example Firewall Script
  I am not going to go too much into the development process for
  firewall rules as they will vary greatly depending on your needs. The ipfw man page or
  the FreeBSD
  docs page are both great places to find specific implementation
  details or how to go about something I may not cover in this blog
  post.
  To setup IPFW rules, we need to set the firewall_script
  variable in /etc/rc.conf, this can be done by either
  running:
  sysrc firewall_script="/etc/ipfw.rules"
  # or running
  echo 'firewall_script="/etc/ipfw.rules"' >> /etc/rc.conf
  This will make IPFW look to a script located at
  /etc/ipfw.rules for what it should do. The script location
  can obviously change to another location if preferred. Let’s see what a
  script for a basic webserver might look like:
  #!/bin/sh
  ipfw -q -f flush
  cmd="ipfw -q add"
  pif="vtnet0"
  
  # Allow SSH traffic
  $cmd 00100 allow tcp from any to me 22 in via $pif limit src-addr 2
  
  # Allow bi-directional pings
  $cmd 00200 allow icmp from any to any via $pif keep-state
  
  # Allow DNS via UDP and TCP
  $cmd 00300 allow udp from me to any 53 via $pif keep-state
  $cmd 00350 allow tcp from me to any 53 via $pif keep-state
  
  # Allow Receiving HTTP(S) 1, 2, and 3 (QUIC)
  $cmd 00400 allow tcp from any to me 80 via $pif keep-state
  $cmd 00400 allow tcp from any to me 443 via $pif keep-state
  $cmd 00400 allow udp from any to me 80 via $pif keep-state
  $cmd 00400 allow udp from any to me 443 via $pif keep-state
  
  # Allow Serving HTTP(S) 1, 2, and 3 (QUIC)
  $cmd 00400 allow tcp from me to any 80 via $pif keep-state
  $cmd 00400 allow tcp from me to any 443 via $pif keep-state
  $cmd 00400 allow udp from me to any 80 via $pif keep-state
  $cmd 00400 allow udp from me to any 443 via $pif keep-state
  
  $cmd 65535 deny all from any to any in via $pif
  Admittedly, this is a very, very basic firewall script that will
  probably not cover many people’s use-case. For one, SSH is allowed from
  any address, which makes some people nervous. If this is the case, ‘any’
  can be replaced with a specific IP or IP range. Additionally, there is
  no logging enabled. That can be enabled by adding log after
  the allow or deny keyword. For example:
  #!/bin/sh
  ipfw -q -f flush
  cmd="ipfw -q add"
  pif="vtnet0"
  
  # Allow SSH from one of Google's public IPs
  $cmd 00100 allow tcp from 173.194.219.102 to me 22 in via $pif limit src-addr 2
  
  # Allow SSH from any IP in 192.168.122.0/24
  $cmd 00100 allow tcp from 192.168.122.0/24 to me 22 in via $pif limit src-addr 2
  
  # Allow bi-directional pings
  $cmd 00200 allow icmp from any to any via $pif keep-state
  
  # Allow DNS via UDP and TCP
  $cmd 00300 allow udp from me to any 53 via $pif keep-state
  $cmd 00350 allow tcp from me to any 53 via $pif keep-state
  
  # Allow Receiving HTTP(S) 1, 2, and 3 (QUIC)
  $cmd 00400 allow tcp from any to me 80 via $pif keep-state
  $cmd 00400 allow tcp from any to me 443 via $pif keep-state
  $cmd 00400 allow udp from any to me 80 via $pif keep-state
  $cmd 00400 allow udp from any to me 443 via $pif keep-state
  
  # Allow Serving HTTP(S) 1, 2, and 3 (QUIC)
  $cmd 00400 allow tcp from me to any 80 via $pif keep-state
  $cmd 00400 allow tcp from me to any 443 via $pif keep-state
  $cmd 00400 allow udp from me to any 80 via $pif keep-state
  $cmd 00400 allow udp from me to any 443 via $pif keep-state
  
  $cmd 65535 deny log all from any to any in via $pif
  There are a few other notes about these firewall rules. I do
  recommend accepting incoming web traffic, because blocking it will
  prevent pkg from operating properly. If you were
  particularly paranoid you could remove the UDP rules for web traffic but
  that would prevent QUIC from working as it is a UDP protocol.
  Additionally, DNS is normally a UDP protocol, but will occasionally use
  TCP. Blocking TCP traffic on port 53 would probably be fine,
  but for most people I do not see a problem with allowing it through.
  Additionally, you could block it for only certain IP addresses
  (i.e. 9.9.9.9) and that would reduce the attack surface.
  From here, we simply enable our firewall:
  sysrc firewall_enable=1
  # or
  echo firewall_enable=1 >> /etc/rc.conf
  and start the service (or you can reboot the machine if that’s
  somehow easier):
  service ipfw start
  You now have a working firewall, you can test that things are
  actually being blocked by using tools such as telnet or
  nc to try to connect to things using ports that should be
  blocked. For example, you should not be able to connect to a mail server
  with the above configuration.
  Dummynet
  Dummynet is a traffic shaping and network testing tool that is
  closely tied into IPFW. With Dummynet, it is possible to add latency,
  bandwidth limits, change scheduling algorithms, introduce packet loss,
  set queue size, and overall introduce limitations or other chaos into
  the network. Dummynet generally operates on either pipes or queues (or
  both). In general, pipes are for setting hard limitations, whereas
  queues can determine how flows will share available bandwidth.
  Dummynet has a lot of stuff it can do, but in practice it is most
  often used for testing networks, so that is going to be what we will do
  in this post. For more information on the specifics or advanced usage of
  the tool, I might recommend the dnctl man
  page or the IPFW man page.
  For our example network, I wanted to emulate a connection to Mars,
  however, Dummynet will not allow for a delay over 10,000. So, instead I
  set the delay for 9,000 and a bandwidth of 2Mbp/s. We can create that
  with the following dummynet rules:
  # Must run as root
  
  # This command sets up the pipes for dummynet to set limits on
  ipfw add 01000 pipe 1 ip from 192.168.122.65 to 192.168.122.1
  ipfw add 01001 pipe 2 ip from 192.168.122.1 to 192.168.122.65
  
  # Actually setting limitations
  ipfw pipe 1 config delay 9000 bw 2Mbp/s
  ipfw pipe 2 config delay 9000 bw 2Mbp/s
  Now we can use tools like iperf3, ping, or sending files via scp to
  verify bandwidth and latency. Though, at those specs, most of these
  tools are fairly painful to use. Playing with the delays, bandwidth and
  queuing makes for an interesting experiment as well as testing things
  like websites in low resource environments. For example, seeing how your
  website loads on a satellite connection, or seeing how terrible SSH is
  with a 9,000ms delay.
  More Resources
  This is the part of the blog post where I admit that I am standing on
  the shoulders of giants for learning about these tools, and if this blog
  post does not cover a particular topic, or you are having trouble
  understanding it please look into the following sources (or check the
  man pages).
  
  Dummynet:
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-04-14.html
  		https://foxide.xyz/projects/2025-04-14.html
  		Sun, 14 Apr 2024 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  Over the weekend, rather than working on the blog post topic I
  originally had in mind, I spent some time trying to figure out how to
  benchmark and optimize my website. Because of that, I thought I might do
  a small write-up for anyone that is interested and does not already know
  a better way to do this. I also want to stress, I am by no means a web
  dev nor do I have in-depth knowledge of web site or web server
  optimization; these are simply things I found that seemed helpful and
  improved my “score” on the benchmarks.
  Benchmarking Performance
  It is always good practice to do benchmarks before
  making adjustments to anything. Without the baseline, it is difficult to
  know what has improved if anything. So, we are going to take a
  preliminary benchmark using bombardier and PageSpeed Insights. My starting
  point is having HTTP/2 and HTTP/3 setup on my nginx web server running
  on a FreeBSD 14.2 host. The nginx config is largely default except what
  has to be changed for my sites, and obviously for enabling HTTP/2 and
  HTTP/3. If you want to know how to do that, this post will
  walk you through the process.
  Let’s start with the benchmark from bombardier:
  bombadier -d 30s -c 50 -l https://example.com
  This command will run for 30 seconds with 50 connections testing the
  site example.com (please don’t run this on sites you don’t have
  permission for). The -l flag will show latency statistics
  for the connections. The output will look something like:
  bombadier -d 30s -c 50 -l https://foxide.xyz
  Bombarding https://foxide.xyz for 30s using 50 connection(s)
  Done!
  Statistics        Avg      Stdev        Max
    Reqs/sec       775.56     510.93    2466.41
    Latency       64.39ms    16.08ms   581.82ms
    Latency Distribution
       50%    63.07ms
       75%    66.13ms
       90%    68.88ms
       95%    71.91ms
       99%    81.14ms
    HTTP codes:
      1xx - 0, 2xx - 23317, 3xx - 0, 4xx - 0, 5xx - 0
      others - 0
    Throughput:     1.51MB/s
  The above statistics show information on requests per second,
  latency, as well as the HTTP codes that were received on the requests.
  From the graph above, we can see my site was able to process ~775
  requests per second on average with ~2466 being the peak requests per
  second. For the purposes of this bench mark, we want this number to be
  as big as possible (with only receiving 2xx HTTP codes) meaning we were
  able to process more traffic. Next, we see the latency statistics with
  the average being ~64.39 milliseconds and the max being ~581.82
  milliseconds; unlike the requests per second, this number should be as
  low as possible.
  In addition to that command-line tool, Firefox as well as Chromium
  have some builtin web development tools that show performance metrics
  among other things. This post is going to use Firefox as it is my
  preferred browser, however, much of the information should translate
  decently well to Chromium based browsers. To open the web tools in
  Firefox, click on the three lines in the toolbar on the right -> More
  tools -> Web Developer Tools, alternatively CTRL+SHift+I can be
  pressed to open the same tools. On the menu that pops up, click
  “Network” and reload the page.
  
  
  
  
  This will show each of the files that are loaded with the web page.
  As you see in this example, my site only loads the html page and the
  stylesheet. Clicking on the file then on the “Timings” tab that shows up
  will show a basic flame graph for the file on the page. For the most
  accurate results, make sure to disable caching.
  
  
  
  
  As you can see it took around 57 milliseconds for the page to load
  without throttling; however, this is an ideal circumstance. We should
  also test load times with throttling to check and see how the site
  performs under different circumstances, bonus points if we test
  different mobile device profiles. To check how our site performs on a
  worse network connection, simply click the throttle button and select
  the level of throttling desired. I am not going to delve too deeply into
  the throttling because while it is good to use for testing, it is a
  fairly straight forward concept and does not really change anything for
  a guide. The only real thing worth noting is that it will point out
  inefficiencies much better. So, if you are having trouble finding the
  inefficiencies, try throttling the loading down really hard to make it
  more obvious.
  The last benchmarking tool that I am going to use is PageSpeed Insights, which is a
  tool made by Google. Normally, I wouldn’t recommend Google-made tools,
  however, if your site is being made public than Google will find it
  eventually. They are also industry leaders in the web field, so their
  tools are probably at least halfway decent for the job, so I am going to
  use it for this post. If you do not want to use it because of its
  author, that is reasonable and skipping this section is appropriate.
  Using the tool is fairly simple, just put the URL of the site in and
  wait for the results. The results have two variations, one for mobile
  and one for desktop with the desktop generally yielding better
  performance (in my experience). Here are the my results for Foxide.xyz:
  
  
  
  
  
  
  
  
  The main performance results that we are concerned about (at least
  for the purposes of this post) are the “Performance” results, and the
  “Best Practices” results (because best practices are always important).
  Since my site is a fairly basic static site, it scores well just because
  it doesn’t have much to load. Scrolling down will also give metrics and
  diagnostics for what might be causing the site to load at the speed it
  is, as well as giving recommendations for fixing it and making it
  perform better.
  Optimizing Delivery
  Now that we have the current stats for our site, we can start making
  tweaks to boost the performance. It is worth pointing out that the type
  of content being served on the server will impact the specifics of how
  to optimize performance. The process for increasing performance on an
  instance of Wordpress is going to be quite a bit different than
  optimizing a static site, additionally it is going to be much easier to
  make a static site load quickly than something like Wordpress. So,
  before working too much on the Nginx optimizations, try to optimize the
  application itself as much as possible. Once that is done, then come
  back and work on improving how Nginx will serve the page.
  The process for implementing any sort of performance improvement is
  to implement one setting at a time and re-test to see if it has the
  impact that you expected. The biggest reason for this, is if you
  implement 100 changes and the performance is worse, it is really
  difficult to tell which setting(s) are causing the degradation. It is
  also worth noting that performance tests can yield different results
  between runs, even without changing anything. So before making heavy
  modifications (especially ones that you do not understand well) run the
  benchmarks and get an average of what is expected, then start changing
  stuff (after also taking a backup of the original configs) and run them
  again.
  After backing up the original Nginx config and noting down the
  average performance benchmarks, we are finally ready to attempt to
  improve performance. For this, I am going to write a pseudo nginx config
  with options that can help performance, as well as some comments that
  help explain what the line does, and some options for how it is
  configured and why I have chosen the value I did. Keep in mind, that
  every workload is going to vary slightly, so while the options I have
  here might work for me, they should be tested and tweaked to your
  workload.
  # The optimal value for the worker_processes option varies depending on a few different things
  # The main one that most people will reference is CPU cores, but there are other things that
  # can affect the optimal value such as hard disks. For most people, setting this option to
  # 'auto' will suffice as nginx will attempt to detect a "best" value on its own.
  worker_processes auto;
  # This option reduces the amount of information that the nginx error log stores
  # Reducing the information stored increases performance by reducing the amount
  # of writes to the disk. This does also decrease visibility into issues though,
  # so be mindful of what your concerns are before changing this option this way.
  error_log /var/log/nginx/error.log crit;
  
  events {
  # The worker connections define how many simultaneous connections can be opened by a worker process.
  # This number includes all connections such as connections to proxy servers as well as other connections.
      worker_connections 4096;
  # This configuration item is the connection processing method.
  # Because the host I am running on is FreeBSD based, kqueue is the more efficient option,
  # however, for Linux it would be epoll. For more information on this option
  # please read the docs located here - https://nginx.org/en/docs/events.html
      use kqueue;
  }
  
  http {
      include mime.types;
      default_type application/octet-stream;
  
      # File caching that will keep the most common files
      # readily available. It can increase performance
      # but will likely need tuning
      open_file_cache max=200000 inactive=20s;
          open_file_cache_valid 30s;
          open_file_cache_min_uses 2;
          on;
  
      # Turning access logs off decreases disk writes, boosting performance
      access_logs off;
  
      # Sendfile is faster than read() and write()
      sendfile on;
  
      # Send headers all at once
      tcp_nopush on;
  
      # Enables unbuffered proxying on TLS connections
      tcp_nodelay on;
  
      # Enabling gzip to compress the amount of data being sent to remote clients
      # as well as various config options surrounding gzip. More info
      # https://nginx.org/en/docs/http/ngx_http_gzip_module.html
      gzip on;
      gzip_min_length 10240;
      gzip_comp_level 2;
      gzip_disabled msie6;
      gzip_proxied pired no-cache no-store private auth;
          gzip_types
      text/css
      text/javascript
      text/plain
      text/xml;
  
      # Sets buffer size for client request bodies.
      # In the event that the request body is larger
      # than the buffer, it is written to a temp file
      # that temp which is two memory pages by default.
      # On 32-bit machines it would be 8K and 64-bit machines
      # it would be 16K
      client_body_buffer_size 16k;
      # This sets the size for the client request header
      # 1K is generally enough here
      client_header_buffer_size 1k;
      # Sets max client body before giving error
      # It is worth noting that most browsers
      # cannot appropriately communicate this error
      client_max_body_size 8m;
      # Sets the max number and size of buffers for reading
      # large client request headers
      large_client_header_buffers 2 1k;
  }
  
  server {
      .
      .
      .
      # This can be added to an existing server block
      # so imagine this section is configured with a site
  
      # This configures client side caching for some file types:
      location /*.(jpg|jpeg|png|gif|ico|css)$ {
          expires 365d;
      }
  }
  Results?
  Enabling the above configuration options, or options similar,
  provided some slight performance improvement to my static site. However,
  I also modified these options for my CryptPad instance; it now loads
  noticeably faster. I would estimate around 15% or better loading speeds
  thanks to these optimizations. Running PageSpeed Insights against my
  CryptPad instance previously got a score in the mid 50s on the
  performance metric, but after enabling the stated options, I am getting
  between 65 and 75 in the performance score on mobile and a 96 on
  Desktop. Using CryptPad on my phone also feels noticeably faster and
  more snappy when interacting with it. Again, as stated in a previous
  paragraph, these options are not something that is going to work the
  same for everyone, but you can modify them to fit your use case and
  significantly boost the responsiveness of your webpage(s).
  Resources
  
  Nginx Tuning:
  Denji
  Nginx Documentation
  
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-05-04.html
  		https://foxide.xyz/projects/2025-05-04.html
  		Sun, 04 May 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  The Domain Name System (DNS) serves as one of the foundational
  protocols and services that allow the Internet to operate as it does
  today. DNS serves as the translation between conventional domain names
  (such as foxide.xyz) and the corresponding IP address (172.245.181.191).
  This has the obvious effect of making servers on the Internet easier to
  find as names tend to be easier to remember than random numbers;
  however, it has the secondary effect of allowing for things like virtual hosting
  which contributed to the boom of websites on the early Internet. To
  think of the modern Internet without DNS is nearly unthinkable, to the
  point where many untrained people would confuse DNS not working with the
  Internet not working. This post is going to teach about some basic DNS
  concepts, troubleshooting, and setting up a DNS server with OpenBSD and Unboound.
  How DNS Works
  As stated above DNS is a foundational protocol that runs the modern
  day Internet, and the services that provide name resolution for the
  Internet have a huge amount of power over what their down stream users
  can and cannot see. Let’s take a look at how DNS works to better
  understand why this is.
  The life of a DNS query is conceptually quite simple; it is just
  asking that query to a series of services until an answer is
  received.
  For a quick oversimplification of the process:
  
  Local machine attempts to resolve by looking at the hosts file and
  any cache on the machine.
  The machine will ask the nearest DNS server.
  The nearest DNS server will send the query upstream.
  Repeat until answer is received.
  
  In practice it operates as follows: First some software asks the
  operating system to resolve a name to an IP address (e.g. foxide.xyz).
  The operating system first checks the ‘hosts’ file
  (/etc/hosts on Unix systems and
  C:\Windows\System32\drivers\etc\hosts on Windows) to see if
  the system’s administrator has declared IP address that should
  correspond to a particular domain name. If the name is declared in the
  hosts file, then the query will use the corresponding IP address as the
  destination for the domain. An example of the hosts file might be:
  127.0.0.1       localhost
  ::1         localhost
  0.0.0.0         foxide.xyz
  In the example above, we can see that localhost (more often referred
  to as the loopback address) was defined as 127.0.0.1 and
  ::1. This corresponds to the reserved loopback addresses
  for IPv4 and IPv6 respectively; reserved in this case means that nothing
  on the Internet will have this address, so it can safely be assigned for
  another purpose. In this case, it is to test that networking is working
  on the local machine in some capacity.
  If the query cannot be resolved via the hosts file, then the
  operating system and/or the web browser will check the cache to see if
  it is able to resolve the query. It is very important to remember that
  modern web browsers are very complex and retain their own DNS cache. The
  purpose behind this is increasing performance when cache is needed,
  however, it can also lead to odd behavior such as a DNS record has
  changed, but the browser is only resolving to the previous address even
  though the operating system resolves to the new address.
  Once the local machine has tried and failed to resolve the query, the
  second step in DNS resolution occurs. The query will then go and ask the
  nearest DNS server to the machine; this will often be the DNS server set
  by the DHCP
  server on the network, but could also be set manually. In many home
  networks, the DHCP server will be the ISPs router that will
  automatically set their preferred public DNS server for each client. In
  most enterprise and home lab settings though, there will be a local DNS
  server to resolve local domain names, then send queries upstream for
  external services. The local services will usually have a Top Level
  Domain (TLD) corresponding to something such as ‘.home’ or ‘.local’. It
  is worth noting that as of 2018 RFC 8375 declared
  ‘.home.arpa’ to be the de facto TLD intended for local network use with
  a unicast DNS server, and RFC 6762 declared
  ‘.local’ to be the de facto TLD for multicast DNS (mDNS) use. An example
  of this use would be something like Bonjour
  50p). For more information, a list of “special use” domain names can be
  found here.
  If the local DNS server does not know where the address is (either by
  deceleration or by caching), then it will ask the upstream DNS server.
  This will generally be a public DNS server and will be configured by the
  network administrator, many people use Google’s DNS (8.8.8.8), however,
  many other options exist:
  
  Cloudflare: 1.1.1.1
  OpenDNS: 208.67.222.222
  Quad9: 9.9.9.9
  
  These DNS services are what provides the Internet experience that
  most users are used to, and without it the Internet might as well be
  gone completely.
  By this point, the query has most likely been answered, as the public
  DNS servers will not only have a record of many different domain
  locations, but they will also keep a cache of commonly used domain name
  locations. However, there are cases in which a domain may not be known
  to these public DNS servers. So, what happens from here?
  Well, basically the same thing that has happened before. The public
  DNS server asks another server. However, this time, it will work in a
  bit of a different direction. It will start by trying to find the DNS
  server that handles the TLD of the domain (for example, the TLD for
  foxide.xyz would be .xyz) to ask where the domain should point to. If
  the DNS server cannot find the server that handles the TLD, it will then
  reach out to one of the 13 Root Servers. These
  are the authoritative DNS servers that ultimately answer for all of the
  TLDs on the Internet; because of their special place in DNS, their TLD
  is technically just a period. It can be thought of an implied trailing
  period on each DNS query that signifies the root server. The root server
  will then tell the DNS server making the query (remember the upstream
  public DNS server for our machine) where to find the authoritative name
  server for the TLD it is looking for. From there, the name server for
  the TLD will give up the IP address that corresponds with the query and
  that will get sent back up to the client that originally requested it.
  The amazing part of all of this, is it happens over the course of
  milliseconds all across the world.
  The next part of understanding DNS is understanding the various
  record types that DNS has to offer!
  DNS Record Types
  While a DNS query may ask where a host’s location is, the record type
  will tell the system what the location is intended for. It can almost be
  though of like the zoning for a building; where the IP address would be
  the building’s address and the record type would be the building’s
  zoning. A brief overview of the most common record types are as
  follows:
  
  A: This record will return an IPv4 address (32-bit), most commonly
  used for mapping hostnames to IP addresses.
  AAAA: This record will return an IPv6 address (128-bit), used in a
  similar manner to IPv4 addresses.
  CNAME: Also called a “canonical name”. This is used to map one name
  to another, most commonly used for things like mapping www.example.com
  to example.com.
  MX: This record declares the list of mail servers that can accept
  mail on behalf of a domain.
  NS: Declares the authoritative name server for a domain.
  PTR: This is often used for reverse DNS lookups.
  TXT: This is a record that was originally intended for arbitrary
  data for humans, but in practice has become a catch all for other DNS
  protocols such as SPF, DKIM, and DMARC.
  
  The above list is far from comprehensive, but should give a general
  idea on the most basic and common record types. For a more complete list
  Wikipedia
  has you covered with lots of information on record types. And if
  that is not enough, you can also look at the Internet Assigned Number
  Authority (IANA)’s website.
  Now that we have a basic understanding of DNS and roughly how it
  works, let’s get a server setup so we can resolve our own queries.
  Lab Setup
  For this project, I am going to be using OpenBSD because it is
  secure, stable, and simple. I am using a virtual machine for testing
  this, however, real hardware should not be much different; the VM is
  using UEFI to emulate a machine made within the past decade though
  OpenBSD is only mildly concerned with that fact.
  Following the installer, and selecting the defaults except for using
  a GPT partition scheme and actually creating a user. Once the system is
  installed and rebooted, I enabled doas using the following
  command:
  # Must be run as root, using su -
  echo "permit persist :wheel" > /etc/doas.conf
  and then installing some optional quality of life packages:
  # Must be run as root, but doas will work now
  pkg_add fish-4.0.2p0 neovim screen-5.0.0
  # Sym Link nvim to vim for muscle memory
  ln -s /usr/local/bin/nvim /usr/local/bin/vim
  # Change shell from ksh to fish
  # This command will be run as your unprivileged user
  chsh -s /usr/local/bin/fish
  These packages are quality of life packages that are completely
  optional and will not really affect anything in this blog post other
  than the above section. The focus of this post will be on the
  unbound DNS server included in OpenBSD base. Below is the
  default configuration file for OpenBSD’s unbound, located at
  /var/unbound/etc/unbound.conf:
  # $OpenBSD: unbound.conf,v 1.21 2020/10/28 11:35:58 sthen Exp $
  
  server:
      interface: 127.0.0.1
      #interface: 127.0.0.1@5353  # listen on alternative port
      interface: ::1
      #do-ip6: no
  
      # override the default "any" address to send queries; if multiple
      # addresses are available, they are used randomly to counter spoofing
      #outgoing-interface: 192.0.2.1
      #outgoing-interface: 2001:db8::53
  
      access-control: 0.0.0.0/0 refuse
      access-control: 127.0.0.0/8 allow
      access-control: ::0/0 refuse
      access-control: ::1 allow
  
      hide-identity: yes
      hide-version: yes
  
      # Perform DNSSEC validation.
      #
      auto-trust-anchor-file: "/var/unbound/db/root.key"
      val-log-level: 2
  
      # Synthesize NXDOMAINs from DNSSEC NSEC chains.
      # https://tools.ietf.org/html/rfc8198
      #
      aggressive-nsec: yes
  
      # Serve zones authoritatively from Unbound to resolver clients.
      # Not for external service.
      #
      #local-zone: "local." static
      #local-data: "mycomputer.local. IN A 192.0.2.51"
      #local-zone: "2.0.192.in-addr.arpa." static
      #local-data-ptr: "192.0.2.51 mycomputer.local"
  
      # Use TCP for "forward-zone" requests. Useful if you are making
      # DNS requests over an SSH port forwarding.
      #
      #tcp-upstream: yes
  
      # CA Certificates used for forward-tls-upstream (RFC7858) hostname
      # verification.  Since it's outside the chroot it is only loaded at
      # startup and thus cannot be changed via a reload.
      #tls-cert-bundle: "/etc/ssl/cert.pem"
  
  remote-control:
      control-enable: yes
      control-interface: /var/run/unbound.sock
  
  # Use an upstream forwarder (recursive resolver) for some or all zones.
  #
  #forward-zone:
  #   name: "."               # use for ALL queries
  #   forward-addr: 192.0.2.53        # example address only
  #   forward-first: yes          # try direct if forwarder fails
  
  # Use an upstream DNS-over-TLS forwarder and do not fall back to cleartext
  # if that fails.
  #forward-zone:
  #   name: "."
  #   forward-tls-upstream: yes       # use DNS-over-TLS forwarder
  #   forward-first: no           # do NOT send direct
  #   # the hostname after "#" is not a comment, it is used for TLS checks:
  #   forward-addr: 192.0.2.53@853#resolver.hostname.example
  The nice thing about the default configuration file is how well
  commented it is; while you may not understand the lines completely, you
  can understand the basics of what they are doing. Then the
  man pages are also always available as a reference for the
  specifics of a configuration option. For a more practical example, here
  is the config that I have setup for this lab with the comments removed
  for readability:
  server:
      interface: 192.168.122.40
  
      outgoing-interface: 192.168.122.40
  
      access-control: 0.0.0.0/0 refuse
      access-control: 127.0.0.0/8 allow
      access-control: 192.168.122.0/24 allow
      access-control: ::0/0 refuse
      access-control: ::1 allow
  
      hide-identity: yes
      hide-version: yes
  
      auto-trust-anchor-file: "/var/unbound/db/root.key"
      val-log-level: 2
  
      aggressive-nsec: yes
  
      local-zone: "home.arpa." static
      local-zone: "122.168.192.in-addr.arpa." static
  
      local-data: "puffy.home.arpa. IN A 192.168.122.40"
      local-data: "fedora.home.arpa. IN A 192.168.122.126"
  
      local-data-ptr: "192.168.122.40 puffy.home.arpa"
      local-data-ptr: "192.168.122.126 fedora.home.arpa"
  
  remote-control:
      control-enable: yes
      control-interface: /var/run/unbound.sock
  
  forward-zone:
      name: "."               # Use for all queries
      forward-addr: 9.9.9.9           # For upstream queries
      forward-first: yes          # try direct if forwarder fails
  I am not going to re-hash each line of the above config as the
  default config already does a pretty good job of explaining what
  everything is. Rather, I am going to just explain what the config file
  is setting up in general, then I will leave the cross referencing to the
  end user as a learning exercise.
  The above config sets the IP address 192.168.122.40 as the incoming
  (listening) address as well as the address is sends queries back out on.
  It also allows traffic from the following networks:
  
  127.0.0.0/8
  192.168.122.0/24
  ::1 (IPv6 localhost)
  
  Then will deny incoming queries from all other networks.
  It will also create a local TLD (home.arpa) for local services to use
  on the 192.168.122.0/24 network. Currently, there are two A records in
  the config with reverse DNS records:
  
  puffy.home.arpa: 192.168.122.40
  fedora.home.arpa: 192.168.122.126
  
  Finally, the last major part is setting the forwarding zone. This
  will control what happens to queries that the DNS server cannot resolve
  by itself and needs to ask for upstream help. It will set quad 9
  (9.9.9.9) as the upstream DNS server.
  There is quite a bit more that unbound can do and this is an
  extremely basic config, however, most people do not need much more than
  A records for local use. If you do, there is always the OpenBSD unbound.conf man
  page as a shining resource, as well as many other online
  resources.
  Troubleshooting DNS
  Because of how integral DNS is to the use of the Internet for the
  vast majority of people, DNS troubleshooting is often taught at the most
  fundamental levels of IT training. Many of the tools and techniques that
  would be used to troubleshoot DNS issues are tools and techniques that
  many people that have done troubleshooting for network problems have
  already done. In the event that someone is reading this and is not
  already familiar with these tools and techniques (did you find the rabbit?) I will go
  over the basics in this section. I will also try to cover some slightly
  more advanced stuff for the people already in the know though. The
  easiest way to explain is going to be using some common scenarios of
  problems and how one might go about dealing with them, so let’s do
  that.
  Before actually getting into the exercises, it is worth noting that I
  am going to actually focus mostly on Windows tools for troubleshooting
  these issues. The reason for that is, practically speaking, in the vast
  majority of cases in the operations side of IT, you will be dealing with
  Windows. When a user calls because a computer isn’t doing the thing, it
  is most likely going to be a Windows machine. So, to work with the tools
  that are most likely to be seen in the field, I am choosing Windows for
  the troubleshooting parts of the exercise.
  Scenario 1: “Can’t get to
  the Internet”
  The first scenario is a common occurrence in the field, as well as a
  common interview question. A user calls and “cannot get to the
  Internet”. When this call comes in, you have no information on what the
  problem actually is. If you don’t believe me, then you
  should read through r/talesfromtechsupport
  and question who you should believe. The correct answer is no one, no
  even yourself; the best you can do is gather information and go through
  the common things.
  
  What does “can’t get to the Internet” actually mean? I have had
  users say something similar, but the issue was one website and was not
  the Internet as a whole. Make sure you understand what is broken and
  what the problem actually is before doing anything. Troubleshooting
  something the user didn’t care about is not only not helpful to anyone,
  but it is wasting the most valuable resource in existence…time.
  Assuming that the user actually cannot get to the Internet, we can
  now start doing actual troubleshooting; start with the basics. Is there
  a network cable plugged into the machine, or are they connected to a
  wireless network. If neither one of these is true, it will be fairly
  difficult to connect to the Internet.
  If a network connection should at least theoretically exist, then it
  is time to start actually asking the computer what might be wrong. Open
  a command prompt (does not have to be administrative command prompt) and
  start running some commands:
  
  :: Remember, this is Windows command prompt, where '::' is a comment
  
  :: First, ping an easy to remember public IP address. Something like 1.1.1.1, 8.8.8.8, or 9.9.9.9 work fine.
  ping 1.1.1.1
  Pinging 1.1.1.1 with 32 bytes of data:
  Reply from 1.1.1.1: bytes=32 time=18ms TTL=56
  Reply from 1.1.1.1: bytes=32 time=17ms TTL=56
  Reply from 1.1.1.1: bytes=32 time=15ms TTL=56
  Reply from 1.1.1.1: bytes=32 time=20ms TTL=56
  
  Ping statistics for 1.1.1.1:
      Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
  Approximate round trip times in milli-seconds:
      Minimum = 15ms, Maximum = 20ms, Average = 17ms
  
  :: Sending a ping to a domain name
  ping example.com
  Ping request could not find host example.com. Please check the name and try again.
  From the above commands, it appears that the “Internet” is working
  just fine, but something appears to be wrong with resolving domain
  names. The next step would be using a tool like nslookup to
  see what DNS is doing with that query:
  nslookup example.com
  DNS request timed out.
      timeout was 2 seconds.
  Server:  UnKnown
  Address:  192.168.122.40
  
  DNS request timed out.
      timeout was 2 seconds.
  DNS request timed out.
      timeout was 2 seconds.
  DNS request timed out.
      timeout was 2 seconds.
  DNS request timed out.
      timeout was 2 seconds.
  From the above command we can see that it appears that the DNS server
  was unreachable, this can be confirmed with a simple ping to the DNS
  server’s IP.
  ping 192.168.122.40
  Pinging 192.168.122.40 with 32 bytes of data:
  Reply from 192.168.122.220: Destination host unreachable.
  Reply from 192.168.122.220: Destination host unreachable.
  Reply from 192.168.122.220: Destination host unreachable.
  Reply from 192.168.122.220: Destination host unreachable.
  
  Ping statistics for 192.168.122.40:
      Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
  Not only does that confirm that DNS is the culprit of the issue, but
  also gives us something else to look for. Why can this client not reach
  the DNS server? In this case it is because I purposely turned the DNS
  server off for the lab, but in the real world it will likely take a bit
  more investigation.
  Scenario 2: Unable
  to resolve local services
  Another fairly common problem is end users are not able to get to
  local services. These service might be something like an Intranet web
  app that the end user cannot access with the domain name that is pointed
  to it. While this issue sounds more complicated, it is actually much
  easier in ways. Step one is still the same, confirm what the problem is
  specifically. Step two, is just make sure the network is working
  elsewhere; can the user get to an external site of some sort
  (e.g. example.com or foxide.xyz). Assuming yes, then it is probably
  something happening locally. Making another assumption that this service
  is mission critical to a large group of people in the company, it is
  probably running otherwise you would have a lot more calls about the
  service. The first step would be to just send a ping to the
  service’s name to confirm that it does in fact fail to connect. We could
  also send a ping to the service’s IP address if it is
  something that we have memorized to confirm that the client machine can
  in fact get to the service. If the client can successfully ping the
  service’s IP, than it is a DNS issue, which means that the
  nslookup command can tell us what might be going on. This
  scenario would happen if the client’s DNS server somehow changed from
  the ‘local’ one that can resolve the local service to a public DNS
  server such as 8.8.8.8. That would allow external sites and services to
  work without a problem, but anything local would not work at all.
  Closing Thoughts
  DNS is a highly important part of the Internet, and as such anyone
  working in the IT field should have a grasp on the basics of DNS and how
  it works, bonus points if you can setup a DNS server. The more an
  individual understands DNS, the more they can break in to the wonderful
  world of doing things on the Internet, and hopefully even doing them
  correctly. Hosting services and having others be able to see the thing
  that you are doing is nearly impossible without some foundational
  knowledge of how to set up the address for the name of the thing. In
  general, it can be thought of as trying to send mail to somewhere
  without a street address; it’s not impossible, but you probably don’t
  want to do it that way.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-06-01.html
  		https://foxide.xyz/projects/2025-06-01.html
  		Sun, 01 Jun 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  The terminal in Linux/BSD/Unix is often seen as an extremely powerful
  and fundamental tool of the system, and with good reason. However, there
  are many cases in which having a plethora of terminal windows opened is
  all but required to accomplish a task; that is where tools such as tmux and GNU screen come in.
  These tools allow for managing terminal windows just like a window
  manager would manage graphical windows. Most people that have been
  using modern Unix-like systems are familiar with tmux; however, screen
  is an older utility that many people often don’t reach for to solve the
  same problems and more.
  What is a
  Terminal Multiplexer and Why is it Useful
  Let’s start at the premise of the tool itself, what does a terminal
  multiplexer do? From Wikipedia’s opening paragraph on the topic:
  
  A terminal mutliplexer is a software application that can be used to
  multiplex
  several separate pseudoterminal-based login sessions inside a single
  terminal display, terminal emulator window, PC/workstation system
  console, or remote login session, or to detach and reattach sessions
  from a terminal. It is useful for dealing with multiple programs from a
  command line interface, and for separating programs from the session of
  th eUnix shell taht started the program, particularly so a remote
  process continues running even when the user is disconnected.
  
  In short, it a piece of software that allows for turning one terminal
  session into multiple, as well as allowing a terminal based application
  to run in a background session without requiring a terminal to be left
  opened. This is something that would be really useful in a time where
  computers were not very powerful and opening another window could be a
  problem, however, it’s 2025 and computers are massively overpowered
  compared to what they were twenty years ago. Why would anyone use a
  terminal multiplexer these days? The main benefit I have from it is the
  organization options that being able to group multiple terminal windows
  into one terminal session provides. If I am working on some writing, I
  can keep all the things related to that writing (SSH sessions, w3m
  sessions, and the text editor) opened all in one terminal window, along
  with being able to name each window to make it easy to figure out where
  I am.
  The other main use would be to keep a semi interactive terminal
  command running in the background; an example of this might be a
  Minecraft server. The server software itself is not one that daemon-izes
  well (at least from what I’ve been told) and it would really be better
  to keep it running properly in the background. Terminal multiplexers
  allow sending a process to the background then easily recalling that
  process in a more intuitive way than something like standard job control.
  Basic Key Bindings
  Before getting into how to actually use GNU screen, I am going to
  start off with a small guide for hotkeys. This will hopefully help
  reduce some of the confusion on how to actually make things happen after
  invoking the screen application.
  The activation hotkey for GNU screen is ctrl+a, or as it is written
  in the documentation, C-a. This hotkey will then prompt screen to listen
  for other keys to do various things depending on which key is pressed.
  For example, the hotkey ‘c’ (so pressing C-a then c) will create another
  window within screen, and the hotkey ’“’ (so C-a then”) would show a
  list of all the windows that are opened in screen at the moment as well
  as letting the user switch to a particular window. There is quite a list
  of keybindings that screen will recognize, however, I will also give a
  small list of some of the ones that I use most often to get people
  started with something a bit less overwhelming than what the man pages
  provides.
  
  C-a “: Present a list of all windows for selection.
  C-a 0…9: Switch to window number 0…9, or the blank window (window
  number corresponds to the number pressed on keyboard).
  C-a Tab: Switch the input focus to the next region.
  C-a C-a: Toggle to the window displayed previously. If this windows
  does no longer exist, other has the same effect as next.
  C-a A: Allow user to enter a title for the current window.
  C-a c: Creates a new window with a shell and switch to that
  window.
  C-a d: Detach screen from this terminal.
  C-a k: Destroy the current window.
  C-a K: Destroy the current window without confirmation.
  C-a S: Split the current region horizontally into two new ones.
  C-a ?: Show key bindings.
  C-a : Kill all windows and terminate screen.
  C-a [: Enter copy/scrollback mode. The scrolling of this mode uses
  vi like bindings for movement, and to copy text, press the ‘enter’ key
  to select where to start the copying text, then move to the end of the
  selection that needs to be copied, and press ‘enter’ again.
  C-a ]: Write the contents of the paste buffer to the stdin queue of
  the current window. To paste the previously copied text, simply use the
  binding, nothing special here.
  C-a |: Split the current region vertically into two new ones.
  
  In large part, I just re-used the verbiage from the man page for
  those commonly used commands, because I found it rather self-explanatory
  with only a few items needing some clarification (at least in my
  opinion).
  Using Screen
  Actually using GNU screen is not terribly difficult especially with
  the assistance of a cheat sheet for hotkeys (whether that be the one
  from above or the built in one). To start the application, simply type
  screen then press enter in a terminal window, and a shell
  will launch in a screen window. From there, any of the hotkeys can be
  invoked to do their corresponding action.
  After some use, the user might want to close the terminal window, but
  keep the session itself alive; In other words, detach from the session.
  That can be done simply by using the ‘C-a d’ hotkey, but then what?
  Screen appears to be closed, but how would one go about re-attaching to
  that session? If only one screen session is active in the background,
  then running screen -d -r will re-attach to the opened
  session. However, if multiple screen sessions are opened, then screen
  will complain with the following:
  There are several suitable screens on:
          6648.pts-1.hostname     (Detached)
          16954.pts-1.hostname    (Detached)
  Type "screen [-d] -r [pid.]tty.host" to resume one of them.
  The two main ways to overcome this are to force screen to choose,
  which will re-attach to the most recently opened session. I want to
  stress here, that most recently opened does not necessarily mean the
  same thing as the most recently attached session. For example, if the
  user opens two sessions, screen will choose the second session even if
  the user re-attached to the first session after creating the second.
  This will remain true until either the second session is closed or a
  third session is opened, in which case screen will default to the first
  or third session respectively.
  The second option to re-attach to the sessions is to give the PID of
  the screen session as an argument as follows:
  # list the screen sessions:
  screen -ls
  There are screens on:
          8428.pts-1.hostname       (Detached)
          6648.pts-1.hostname       (Detached) # We will re-attach to this session
          16954.pts-1.hostname      (Detached)
  3 Sockets in /run/screens/S-user.
  # Then re-attach to the session, 6648 in this example
  screen -d -r 6648
  What do the ‘-d’ and ‘-r’ flags do specifically though? Well,
  according to the man page:
  
  -d|-D [pid.tty.host] does not start screen, but detaches the
  elsewhere running screen session. It has the same effect as typing C-a d
  from screen’s controlling terminal. -D is the equivalent to the power
  detach key. If no session can be detached, this option is ignored. In
  combination with the -r/-R option more powerful effects can be
  achieved:
  -d -r Reattach a session and if necessary detach it first.
  -d -R Reattach a session and if necessary detach or even create it
  first.
  -d -RR Reattach a session and if necessary detach or create it. Use
  the first session if more than one session is available.
  -D -r Reattach a session. If necessary detach and logout remotely
  first.
  -D -R Attach here and now. In detail this means: If a session is
  running, then reattach. If necessary detach and logout remotely first.
  If it was not running create it and notify the user. This is the
  author’s favorite.
  -D -RR Attach here and now. Whatever that menas, just do it.
  Note: It is always a good idea to check the status of your sessions
  by means of screen -list.
  
  For single user systems, the difference between ‘-d -r’ and ‘-D -R’
  should not be very noticeable, however, on multi-user systems
  understanding the difference between the permutations of the ‘-d’, ‘-r’,
  and ‘-RR’ flags may be useful. It is also worth noting that the ‘-list’
  and ‘-ls’ flags do the same thing, ‘-ls’ is just less typing, and thus
  is usually better.
  One last useful thing that GNU screen can do is attaching to serial
  connections. This isn’t something that I do often, however, it is
  extremely useful when I need it. To attach to a serial connection with
  screen, simply run:
  screen /dev/ttyUSB0 
  Screen will automagically negotiate the baud rate and attach to the
  serial connection (assuming that ttyUSB0 is the correct serial
  connection). There are obviously other tools that can do this, but it is
  helpful to have more tools in the bag in the event something else might
  work better. In this case, being able to copy and paste something like a
  switch config via screen might be useful.
  Customizing Screen
  Customizing GNU screen is generally done via an rc file, however, can
  also be done by running commands with ‘C-a :’ within a screen session.
  The rc file for GNU screen is just a file that will tell screen a list
  of commands to run which will then provide the customizations that the
  user desires. For example, my .screenrc file is fairly
  simplistic at time of writing:
  startup_message off
  hardstatus  alwayslastline
  hardstatus string "screen (%n: %t)"
  caption always '%{=7;0}%Y-%m-%d %c %-Lw%{+b 5;0}{%n %t]%{-}%+Lw%=%{-}'
  The specifics of each of these commands, as well as a complete list
  of all available commands, is available on the screen
  man page. It is a rather exhaustive list, so I will not be going
  over it in detail here. In short, my screen rc file turns off the
  startup message that opens in each new screen session (unless you turn
  it off), then use the other three lines to create a kind of simple
  status bar within my screen sessions. However, much more advanced things
  can certainly be done with the exhaustive list of commands that screen
  will recognize.
  Additionally, the keybindings can be re-assigned if they are not to
  your liking. Simply by using the bind
  command. This command will allow binding any key to and command;
  this can be paired with the ‘unbindall’ command to be able to completely
  re-map all of the shortcuts GNU screen can handle.
  Closing Thoughts
  GNU Screen is not the flashiest terminal multiplexer available, not
  even in the open source world, however, it is one that has been tried
  and tested for over 30 years. This isn’t a tool that is going to
  completely revolutionize how you interact with a terminal, or even be a
  tool that is opened on a regular basis, but it is a tool that is nice to
  be installed in the system and have a basic understanding of how to use.
  It can make managing terminal windows much easier in an event where you
  may need to open lots of them, and it certainly makes dealing with
  serial connections suck slightly less. Give it a try and see if it
  improves your workflow on a terminal, or if it is just a hindrance.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-07-06.html
  		https://foxide.xyz/projects/2025-07-06.html
  		Sun, 06 Jul 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  Key re-mapping is a very common practice for a variety of computer
  users. For anything from swapping two keys to be in a more optimal place
  on the keyboard, to completely changing the keyboard layout to something
  like Colemak.
  Additionally, some keyboards require the use of different layers to
  provide all of the functionality that a standard keyboard would provide.
  Layers become more necessary as the keyboards get smaller, for example
  many of these keyboards would need
  multiple layers to be useful for most people. Generally, these
  re-mappings and layers are handled by the keyboard in firmware, or is
  handled by the host operating system. This blog post will be going over
  one of the operating system based solutions for this problem, kMonad.
  What is KMonad
  KMonad is a tool that allows for remapping keys on one or more
  keyboards on the system with the same or different configurations. From
  the project’s GitHub page:
  
  KMonad offers advanced customization features such as layers,
  multi-tap, tap-hold, and much more. These features are usually available
  at the hardware level on the QMK-firmware enabled keyboard. However,
  KMonad allows you to enjoy such features in virtually any keyboard by
  low-level system manipulations.
  
  Setting Up
  Kmonad can be installed via their release page, just
  download the binary and put it in /usr/local/bin on a Linux
  system. Things get slightly more difficult (in my opinion) in the world
  of Microsoft Windows, but ultimately adding the exe to the path is
  all that needs to occur. From there, the application can be run with the
  file of the keyboard layout as an argument…at least in theory. I did
  this without major issues in Void Linux, simply add yourself to the
  input group if not already:
  sudo usermod -aG input ${USER}
  Logout and log back in for the changes to take effect; can confirm
  group membership by running groups in a terminal session as
  your user. From there we just need to start the configuration of the
  keyboard layout file. I keep mind in ~/.config/kmonad/, but
  the location doesn’t really matter if you would prefer somewhere
  else.
  (defcfg
    ;; For Linux
    input  (device-file "/dev/input/by-id/usb-Razer_Razer_Huntsman_Mini_00000000001A-event-kbd")
    output (uinput-sink "My KMonad output")
  
    ;; Comment this is you want unhandled events not to be emitted
    fallthrough true
  
    ;; Set this to false to disable any command-execution in KMonad
    allow-cmd true
  )
  I also setup kmonad on Slackware Linux with slightly more challenges;
  most prominently setting up permissions for /dev/uinput; I
  accomplished that by setting up a udev rule:
  KERNEL=="uinput", MODE="0660", GROUP="input", OPTIONS+="static_node=uinput"
  Upon rebooting the system, /dev/uinput will then be
  owned by the input group and allow anyone in the
  input group to use kmonad. That is not something I had to
  do on my Void Linux machine, but might be helpful to know in the event
  that it is necessary.
  Creating Keyboard Layout
  Now for the part that is actually fun, creating the keyboard layout.
  The first step to set a custom layout is to define the keys that your
  keyboard will be sending to the operating system when they are pressed.
  Below I am using the ten keyless layout template provided
  by the project as an example:
  (defsrc
    esc  f1   f2   f3   f4   f5   f6   f7   f8   f9   f10  f11  f12
    grv  1    2    3    4    5    6    7    8    9    0    -    =    bspc  ins  home pgup
    tab  q    w    e    r    t    y    u    i    o    p    [    ]    \     del  end  pgdn
    caps a    s    d    f    g    h    j    k    l    ;    '    ret
    lsft z    x    c    v    b    n    m    ,    .    /    rsft                 up
    lctl lmet lalt           spc            ralt rmet cmp  rctl            left down rght
  )
  The project also provides several other keyboard templates
  including:
  
  Apple
  Atreus
  Freestyle2
  ThinkPad T430
  ThinkPad x220
  ISO and ANSI 100%
  ISO and ANSI ten keyless
  ISO and ANSI 60%
  
  You can obviously create a new template if none of those work for
  you; the only really syntax requirements are that the keyboard layout is
  between the defsrc and the closing parenthesis, each key is
  separated by spaces, and each row is on its own row in the layout.
  Making an attempt to make the keyboard layout look like the physical
  keyboard is not a requirement, but can make working with it a bit
  easier.
  Once the defsrc is made, we then need to create the
  layers that we want on the keyboard; there is no limit to the quantity
  of layers on the keyboard, but the layout does have to match the
  defsrc. The way I usually make a new layer is just to copy
  the defsrc and change it from defsrc to
  deflayer followed by the layer name. This might be an
  example of a basic first layer with a modification:
  (deflayer qwerty
    esc  f1   f2   f3   f4   f5   f6   f7   f8   f9   f10  f11  f12
    grv  1    2    3    4    5    6    7    8    9    0    -    =    bspc  ins  home pgup
    tab  q    w    e    r    t    y    u    i    o    p    [    ]    \     del  end  pgdn
    esc  a    s    d    f    g    h    j    k    l    ;    '    ret
    lsft z    x    c    v    b    n    m    ,    .    /    rsft                 up
    lctl lmet lalt           spc            ralt rmet cmp  rctl            left down rght
  )
  As you can see, the caps lock key was replaced with escape in that
  layer. Even with just this basic functionality, it can already be used
  to entirely change the keyboard layout from something like qwerty to dvorak
  or colmak.
  If the goal was to create a layer called ‘colmak’ with a colmak keyboard
  layout, that might look as follows:
  (deflayer colmak
    esc  f1   f2   f3   f4   f5   f6   f7   f8   f9   f10  f11  f12
    grv  1    2    3    4    5    6    7    8    9    0    -    =    bspc  ins  home pgup
    tab  q    w    f    p    g    j    l    u    y    ;    [    ]    \     del  end  pgdn
    caps a    r    s    t    d    h    n    e    i    o    '    ret
    lsft z    x    c    v    b    k    m    ,    .    /    rsft                 up
    lctl lmet lalt           spc            ralt rmet cmp  rctl            left down rght
  )
  Kmonad will automatically pick the first deflayer
  statement as the default when it is started; to be able to switch layers
  and have more complex functionality than switching key behavior, a
  defalias statement will need to be made.
  Defalias
  This section is where the magic of kmonad really comes into play.
  Keys can be setup to have different functionality depending on whether
  the key was pressed or held, or have different functionality depending
  on how many times the key was pressed. I am just going to go over the
  basics for things I have implemented with my keyboard; there is a tutorial
  layout that goes over each of the functions in depth. Use that for a
  more complete reference for what is possible with the tool as well as
  how to go about accomplishing it.
  The two main functions I use are the ‘layer-switch’ and the
  ‘tap-hold’ functions. Layer-switch does basically what it sounds like,
  simply switching to a different layer. The main layer I use this with is
  to re-implement a numpad on my keyboard that does not already have one.
  The actual layout for that isn’t particularly important, but the
  defalias would look something like this:
  (defalias
      default (layer-switch qwerty)
      numlayer (layer-switch numpad)
      num (tap-hold 300 @numlayer ralt)
      cap (tap-hold 300 esc caps)
  )
  As stated previously, the ‘switch-layer’ function is simply just to
  switch to a different layer defined in a ‘deflayer’ statement. So when
  the key bound to that function is pressed, the new layer will be
  activated. The ‘tap-hold’ function is a bit more interesting; when the
  tap-hold function is written, it is given a time in milliseconds (300
  has been a good timing for me personally), then the behavior on a tap,
  then finally the behavior when the button is held for more than the
  timing. Each of the functions are given a name that can be bound to a
  key. For example, in the above defalias statement, binding the ‘cap’
  function (that acts like escape on a press but caps lock on hold) could
  be defined on a key with @cap. You can also see that we can
  combine functions together in multiple functions to get more complex
  behavior; I have a ‘layer-switch’ function setup that is used in a
  ‘tap-hold’ function so that if the key is pressed it will switch to the
  numpad layer, but if it is held, it will act as a right alt key.
  Again, I am using just the most basic functionality of this tool and
  it has completely changed how I interact with my keyboard. Kmonad is
  capable of doing much, much more than I am describing here.
  Outstanding Challenges
  While I do like this tool, and I am certainly going to continue to
  use it, I have had two minor challenges with it. The first one is that
  the key remapping is not always detected by software correctly. I have
  only had this happen in games, and it very well could be something that
  Proton is doing rather than kmonad, but it has certainly happened where
  I had to force quit a game I was playing because I couldn’t open the
  pause menu. This can easily be worked around by having a “gaming” layer
  that is mostly the default layout with one unimportant key remapped to
  switch to the other layers.
  The second challenge is that I have not found a good way to run it on
  FreeBSD yet; it is software that might run in the Linuxulator,
  but it did not compile naively. This is something that I have a hard
  time griping at too hard though. It would be nice to have FreeBSD
  support, but the devs certainly are not bound to my whims. And
  ultimately it is open source software, if I wanted FreeBSD support bad
  enough, I could make a PR for it.
  Closing Thoughts
  Kmonad is an overall amazing tool for remapping keyboards that is
  much more powerful than something like setxkbmap, which is
  what I was using previously to rebind my caps lock key to escape. This
  tool allows me to do quite a bit more, also making using small keyboards
  much more pleasant. Normally I would put a ‘more resources’ section
  here, but the documentation in the projects GitHub page is quite good
  and explains things fairly well, so I would recommend trying that out
  before doing much digging around online.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-07-20.html
  		https://foxide.xyz/projects/2025-07-20.html
  		Sun, 20 Jul 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  The Linux, and more generally Unix, ecosystems have prided themselves
  on not needing graphical applications to do most tasks. While that is
  true generally, there are times in which it might be nice to have a
  graphical application or entire desktop running on a remote machine.
  This could be useful for things like remote development or running a
  graphical environment on a headless
  system such as a raspberry pi.
  This blog post is going to cover how to setup and run X11
  applications over SSH as well as how to tunnel VNC connections over SSH
  making using VNC over a hostile network (like the Internet) less risky.
  While everything that will be covered in this is probably possible one
  way or another on Wayland, I will not
  be covering or even speculating on how one might go about it; however,
  that would be a great topic for someone to cover in their own blog!
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-08-01.html
  		https://foxide.xyz/projects/2025-08-01.html
  		Fri, 01 Aug 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  One of the biggest gripes I have had with Windows is the lack of
  window management options. Secondary to this is the gripe that the
  operating system relies extremely heavily on the mouse, and while
  Windows does provide keyboard shortcuts for a
  lot of things, It isn’t quite as cohesive and straightforward as
  using one of the various tiling window managers for Linux and the BSDs,
  or even as nice as something like Yabai for MacOS.
  Recently though, I saw this
  video that piqued my interest (mostly the thumbnail) going over GlazeWM, a tiling window
  manager for Windows.
  Getting Started
  To get started with GlazeWM, simply choose one of the installation
  options on the README, I chose the winget
  option, however, it can also be installed via Chocolatey, Scoop, or by
  installing the release just like
  a normal Windows program.
  I have not tested all of the install methods, but the winget
  installation also automatically installs Zebar as well as GlazeWM;
  according to the documentation the release install has a checkbox to
  install Zebar if desired. Zebar is a status bar, similar to something
  like i3blocks, bumblebee
  status, or slstatus, but for use
  with Windows. That’s basically it; we can run GlazeWM by simply running
  the ‘GlazeWM’ application. To have GlazeWM automatically start on sign
  in, create a shortcut for the application (can be done by right clicking
  and hitting ‘Create shortcut’); then placing that shortcut in the
  startup folder for your user. The easiest way to get to the startup
  folder is to type in shell:startup in the run menu (win+r),
  then hit enter. This should open a File Manager window, executable file
  or shortcut in that folder will start automatically upon login.
  Actually using it should be fairly nice and familiar for anyone that
  uses tiling window managers on other systems. I’m not going to go over
  all of the various keyboard shortcuts, but there is a cheatsheet
  on their GitHub repo; additionally, if those key bindings don’t work for
  some reason, they are customizable in the config file.
  Customizing GlazeWM
  Changing customization options of Glaze is as simple as editing some
  YAML, the default config file is located at
  %USERPROFILE%.glzr.yaml. Custom locations can be specified
  using the –config flag or setting the
  GLAZEWM_CONFIG_PATH environment variables.
  The default configuration is rather sane and comfortable, especially
  considering the starting point. I did, however, change some of the
  options, such as transparency and border color. Additionally, I changed
  the ‘launch terminal’ command to launch WSL instead, and rebound the
  default ‘launch terminal’ command to Alt+Shift+Enter, as the Windows
  command prompt is occasionally useful. From there, the big thing is
  adjusting to the new capabilities that the window manager provides.
  Minor Annoyances
  While the project, at least in my opinion, is leaps and bounds ahead
  of what the vanilla experience provides, there are some annoyances that
  I have with GlazeWM.
  The chief one is that there is not a straightforward way to manage
  the numbered workspace when using multiple monitors. This issue is not
  one that would affect everyone, and I think it would not be particularly
  annoying with two monitors, however, I am using three monitors on my
  Windows machine which can make managing all of the workspaces a bit
  tricky because they are sequential with no obvious way to move one
  number to another monitor. There does appear to be a PR that might fix
  this issue though.
  The other issue, which is only slightly annoying, is that I cannot
  maximize multiple windows at a time and have them stack on top of each
  other. This is quite a useful feature when I want to quickly switch
  between windows, but I need the entirety of the screen’s real-estate.
  Looking at my previous complaint, that is what the multiple monitors
  would be for. While that is accurate, this would be a more serious
  problem on a laptop with only one screen. The current way to support
  this is to minimize windows, which works but is slightly more cumbersome
  (at least in my opinion) than being able to stack the windows on top of
  each other.
  The other annoyance that I have, which is extremely minor, is the
  lack of a “stacking” mode. Glaze has a mode for the Windows to float or
  to minimize windows as well as fullscreening a window, but does not have
  a stacking mode similar to i3, where only the active window is
  displayed, but is not necessarily “fullscreen”. While this feature
  sounds similar to fullscreen, it differs slightly in the fact of another
  window can be made to be the active window rather dynamically to easily
  swap between windows.
  Closing Thoughts
  Despite the minor criticisms I have of the projects current
  capabilities, I will certainly be trying to use it and follow the
  project as it improves. Glaze makes using Windows suck way less, and
  removes many of the complains that I had about using it in the first
  place. Replacing the terminal launcher with WSL almost makes me forget
  that I am using Windows. If you are a Windows user interested in tiling
  window managers, or a Linux/BSD user having to use Windows, I would
  recommend giving this project a try.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-08-24.html
  		https://foxide.xyz/projects/2025-08-24.html
  		Sun, 24 Aug 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  I am setting up a new server for my house, and thought I would take
  the opportunity to document setting up Bind 9 with ad blocking. Currently,
  my DNS is setup with Bind, but I did not make any notes or write
  anything down at the time of setting it up. Since I am re-doing much of
  that work anyway, I thought I might as well take some notes for myself
  and anyone else struggling with Bind like I know I did, both in the past
  and with this current iteration.
  Basic OS
  I went with a very standard FreeBSD bootonly install with ZFS on
  root. Rebooting into the newly installed system, I installed some basic
  things that I want on the system such as neovim,
  doas, and git:
  pkg update
  pkg install -y neovim doas git
  # Symlinking nvim to vim because muscle memory
  ln -s /usr/local/bin/nvim /usr/local/bin/vim
  I then changed the configuration of the network adapter from DHCP to
  a static IP. This is best done in /etc/rc.conf to be
  persistent between reboots. While I was editing my rc.conf
  file, I went ahead and organized the parameters to be a bit more
  logical, at least for me:
  # Standard Host Config
  clear_tmp_enable="YES"
  syslogd_flags="-ss"
  sendmail_enable="NONE"
  powerd_enable="YES"
  # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
  dumpdev="AUTO"
  zfs_enable="YES"
  moused_nondefault_enable="NO"
  
  # Networking Stuff
  hostname="server"
  # ifconfig_re2="dhcp"
  ifconfig_re2="inet 172.16.0.40 netmask 255.255.255.0"
  defaultrouter="172.16.0.1"
  
  # Setting Time
  sshd_enable="YES"
  ntpd_enable="YES"
  ntpd_sync_on_start="YES"
  
  # Bind Stuff
  
  # Bhyve Stuff
  kld_list="nmdm vmm"
  
  # Jails
  cloned_interfaces="lo1"
  This file will be modified a few times during the deployment of the
  server, but that will be a later me problem. For now the lines that are
  relevant are the ones under the “Networking Stuff” section. In
  order:
  
  hostname = "server": Sets the hostname to ‘server’
  #ifconfig_re2="dhcp": commented out, but would set up
  DHCP on interface re2 if uncommented
  ifconfig_re2="inet 172.16.0.40 netmask 255.255.255.0":
  sets a static IP of 172.16.0.40 with a /24 netmask.
  defaultrouter="172.16.0.1": sets the gateway to
  172.16.0.1.
  
  Setting Up Bind
  pkg install -y bind-tools bind920
  rndc-confgen -a
  The bind-tools package is what provides utilities such
  as nslookup and dig, which are not strictly
  required, but make troubleshooting much easier. After getting the
  packages installed, we have to get the server configured:
  // Comment out the following line to enable named for all interfaces
  // listen-on      { 127.0.0.1; };
  // alternatively, an address can be specified for a listener address
  ...
  
  // Enables recursive DNS
  recursion yes
  // Stealing the following comments from the original file:
  // If you've got a DNS server around at your upstream provider, enter
  // its IP address here, and enable the link below. This will make you
  // benefit from its cache, thus reduce overall DNS traffic in the internet.
  forwarders {
      9.9.9.9;  // Using quad 9
      8.8.8.8;  // Google DNS
      1.1.1.1;  // Cloudflare
      // etc
  }
  
  // If you only want the DNS server to forward queries,
  // then the following line should be uncommented.
  // forward only;
  Enabling recursive DNS is generally recommended as it caches the
  results of DNS queries, boosting performance. We will be coming back to
  the named.conf file to enable the zones, however, let’s
  look at the zone files for now.
  Zone Files
  While the zone files can technically be stored anywhere on the
  system, I tend to make a master directory in the
  named configuration directory
  (/usr/local/etc/namedb). Alternatively, the FreeBSD
  implementation has a default primary directory for the
  localhost and ‘empty’ zone files.
  Here is an example zone file for the domain
  example.home.arpa, with comments (starting with ‘;’) to
  better explain what each line is:
  $ORIGIN example.home.arpa.  ; Base domain name for the zone file
  $TTL 3H                     ; default time to live for the zone
  @       IN      SOA     example    admin.example.home.arpa. ( ; Setting the SOA record for the zone, this is a requirement for the zone configureation to be valid
                                  6       ; Serial, number must increase on each edit for changes to be loaded. If number decreses, bind will give an error on the zone
                                  3h      ; Refresh
                                  1h      ; Retry
                                  1w      ; Expirey
                                  1h )    ; Minimum TTL
  
          IN      A       172.16.0.40             ; A record for main domain
  @       IN      NS      ns1.example.home.arpa.  ; Name Server record for domain (recommended, but not strictly required)
  ns1     IN      A       172.16.0.40             ; A record for Name Server record (required for NS record to be valid)
  Any domain can be added to these zone files, however, for local
  domains it is generally recommended, per RFC 8375 to use
  .home.arpa for local domains rather than something like
  .home, .local, or .lan. This is
  because the .home.arpa domain is specifically reserved for
  that purpose, and thus will not run into collision issues with external
  services.
  Now let’s setup a reverse zone for our domain:
  $TTL 3h
  @ SOA example.home.arpa. admin.exmaple.home.arpa. (
                  2       ; Serial
                  1d      ; Refresh
                  12h     ; Retry
                  1w      ; Expire
                  3h )    ; TTL
  
          NS      ns1.home.arpa.
  
  40      PTR     ns1.home.arpa.
  40      PTR     home.arpa.
  While this file looks very similar, it is not exactly the same, with
  the most notable feature being the lack of full IP addresses. Rather, it
  will only contain the last octet for each of the domain names, the rest
  of the address will actually come from the named.conf file
  when we enable the zone(s) and reverse zone(s). I added those entries
  right above the entries for the various RFCs:
  /* Serving the following zones locally will prevent any queries
     for these zones leaving your network and going to the root
     name servers. This has two significat advantages:
     1. Faster local resolution for your users
     2. No spurious traffic will be sent from your network to the roots
  */
  // Custom Forward Zone Files
  zone "example.home.arpa"    { type primary; file "/usr/local/etc/namedb/master/example-forward.db"; };
  // Custom Reverse Zone Files
  zone "0.16.172.in-addr.arpa" { type primary; file "/usr/local/etc/namedb/master/local-reverse.db"; };
  ...
  // Comment (or change) the existing entry for '0.16.172.in-addr.arpa'
  // zone "0.16.172.in-addr.arpa" { type primary; file "/usr/local/etc/namedb/primary/empty.db"; };
  ...
  We can check for mis-configurations using the
  named-checkconf and named-checkzone utilities
  as follows:
  # Checking for configuration issues (no output means no configuration issues were found)
  named-checkconf
  # Checking for configuration issues in the forward zone file
  named-checkzone example.home.arpa master/example-forward.db
  zone example.home.arpa/IN: loaded serial 6
  OK
  # Checking for configuration issues in the reverse zone file
  named-checkzone 40.0.16.172.example.home.arpa master/local-reverse.db
  zone 40.0.16.172.example.home.arpa/IN: loaded serial 2
  OK
  Now that everything is configured, let’s enable the service and see
  if it works:
  # I tend to prefer to not fully enable the service until it is tested
  # the one start command will allow it to run as if it were enabled,
  # without it actually being enabled
  service named onestart
  
  # To fully enable the service:
  sysrc named_start=YES
  service named start
  Now we need to add the DNS server to its own
  /etc/resolv.conf file (and I generally remove other
  namesevers):
  # The search line reaches out to the domain to check for connectivity to the Internet
  search freebsd.org
  nameserver 127.0.0.1
  # Can also use the machine's actual network address
  nameserver 172.16.0.40
  Then we test:
  # Checking to make sure the machine can get to the Internet,
  # as well as checking which name server it is using:
  nslookup foxide.xyz
  Server:         172.16.0.40
  Address:        172.16.0.40#53
  
  Non-authoritative answer:
  Name:   foxide.xyz
  Address: 172.245.181.191
  
  nslookup example.home.arpa
  # Now for our local domain
  Server:         172.16.0.40
  Address:        172.16.0.40#53
  
  Name:   example.home.arpa
  Address: 172.16.0.40
  
  # Now for the reverse address
  nslookup 172.16.0.40
  40.0.16.172.in-addr.arpa        name = ns1.example.home.arpa.
  40.0.16.172.in-addr.arpa        name = example.home.arpa.
  Assuming all the tests passed, the DNS server is basically setup, and
  more zones can be added as desired, each new domain will need a separate
  zone file, however, the reverse zones can live in one file as long as
  they are within the same subnet. I do also want to point out that the
  various config checks will not check that the paths within the config
  are valid, so, if the local addresses are not working, check that. I
  spend at least 30 minutes troubleshooting only to find that I missed a
  ‘b’ in the file path.
  Ad Blocking
  Now that we have the standard DNS stuff setup and working, let’s move
  on to ad blocking. This is something that can be done with things like
  pi-hole, but I want to block ads and
  use Internet grade DNS. Thankfully, this is a problem that has been tackled
  before. The following script was modified from the linked articled
  by bgstack15, which was adapted from Mueller-ma.
  That script had some other features such as block lists and allow lists,
  however, that is not a feature I care to use currently. Maybe I will
  update the script if I decide I need and/or want that feature. For now,
  let’s take a look at the script:
  #!/bin/sh
  
  # Create a workspace in /tmp to keep organized
  mkdir /tmp/adblock
  cd /tmp/adblock
  
  # Setting some variables
  blockedfile="/usr/local/etc/namedb/primary/empty.db"
  zonefile="/usr/local/etc/namedb/named.conf"
  
  # Fetching the desired domain list,
  # checkout https://github.com/StevenBlack/hosts for all options
  fetch https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o StevenBlack-hosts
  
  # The original lists were meant to go in /etc/hosts, this prepares them to be turned into a zone entry
  < StevenBlack-hosts awk '/^0\.0\.0\.0/ && ! /127.0.0.1|255.255.255.255|::1/ {print $2}' >> hosts
  # This removes duplicates as well as commented lines
  < hosts grep -vE '^\s*($|#)' | sort | uniq > uniq_hosts
  # This turnes each domain into a zone entry
  < uniq_hosts sed -r "s:(.*):zone \"\1\" { type master; file \"${blockedfile}\"; };:" > zone
  # Copies the current named.conf minus adblocking entires for safety
  sed "/Adblocking/q" ${zonefile} >> ${zonefile}.bk
  # Moves just the normal configs back
  cp ${zonefile}.bk ${zonefile}
  # Add the adblocking zones to named
  cat zone >> ${zonefile}
  # Restart service for changes to take effect
  service named restart
  
  # Cleanup
  rm /tmp/adblock/*
  The only manual modification required for the named.conf
  file is adding a comment containing the word “Adblocking” at the end of
  the file. This is used to remove old entries that were being block via
  the sed command in the script.
  It is generally best practice to copy the script to either the
  /usr/local/bin or /usr/local/sbin directories,
  then from there the script can be run via cron or other automation. The
  source repo appears to be updated fairly often, so I plan on pulling a
  new file every 2 weeks. It can also be run manually if needed
  though.
  General Notes
  This section is a random collection of thoughts that didn’t really
  fit in anywhere else.
  Issues with some tooling
  The rndc
  tool that is intended to manage bind is quite neat, however, I expect
  that for most small bind instances, it is quite overkill. I also found
  that it inconsistently works; there were many times in which I tried to
  run rndc reload, but the command would fail. To get around
  this, I had to run service named onerestart; this isn’t
  particularly a problem as the workaround is was I would have done
  without the other tool anyway, but it does make me want to use the tool
  less. It is also possible there was a mis-configuration on my part, or
  something else I did that made the tool operate this way. Certainly feel
  free to try the tool, but issues may arise during that process.
  Getting around ISP routers
  I was in a situation where my router would not allow changing DNS
  servers. There are a few different solutions such as putting the router
  in bridge mode and bringing APs for WiFi, a router, and a firewall, but
  that was not something I could practically do right now. Another
  solution was buy a better router. While this solution is good and much
  less expensive than creating an enterprise grade network for my house,
  it is not free, and not something I was interesting in spending the
  money on right now. My solution ended up being creating a DHCP range of
  one static address, then implementing a DHCP server (on the same server
  as bind) that would set the DNS server as I wanted it. This got around
  the problem for all the machines I cared about.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-10-05.html
  		https://foxide.xyz/projects/2025-10-05.html
  		Sun, 05 Oct 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  Recently I have been seeing videos on small ortholinear keyboards,
  and have been considering getting one. The big thing that most keyboards
  are lacking for me though is some sort of a pointing device. My
  preference is a Trackpoint, however, a trackball or touchpad would work
  as well, just as long as I can keep my hands as close to the home row as
  possible. I did come across a website that does sell both kits and
  complete keyboards with pointing devices as an optional add-on. That’s
  great because I am not going to put in the effort to build my own.
  The issue with going with one of those smaller keyboards is the lack
  of keys though; before purchasing one, I wanted to see how difficult
  and/or painful it would be to use less keys on my keyboard. This brought
  me to home row
  mods, and this post is mostly denoting my experience with them.
  Things like what does and does not work for me, problems I have run
  into, and things I would still like to improve.
  What are home row mods
  Home row mods refers to adding the functions of the modifier keys to
  the home row. This reduces the need to stretch the fingers in awkward
  ways as well as moving the commonly used keys away from the weaker
  fingers (such as the pinky) and adding the functionality to stronger
  fingers like the index. It also has the added side benefit of being able
  to get more functionality out of fewer keys, which is helpful for small
  keyboards with less than 50 keys. Unfortunately, not all keyboards have
  the capability of doing this out of the box, which is the situation I
  was in. So I had to implement a software solution, and the software I
  chose for this was kmonad since I
  already know how to use it. I am not going to explain how to use the
  software to actually implement this, rather I am going to be talking
  about my experience with actually using home row mods. For the “how-to”
  guide on using kmonad, read my previous post on the topic, or look at
  the tutorial
  layout for the project.
  Setup
  The keyboards that I am wanting to get are have QMK firware, but I do not have any keyboards
  that are QMK enabled. So, I will have to emulate much of the
  functionality with other software. It is also available on Windows and
  Linux which makes it easy enough to implement on enough devices to
  really live with for a few weeks and find out how well or poorly I can
  handle the lack of keys I am thinking I want to buy into.
  I am not going to go into the setting up of kmonad itself, for that
  read my previous post on the
  topic. This time I am going to go over the home row mods that I am
  using, what I like and dislike about them. Then finally, things that I
  would like to improve to reduce finger movement and generally use less
  keys.
  For the purposes of this blog post, let’s assume the following
  keyboard layout:
    esc  1    2    3    4    5    6    7    8    9    0    -    =    bspc
    tab  q    w    e    r    t    y    u    i    o    p    [    ]    \
    caps   a    s    d    f    g    h    j    k    l    ;    '       ret
    lsft      z    x    c    v    b    n    m    ,    .    /         rsft
    lctl lmet lalt                   spc             ralt  func menu rctl
  This is a quite rough re-creation of the Razer Hunstman Mini layout.
  Now let’s start by creating a base layer for the keyboard use:
  ;; Layer 1
    grav 1    2    3    4    5    6    7    8    9    0    -    =    bspc
    tab  TH-q    w    e    r    t    y    u    i    o    p    [   ]  TH-bspc
    TH-1  TH-a TH-s TH-d  TH-f  g   h   TH-j TH-k TH-l  TH-;  '      ret
    lsft      z    x    c    v    b    n    m    ,    .    /         L2
    lctl lmet lalt                   spc             L1  rmet cmp  rctl
  The keyboard layout above is fairly normal; however, the home row
  keys, as well as some other keys, have additional functionality
  described below.
  
  L1: Layer 1
  L2: Layer 2
  TH-1: Tap - Escape, hold - caps lock
  TH-a: Tap - a, hold left shift
  TH-s: Tap - s, hold left control
  TH-d: Tap - d, hold left meta (windows key)
  TH-f: Tap - f, hold left alt
  TH-j: Tap - j, hold right alt
  TH-k: Tap - k, hold right meta
  TH-l: Tap - l, hold right control
  TH-; Tap - semi colon, hold right shift
  TH-bspc: Tap - Backspace, hold
  
  
  TH-q: Tap - q, hold tab
  
  This is the default layer I have been using; I expected the layout to
  be quite difficult to get used to and highly annoying. Getting used to
  it has been surprisingly easy though; the biggest problem I have had is
  getting used to using my right hand for using shift. It is much easier
  with the better position of the key, but is still not quite second
  nature yet. Past that, it is more minor issues such as getting used to
  the timing of holding the keys down, or technical issues with various
  bits of software.
  Now that I’ve gone over the main layer, let’s take a look at the
  other two layers that I am using
  ;; Layer 2
    grav 1    2    3    4    5    6    7    8    9    0    -    =    bspc
    tab TH-q    w    e    r    t    y    4    5    6    *    [    ]    \
    TH-1  TH-a TH-s  TH-d  TH-f  g   h    1    2    3    -    '       ret
    lsft      z    x    c    v    b    n    0    ,    .    /         L2
    lctl lmet lalt                   spc             default func menu rctl
  This layer’s big feature is swapping the keys on the right side of
  the keyboard for a number pad; the layout for the number pad is based
  around the layout that old
  ThinkPads had. It’s quite convenient and easy enough to learn to
  use.
  The last layer is for using keys that I need far less, such as the
  arrow keys or other navigational keys.
  ;; Layer 3
    esc 1    2    3    4    5    6    7    8    9    0    -    =    bspc
    tab  q    w    e    r    t   home  pgdn  pgup  end     p    [    ]    \
    caps   a   s    d    f     g   left down  up  right   ;    '       ret
    lsft      z    x    c    v    b    n    m    ,    .    /         default
    lctl lmet lalt                   spc             L1     func menu rctl
  I also use it for when I am playing games as some games do not do
  well with remapped keys or the tap-hold scheme.
  Gripes
  Changing to using a keyboard this way has been quite nice, and for
  the most part the gripes I have are problems more with the
  implementation rather than actually problems with the setup. For
  example, on my Windows computer, kmonad often runs into problems where
  it thinks the CTRL key is being held down after signing in. For the most
  part, actually using home row mods has been fairly pleasant and I am
  quickly getting used to it. So much so, that I am already having a
  difficult time switching back to the standard layout on the keyboard. I
  do want to find a more elegant way to deal with numbers that do not
  necessarily require a number row though, as the keyboard that I am
  wanting to get does not have a row for numbers. I could use the number
  layer for that, but I feel like it will be extremely difficult to learn
  where the symbols (things like $ and #) are on the keyboard that way.
  Reading on some forums, it seems like it is less of a problem if you use
  a layer to bind the top row of the keyboard for a number row, but then
  that means switching layers for numbers and that would really suck for
  changing workspaces on DWM.
  Closing Notes
  Overall, the idea of home row mods is one that I really like as it
  requires moving my hands around the keyboard significantly less,
  however, I could also see it not being for everyone. It does take some
  getting used to, and the initial switch requires a bit of thought when
  you are wanting to actually use the keys because you aren’t used to it
  yet. For those wanting to try it, I would recommend trying it on a more
  normal keyboard with something like kmonad and slowly get used to it
  with that rather than just jumping into something like a tiny
  ortholinear keyboard. This will help you determine if this is a setup
  that could work for you rather than spending hundreds of
  dollars on a keyboard that you end up hating.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-10-22.html
  		https://foxide.xyz/projects/2025-10-22.html
  		Wed, 22 Oct 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  Password managers are all but required software for working on a
  computer in today’s age. While many major organizations recognize the
  problems with password based authentication and are actively trying to
  improve the landscape with things like multifactor
  authentication or even passkeys, which are an
  extension of MFA designed to be passwordless and phishing resistant.
  However, until there is a more widely adopted standard for
  authentication, most people will continue to rely on username and
  password based authentication. This causes a big problem with having to
  remember usernames and passwords; Nothing up to this point should be a
  surprise, and there are many products, both proprietary and open, to
  solve this problem:
  
  Keepass
  1Password
  Roboform
  Bitwarden
  
  Keepass is the password manager I have been using for my personal
  passwords, however, I have an issue of forgetting to sync the password
  file into my cloud storage to be used on other devices when needed. This
  is a problem that is easy enough to solve by having a client
  automatically mount my Nextcloud somewhere on my local computer rather
  than having to login and upload it manually, but that is not something I
  am particularly interested in doing just for my passwords as most of my
  other important files can be managed with something like Git, SCP, or
  Rsync. Thankfully, gnu pass
  exists and works well with that workflow, while following the Unix
  philosophy and working in a terminal.
  How does it work?
  GNU Pass encrypts your data with a PGP key that you create with GNU Privacy Guard, from there you can
  access the stored passwords by unlocking the PGP key. Pass also includes
  commands to interact with git to make it easier to backup the files;
  then simply transfer the GPG keys and the pass vault can be unlocked and
  used on another device.
  Getting Started
  For this post, I am doing with Linux Mint Debian
  Edition since they had a release semi-recently, on top of I haven’t
  used Linux Mint in a while. After a standard install and update cycle on
  the machine, GNU Pass can be installed via the following command:
  # -y will prevent asking for confirmation
  sudo apt install -y pass
  Then we can initialize the password vault by:
  # First generate GPG key if you do not have one
  gpg --full-generate-key
  
  # Replace 'john.smith@example.com' with
  # the email address associated with your key
  pass init john.smith@example.com
  
  # Alternatively, you can just use the key-id found by running this:
  gpg --list-secret-keys
  
  # then insert in the command
  pass init ${KEY-ID}
  Now let’s start adding some keys:
  pass add example.com
  Enter password for example.com
  Retype password for example.com
  While this is a great way to store the password, categorization of
  the passwords would be nice, as well as being able to store the
  usernames. Luckily, the developer has already thought of this. We can
  specify a directory to separate logins that we can use to categorize
  things, then we can also add usernames to store those as well:
  # This would create an entry in the 'cat1' category for a user called 'user'
  pass insert cat1/docs.example.com/user
  mkdir: created directory '/home/user/.password-store/cat1'
  mkdir: created directory '/home/user/.password-store/cat1/docs.example.com'
  Enter password for docs.example.com
  Retype password for docs.example.com
  Now that we have that password, we can show the various passwords in
  our vault with:
  pass show
  Password Store
  └── cat1
      └── git.example.com
          └── user
  Then to retrieve the password, we can either list it or copy it to
  the clipboard. Commands for each of those below:
  # To show the password in the terminal
  pass show cat1/docs.example.com/user
  password
  
  # To copy the password to clipboard
  pass -c cat1/docs.example.com/user
  Copied cat1/docs.example.com/user to clipboard. Will clear in 45 seconds.
  Pass is also capable of generating strong random passwords by using
  the generate command:
  [master (root-commit) abfb5eb] Add generated password for cat2/media.example.com/user.
   4 files changed, 2 insertions(+)
   create mode 100644 .gitattributes
   create mode 100644 .gpg-id
   create mode 100644 cat2/media.example.com/user.gpg
  The generated password for cat2/media.example.com/user is:
  &zm>CnQ98+^e(oyz]^]sA+SN1
  Finally, we can then use Git to version control the password files
  and store them securely (since it’s encrypted with GPG) in Git. We
  simply initialize the git repo by running pass git init,
  then when pass creates a new password, a commit will automatically be
  generated; from there run pass git push (assuming the
  origin has already been set for the repo) to send it up to your forge of
  choice.
  Switching and Considerations
  If you are wanting to try or use GNU Pass, check out the website and the Arch
  Wiki, the website specifically shows a lot of extensions and add-ons
  that can be used to interact with other systems, such as web browsers,
  to make this tool a bit easier to use. In addition, it also shows some
  tools to automatically export your passwords from another system to GNU
  pass rather than having to manually enter them.
  The main consideration for me with GNU Pass is whether to trust
  storing the passwords on something like Github or Codeberg. I don’t have
  enough knowledge to give any legitimate criticism to GPG or PGP overall,
  there are certainly people
  online that do take issue with PGP. However, much like anything else
  on the Internet, there are people that have opposing
  positions. GPG has been a trusted mechanism for thousands of people
  online in various more dangerous situations than I am in right now,
  however, before using this software, you should come to terms with how
  much you do or do not trust GPG. That brings the next question though,
  if you do not trust GPG or PGP in general, what system do you trust for
  guarding your passwords?
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-11-02.html
  		https://foxide.xyz/projects/2025-11-02.html
  		Sun, 02 Nov 2025 00:00:00 -0400 
  		
  		
  		 
  		 
  			
  Abstract
  I am going to be making a series of blog posts going over IPv6,
  everything from some basic theory to actually getting it implemented
  into your network. This is the first post on the topic, and will mostly
  be going over the basic concepts of IPv6 as an introduction for anyone
  unfamiliar with IPv6; additionally, it will also be sharing the
  resources I have found for studying and learning IPv6.
  Before going to far into this post, I want to preface it by saying
  this post is intended to be a primer for IPv6 and thus will not be
  complete. For more in-depth and thoroughly explained information on the
  subject, the main reference I am using to write this post is a book
  called Book
  6 which has a lot more in-depth information than what I will
  include here. For those wanting to truly understand IPv6 in great
  detail, I would recommend giving that a read; the book has a lot of
  history and good practices when working with IPv6, complete with
  references on RFCs and case studies.
  Where is IPv5?
  Before really getting into anything to do directly with IPv6, let’s
  talk about IPv5. Specifically, where is version 5? IPv5 was a real
  protocol called Internet Stream
  Protocol, which was intended to be used for things like audio and
  video calls over the less than reliable Internet connections of the
  time.
  Because the protocol was completely incompatible with IPv4, a new
  designation was required and 5 was the next available number. Eventually
  as Internet connections got faster and more reliable, ST became
  unnecessary and never saw real world use.
  When IPv6 was defined later that year in RFC 1883, the headers
  were also completely incompatible with previous versions of IP, thus
  requiring a new designation number with the next available one being
  6.
  Why Learn IPv6?
  So, why should you learn IPv6? put bluntly, most people’s answer on
  this is because they have to. Dealing with an IPv6 network is often
  spoken of as fondly as printers are among IT workers; given the massive
  annoyance that IPv6 is, why would you take the time to learn it if you
  don’t have to? The main reason I am putting in the effort to learn it is
  to better understand IP schemes on the open Internet; IPv6 adoption has
  grown massively over the last five years, and many countries have goals
  of having IPv6 majority within the next five to ten years. While it is
  unlikely that I will be working in a position that understanding IPv6
  will be critical, it will not generally be problematic to bring such
  knowledge with me.
  IPv6 Basics
  So, why do people complain about learning IPv6 so much? IPv6 does not
  change fundamental networking concepts such as encapsulation and the OSI Model are the
  same as in IPv4; however, the implementation details between the two are
  very different. The core change that IPv6 brings over IPv4 is that
  addresses are 128 bit rather than 32 bit. Because of the sheer quantity
  of usable addresses in IPv6 vs IPv4
  (340282366920938463463374607431768211456 vs 4294967296), the format
  changed from decimal
  notation to hexadecimal
  notation. By itself that change is not awful, however, the
  implications of this change cause a lot of headaches when trying to
  translate the ideas from IPv4 to IPv6. In general, IPv6 has a direct
  analog, if not re-implementation, of IPv4 concepts, but the details of
  implementing and maintaining them are quite different and often allow
  for more choice by the network operator.
  Addressing
  As stated above, IPv6 addresses use hexadecimal format for
  representing addresses, for example:
  2001:0db8:aaaa:0000:0000:0000:9876:cafe
  This address looks very annoying to deal with because it is quite
  long, and many of the characters are tedious to type (specifically the
  colons). While the intent is that IPv6 addresses should be copied and
  pasted, but on occasion that is not possible for one reason or another.
  Thankfully, IPv6 has some compression mechanisms that make dealing with
  addresses slightly less terrible. The first mechanism is to drop leading
  zeros in each chunk. Applying this to our previous address would change
  it to:
  2001:db8:aaaa:0:0:0:9876:cafe
  We can further compress the address with the second mechanism, which
  is to replace a consecutive grouping of zeros with a double colon like
  so:
  2001:db8:aaaa::9876:cafe
  The double colon can be used once per address, and can shorten as
  many or as few zeros as desired. Being able to only use it once per
  address is to avoid confusion on how many bits are zeros, as it would be
  impossible to be able to tell with 100% certainty how many consecutive
  chunks are zeros.
  The details for representing IPv6 addresses is described in detail in
  RFC 5952 in section
  4; in short that section describes:
  
  Removing leading zeros in chunks
  Use of the double colon to shorten addresses as much as possible by
  using it on the longest running chunk of zeros. The double colon must
  not be used on just one chunk of zeros.
  Use the lower case forms of characters a-f in an IPv6 address
  
  Special Purpose IPv6
  Addresses
  Much like IPv4, some addresses and address ranges have a special
  reservation, and cannot be used for another purpose.
  
  ::1 - Local host address. Unlike IPv4 which uses 127.0.0.0/8, IPv6
  allocates exactly 1 local host address.
  2001:db8::/32 and 3fff::/20 - Reserved for examples. The first
  address is recommended for most examples, however, for examples that
  need shorter than a /32 prefix, the second one has been reserved.
  
  Address Types
  The next detail to cover is address types and address prefixing. All
  IPv6 addresses are broken into two parts, the prefix and the Interface
  Identifier (IID). The prefix is the part of the address that identifies
  the subnet of the network; this is very similar to CIDR
  Notation for representing address ranges in IPv4. The IID is the
  part of the address that represents the specific device on the network.
  For privacy reasons (that being having an address that is not easily
  guessable) it is generally recommended that the IID be made of
  pseudo-random bits. One method of generating these pseudo-random bits
  was based around the MAC address of the device, however, that has since
  been deprecated.
  Unicast Addresses
  The most common type of addresses in the world of IPv6 is Routable
  Unicast Addresses; these tend to come in two varieties Globally Unique
  Addresses (GLA) and Unique Local Addresses (ULA) GUAs are intended to be
  publicly facing over the opened Internet and will usually belong to one
  of two glasses: Provider Independent (PI) and Provider Assigned (PA).
  Between the two, PA prefixes are usually preferred because all of the
  prefixes for a particular customer can be aggregated in one BGP-4
  announcement, while each new PI prefix will add to the global routing
  tables and reducing performance.
  ULAs are intended to be used locally and have the following
  benefits:
  
  They are self-allocated by a particular network for its own internal
  use.
  They are all under a /48 prefix that includes a locally assigned
  pseudo-random 40 bit part
  The MUST NOT be routed over the open Internet, so
  to remain private.
  
  ULA prefixes usually start with ‘fd’, with many networks using
  fd00::/48. This is not good practice though because the merging of two
  such networks will break things.
  Anycast Addresses
  Anycast addresses are, for all intents and purposes, identical to
  unicast addresses. Thus any GUA or ULA can be treated as an anycast
  address. A special case for this is an address with the IID set to zero.
  This addresses is for use by the subnet-router anycast address. For
  example:
  2001:db8:23:13fe::/64
  Link Local Addresses
  Link Local Addresses (LLA) are strictly for use within a local
  network and will never be forwarded by a router, but will be forwarded
  by a layer 2 switch. This function is essential to reach the first-hop
  router. LLAs are specific to their interfaces; hosts with multiple
  interfaces will have different addresses on each one.
  LLAs are shown by the zone at the end of the address, an example
  follows:
  fe80::1234:0:abcd:13:fefe%eth0
  Embedded IPv4 Addresses
  It is possible to embed IPv4 addresses into IPv6 addresses in certain
  circumstances, but is a bit outside the depth of this post. For more
  information on the topic, refer to RFC
  4038.
  Multicast Addresses
  All IPv6 multicast addresses are under the ff00::/8 prefix. That is,
  they all start with ff with the next 8 bits having a special meaning.
  These meanings are covered in subsection 2.7 of RFC 4291. #
  Address Configuration IPv6 has three main mechanisms for configuring
  addresses for end devices:
  
  Static Address Configuration
  Dynamic Host Configuration Protocol v6 (DHCPv6)
  StateLess Address AutoConfiguration (SLAAC)
  
  The static address configuration (manually setting an IP on each
  device) and DHCP operate in a very similar way to how they did in IPv4.
  SLAAC, however, is new and operates quite a bit differently to the other
  address configuration methods. Unfortunately, it is also one that cannot
  be ignored as it is very widely used by many end devices; for example,
  the Android operating system has no support for DHCPv6, only allowing
  for obtaining an IPv6 address via SLAAC.
  SLAAC was first introduces in RFC 1971 and was
  created because during the creation of IPv6, DHCP was not commonly
  deployed on networks the same way it is today. Additionally, it was
  intended to make configuring networking devices on simple networks much
  easier by not requiring a separate configuration protocol. This results
  in things like neighbor discovery and router advertisement being a
  requirement for even complex IPv6 networks.
  For SLAAC to work, the first-hop router on the network
  MUST listen to the link local all-routers multicast
  address, defined as ff02::2. New nodes will send a Router Solicitation
  ICMPv6 messages to that address. From there, each router will respond
  with a router advertisement; these router advertisements are rather
  complex, but are defined in RFC 4861 for
  those interested.
  SLAAC is something that must be supported by all IPv6 nodes in the
  event that they find themselves on a network where it is the only method
  of acquiring an address.
  Key Differences
  For a TL;DR, these are some of the key take aways re-phrased from Book
  6 in the ‘IPv6’ primary differences from IPv4 section:
  
  IPv6 uses 128 bit addresses compared to IPv4’s 32 bit.
  Network Address Translation (NAT) is commonly used on most IPv4
  LANs; IPv6 does have the ability to do NAT, however it is discouraged by
  the Internet Engineering Task Force (IETF), and does not have a
  specified standard.
  Multiple IPv6 addresses can exist on one interface, while IPv4 only
  allows one address per interface
  IPv6 has SLAAC in addition to DHCP and manually setting
  addresses
  Multicast is mandatory for the operation of IPv6, while it is not
  needed in IPv4.
  
  For much more in-depth material on learning IPv6, that is a great
  resource to get started. For even more information, going through the
  RFCs outlined in that book is also a wonderful way to learn more about
  the standards, and how the technology is supposed to work. Often times,
  the RFCs also include specific recommendations on how things should be
  managed and best practices for dealing with many of the complexities in
  these networks.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2025-11-25.html
  		https://foxide.xyz/projects/2025-11-25.html
  		Tue, 25 Nov 2025 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  The landscape of authentication has changed a lot over the past
  couple of decades; traditionally companies would setup user accounts on
  some sort of a local authentication service such as Active Directory and
  manage their user accounts and permissions there, however, today’s world
  is much more interconnected and security conscious than it used to be.
  Because of this many companies have moved on to Identification as a
  Service (IDaaS) products, such as Microsoft
  Entra and Google Cloud
  Identity. These products are built on top of open protocols such as
  OAuth
  and OpenID,
  and in the world of business, these products are vital in providing the
  three A’s of security, Authentication, Authorization, and Accountability
  to help keep companies secure from threat actors. What about the humble
  home-laber though? Yes, both Entra
  and Cloud
  Identity have a free tier, but those are proprietary online services
  that cannot be self-hosted. Thankfully there are tools that can be
  self-hosted so that you can become your own OAuth and IDaaS provider,
  the one I am going to cover is Keycloak, however there are other
  products that provide roughly the same capabilities.
  It is worth pointing out that there are a few topics in this project
  that are fairly big, as well as the project itself. Because of that, I
  am going to break this up into multiple parts to make writing a bit
  easier (potentially reading easier too). This part is going to go over
  the basics of OAuth and friends, what they do, how they are related, and
  the most basic setup of Keycloak to use it with another application.
  What are OpenID and OAuth?
  While understanding these concepts aren’t strictly required for
  setting up and running an OAuth system, I do believe that not having at
  least a basic understanding of those concepts is a disservice when doing
  so. Even beyond this project, these concepts create the foundation of
  modern identity management systems that are widely used in the industry
  today; learning and understanding how they work will be key for
  progressing within the field of IT, especially within security and
  operations roles.
  Let’s start by going over OAuth 2.0, which
  counter intuitively (at least to me) is an authorization protocol, not
  an authentication protocol. Specifically, let’s start by going over the
  problem that OAuth is attempting to solve:
  
  In the traditional client-server authentication model, the client
  requests an access-restricted resource (protected resrouce) on the
  server by authenticating with the server using the resource owner’s
  credentials. In order to provide third-party applications access to
  restricted resources, the resource owner shares its credentials with the
  third party. This creates several problems and limitations:
  
  Third-party applications are required to store the resource owner’s
  credentials for future use, typically a password in clear-text
  Servers are required to support password authentication, despite the
  security weaknesses inherent in passwords.
  Third-party applications gain overly broad access to the resource
  owner’s protected resources, leaving resource owners without any ability
  to restrict duration or access to a limited subset of resources.
  Resource owners cannot revoke access to an individual third party
  without revoking access to all third parties, and must do so by changing
  the third party’s password.
  Compromise of any third-party application results in compromise of
  the end-user’s password and all of the data protect by that
  password.
  
  
  
  RFC 6749
  
  To summarize, traditional access and authentication (usually
  passwords) are inseparable which causes problems when needing to revoke
  access to a resource or when dealing with a compromised system. OAuth’s
  answer to this problem is adding an authorization layer after the user
  authenticates, but before gaining access to server resources. Rather
  than authentication inherently granting access, the authentication will
  grant a token that will be accepted in place of authentication
  credentials. Think about this like buying tickets to a fair. You buy
  tickets at the gate, then use the tickets to gain access to resources
  (rides, games, etc.) within the fair. The most obvious benefit is being
  able to separate the authentication from the access to resources and
  removing the need for usernames and passwords to be stored on the
  servers, increasing security.
  However, as stated above, OAuth 2.0 does not actually handle
  authorization. So, what do you add to it to provide proper identity
  management? This is where OpenID
  Connect comes in; OpenID Connect (usually referred to as OpenID) was
  created specifically to be used with OAuth. From the OpenID spec:
  
  OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0
  protocol. It enables Clients to verify the identity of the End-User
  based on the authentication performed by an Authorization Server, as
  well as to obtain basic profile information about the End-User in an
  interoperable and REST-like manner.
  
  Combining these two protocols together allows for centrally managing
  users to use resources either hosted by your infrastructure or
  third-parties. This foundation enables Single Sign-On
  (SSO) and allows for users to sign-in to multiple services using one
  account, such as their Google, Microsoft, Apple, or Facebook
  account.
  Basic Setup
  Now that we have gone over the basics of what OAuth 2 and OpenID are,
  let’s actually do something. I am doing this project on a FreeBSD 15.0
  machine, however, the service is written in Java and should be fairly
  portable across operating systems. After install of OS, install
  dependencies and download the application files:
  # Dependency for Keycloak
  pkg install -y openjdk21
  fetch https://github.com/keycloak/keycloak/releases/download/26.5.0/keycloak-26.5.0.zip
  cd keycloak-26.5.0
  bin/kc.sh start-dev
  Now visit the IP of the server at port 8080, for example my server is
  located at 192.168.122.146, so I would go to
  http://192.168.122.146:8080. We unfortunately run into a problem here,
  when running the kc.sh script to setup the admin account it
  complains if we are not accessing it over localhost, which I am not in
  this case. There are a few different ways to handle this, but the
  simplest is going to forward X11 over ssh and run Firefox (or another
  browser) on the server. I explain this process in this blog post.
  Additionally, you will probably want to open multiple ssh sessions or
  use a terminal multiplexer such as screen to more
  easily run the kc.sh script and a remote Firefox.
  Now that we have are remote Firefox setup and can access the web
  interface over localhost, let’s try re-running the kc.sh
  script:
  cd keycloak-26.5.0 # Only necessary if not already in the directory
  bin/kc.sh start-dev
  After creating the initial admin account, we can start managing the
  ‘realms’, which is Keycloak’s term for what might be call a tenant in
  something like Microsoft Entra. This process is documented decently well
  in the getting
  started guide, however a quick rundown is:
  
  Login to the Admin Console
  Click “Manage Realms” and “Create realm” at the top
  Then fill out the required information for the realm
  Click the newly created realm to make it the “Current realm”
  
  From there, we can go to the ‘Realm Settings’ tab and modify the
  permissions and whatnot for the realm. For now, I am going to leave the
  realm settings alone, but this is where we would set things like
  allowing user registration. I am also going to go ahead and create some
  test users for when it is time to test out using Keycloak as an
  authentication mechanism. To do that, make sure the newly created realm
  (remember the ‘Master’ realm is intended to be used for administrative
  purposes, not for client use) is active, and go to ‘Users’ in the left
  sidebar and ‘Create new user’. Enter the user’s information (username is
  the only required field), then after the user is created click on
  ‘Credentials’ in the top banner. This is where we can set a password for
  the user to be able to authenticate; we can then try to login to the
  realm using that new account. To get to the realm’s console (which is
  different than the admin console so going to http://${DOMAIN}:8080 won’t
  work) is located at:
  http://DOMAIN : 8080/realms/{REALM}/account
  Where ${DOMAIN} is the domain name or IP address of the
  Keycloak server and ${REALM} is the name of the realm that
  was just created. It would also be beneficial for later steps to setup
  some sort of name resolution for the Keycloak server. This can be done
  pretty easily by adding an entry to /etc/hosts on Unix-like
  OSs or in the C:\Windows\system32\drivers\etc\hosts file in
  Windows. Alternatively, setting up DNS for the server would work as well
  (but is outside the scope of this project). Additionally, setting up
  name resolution on the server that will be using Keycloak will also be
  important later on.
  After confirming we can authenticate into our newly created realm,
  let’s setup a connection to be able to use this account to access other
  resources.
  Authenticating Forgejo
  with Keycloak
  Now we can finally do something cool and setup authentication to a
  service using Keycloak. For testing this, I decided to go with Forgejo
  as it is fairly easy to setup and has a nice interface for connecting
  external authentication services. I’m not going to go through the setup
  of Forgejo itself, just install like normal and create the initial admin
  account so we can login and start working with the service.
  We are then going to go back over to Keycloak and setup an
  Keycloak has a tool for
  testing the authentication, however, it’s much more fun to set up an
  application and see it work with Keycloak as an authentication method.
  Before we can actually integrate Keycloak as an authentication service
  though, we need to setup the OpenID client for it in Keycloak. To set up
  that client, login to Keycloak and select the realm that was created
  earlier to make it the active realm, then click ‘Clients’. Click the
  ‘Create client’ button which will take you into a menu to assist with
  creating that new client. You are only required to select a client type
  and give the client an ID, however, it is usually best practice to also
  give at least a description to assist future admins in understanding
  what that client does. In this instance, we will leave the client type
  as ‘OpenID Connect’ and give the client a simple name such as forgejo,
  then click next.
  The next screen will assist us in configuring the specific
  capabilities for the client. This can determine which features of the
  OAuth standard that this client will be capable of using for
  authorization. The only default setting we need to change here is
  turning the client authentication on, this will allow us to generate a
  client secret to add to Forgejo to be able to authenticate users. We
  also need to choose a Proof Key for Code Exchange (PKCE) method, I went
  with S256 as it is more secure and there is not really a reason to go
  with the less secure option here.
  At the login settings screen, we tell Keycloak more about the service
  that will be using on the client, i.e. telling Keycloak about Forgejo.
  Enter the root and home url, in my case both are:
  http://192.168.122.99:3000, then we also need to enter the “Valid
  redirect URIs” and “Valid post logout redirect URIs”. Forgejo will tell
  you what these should be, so let’s go back into Forgejo with out
  previously created admin account and begin the setup process over
  there.
  Once you get into Forgejo as an Admin go to your profile → Site
  administration → Identity & access → Authentication sources and ‘Add
  authentication source’. Change the authentication type to OAuth2 and the
  OAuth2 provider to OpenID Connect, the menu will start asking for some
  information. Ignore that for now, and scroll to the bottom and grab the
  callback/redirect URL under the ‘OAuth2 authentication’ section. The
  line you are looking for should read:
  
  When registering a new OAuth2 authentication, the callback/redirect
  URL should be: http://192.168.122.99:3000/user/oauth2//callback
  
  With a potentially different URL of course.
  Copy and paste that into the “Valid redirect URIs” and “Valid post
  logout redirect URIs” boxes. Once that is finished, click save and you
  will be redirected to the newly made client’s OpenID entry. In that
  menu, we can modify the settings we previously set if we want to change
  something, however, what we are currently wanting is under the
  ‘credentials’ tab. This tab is where we get our secret to be able to
  authenticate Forgejo as something that should be able to use Keycloak
  for user information. Copy the secret and paste it into the ‘Client
  secret’ in the Forgejo authentication setup menu, additionally type or
  paste the client ID that we created in the Client ID text field. Finally
  we need to add the OpenID Connect Auto Discovery URL. This URL is found
  at the bottom of the ‘Realm settings’ menu, the one we want is the
  ‘OpenID Endpoint Configuration’; a big gotcha I ran into was not having
  the domain name for the Keycloak server properly configured on the
  Forgejo server. That is why I recommended putting an entry in
  /etc/hosts, at least until we get Keycloak in a more
  production ready state. Paste that URL into Forgejo as the Auto
  Discovery URL, then make sure the authentication source is active and
  ‘Add authentication source’.
  From there, simply try to sign-in using the previously created
  Keycloak user. Assuming it works, we have a very basic, not production
  ready OAuth 2 server. The next post I plan on writing about how to get
  Keycloak in a more production ready state and potentially how to get
  passkeys to work.
  Resources
  OAuth and OpenID feel like a massive topic and running a server that
  provides those things certainly feels like it takes quite a bit to get
  it right. If this is something that you are seriously wanting to do, I
  recommend doing some learning on not only the Keycloak service, but also
  the underlying protocols. From there, start by creating a test lab
  before putting it into production because there have been quite a few
  things that I am still working on figuring out, such as how to create
  admin users for a realm rather than doing everything via the master
  realm.
  For those interested in learning more, here are some resources that I
  have used to learn about this and related topics.
  
  Auth0 - This site
  provides a very entry level overview of the topics and how they piece
  together. It also provides some references on deeper learning for people
  that want more.
  Keycloak
  Getting Started - Getting started guide that I have loosely been
  following to get this working
  Keycloak’s landing page
  for guides
  
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2026-01-22.html
  		https://foxide.xyz/projects/2026-01-22.html
  		Thu, 22 Jan 2026 00:00:00 -0500 
  		
  		
  		 
  		 
  			
  Abstract
  Secure Shell
  Protocol (SSH) is one of the most important communication protocols
  on the Internet; it is what allows for remote administration of a large
  majority of Unix and Unix-like operating systems across the open
  Internet. However, can also be used to create both forward and reverse
  tunnels to bind ports either to or from one machine to another. This
  might seem like something silly to do, but it can be extremely useful
  for adding a security layer to an insecure protocol or adding a layer of
  security when exposing local services to the Internet. While I have
  lightly touched on this topic in a previous blog
  post, this blog post is intended to dive a bit deeper into the topic
  to hopefully explain how to use the tunnels, and some use-cases that
  might benefit from them.
  Setup
  The setup for this project is fairly simple and only requires OpenSSH client and server. Another
  utility that might be useful for daemonizing the tunnel would be AutoSSH; the utility is
  best described from their GitHub description:
  
  autossh is a program to start a copy of ssh and monitor it,
  restarting it as necessary should it die or stop passing traffic. The
  original idea and the mechanism were from rstunnel (Reliable SSH
  Tunnel). With version 1.2 the method changed: autossh now uses ssh to
  construct a loop of ssh forwardings (one from local to remote, one from
  remote to local), and then sends test data that it expects to get back.
  (The idea is thanks to Terrence Martin.)
  
  AutoSSH will monitor the connection and try to re-open it upon
  failure. This is exactly what you would want for creating a service to
  make the tunnel persistent, however, we will come back to this utility
  later.
  Creating Basic Tunnels
  For setting up and testing these tunnels, we really just need two
  machines (VMs, containers, or jails work just fine for this). One
  machine will act as the “remote” and the other will act as the “local”;
  they will need the OpenSSH server and client respectively. Then for the
  tunnels to properly work, the following setting needs to be changed on
  the server in the sshd_config file (usually located at
  /etc/ssh/sshd_config).
  # Change from this line
  # GatewayPorts no
  
  # To this line
  GatewayPorts yes
  and finally restart the sshd service.
  # systemD Distros
  systemctl restart sshd
  
  # Void Linux
  sv restart sshd
  
  # FreeBSD and NetBSD
  service sshd restart
  
  # OpenBSD
  rcctl restart sshd
  Now our system is ready to securely forward ports.
  Forward SSH Tunnel
  Let’s look at a really basic example of tunneling a VNC connection to
  a local machine:
  ssh -L 3000:localhost:5900 ${USER}@${HOST}
  This command will set up an SSH connection and tunnel port 5900 on
  the remote machine to port 3000 on the local machine. Then, if you want
  to try to connect to the machine via VNC, you connect to
  localhost:3000 rather than
  ${REMOTE_ADDRESS}:5900. Fairly simple, and a much better
  way to connect to VNC over the opened Internet; however, it is not
  limited to just VNC. Rather we can tunnel any TCP port on the remote
  machine to a port on our local machine. From the ssh man
  page:
  
  -L [bind_address:]port:host:hostport -L
  [bind_address:]port:remote_socket -L local_socket:host:hostport -L
  local_socket:remote_socket Specifies that connections to the given TCP
  port or Unix socket on the local (client) host are to be forwarded to
  the given host and port, or Unix socket, on the remote side. This works
  by allocating a socket to listen to either a TCP port on the local side,
  optionally bound to the specified bind_address, or to a Unix socket.
  Whenever a connection is made to the local port or socket, the
  connection is forwarded over the secure channel, and a connection is
  made to either host port hostport, or the Unix socket remote_socket,
  from the remote machine.
  
  The documentation is fairly clear about what is actually occurring
  here, and the limitation of TCP ports makes sense since SSH is a TCP
  only protocol. While it is possible to do UDP over SSH, it is more
  complicated and not something that I am going to delve into on this
  post. The part that was unclear, at least for me, is the order the ports
  should go in on the command. In short, for the forward SSH tunnels (what
  the -L flag is used for) it is supposed to be local port,
  than remote port.
  # Generalized example
  ssh -L ${LOCAL_PORT}:localhost:${REMOTE_PORT} ${REMOTE_IP}
  
  # Forwarding remote port 8080 to local port 3000
  ssh -L 3000:localhost:8080 192.168.122.146
  The forward tunnel’s use-case is to secure vulnerable or sensitive
  remote services. Some examples might be things like VNC, a WordPress
  admin portal, or any other service that a bit too sensitive or insecure
  to share over the opened Internet.
  Reverse SSH Tunnel
  Now let’s look at the reverse tunnels. Again, from the ssh man
  page:
  
  -R [bind_address:]port:host:hostport -R
  [bind_address:]port:local_socket -R remote_socket:host:hostport -R
  remote_socket:local_socket -R [bind_address:]port Specifies that
  connections to the given TCP port or Unix socket on the remote (server)
  host are to be forwarded to the local side.
  This works by allocating a socket to listen to either a TCP port or
  to a Unix socket on the remote side. Whenever a connection is made to
  this port or Unix socket, the connection is forwarded over the secure
  channel, and a connection is made from the local machine to either an
  explicit destination specified by host port hostport, or local_socket,
  or, if no explicit destination was specified, ssh will act as a SOCKS
  4/5 proxy and forward connections to the destinations requested by the
  remote SOCKS client.
  
  Again, the documentation is fairly clear on what is going on, and
  again the limitation of TCP connections makes sense as SSH is still a
  TCP protocol. The main difference is the order the ports will be
  declared in the command, it will go the remote port, than the local
  port.
  # Generalized example
  ssh -R ${REMOTE_PORT}:localhost:${LOCAL_PORT}
  
  # Forwarding data from local port 8000 to the remote port 3000
  ssh -R 3000:localhost:8000 ${USER}@192.168.122.99
  This type of tunnel is going to be best used for something like
  securely exposing local services to the Internet. It can be thought of
  as an alternative to a service like a Cloudflare
  tunnel or a Tailscale
  Funnel, but are not cloud based services and can be fully
  self-managed.
  Creating Tunnels as Services
  While the previous commands are easy enough, there are a few things
  that can be polished a bit. The first part is after the SSH tunnel is
  established, we don’t really have a reason to have a shell. Thankfully,
  the open SSH client already has an option for this, -N.
  From the man page:
  
  -N Do not execute a remote command. This is useful for just
  forwarding ports. Refer to the description of SessionType in
  ssh_config(5) for details.
  
  Perfect, just add that flag to any of the previous commands, and it
  will only set up the tunnel, rather than starting a shell.
  # This command
  ssh -R 3000:localhost:8000 ${USER}@192.168.122.99
  
  # Becomes this command
  ssh -N -R 3000:localhost:8000 ${USER}@192.168.122.99
  
  # Though order is not important, this command is equivalent
  ssh -R 3000:localhost:8000 ${USER}@192.168.122.99 -N
  The next issue is being able to set up these tunnels as services and
  maintaining that connection. SSH doesn’t have a built in way to monitor
  and maintain the connection. We could certainly use something like the
  daemon
  utility in FreeBSD, but if the connection died, it would not be able to
  automatically recover.
  This is where a tool like autossh comes in. From
  the man
  page:
  
  autossh is a program to start a copy of ssh and monitor it,
  restarting it as necessary should it die or stop passing traffic.
  
  
  The original idea and the mechanism were from rstunnel (Reliable SSH
  Tunnel). With version 1.2 the method changed: autossh now uses ssh to
  construct a loop of ssh forwardings (one from local to remote, one from
  remote to local), and then sends test data that it expects to get back.
  (The idea is thanks to Terrence Martin.)
  
  The main thing that is different with autoSSH is that we also need to
  provide a port that is separate for monitoring the connection. This
  means the stated port will need to be able to receive traffic on
  whatever firewall is on the remote server. Here is an example of an
  autoSSH command:
  # Monitor on port 6500, then normal SSH commands and flags after
  autossh -M 6500 -N -R 3000:localhost:8000 192.168.122.146
  Additionally, it has a flag for running in the background,
  -f; so our previous command can be changed into:
  # Monitor on port 6500, then normal SSH commands and flags after
  autossh -f -M 6500 -N -R 3000:localhost:8000 192.168.122.146
  Now we need to make sure that the service user can login without a
  password using SSH keys. For this, simply generate the key using
  ssh-keygen then copy the contents of the generated public
  key into /home/${SERVICE_USER}/.ssh/authorized_keys; test
  using SSH to confirm it worked. If you had to enter a password, it will
  not work as there is not a good way to make a service do that.
  Finally, we would just need to create a service. I put the following
  in /etc/systemd/system/tunneld.service and it worked for my
  testing:
  [Unit]
  Description=AutoSSH tunnel service
  After=network.target
  
  [Service]
  User=${SERVICE_USER}
  Group=${SERVICE_GROUP}
  Environment="AUTOSSH_GATETIME=0"
  ExecStart=/usr/bin/autossh -M 6500 -i /home/${SERVICE_USER}/.ssh/id_ed25519 -N -R 8000:localhost:8096 ${SERVICE_USER}@192.168.122.146
  Restart=always
  
  [Install]
  WantedBy=multi-user.target
  Obviously replace ${SERVICE_USER} and
  ${SERVICE_GROUP} with the relevant service user and group
  for the service. After that, run the following commands to enable and
  start the service:
  # Run as root to enable service
  systemctl enable tunneld
  
  # Start service
  systemctl start tunneld
  Obviously creating services on other init systems are possible, but
  will be left as an exercise for the reader.
  
  			]]> 
  		 
  		https://foxide.xyz/projects/2026-02-18.html
  		https://foxide.xyz/projects/2026-02-18.html
  		Wed, 18 Feb 2026 00:00:00 -0500