Abstract

Modern Unix-like operating systems have a variety of ways of getting software. The most common method is probably the package manager of the operating system you are using; however, some other options are:

Containers, such as Snaps, Flatpaks, or AppImages
Build recipes, such as ports for the BSDs or the AUR for Arch Linux
Downloading the binaries directly, such as with Chrome

In addition to those above options, there is the option to build from source (at least with software the source is available). While doing this may not be extremely useful for many desktop Linux users, many jobs that involve working with Linux will want some experience in building (and maybe patching) software to run on the system. This article is a quick introduction into some common ways to get , compile, and patch source code. It also aims to cover the logic behind each of these steps for the case that they WILL fail, as building software from source is generally not a something that can be copy and pasted from one application’s source to another. As an example for this write up, I am going to be using a fairly simple project that is relatively easy to build. The project is a redshifting program called SCT, and uses a bit of C to change the color temperature of your screen.

I also want to go ahead and thank my co-worker, William, for helping proof-read this post ahead of time as he pointed out some technical issues that I did not initially catch; specifically, that I should mention the required dependencies for SCT to compile and take some guesswork out, as well as some vague instructions towards the beginning of the post.

Obtaining Source Code

The first hurdle in dealing with source code is actually obtaining it. As far as the ‘how’, it really depends on what you are trying to build. That being said, many projects have either a read-only or an unofficial mirror on sites like GitHub.

For this example, there are two ways to go about getting the source code. Option 1: Click on the ‘Code’ button towards the top right of the GitHub page, then click ‘Download ZIP’ at the bottom of the small dialog window, or Option 2: use git to download the software. Both options work, however, using git will allow you to keep track of software changes (future updates, or custom patches added by you) easier than the ZIP file will.

# One option for opening the ZIP archieve if that was the chosen method
7z x sct-master.zip

# Alternatively, the command if git was used
git clone https://github.com/faf0/sct

From there, simply cd into the sct directory and we can begin the next step.

Building the software

Before building the software, we need to make sure we have all the required dependencies; I am not going to list the command to install the required dependencies for all Linux distros ever, but I will provide what you would need for Debian/Ubuntu/Mint. If you do not use one of those, you can search for similar package names in your package manager of choice and you should find it. Debian also does not, at least by default, include a compiler or make, so that has to be installed as well.

apt install -y libx11-dev libxrandr-dev make gcc

Now that we have the required libraries, we have to compile (or build) the software. This step is where things can vary heavily depending on a lot of different factors such as build system, language used, dependencies, target operating system, etc. Most projects will have instructions in their README file on what the dependencies are and roughly how to build it. However, it is important to read it very carefully as minor details can end up being the difference between successfully compiled software and hours of incredibly frustrating troubleshooting as to what is causing it to SEGFAULT. SCT happens to be a fairly easy software to build and can simply be built with make in the project directory. Once the software is built, it can be installed by running make install (again in the project directory). Wonderful, but what did we learn? Really nothing… So, let’s backup, what is make and why does it install stuff on my system?

The wonderful world of Makefiles

Make is a command line tool that is often used to automate building software, although it is not limited to just building software. Going over the ins-and-outs of all the things make can do is well outside the scope of this one blog post, however, if the reader is interested in learning more about the topic the GNU Make man page is a great place to start. This post will go into some general ideas to get started with reading a basic Makefile (configuration file for what make will do when invoked).

Getting back to the actually getting code compiled, let’s begin by looking at what the manual compilation command looks like, as that is what make will end up doing:

gcc -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include xsct.c -o xsct -L /usr/X11R6/lib -lX11 -lXrandr -lm -s

That looks super annoying to type each time you want to compile something; thankfully make can automate that. Let’s look at the Makefile to see what is going on in there:

CC ?= gcc
CFLAGS ?= -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include
LDFLAGS ?= -L /usr/X11R6/lib -s
PREFIX ?= /usr
BIN ?= $(PREFIX)/bin
MAN ?= $(PREFIX)/share/man/man1
INSTALL ?= install

PROG = xsct
SRCS = src/xsct.c

LIBS = -lX11 -lXrandr -lm

$(PROG): $(SRCS)
    $(CC) $(CFLAGS) $^ -o $@ $(LDFLAGS) $(LIBS)

install: $(PROG) $(PROG).1
    $(INSTALL) -d $(DESTDIR)$(BIN)
    $(INSTALL) -m 0755 $(PROG) $(DESTDIR)$(BIN)
    $(INSTALL) -d $(DESTDIR)$(MAN)
    $(INSTALL) -m 0644 $(PROG).1 $(DESTDIR)$(MAN)

uninstall:
    rm -f $(BIN)/$(PROG)
    rm -f $(MAN)/$(PROG).1

clean:
    rm -f $(PROG)

Anyone familiar with shell scripting may be able read and kind of see what the file is doing. However, not everyone is already used to shell scripting, so let’s add some comments to make it a bit more clear for those that are not as familiar:

# This section is just setting variables that will be used later
# Variables set with just '?=' are conditional variables
# that only matter if the variable is not already set by the environment.
# variables set with '=' are ones that will be used verbatim

# This variable is just setting the compiler, gcc in this case
CC ?= gcc

# This variable is setting some C compilation flags
CFLAGS ?= -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include
LDFLAGS ?= -L /usr/X11R6/lib -s

# This variable is setting the prefix for where things should be installed
# This doesn't /really/ matter, but on BSD systems, would probably
# be changed to `/usr/local`
PREFIX ?= /usr
# The $(PREFIX) is how the variable can be recalled
# In this case, it is just being called while setting another
# variable, which is quite common in Makefiles
BIN ?= $(PREFIX)/bin
MAN ?= $(PREFIX)/share/man/man1
# Install is a program in Linux and many other Unix-like systems
# You can check whether install is a program on your machine
# by running `which install`, my computer shows it is installed
# at `/usr/local/bin/install`
INSTALL ?= install

PROG = xsct
SRCS = src/xsct.c

# This variable is setting the libraries that must be invoked
# during compilation, in this case:
# X11 libraries, Xrandr libraries, and math libraries
LIBS = -lX11 -lXrandr -lm

# This part of the Makefile is where the compilation happens
# I will go through the specifics of this in a minute
$(PROG): $(SRCS)
    $(CC) $(CFLAGS) $^ -o $@ $(LDFLAGS) $(LIBS)

# This section is describing what `make` should do if
# `make install` is called
install: $(PROG) $(PROG).1
    $(INSTALL) -d $(DESTDIR)$(BIN)
    $(INSTALL) -m 0755 $(PROG) $(DESTDIR)$(BIN)
    $(INSTALL) -d $(DESTDIR)$(MAN)
    $(INSTALL) -m 0644 $(PROG).1 $(DESTDIR)$(MAN)

# This section is describing what `make` should do if
# `make uninstall` is called
uninstall:
    rm -f $(BIN)/$(PROG)
    rm -f $(MAN)/$(PROG).1

clean:
    rm -f $(PROG)

Often times, a large majority of a Makefile is just getting variables set properly, this example is no different. Let’s substitute the variables in the section that builds the software so it is a bit more straightforward to read:

xsct: src/xsct.c
    gcc -Wall -Wextra -Werror -pedantic -std=c99 -O2 -I /usr/X11R6/include src/xsct.c -o xsct -L /usr/X11R6/lib -lX11 -lXrandr -lm

After all of the variables are replaced with what they will evaluate to (I know I glossed over the $^ and $@, I will come back to those), the build command in the Makefile looks almost identical to the manual build command.

Applying and Creating Patches for Software

When working with source code, you may eventually come across something that you want changed, modified, or removed. With proprietary applications this is next to impossible, however, with open source projects it is allowed and even encouraged. It is worth pointing out that while patch management is not required if the changes are only for you; utilizing version control and patching tools will make this process much easier even if you do not plan on contributing your changes back to the project (though you should if they are relevant).

Using Git

There are a lot of ways to go about version control and managing patches, however, I am only going to go over two. The first one will be some basic use of Git, and the second one will be more traditional Unix tools. I am going to start with Git as it is the industry standard for version control and managing patches by far. So, going back to the SCT project, let’s download it using git if we have not already done so (may need to install git on your machine). After that, we are going to make a new branch that we can make changes to.

# 'clone' or download the repository (and history) using git
git clone https://github.com/faf0/sct
cd sct
# Add new branch 'new-feature'
git branch new-feature
# Switch to branch 'new-feature'
git checkout new-feature
# Alternatively, you can create and switch to the new branch 'new-feature'
# in one command by running
git checkout -b new-feature

Now that we have a new branch, we can begin making modifications to the code. Just to make a simple example of this, I will be adding a line to the README.md file. To get the diff of the changes I have made versus what the master branch has, simply run:

git diff master

The results of this, at least for the change I made, are as follows:

diff --git a/README.md b/README.md
index afc027f..00b24df 100644
--- a/README.md
+++ b/README.md
@@ -21,6 +21,7 @@ Minor modifications were made in order to get sct to:
 - return `EXIT_SUCCESS`

 # Installation
+This is a new line to the README file.

 ## Make-based

To make a proper patch of the changes that were made, we need to commit the changes to the new branch, then make a patch file. A ‘commit’ is the way of telling git that you are happy with the changes and you would like them to be submitted into the tree. Essentially, this is the way to ‘save’ the file and be able to track those specific changes in the version history. Making a commit and a patch file can be done by running:

# This command adds the new or modified files to be commited
git add README.md
# This command makes the commit, the argument '-m' will make
# the commit message inline. Without this argument
# git will open your $EDITOR for the commit message to be
# typed out
git commit -m "Some helpful message"
# This command will format the patch file and compare
# the current commit branch to the master branch
git format-patch origin/master

The output of the last command will be a file that contains the diff as well as the commit message for the commits ahead of the master branch. Essentially, just describing the differences between the current branch and the master branch. From there, that patch file can be stowed away, emailed, or otherwise shared. Now let’s see how to apply that patch to the code base:

# Switches back to the master branch to 'undo' our changes
git checkout master
# Creating a test branch, it is bad practice to modify the master branch directly
git checkout -b apply-test
# Then apply the patch, it is worth noting that the .patch file
# may have a different name that what I have here
git apply 0001-Test-change.patch

This is the absolute basics of dealing with patches in git; there is a lot about git that I did not cover, because it would make this blog post a medium length novel. So, rather than boring you, I will just provide places to get more information on working with Git and open source code:

Git documentation
How to contribute to open source
Many projects have a ‘how to contribute’ section in their documentation

Manual way of dealing with patches

In the interest of history, I am also going to give a quick overview of how to deal with patches without using git. This is not something I recommend doing without a reason as it is quite a bit more painful, at least in my opinion. It also does not transfer to many open source projects the same way that learning git does; however, it is important to keep the old ways in mind.

So, let’s go back before any changes were made to any files for the application. Before we actually work on any changes, we need to copy the file(s) that we are going to work on to another file; generally, the naming convention is to just append the filename with ‘.orig’, but anything will work as long as you keep up with it. After making the desired changes (will be same changes as the previous section) you can run the following command:

# The '-u' flag is to make the output of the diff
# look more similar to the git output, I find it easier
# read than the default output that diff uses
diff -u README.md.orig README.md
# To save the output into a file, simply use the Unix file redirection
diff -u README.md.orig README.md > README.md.patch

The patch file can then be sent and shared around similar to the patch file using git. Now how do we apply these patches to a code base? With a utility called patch, that will likely not be on the machine by default, thus will need to be installed. Once installed, patches can be applied as follows:

# Notice the '<' rather than the '>', if the '>' is used, the program will just hang
patch < README.md.patch README.md

Closing thoughts

Working with source code can be intimidating at times, especially if you are not used to doing it. Hopefully, this post helps alleviate some of the fears and concerns with working with source code, and with making patches to software.