When people discuss the types of problems they must often cope with when working in the cloud,
the "noisy neighbor" is often near the top of the list.
The basic problem this term refers to is that other applications running
on the same physical system as yours can have a noticeable impact on your performance and resource availability.
Virtual machines have the advantage that you can easily and very tightly control how much memory and CPU,
among other resources, are allocated to the virtual machine.
When using Docker, you must instead leverage the cgroup functionality
in the Linux kernel to control the resources that are available to a Docker container.
—Sean P. Kane & Karl Matthias, Docker Up & Running, 2nd Edition
Docker is widely used to encapsulate services and applications that run as daemons, i.e., that run in the background without being attached to a terminal. This is useful for a wide variety of applications, such as web apps, databases, log servers, and many more. Today, you will build a simple filesystem event logging server, contained in a Docker image, then run it as a background service. This will enable you to experiment with the various management functions that Docker provides.
In this studio, you will:
inotify, writing events to a given file instead of standard outputcgroupsPlease complete the required exercises below. We encourage you to please work in groups of 2 or 3 people on each studio (and the groups are allowed to change from studio to studio) though if you would prefer to complete any studio by yourself that is allowed.
As you work through these exercises, please record your answers, and when finished upload them along with the relevant source code to the appropriate spot on Canvas.
Make sure that the name of each person who worked on these exercises is listed in the first answer, and make sure you number each of your responses so it is easy to match your responses with each exercise.
As the answer to the first exercise, please list the names of the people who worked together on this studio.
First, you will write a basic filesystem event logging program that Docker will encapsulate into a background service.
On your Raspberry Pi, write a user-space C program that takes two command line arguments.
If it is run with fewer or more arguments it should generate a helpful usage message showing how to run it correctly,
then enter an infinite loop.
The program should initialize an inotify instance,
then add a watch for all events involving the directory passed to it as the first command line argument.
Instead of printing inotify events to standard output,
the program will instead print events to a file specified by the second command-line argument.
It should attempt to open the file for writing,
and print an error message and enter an infinite loop if this does not succeed.
It should, in a loop, read from the inotify file descriptor,
iterate through all the inotify_events
that were obtained by each read and print out each event's
watch descriptor, mask, cookie, name field length, and (if that length is non-zero) name.
For each event the program also should also print out a descriptive message for each
inotify event type that is set in the event's mask.
NOTE: Any errors that this program encounters should trigger an informative error message, but instead of exiting, the program should enter an infinite loop. This would typically not be a good practice in a production environment, but it will enable the container to stay active even if the program produces an error, which is the desired behavior for the purposes of this studio.
HINT: Reuse code from the Observing File System Events studio.
This program should be similar to the one you wrote for that studio,
though no multiplexing is needed for this studio.
Also, consider using the fprintf() (with fflush()) function instead of printf()
to output to the specified log file.
Create a directory to monitor, then test your program: compile and run it in one terminal window, and then, in a second terminal window, create a file in the watched directory. As the answer to this exercise, please show the contents of the log file that your monitoring program produced.
Like in the previous studio, create a directory for your image. Inside of that directory, create an empty Dockerfile, then a subdirectory for your application. Copy your program into that subdirectory.
Your Dockerfile is going to inherit from the alpine-gcc
image you created in the previous studio.
This time, however, you're going to construct the Dockerfile
so that it uses the conventional practice of always inheriting the latest version of an image.
To do so, you will create a tag for the image that serves as an alias to the latest version.
To see all of the images installed on your system that have the name "alpine-gcc", issue the command:
docker image list "alpine gcc"
Identify the latest version of the image (it's likely that there's only one listing, i.e., the "v0" that you created last time). Then, issue the command:
docker tag alpine-gcc:<TAG> alpine-gcc:latest
(Replace <TAG> with the listed tag value.)
Now, open your Dockerfile, and add the following lines (replacing the names of your program and its subdirectory as appropriate):
FROM alpine-gcc:latest as builder
COPY app /app
RUN gcc /app/inotify-logger.c -o /inotify-logger
This should look familiar, but notice the addition of the "as builder" directive. This allows your Dockerfile to define a multi-stage image build. The lines you've provided comprise the first image stage, named "builder".
Now, add the following additional lines to your Dockerfile:
FROM alpine:latest
COPY --from=builder /inotify-logger /inotify-logger
CMD /inotify-logger
Notice that the first stage inherits from the alpine-gcc image,
which, as you saw in the previous studio, is relatively large and contains many files.
The second stage (which defines the final image that is produced)
will derive from alpine; since this image doesn't have
gcc or a C library, it is much lighter-weight.
The second stage copies the compiled hello-world
binary from the heavier builder image,
then creates a lighter image that just contains the base Alpine Linux and the binary.
It then launches the binary as its default command.
Build your image:
docker build -t inotify-logger:v0 .
Then, run it in the background using the -d flag:
docker run -d inotify-logger:v0
Now, you can view the running container, and the name that Docker automatically generated for it,
using the docker ps command.
Unless you specify a name for your container
(which you can do with the docker run --name parameter),
Docker will automatically generate a name for your container,
consisting of an adjective, then the last name of a famous (and possibly historical) person.
Once you've identified your container, export its files:
docker export <CONTAINERNAME> -o mycontainer.tar
Look at the contents of the file with the following command:
tar -tvf mycontainer.tar | less
As the answer to this exercise, please (1) show the output of the docker ps command,
(2) report how many files are in the container
(e.g., by piping tar -tvf mycontainer.tar into the wc wordcount utility),
then (3) report the size of the container (i.e., the size of the exported tar file).
Please also (4) say how that compares to the size of the container you created in the previous studio.
Once you've finished this exercise, kill the container:
docker kill <CONTAINERNAME>
(You can also use the container ID instead of the container name as the target of docker kill.)
Your filesystem event monitoring service needs a way to access the directory it is supposed to monitor.
When you run your container, you can issue a command to Docker
to bind-mount the directory you created earlier for your program to watch into the container,
by using the -v flag. This flag must be followed by a string parameter,
which takes the form /host/directory:/container/directory.
If you want the mount to be read-only, add :ro as a suffix to this string.
Launch a container of your image, again in the background, but this time (1) bind-mount the directory you created into a location at the root of your container, specifying a read-only mount, then (2) provide appropriate arguments to your container so that your program monitors this mount and writes output to a file you specify at the root of the container. For example, the command might look like:
docker run -d -v /home/pi/studio12/monitor:/monitor:ro inotify-logger:v0 /inotify-logger /monitor /log.txt
Notice that, even though you specified a default container command in the Dockerfile for this image,
the docker run command allows you to override it,
so you can provide the appropriate command-line arguments to your program.
Now that your container is running in the background, navigate into the directory that is being monitored, and trigger a simple filesystem event (e.g. create a new file, modify an existing file, etc.).
To view the contents of the log file,
you will need to enter the container.
First, retrieve the container name or ID with docker ps.
Then, use the docker exec command to launch a shell in the container:
docker exec -it CONTAINERNAME /bin/sh
From the shell inside the container, print the contents of the log file that your program has generated. As the answer to this exercise, please show those contents.
Keep your container running for the next exercise!
HINT: if the log file is not where you expect it to be, or it doesn't contain the events you expected to see, it's possible that your program has generated an error. Even though your container was launched in the background, disconnected from any terminal, any error messages it printed would have been captured by Docker's log. To view the Docker log for your container, issue the command:
docker logs CONTAINERNAME
Docker is an example of a client-server architecture:
when you run a command, the Docker client communicates to the Docker server
(which may not be on the same machine, although, for this studio, it is).
This means that if there is a problem with the daemon that handles requests from Docker clients,
you may not be able to connect to the container using the docker exec command.
However, a Docker container is still a container in the traditional sense:
you can join it with the setns() syscall.
The nsenter command-line utility provides a wrapper for the setns() syscall.
It allows you to execute a command in the namespace(s) of a given process.
To retrieve the PID of your container's "init" process,
issue the following command:
docker inspect --format \{{.State.Pid\}} CONTAINERNAME
Once you have retrieved the PID, launch a shell in the container with the command:
sudo nsenter --target PID --mount --uts --ipc --net --pid /bin/sh
Once you have joined the container this way, verify the contents of the log file match what you saw in the previous exercise. As the answer to this exercise, again show the contents of the log file.
Once you've finished this exercise, kill the container:
docker kill <CONTAINERNAME>
A Docker container's filesystem is, by default, transient: when your container is killed, the log file it generated disappears. To create persistent storage, and allow storage to be shared among containers, you can create Docker volumes. Issue the following command to create a volume named "logstorage":
docker volume create logstorage
Verify that the volume has been created by issuing the following command, which lists all volumes on the Docker server:
docker volume list
A Docker container can be launched that contains both bind mounts and mounted volumes.
To mount a volume, you use the --mount flag,
which must be followed by a string parameter
taking the form source=volumename,target=/container/directory.
Launch your image with the monitored directory bind-mounted into the container,
and mount the logstorage volume you created. Pass the appropriate command
so that your application writes its log to the mounted volume.
For example, the command might look like:
docker run -d -v /home/pi/studio12/monitor:/monitor:ro --mount source=logstorage,target=/logstorage inotify-logger:v0 /inotify-logger /monitor /logstorage/log.txt
With your container running in the background,
again navigate into the directory that is being monitored
and trigger a simple filesystem event.
Then se either the docker exec command or the
nsenter utility to launch a shell in the container.
From the shell inside the container, print the contents of the log file
that your program has generated.
As the answer to this exercise, please show those contents.
Also show the output of the docker volume list command.
Make a copy of the directory containing your Dockerfile and program.
Modify the copy of your program so that,
after it parses each inotify_event struct,
it attempts to allocate several megabytes of memory.
Build an image based on this new program, then run it in the background as you did in the previous exercise. Issue the command:
docker stats CONTAINERNAME
This will show you several statistics about your container, including its CPU and memory usage. It also reports the memory limit that has been applied to the container; since no limit was specified, the limit should be equal to the system's available memory.
Now, open another terminal window (keeping the statistics feed open in the first window). Navigate into the monitored directory on the host, trigger a filesystem event, then observe how much the container's memory usage increases.
Next, constrain your containers memory to a maximum value such that two more triggered filesystem events
will cause it to reach this threshold. You can use the docker update command.
A memory constraint can be defined with the --memory="" flag,
which takes a string parameter that includes a unit (one of b, k, m, or g).
You will also need to set a total memory limit, including swap space,
to a value equal to the memory constraint, using the --memory-swap="" flag.
So, your command might look like:
docker update --memory="8M" --memory-swap="8M" CONTAINERNAME
Observe the change to the output of docker stats in the first terminal window.
In your second terminal window, where you just set the limit,
you will want to verify that the limit was actually applied with the corresponding
cgroup interface file. Issue the following command to get the PID
of the container's init process:
docker inspect --format \{{.State.Pid\}} CONTAINERNAME
Then, find its cgroup directory by inspecting the
/proc/PID/cgroup file.
Navigate to the listed subdirectory of the /sys/fs/cgroup
filesystem. Then, inspect the contents of the memory.max
file.
As the answer to this exercise, please (1) write the docker update
command that you issued to set the memory limit,
(2) show the current output of docker stats,
and (3) copy the contents of memory.max.
Keep your container, and the terminal window with the docker stats output,
running for the next exercise!
With docker stats still producing output in the first terminal window,
and with the second terminal window still open,
open a third terminal window.
In the second terminal window, issue the command:
docker events
In the third window, navigate back to the directory that is being monitored.
Trigger several filesystem events in that directory,
observing how the output of docker stats changes each time.
Continue doing this until the threshold has been passed and the container process is killed.
As the answer to this exercise, please (1) report the final output of docker stats
and (2) show any output from docker events.
Page updated Thursday, March 3, 2022, by Marion Sudvarg and Chris Gill.