• It feels like a VM, but it is not.
• Uses the host Kernel.
• Cannot boot a different Operating System.
• Can’t have its own modules.
• Does not need “init” as PID (Process id) as “1”

LXC (LinuX Containers) technology was developed long ago and is an Operating System level virtualization technology. This was existing from the days of BSD and System-V Release 4 (Popular Unix flavors during 1980-90’s). But until recently, no one new how much it can help us in saving in terms of resource utilization. Because of this technology change, all enterprises are moving towards adoption of virtualization (be it Cloud or be it Docker containers). This also helped in better management of OpEX(Operational expenditures) and CaPEX(Captial expenditures) costs. Using this technique, we can create and run multiple and isolated Linux virtual environments on a single Linux host machine (called control host). LXC mainly uses Linux’s cgroups and namespaces functionalities, which were introduced in version 2.6.24(kernel version) onwards. In parallel many advancements in hypervisors happened like that of KVM, QEMU, Hyper-V, ESXi etc. Especially KVM (Kernel Virtual Machine) which is core of Linux OS, helped in this kind of advancement.

Difference between LXC and LXD is that LXC is the original and older way to manage containers but it is still supported, all commands of LXC starts with “lxc-“ like “lxc-create” & “lxc-info“, whereas LXD is a new way to manage containers and lxc command is used for all containers operations and management.

All of us know that “Docker” utilizes LXC and was developed using Go language, cgroups, namespaces and finally the Linux Kernel itself. Complete Docker has been built and developed using LXC as the basic foundation block. Docker is completely dependent on underlying infrastructure & hardware and using the Operating System as the medium. However, Docker is a portable and easily deployable container engine; all its dependencies are run using a virtual container on most of the Linux based servers. Groups, and Namespaces are the building block concepts for both LXC and Docker containers. Following are the brief description of these concepts.

C Groups (Control Groups)

With Cgroups each resource will have its own hierarchy.

• CPU, Memory, I/O etc will have their own control group hierarchy. Following are various characterics of Cgroups,
•  Each process is in each node
• Each hierarchy starts with one node
• Initially all processes start at the root node. Therefore “each node” is equivalent to “group of processes”.
• Hierarchies are independent, ex: CPU, Block I/O, memory etc

As explained earlier there are various Cgroup types as listed below,

1) Memory Cgroups

a) Keeps track of pages used by each group.

b) File read/write/mmap from block devices

c) Anonymous memory(stack, heap etc)

d) Each memory page is charged to a group

e) Pages can be shared across multiple groups

2) CPU Cgroups

a) Track users/system cpu time

b)  Track usage per CPU

c) Allows set to weights

d) Can’t set cpu limits

3) Block IO Cgroup

a) Keep track of read/write(I/O’s)

b) Set throttle (limits) for each group (per block device)

c) Set real weights for each group (per block device)

4) Devices Cgroup

a) Controls what the group can do on device nodes

b) Permission include /read/write/mknode

5) Freezer Cgroup

a) Allow to freeze/thaw a group  of processes

b) Similar to SIGSTOP/SIGCONT

c) Cannot be detected by processes

NameSpaces

Namespaces provide processes with their own system view. Each process is in name space of each type.

There are multiple namespaces like,

• PID – Process within a PID name space only see processes in the same PID name space
• Net – Processes within a given network namespace get their own private network stack.
• Mnt – Processes can have their own “root” and private “mount” points.
• Uts –  Have container its own hostname
• IPC – Allows processes to have own IPC semaphores, IPC message queues and shared memory
• USR – Allows to map UID/GID

Installation and configuration of LXD containers

To have LXD installed on Ubuntu system (18.04 LTS) , we can start with LXD installation using below apt command

shashi@linuxtechi:~$ sudo apt update
shashi@linuxtechi:~$ sudo apt install lxd -y

Once the LXD is installed, we can start with its initialization as below, (most of the times use the default options)

shashi@linuxtechi:~$ sudo lxd init

Once the LXD is initialized successfully, run the below command to verify information

shashi@linuxtechi:~$ sudo lxc info | more

Use below command to list if there is any container is downloaded on our host,

shashi@linuxtechi:~$ sudo lxc image list
+-------+-------------+--------+-------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+------+------+-------------+
shashi@linuxtechi:~$

Quick and easy way to start the first container on Ubuntu 18.04 (or any supported Ubuntu flavor) use the following command. The container name we have provided is “shashi”

shashi@linuxtechi:~$ sudo lxc launch ubuntu:18.04 shashi
Creating shashi
Starting shashi
shashi@linuxtechi:~$

To list out what are the LXC containers that are in the system

shashi@linuxtechi:~$ sudo lxc list
+--------+---------+-----------------------+-----------------------------------------------+------------+-----------+
|  NAME  |  STATE  |         IPV4          |                     IPV6                      |    TYPE    | SNAPSHOTS |
+--------+---------+-----------------------+-----------------------------------------------+------------+-----------+
| shashi | RUNNING | 10.122.140.140 (eth0) | fd42:49da:7c44:cebe:216:3eff:fea4:ea06 (eth0) | PERSISTENT | 0         |
+--------+---------+-----------------------+-----------------------------------------------+------------+-----------+
shashi@linuxtechi:~$

Other Container management commands for LXD are listed below :

Note: In below examples, shashi is my container name

How to take bash shell of your LXD Container?

shashi@linuxtechi:~$ sudo lxc exec shashi bash
root@shashi:~#

How Stop, Start & Restart LXD Container?

shashi@linuxtechi:~$ sudo lxc stop shashi
shashi@linuxtechi:~$ sudo lxc list
+--------+---------+------+------+------------+-----------+
|  NAME  |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+--------+---------+------+------+------------+-----------+
| shashi | STOPPED |      |      | PERSISTENT | 0         |
+--------+---------+------+------+------------+-----------+
shashi@linuxtechi:~$
shashi@linuxtechi:~$ sudo lxc start shashi
shashi@linuxtechi:~$ sudo lxc restart shashi

How to delete a LXD Container?

shashi@linuxtechi:~$ sudo lxc stop  shashi
shashi@linuxtechi:~$ sudo lxc delete  shashi
shashi@linuxtechi:~$ sudo lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
shashi@linuxtechi:~$

How to take snapshot of LXD container and then restore it?

Let’s assume we have pkumar container based on centos7 image, so to take the snapshot use the following,

shashi@linuxtechi:~$ sudo lxc snapshot pkumar pkumar_snap0

Use below command to verify the snapshot

shashi@linuxtechi:~$ sudo lxc info  pkumar  | grep -i Snapshots -A2
Snapshots:
  pkumar_snap0 (taken at 2019/08/02 19:39 UTC) (stateless)
shashi@linuxtechi:~$

Use below command to restore the LXD container from their snapshot

Syntax:

$ lxc restore {container_name} {snapshot_name}

shashi@linuxtechi:~$ sudo lxc restore pkumar pkumar_snap0
shashi@linuxtechi:~$

How to delete LXD container snapshot?

$ sudo lxc delete <container_name/snapshot_name>

How to set Memory, CPU and Disk Limit on LXD container?

Syntax to set Memory limit:

# lxc config set <container_name> limits.memory <Memory_Size>KB/MB/GB

Syntax to set CPU limit:

# lxc config set <container_name>  limits.cpu {Number_of_CPUs}

Syntax to Set Disk limit:

# lxc config device set <container_name> root size <Size_MB/GB>

Note: To set a disk limit (it requires btrfs or ZFS filesystem)

Let’s set limit on Memory and CPU on container shashi using the following commands,

shashi@linuxtechi:~$ sudo lxc config set shashi limits.memory 256MB
shashi@linuxtechi:~$ sudo lxc config set shashi limits.cpu 2

Install and configure LXC container (commands and operations)

To install lxc on your ubuntu system, use the beneath apt command,

shashi@linuxtechi:~$ sudo apt install lxc -y

In earlier version of LXC, the command “lxc-clone” was used and later it was deprecated. Now, “lxc-copy” command is widely used for cloning operation.

Note: To get “lxc-copy” command working, use the following installation steps,

shashi@linuxtechi:~$ sudo apt install lxc1 -y

Creating Linux Containers using the templates

LXC provides ready-made templates for easy installation of Linux containers. Templates are usually found in the directory path /usr/share/lxc/templates, but in fresh installation we will not get the templates, so to download the templates in your local system , run the beneath command,

shashi@linuxtechi:~$ sudo apt install lxc-templates -y

Once the lxc-templates are installed successfully then templates will be available,

shashi@linuxtechi:~$ sudo ls /usr/share/lxc/templates/
lxc-alpine     lxc-centos    lxc-fedora         lxc-oci           lxc-plamo      lxc-sparclinux    lxc-voidlinux
lxc-altlinux   lxc-cirros    lxc-fedora-legacy  lxc-openmandriva  lxc-pld        lxc-sshd
lxc-archlinux  lxc-debian    lxc-gentoo         lxc-opensuse      lxc-sabayon    lxc-ubuntu
lxc-busybox    lxc-download  lxc-local          lxc-oracle        lxc-slackware  lxc-ubuntu-cloud
shashi@linuxtechi:~$

Let’s Launch a container using template,

Syntax: lxc-create -n <container_name> lxc -t <template_name>

shashi@linuxtechi:~$ sudo lxc-create -n shashi_lxc -t ubuntu
………………………
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Current default time zone: 'Etc/UTC'
Local time is now:      Fri Aug  2 11:46:42 UTC 2019.
Universal Time is now:  Fri Aug  2 11:46:42 UTC 2019.

##
# The default user is 'ubuntu' with password 'ubuntu'!
# Use the 'sudo' command to run tasks as root in the container.
##
………………………………………
shashi@linuxtechi:~$

Once the complete template is created, we can login into this console using the following steps

shashi@linuxtechi:~$ sudo lxc-start -n shashi_lxc -d
shashi@linuxtechi:~$ sudo lxc-console -n shashi_lxc

Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself

Ubuntu 18.04.2 LTS shashi_lxc pts/0

shashi_lxc login: ubuntu
Password:
Last login: Fri Aug  2 12:00:35 UTC 2019 on pts/0
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-20-generic x86_64)
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@shashi_lxc:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G         23M        3.8G        112K        8.7M        3.8G
Swap:          1.9G        780K        1.9G
ubuntu@shashi_lxc:~$ grep -c processor /proc/cpuinfo
1
ubuntu@shashi_lxc:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G  7.4G   31G  20% /
ubuntu@shashi_lxc:~$

Now logout or exit from the container and go back to host machine login window. With the lxc-ls command we can see that shashi-lxc container is created.

shashi@linuxtechi:~$ sudo lxc-ls
shashi_lxc
shashi@linuxtechi:~$

“lxc-ls -f” command provides details with ip address of the container and the same is as below,

shashi@linuxtechi:~$ sudo lxc-ls -f
NAME       STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED
shashi_lxc RUNNING 0         -      10.0.3.190 -    false
shashi@linuxtechi:~$

“lxc-info -n <container_name>” command provides with all the required details along with State, ip address etc.

shashi@linuxtechi:~$ sudo lxc-info -n shashi_lxc
Name:           shashi_lxc
State:          RUNNING
PID:            6732
IP:             10.0.3.190
CPU use:        2.38 seconds
BlkIO use:      240.00 KiB
Memory use:     27.75 MiB
KMem use:       5.04 MiB
Link:           vethQ7BVGU
 TX bytes:      2.01 KiB
 RX bytes:      9.52 KiB
 Total bytes:   11.53 KiB
shashi@linuxtechi:~$

How to Start, Stop, Restart and Delete LXC containers

$ lxc-start -n <container_name>
$ lxc-stop -n <container_name>
$ lxc-destroy -n <container_name>

LXC Cloning operation

Now the main cloning operation to be performed on the LXC container. The following steps are followed

As described earlier LXC offers a feature of cloning a container from the existing container, by running the following command to clone an existing “shashi_lxc” container to a new container “shashi_lxc_clone”.

Note: We have to make sure that before starting the cloning operation, first we have to stop the existing container using the “lxc-stop” command.

shashi@linuxtechi:~$ sudo lxc-stop -n shashi_lxc
shashi@linuxtechi:~$ sudo lxc-copy -n shashi_lxc -N shashi_lxc_clone
shashi@linuxtechi:~$ sudo lxc-ls
shashi_lxc       shashi_lxc_clone
shashi@linuxtechi:~$

Now start the cloned container

shashi@linuxtechi:~$ sudo lxc-start -n shashi_lxc_clone
shashi@linuxtechi:~$ sudo lxc-ls -f
NAME             STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED
shashi_lxc       STOPPED 0         -      -          -    false
shashi_lxc_clone RUNNING 0         -      10.0.3.201 -    false
shashi@linuxtechi:~$

With the above set of commands, cloning operation is done and the new clone “shashi_lxc_clone” got created. We can login into this lxc container console with below steps,

shashi@linuxtechi:~$ sudo lxc-console -n shashi_lxc_clone

Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
Ubuntu 18.04.2 LTS shashi_lxc pts/0

shashi_lxc login:

LXC Network configuration and commands

We can attach to the newly created container, but to remotely login into this container using SSH or any other means, we have to do some minimal configuration changes as explained below,

shashi@linuxtechi:~$ sudo lxc-attach -n shashi_lxc_clone
root@shashi_lxc:/#
root@shashi_lxc:/# useradd -m shashi
root@shashi_lxc:/# passwd shashi
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
root@shashi_lxc:/#

First install the ssh server using the following command so that smooth “ssh” connect can be established.

root@shashi_lxc:/# apt install openssh-server -y

Now get the IP address of the existing lxc container using the following command,

root@shashi_lxc:/# ip addr show eth0|grep inet
    inet 10.0.3.201/24 brd 10.0.3.255 scope global dynamic eth0
    inet6 fe80::216:3eff:fe82:e251/64 scope link
root@shashi_lxc:/#

From the host machine with a new console window, use the following command to connect to this container over ssh

shashi@linuxtechi:~$ ssh 10.0.3.201
shashi@10.0.3.201's password:
$

Now, we have logged in a container using ssh session.

LXC process related commands

shashi@linuxtechi:~$ ps aux|grep lxc|grep -v grep

LXC snapshot operation

Snapshotting is one of the main operations which will help in taking point in time snapshot of the lxc container images. These same snapshot images can be used later for further use.

shashi@linuxtechi:~$ sudo lxc-stop -n shashi_lxc
shashi@linuxtechi:~$ sudo lxc-snapshot -n shashi_lxc
shashi@linuxtechi:~$

The snapshot path can be located using the following command.

shashi@linuxtechi:~$ sudo lxc-snapshot -L -n shashi_lxc
snap0 (/var/lib/lxc/shashi_lxc/snaps) 2019:08:02 20:28:49
shashi@linuxtechi:~$

Conclusion:

LXC, LinuX containers is one of the early container technologies. Understanding the concepts and learning about LXC will help in deeper understanding of any other containers like Docker Containers. This article has provided deeper insights on Cgroup and Namespaces which are also very much required concepts for better understanding of Containers and like. Many of the LXC operations like cloning, snapshotting, network operation etc are covered with command line examples.

Challenges like number of open files in any of the production environment has become common now a day. Since many applications which are Java based and Apache based, are getting installed and configured, which may lead to too many open files, file descriptors etc. If this exceeds the default limit that is set, then one may face access control problems and file opening challenges. Many production environments come to standstill kind of situations because of this.

Luckily, we have “ulimit” command in any of the Linux based server, by which one can see/set/get number of files open status/configuration details. This command is equipped with many options and with this combination one can set number of open files. Following are step-by-step commands with examples explained in detail.

View Current Open File Limit in Linux

To get open file limit on any Linux server, execute the following command,

[root@ubuntu ~]# cat /proc/sys/fs/file-max
146013

The above number  shows that user can open ‘146013’ file per user login session.

[root@centos ~]# cat /proc/sys/fs/file-max
149219
[root@debian ~]# cat /proc/sys/fs/file-max
73906

This clearly indicates that individual Linux operating systems have different number of open files. This is based on dependencies and applications which are running in respective systems.

ulimit command

As the name suggests, ulimit (user limit) is used to display and set resources limit for logged in user.When we run ulimit command with -a option then it will print all resources’ limit for the logged in user. Now let’s run “ulimit -a” on Ubuntu / Debian and RHEL systems,

Ubuntu / Debian System,

shashi@Ubuntu ~}$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 5731
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024      
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5731
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

RHEL /CentOS  System

shashi@centos ~}$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 5901
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5901
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

As we can be seen here different OS have different limits set. All these limits can be configured/changed using “ulimit” command.

To display the individual resource limit then pass the individual parameter in ulimit command, some of parameters are listed below:

• ulimit -n  –> It will display number of open files limit
• ulimit -c –> It display the size of core file
• umilit -u –> It will display the maximum user process limit for the logged in user.
• ulimit -f –> It will display the maximum file size that the user can have.
• umilit -m –> It will display the maximum memory size for logged in user.
• ulimit -v –> It will display the maximum memory size limit

Use below commands check hard and soft limits for number of open file for the logged in user

shashi@Ubuntu ~}$ ulimit -Hn
1048576
shashi@Ubuntu ~}$ ulimit -Sn
1024

How to fix the problem when limit on number of maximum files was reached ?

Let’s assume our Linux server has reached the limit of maximum number of open files and want to extend that limit system wide, for example we want to set 100000 as limit of number of open files.

Use sysctl command to pass fs.file-max parameter to kernel on the fly, execute beneath command as root user,

root@ubuntu~]# sysctl -w fs.file-max=100000
fs.file-max = 100000

Above changes will be active until the next reboot, so to make these changes persistent across the reboot, edit the file /etc/sysctl.conf and add same parameter,

root@ubuntu~]# vi /etc/sysctl.conf
fs.file-max = 100000

save and exit file,

Run the beneath command to make above changes into effect immediately without logout and reboot.

root@ubuntu~]# sysctl -p

Now verify whether new changes are in effect or not.

root@ubuntu~]# cat /proc/sys/fs/file-max
100000

Use below command to find out how many file descriptors are currently being utilized:

[root@ansible ~]# more /proc/sys/fs/file-nr
1216    0       100000

Note: Command “sysctl -p” is used to commit the changes without reboot and logout.

Set ulimit (User level resource limit) via limit.conf file 

“/etc/sysctl.conf” file is used to set resource limit system wide but if you want to set resource limit for specific user like Oracle, MariaDB and Apache then this can be achieved via “/etc/security/limits.conf” file.

Sample limit.conf is shown below,

root@ubuntu~]# cat /etc/security/limits.conf

Let’s assume we want to set hard and soft limit on number of open files for linuxtechi user and for oracle user set hard and soft limit on number of open process, edit the file “/etc/security/limits.conf” and add the following lines

# hard limit for max opened files for linuxtechi user
linuxtechi       hard    nofile          4096
# soft limit for max opened files for linuxtechi user
linuxtechi       soft    nofile          1024

# hard limit for max number of process for oracle user
oracle           hard    nproc          8096
# soft limit for max number of process for oracle user
oracle           soft    nproc          4096

Save & exit the file.

Note: In case you want to put resource limit on a group instead of users, then it can also be possible via limit.conf file, in place of user name , type @<Group_Name> and rest of the items will be same, example is shown below,

# hard limit for max opened files for sysadmin group
@sysadmin        hard         nofile            4096 
# soft limit for max opened files for sysadmin group
@sysadmin        soft         nofile            1024

Verify whether new changes are in effect or not,

~]# su - linuxtechi
~]$ ulimit -n -H
4096
~]$ ulimit -n -S
1024

~]# su - oracle
~]$ ulimit -H -u
8096
~]$ ulimit -S -u
4096

Note: Other majorly used command is “lsof” which is used for finding out “how many files are opened currently”. This command is very helpful for admins.

Conclusion

As mentioned in the introduction section “ulimit” command is very powerful and helps one to configure and make sure application installations are smoother without any bottlenecks. This command helps in fixing many of the number of file limitations in Linux based servers.

Monitoring docker container images

shashi@linuxtechi ~}$ docker ps -a
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS   PORTS                             NAMES
9ab9aa72f015        ubuntu                "/bin/bash"         14 seconds ago      Exited (0) 12 seconds ago                  suspicious_shannon
305369d3b2bb        centos                "/bin/bash"         24 seconds ago      Exited (0) 22 seconds ago                  admiring_mestorf
9a669f3dc4f6        portainer/portainer   "/portainer"        7 minutes ago       Up 7 minutes   0.0.0.0:9000->9000/tcp      trusting_keller

Including the portainer(which is a docker container image), all the exited and present running docker images are displayed. Below screenshot from Portainer GUI displays the same.

Monitoring events

Click on the “Events” option from the portainer webpage as shown below.

Various events that are generated and created based on docker-container activity, are captured and displayed in this page

Now to check and validate how the “Events” section works. Create a new docker-container image redis as explained below, check the docker ps –a status at docker command-line.

shashi@linuxtechi ~}$ docker ps -a
CONTAINER ID        IMAGE                 COMMAND                  CREATED              STATUS         PORTS                    NAMES
cdbfbef59c31        redis                 "docker-entrypoint.s…"   About a minute ago   Up About a minute         6379/tcp                 angry_varahamihira
9ab9aa72f015        ubuntu                "/bin/bash"              10 minutes ago       Exited (0) 10 minutes ago                            suspicious_shannon
305369d3b2bb        centos                "/bin/bash"              11 minutes ago       Exited (0) 11 minutes ago                            admiring_mestorf
9a669f3dc4f6        portainer/portainer   "/portainer"             17 minutes ago       Up 17 minutes         0.0.0.0:9000->9000/tcp   trusting_keller

Click the “Event List” on the top to refresh the events list,

Now the event’s page also updated with this change,

Host status

Below is the screenshot of the portainer displaying the host status. This is a simple window showing-up. This shows the basic info like “CPU”, “hostname”, “OS info” etc of the host linux machine. Instead of logging- into the host command-line, this page provides very useful info on for quick glance.

Dashboard in Portainer

Until now we have seen various features of portainer based under “Local” section. Now jump on to the “Dashboard” section of the selected Docker Container image.

When “EndPoint” option is clicked in the GUI of Portainer, the following window appears,

This Dashboard has many statuses and options, for a host container image.

1) Stacks: Clicking on this option, provides status of any stacks if any. Since there are no stacks, this displays zero.

2) Images: Clicking on this option provides host of container images that are available. This option will display all the live and exited container images

For example create one more “Nginx” container and refresh this list to see the updates.

shashi@linuxtechi ~}$  sudo docker run nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
27833a3ba0a5: Pull complete
ea005e36e544: Pull complete
d172c7f0578d: Pull complete
Digest: sha256:e71b1bf4281f25533cf15e6e5f9be4dac74d2328152edf7ecde23abc54e16c1c
Status: Downloaded newer image for nginx:latest

The following is the image after refresh,

Once the Nginx image is stopped/killed and docker container image will be moved to unused status.

Note:-One can see all the image details here are very clear with memory usage, creation date and time. As compared to command-line option, maintaining and monitoring containers from here it will be very easy.

3) Networks: this option is used for network operations. Like assigning IP address, creating subnets, providing IP address range, access control (admin and normal user) . The following window provides the details of various options possible. Based on your need these options can be explored further.

Once all the various networking parameters are entered, “create network” button is clicked for creating the network.

4) Container: (click on container) This option will provide the container status. This list will provide details on live and not running container statuses. This output is similar to docker ps command option.

From this window only the containers can be stopped and started as need arises by checking the check box and selecting the above buttons. One example is provided as below,

Example, Both “CentOS” and “Ubuntu” containers which are in stopped state, they are started now by selecting check boxes and hitting “Start” button.

Note: Since both are Linux container images, they will not be started. Portainer tries to start and stops later. Try “Nginx” instead and you can see it coming to “running”status.

5) Volume: Described in Part-I of Portainer Article

Setting option in Portainer

Until now we have seen various features of portainer based under “Local” section. Now jump on to the “Setting” section of the selected Docker Container image.

When “Settings” option is clicked in the GUI of Portainer, the following further configuration options are available,

1) Extensions: This is a simple Portainer CE subscription process. The details and uses can be seen from the attached window. This is mainly used for maintaining the license and subscription of the respective version.

2) Users: This option is used for adding “users” with or without administrative privileges. Following example provides the same.

Enter the selected user name “shashi” in this case and your choice of password and hit “Create User” button below.

Similarly the just now created user “shashi” can be removed by selecting the check box and hitting remove button.

3) Endpoints: this option is used for Endpoint management. Endpoints can be added and removed as shown in the attached windows.

The new endpoint “shashi” is created using the various default parameters as shown below,

Similarly this endpoint can be removed by clicking the check box and hitting remove button.

4) Registries: this option is used for registry management. As docker hub has registry of various images, this feature can be used for similar purposes.

With the default options the “shashi-registry” can be created.

Similarly this can be removed if not required.

5) Settings: This option is used for the following various options,

• Setting-up snapshot interval
• For using custom logo
• To create external templates
• Security features like- Disable and enable bin mounts for non-admins, Disable/enable privileges for non-admins, Enabling host management features

Following screenshot shows some options enabled and disabled for demonstration purposes. Once all done hit on “Save Settings” button to save all these options.

Now one more option pops-up on “Authentication settings” for LDAP, Internal or OAuth extension as shown below”

Based on what level of security features we want for our environment, respective option is chosen.

That’s all from this article, I hope these Portainer GUI articles helps you to manage and monitor containers more efficiently.  Please do share your feedback and comments.

Based on the above scenarios, there was in need of one light weight and robust tool requirement was growing. So Portainer.io addressed this. “Portainer.io“,(Latest version is 1.20.2) the tool is very light weight(with 2-3 commands only one can configure it) and has become popular among Docker users.

This tool has advantages over other tools; some of these are as below,

• Light weight (requires only 2-3 commands to be required to run to install this tool) {Also installation image is only around 26-30MB of size)
• Robust and easy to use
• Can be used for Docker monitor and Build
• This tool provides us a detailed overview of your Docker environments
• This tool allows us to manage your containers, images, networks and volumes.
• Portainer is simple to deploy – this requires just one Docker command (can be run from anywhere.)
• Complete Docker-container environment can be monitored easily

Portainer is also equipped with,

• Community support
• Enterprise support
• Has professional services available(along with partner OEM services)

Functionality and features of Portainer tool are,

1. It comes-up with nice Dashboard, easy to use and monitor.
2. Many in-built templates for ease of operation and creation
3. Support of services (OEM, Enterprise level)
4. Monitoring of Containers, Images, Networks, Volume and configuration at almost real-time.
5. Also includes Docker-Swarm monitoring
6. User management with many fancy capabilities

Read Also : How to Install Docker CE on Ubuntu 16.04 / 18.04 LTS System

How to install and configure Portainer.io on Ubuntu Linux / RHEL / CentOS 

Note: This installation is done on Ubuntu 18.04 but the installation on RHEL & CentOS would be same. We are assuming Docker CE is already installed on your system.

shashi@linuxtechi:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic
shashi@linuxtechi:~$

Create the Volume for portainer

shashi@linuxtechi:~$ sudo docker volume create portainer_data
portainer_data
shashi@linuxtechi:~$

Launch and start Portainer Container using the beneath docker command,

shashi@linuxtechi:~$ sudo docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
Unable to find image 'portainer/portainer:latest' locally
latest: Pulling from portainer/portainer
d1e017099d17: Pull complete
0b1e707a06d2: Pull complete
Digest: sha256:d6cc2c20c0af38d8d557ab994c419c799a10fe825e4aa57fea2e2e507a13747d
Status: Downloaded newer image for portainer/portainer:latest
35286de9f2e21d197309575bb52b5599fec24d4f373cc27210d98abc60244107
shashi@linuxtechi:~$

Once the complete installation is done, use the ip of host or Docker using port 9000 of the Docker engine where portainer is running using your browser.

Note: If OS firewall is enabled on your Docker host then make sure 9000 port is allowed else its GUI will not come up.

In my case, IP address of my Docker Host / Engine is “192.168.1.16” so URL will be,

http://192.168.1.16:9000

Please make sure that you enter 8-character passwords. Let the admin be the user as it is and then click “Create user”.

Now the following screen appears, in this select “Local” rectangle box.

Click on “Connect”

Nice GUI with admin as user home screen appears as below,

Now Portainer is ready to launch and manage your Docker containers and it can also be used for containers monitoring.

Bring-up container image on Portainer tool

Now check the present status, there are two container images are already running, if you create one more that appears instantly.

From your command line kick-start one or two containers as below,

shashi@linuxtechi:~$ sudo docker run --name test -it debian
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
e79bb959ec00: Pull complete
Digest: sha256:724b0fbbda7fda6372ffed586670573c59e07a48c86d606bab05db118abe0ef5
Status: Downloaded newer image for debian:latest
root@d45902e717c0:/#

Now click Refresh button (Are you sure message appears, click “continue” on this) in Portainer GUI, you will now see 3 container images as highlighted below,

Click on the “containers” (in which it is red circled above), next window appears with “Dashboard Endpoint summary”

In this page, click on “Containers” as highlighted in red color. Now you are ready to monitor your container image.

Simple Docker container image monitoring

From the above step, it appears that a fancy and nice looking “Container List” page appears as below,

All the container images can be controlled from here (stop, start, etc)

1) Now from this page, stop the earlier started {“test” container (this was the debian image that we started earlier)}

To do this select the check box in front of this image and click stop button from above,

From the command line option, you will see that this image has been stopped or exited now,

shashi@linuxtechi:~$ sudo docker container ls -a
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS                       PORTS                    NAMES
d45902e717c0        debian                "bash"              21 minutes ago      Exited (0) 49 seconds ago                             test
08b96eddbae9        centos:7              "/bin/bash"         About an hour ago   Exited (137) 9 minutes ago                            mycontainer2
35286de9f2e2        portainer/portainer   "/portainer"        2 hours ago         Up About an hour             0.0.0.0:9000->9000/tcp   compassionate_benz
shashi@linuxtechi:~$

2) Now start the stopped containers (test & mycontainer2) from Portainer GUI,

Select the check box in front of stopped containers, and the click on Start

You will get a quick window saying, “Container successfully started” and with running state

Various other options and features are explored as below step-by-step

1) Click on “Images” which is highlighted, you will get the below window,

This is the list of container images that are available but some may not running. These images can be imported, exported or uploaded to various locations, below screen shot shows the same,

2) Click on “volumes” which is highlighted, you will get the below window,

3) Volumes can be added easily with following option, click on add volume button, below window appears,

Provide the name as “myvol” in the name box and click on “create the volume” button.

The newly created volume appears as below, (with unused state)

Conclusion:

As from the above installation steps, configuration and playing around with various options you can see how easy and fancy looking is Portainer.io tool is. This provides multiple features and options to explore on building, monitoring docker container. As explained this is very light weight tool, so doesn’t add any overload to host system. Next set-of options will be explored in part-2 of this series.

Read Also: Monitor and Manage Docker Containers with Portainer.io (GUI tool) – Part-2

Docker is playing very critical role as both a packaging format for the applications and a unifying interface and methodology that enables the application team/s to own the Docker-formatted container images (Note: It is assumed that including all dependencies). This has made Docker as one of the leading front runners in microservices and microservices based adaption and usage.

Many of the Docker and Docker based utilities helps us to bring greater efficiencies of scalability and performance by shrinking application footprints through Dockerized containers. In many cases system-level dependencies are reduced to bare minimum, which helps in bringing down memory usage in terms of MB’s (Mega bytes).

All these aspects helped in making Docker as one of the leading container based utility. Docker has many command-line utilities and capabilities. Docker Daemon can provide many of these additional facilities and which makes Docker configuration easier.

Read Also :How to Install Docker (Community Edition) on Ubuntu 16.04 / 18.04 LTS

Hope the reader/user of this article knows how to spin-off docker-container on any given Linux environment. Assuming this, some of the following facilities are described,

Some of these facilities can be used for the following functions,

• Check Docker server information
• Download docker images and updates
• Inspect containers
• View and monitor logs
• Monitor various statistics and details

1) How to check present docker version

Knowing the docker version is one of the very important aspects as depending on this, many of the version based decisions are taken.

shashi@linuxtechi:~$ sudo docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:43:57 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:10:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false
shashi@linuxtechi:~$

This above output provides both server and client API versions as same and OS and Architecture version also should be same. If there is any mismatch in client and server based versions, client-server communication will fail. One has to make sure what the supporting versions are and take decisions accordingly.

2) Capturing and analyzing server information

Using docker info we can also find the following information. Some of the useful information we can gather are something like, which server is running as backend, which kernel version it is, which OS and Docker root directory etc

shashi@linuxtechi:~$ sudo docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 18.09.5
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-20-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.947GiB
Name: linuxtechi
ID: VRUY:AWXX:7JWE:YWU7:X4QW:TNKE:6H26:PNRR:QFGI:XYRQ:QUXF:MTXC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
shashi@linuxtechi:~$

All the above information is based on how the Docker Daemon is set-up, underlying OS version and file system type. All these can be captured using the following set-of the commands,

Execute the below command to get OS Name, its version and Code name

shashi@linuxtechi:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"
shashi@linuxtechi:~$

or

shashi@linuxtechi:~$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
shashi@linuxtechi:~$

Execute the beneath command to get the file system details :

shashi@linuxtechi:~$ mount | grep "^/dev"
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
shashi@linuxtechi:~$

3) Docker daemon

The docker daemon plays a very crucial role in docker environment as whole.

Without a proper daemon, the complete docker system will be useless. One can verify the daemon status using the following command,

Note:- Assuming proper docker installation is done

shashi@linuxtechi:~$ sudo service docker status

If the docker service is running, then output of above command should be something like below:

In case docker service is not running then use below command to start it

shashi@linuxtechi:~$ sudo systemctl start docker
or
shashi@linuxtechi:~$ sudo service docker start
shashi@linuxtechi:~$

Use below “docker ps” command to list the running containers

shashi@linuxtechi:~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
shashi@linuxtechi:~$

To list all running and stopped containers use “docker ps -a”

shashi@linuxtechi:~$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
497e6733d760        ubuntu              "bash"              19 minutes ago      Exited (0) 2 minutes ago                        goofy_morse
0862fe109f96        hello-world         "/hello"            19 minutes ago      Exited (0) 19 minutes ago                       vibrant_shannon
shashi@linuxtechi:~$

Docker default root directory  is “/var/lib/docker”

shashi@linuxtechi:~$ sudo ls -l /var/lib/docker
total 48
drwx------  2 root root 4096 Apr 14 07:00 builder
drwx------  4 root root 4096 Apr 14 07:00 buildkit
drwx------  4 root root 4096 Apr 14 07:09 containers
drwx------  3 root root 4096 Apr 14 07:00 image
drwxr-x---  3 root root 4096 Apr 14 07:00 network
drwx------ 16 root root 4096 Apr 14 07:27 overlay2
drwx------  4 root root 4096 Apr 14 07:00 plugins
drwx------  2 root root 4096 Apr 14 07:27 runtimes
drwx------  2 root root 4096 Apr 14 07:00 swarm
drwx------  2 root root 4096 Apr 14 07:27 tmp
drwx------  2 root root 4096 Apr 14 07:00 trust
drwx------  2 root root 4096 Apr 14 07:00 volumes
shashi@linuxtechi:~$

The docker daemon if not started can be invoked using the following command,

shashi@linuxtechi:~$ sudo dockerd

Output of above dockerd command will be something like below:

Read Also : How to install Docker on CentOS 7

4) Downloading docker container image and inspecting the container

shashi@linuxtechi:~$ sudo docker pull ubuntu:latest
Using default tag: latest
latest: Pulling from library/ubuntu
898c46f3b1a1: Pull complete
63366dfa0a50: Pull complete
041d4cd74a92: Pull complete
6e1bee0f8701: Pull complete
Digest: sha256:017eef0b616011647b269b5c65826e2e2ebddbe5d1f8c1e56b3599fb14fabec8
Status: Downloaded newer image for ubuntu:latest
shashi@linuxtechi:~$
latest: Pulling from library/ubuntu898c46f3b1a1: Pull complete
63366dfa0a50: Pull complete
041d4cd74a92: Pull complete6e1bee0f8701: Pull complete
Digest: sha256:017eef0b616011647b269b5c65826e2e2ebddbe5d1f8c1e56b3599fb14fabec8Status: Downloaded newer image for ubuntu:latest

Launching a container, example is shown below

shashi@linuxtechi:~$ sudo docker run -d -t ubuntu /bin/bash
58c023f0f5689ff08b858221ca10c985936a8c9dd91d08e84213009facb64724
shashi@linuxtechi:~$
shashi@linuxtechi:~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
58c023f0f568        ubuntu              "/bin/bash"         27 seconds ago      Up 26 seconds                           boring_dijkstra
shashi@linuxtechi:~$

Let’s inspect this container using the following command,

shashi@linuxtechi:~$ sudo docker inspect 58c023f0f568
[
    {
        "Id": "58c023f0f5689ff08b858221ca10c985936a8c9dd91d08e84213009facb64724",
        "Created": "2019-04-14T06:55:26.289022884Z",
        "Path": "/bin/bash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 15538,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-04-14T06:55:27.142274111Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:94e814e2efa8845d95b2112d54497fbad173e45121ce9255b93401392f538499",
        "ResolvConfPath": "/var/lib/docker/containers/58c023f0f5689ff08b858221ca10c985936a8c9dd91d08e84213009facb64724/resolv.conf",
………………………………………………

Note:- The complete output of this command is not shown here, because it’s output is too large

5) Going inside the running container image

As one is aware of Docker used originally everything based on LXC backend, lxc-attach command of linux was used for quite some time. But once docker was built as a standalone package and started using “libcontainer” as the default backend, the “docker exec” command usage became popular

Below set of commands explain on how to login into container using command line,

:~$ sudo docker  exec -t <Container_id>  <Commands>

shashi@linuxtechi:~$ sudo docker  exec -t 58c023f0f568 ls -a
.   .dockerenv  boot  etc   lib    media  opt   root  sbin  sys  usr
..  bin         dev   home  lib64  mnt    proc  run   srv   tmp  var
shashi@linuxtechi:~$ sudo docker  exec -t 58c023f0f568 ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 06:55 pts/0    00:00:00 /bin/bash
root        20     0  0 07:17 pts/1    00:00:00 ps -ef
shashi@linuxtechi:~$

The alternate way to run commands inside the container is, first login to container using “docker attach” and then execute commands

:~$ sudo docker attach <Container_id>

shashi@linuxtechi:~$ sudo docker attach 58c023f0f568
root@58c023f0f568:/#

There are some situations where you want to launch a container, attaching a volume to it and when we exit from container it should deleted automatically, example is shown below

shashi@linuxtechi:~$ sudo docker run -it --rm -v /usr/local/bin:/target jpetazzo/nsenter bash
root@73c72922f87e:/src# df -h /target/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        18G  5.6G   12G  34% /target
root@73c72922f87e:/src# exit
exit
shashi@linuxtechi:~$

Now verify whether docker is removed or not automatically when we exit from the container, run “docker ps”

shashi@linuxtechi:~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
58c023f0f568        ubuntu              "/bin/bash"         About an hour ago   Up About an hour                        boring_dijkstra
shashi@linuxtechi:~$

6) Monitoring Docker

Use “docker stats” command to display resource utilization of all the containers,

shashi@linuxtechi:~$ sudo docker stats
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
58c023f0f568        boring_dijkstra     0.00%               1.059MiB / 1.947GiB   0.05%               4.75kB / 0B         4.74MB / 0B         1
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
58c023f0f568        boring_dijkstra     0.00%               1.059MiB / 1.947GiB   0.05%               4.75kB / 0B         4.74MB / 0B         1
shashi@linuxtechi:~$

7) Docker Events

Many a times capturing docker events will be crucial as this will provide information on various docker operations and events taking place. Below is the snapshot of the same.

shashi@linuxtechi:~$ sudo docker events
2019-04-14T09:29:07.636990738+01:00 image pull wordpress:latest (name=wordpress)
2019-04-14T09:29:46.936676431+01:00 volume create 36187e0a44d277439fea0df2446fc44987fa814c52744091929e5c81bd8134e5 (driver=local)
2019-04-14T09:29:46.998798935+01:00 container create b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058 (image=wordpress, name=friendly_heisenberg)
2019-04-14T09:29:47.000202026+01:00 container attach b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058 (image=wordpress, name=friendly_heisenberg)
2019-04-14T09:29:47.209257002+01:00 network connect 18dd93c3c6fc9ce51a98f7d2359b319db251efcae6b991157965ef727a580702 (container=b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058, name=bridge, type=bridge)
2019-04-14T09:29:47.239846902+01:00 volume mount 36187e0a44d277439fea0df2446fc44987fa814c52744091929e5c81bd8134e5 (container=b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058, destination=/var/www/html, driver=local, propagation=, read/write=true)
2019-04-14T09:29:47.942997316+01:00 container start b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058 (image=wordpress, name=friendly_heisenberg)
2019-04-14T09:29:47.944521098+01:00 container resize b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058 (height=39, image=wordpress, name=friendly_heisenberg, width=130)
2019-04-14T09:29:59.829378089+01:00 container die b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058 (exitCode=0, image=wordpress, name=friendly_heisenberg)
2019-04-14T09:30:00.147435896+01:00 network disconnect 18dd93c3c6fc9ce51a98f7d2359b319db251efcae6b991157965ef727a580702 (container=b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058, name=bridge, type=bridge)
2019-04-14T09:30:00.845336887+01:00 volume unmount 36187e0a44d277439fea0df2446fc44987fa814c52744091929e5c81bd8134e5 (container=b97f6a565c518eb1464cf81e6e09db1acfd84a0fdcbaea94255f1f182d79c058, driver=local)
………………

Conclusion

The combination of all these commands and utilities are very important in making docker and container a successful microservices based environment. As many of the microservices architectures need these kinds of utilities to debug, understand and learn more about them for every day usage, I hope this article will play role in helping such cases.

Read Also : How to install and use docker-compose to deploy containers in CentOS 7

Most of these concepts are explained using programming examples. In this article, I will be covering what are fork, vfork, exec and wait system calls, their distinguishing characters and how they can be better used.

fork()

fork(): System call to create a child process.

shashi@linuxtechi ~}$ man fork

This will yield output mentioning what is fork used for, syntax and along with all the required details.

The syntax used for the fork system call is as below,

pid_t fork(void);

Fork system call creates a child that differs from its parent process only in pid(process ID) and ppid(parent process ID). Resource utilization is set to zero. File locks and pending signals are not inherited. (In Linux “fork” is implemented as “copy-on-write()“).

Note:- “Copy on write” -> Whenever a fork() system call is called, a copy of all the pages(memory) related to the parent process is created and loaded into a separate memory location by the Operating System for the child process. But this is not needed in all cases and may be required only when some process writes to this address space or memory area, then only separate copy is created/provided.

Return values:-  PID (process ID) of the child process is returned in parents thread of execution and “zero” is returned in child’s thread of execution. Following is the c-programming example which explains how fork system call works.

shashi@linuxtechi ~}$ vim 1_fork.c
#include<stdio.h>
#include<unistd.h>
Int main(void)
{
printf("Before fork\n");
fork();
printf("after fork\n");
}
shashi@linuxtechi ~}$ 
shashi@linuxtechi ~}$ cc 1_fork.c
shashi@linuxtechi ~}$ ./a.out
Before fork
After fork
shashi@linuxtechi ~}$

Whenever any system call is made there are plenty of things that take place behind the scene in any unix/linux machines.

First of all context switch happens from user mode to kernel(system) mode. This is based on the process priority and unix/linux operating system that we are using. In the above C example code we are using  “{” opening curly brace which is the entry of the context and “}” closing curly brace is for exiting the context. The following table explains context switching very clearly.

vfork()

vfork –> create a child process and block parent process.

Note:- In vfork, signal handlers are inherited but not shared.

shashi@linuxtechi ~}$ man vfork

This will yield output mentioning what is vfork used for, syntax and along with all the required details.

pid_t vfork(void);

vfork is as same as fork except that behavior is undefined if process created by vfork either modifies any data other than a variable of type pid_t used to store the return value p of vfork or calls any other function between calling _exit() or one of the exec() family.

Note: vfork is sometimes referred to as special case of clone.

Following is the C programming example for vfork() how it works.

shashi@linuxtechi ~}$ vim 1.vfork.c
#include<stdio.h>
#include<unistd.h>
Int main(void)
{
printf("Before fork\n");
vfork();
printf("after fork\n");
}
shashi@linuxtechi ~}$ vim 1.vfork.c
shashi@linuxtechi ~}$ cc 1.vfork.c
shashi@linuxtechi ~}$ ./a.out
Before vfork
after vfork
after vfork
a.out: cxa_atexit.c:100: __new_exitfn: Assertion `l != NULL' failed.
Aborted

Note:– As explained earlier, many a times the behaviour of the vfork system call is not predictable. As in the above case it had printed before once and after twice but aborted the call with _exit() function. It is better to use fork system call unless otherwise and avoid using vfork as much as possible.

Differences between fork() and vfork()

Vfork() behaviour explained in more details in the below program.

shashi@linuxtechi ~}$ cat vfork_advanced.c
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
int main()
{
    int n =10;
    pid_t pid = vfork(); //creating the child process
    if (pid == 0)          //if this is a chile process
    {
        printf("Child process started\n");
    }
    else//parent process execution
    {
        printf("Now i am coming back to parent process\n");
    }
    printf("value of n: %d \n",n); //sample printing to check "n" value
    return 0;
}
shashi@linuxtechi ~}$ cc vfork_advanced.c
shashi@linuxtechi ~}$ ./a.out
Child process started
value of n: 10
Now i am coming back to parent process
value of n: 594325573
a.out: cxa_atexit.c:100: __new_exitfn: Assertion `l != NULL' failed.
Aborted

Note: Again if you observe the outcome of vfork is not defined. Value of “n” has been printed first time as 10, which is expected. But the next time in the parent process it has printed some garbage value.

wait()

wait() system call suspends execution of current process until a child has exited or until a signal has delivered whose action is to terminate the current process or call signal handler.

pid_t wait(int * status);

There are other system calls related to wait as below,

1) waitpid(): suspends execution of current process until a child as specified by pid arguments has exited or until a signal is delivered.

pid_t waitpid (pid_t pid, int *status, int options);

2) wait3(): Suspends execution of current process until a child has exited or until signal is delivered.

pid_t wait3(int *status, int options, struct rusage *rusage);

3) wait4(): As same as wait3() but includes pid_t pid value.

pid_t wait3(pid_t pid, int *status, int options, struct rusage *rusage);

exec()

exec() family of  functions or sys calls replaces current process image with new process image.

There are functions like execl, execlp,execle,execv, execvp and execvpe are used to execute a file.

These functions are combinations of array of pointers to null terminated strings that represent the argument list , this will have path variable with some environment variable combinations.

exit()

This function is used for normal process termination. The status of the process is captured for future reference. There are other similar functions exit(3) and _exit()., which are used based on the exiting process that one is interested to use or capture.

Conclusion:-

The combinations of all these system calls/functions are used for process creation, execution and modification. Also these are called “shell” spawning set-of functions. One has to use these functions cautiously keeping in mind the outcome and behaviour.

Some small details will help us a lot in debugging and troubleshooting many challenging situations if we know well in advance about these commands and related concepts.

In this topic, I will be covering what are links, different types, distinguishing characters and how they can be better used along with concepts required.

By executing “man ln” command you can see that this says “make link between files” and doesn’t say about soft or hard links.

shashi@linuxtechi ~}$ man ln

Similarly the command “man link” describes as “call link function to create a file”.

Soft-link :

Soft link as the name suggests is a just a new link created to the new file. In this case the new file’s inode number will be pointing to the old file.

Hard Link :

In this case the old and new file both will be pointing to same inode number.

Symbolic Link :

In some Unix/Linux flavors both the symbolic and soft links are treated as same. But actual difference is that both the inode numbers of new and old file will be pointing to a new inode number. This will be completely depending on the implementation.

Note 1:- In many cases symbolic and soft link terminologies are used interchangeably. But one has to be aware of when to use what.

Usage of “ln” command for creating hard links and soft links

Creating Hard Link

1) “man ln” command will provide the following output.

shashi@linuxtechi ~}$ man ln
ln  - make links between files

2) When “ln” command is passed without any arguments, “missing file operand” error is thrown.

shashi@linuxtechi ~}$ln
ln: missing file operand
Try 'ln --help' for more information.

3) Creating the hard link between two files. First check for any existing files if any, else create the file/files freshly and then link them. Following  is  the step-by-step approach,

shashi@linuxtechi ~}$ ls -la
total 24
drwx------  4 root root 4096 Feb  6 15:23 .
drwxr-xr-x 23 root root 4096 Jan 25 16:39 ..
-rw-r--r--  1 root root 3122 Jan 25 16:41 .bashrc
drwx------  2 root root 4096 Feb  6 15:23 .cache
-rw-r--r--  1 root root    0 Jan 25 16:41 .hushlogin
-rw-r--r--  1 root root  148 Aug 17  2015 .profile
drwxr-xr-x  2 root root 4096 Jan 25 16:41 .ssh

i)  Create the file using “touch” command

shashi@linuxtechi ~}$ touch 123.txt
shashi@linuxtechi ~}$ ls -l
total 0
-rw-r--r-- 1 root root 0 Feb  6 15:51 123.txt

ii) Enter contents into the file using “cat” command, and hit “ctrl+c” to save and exit.

shashi@linuxtechi ~}$ cat > 123.txt
Welcome to this World!
^C

iii) Create the hard link between the files “123.txt” and “321.txt”. In this case the “123.txt” already existed with contents “Welcome to this World”.

shashi@linuxtechi ~}$ ln 123.txt 321.txt
shashi@linuxtechi ~}$ ls -l
total 8
-rw-r--r-- 2 root root 23 Feb  6 15:52 123.txt
-rw-r--r-- 2 root root 23 Feb  6 15:52 321.txt

iv) Check the inode(index node) numbers of the files. For both the files the inode number is same and which is 794583. Also check the contents of new file “321.txt” which is also same as “123.txt”.

shashi@linuxtechi ~}$ ls -li
total 8
794583 -rw-r--r-- 2 root root 23 Feb  6 15:52 123.txt
794583 -rw-r--r-- 2 root root 23 Feb  6 15:52 321.txt
$ cat 321.txt
Welcome to this World!

Note 2:- Inode number is a unique index number generated for any file that is being created in Linux/unix Operating systems. These inode numbers are stored in directory/file attributes in /proc directory. But in case of links these inode numbers are shared b/w the files and only path is updated in file table.

v) Create one more file called “456.txt” and link this using ln command to “321.txt”. Now all the three files have the same inode numbers. Contents of “456.txt” will be as same as that of original file.

shashi@linuxtechi ~}$  ls -li
total 12
794583 -rw-r--r-- 3 root root 23 Feb  6 15:52 123.txt
794583 -rw-r--r-- 3 root root 23 Feb  6 15:52 321.txt
794583 -rw-r--r-- 3 root root 23 Feb  6 15:52 456.txt
$ cat 456.txt
Welcome to this World!
shashi@linuxtechi ~}$   ls -l
total 12
-rw-r--r-- 3 root root 23 Feb  6 15:52 123.txt
-rw-r--r-- 3 root root 23 Feb  6 15:52 321.txt
-rw-r--r-- 3 root root 23 Feb  6 15:52 456.txt

vi)  When the source file or any of these files are removed, it will not impact other files. The source file can be removed using “rm” command. Contents of other file will not have any impact either.

shashi@linuxtechi ~}$ rm 123.txt
shashi@linuxtechi ~}$ ls -l
total 8
-rw-r--r-- 2 root root 23 Feb  6 15:52 321.txt
-rw-r--r-- 2 root root 23 Feb  6 15:52 456.txt
shashi@linuxtechi ~}$ ls -li
total 8
794583 -rw-r--r-- 2 root root 23 Feb  6 15:52 321.txt
794583 -rw-r--r-- 2 root root 23 Feb  6 15:52 456.txt
shashi@linuxtechi ~}$ cat 456.txt
Welcome to this World!

vii) Hard link creation across directories is not allowed.

shashi@linuxtechi ~}$ls -l
total 8
-rw-r--r-- 2 root root 23 Feb  6 15:52 321.txt
-rw-r--r-- 2 root root 23 Feb  6 15:52 456.txt
shashi@linuxtechi ~}$ mkdir abc
shashi@linuxtechi ~}$ ln abc def
ln: abc: hard link not allowed for directory

viii) Any changes to the content of one file will impact and change the content of other file accordingly, following are the steps explained,

shashi@linuxtechi ~}$ vi 321.txt
Welcome to this World!
You are welcomed to this new world
:wq
shashi@linuxtechi ~}$ ls -l
total 12
-rw-r--r-- 2 root root   59 Feb  6 16:24 321.txt
-rw-r--r-- 2 root root   59 Feb  6 16:24 456.txt
drwxr-xr-x 2 root root 4096 Feb  6 16:18 abc
shashi@linuxtechi ~}$ cat 456.txt
Welcome to this World!
You are welcomed to this new world

Creating Soft Link:

1)  Create the file “src.txt” using “touch” command and enter contents as “Hello World” using cat command and then hit “ctrl+c” to save and exit.

shashi@linuxtechi ~}$ touch src.txt
shashi@linuxtechi ~}$ cat > src.txt
Hello World
^C
shashi@linuxtechi ~}$ ls -l
total 4
-rw-r--r-- 1 root root 12 Feb  6 16:32 src.txt

2)  Create the destination file as “dst.txt” and using “ln -s” command line options create the symbolic link (also called as soft link). Check the contents of “dst.txt” file and same contents as that of “src.txt” can be seen.

shashi@linuxtechi ~}$ ln -s src.txt dst.txt
shashi@linuxtechi ~}$  ls -l
total 4
lrwxrwxrwx 1 root root  7 Feb  6 16:33 dst.txt -> src.txt
-rw-r--r-- 1 root root 12 Feb  6 16:32 src.txt
shashi@linuxtechi ~}$  cat dst.txt
Hello World

3)  In case of symbolic links the inode number of source file and destination file differs. Also, in the permissions letter “l” appears indicating that these are links. “dst.txt–>src.txt” will be the new link established now.

shashi@linuxtechi ~}$  ls -li
total 4
794584 lrwxrwxrwx 1 root root  7 Feb  6 16:33 dst.txt -> src.txt
794583 -rw-r--r-- 1 root root 12 Feb  6 16:32 src.txt

4) Symbolic creation for directories is allowed. This is explained as below in the steps

shashi@linuxtechi ~}$ mkdir abc
shashi@linuxtechi ~}$ ln -s abc def
$ ls -l
total 8
drwxr-xr-x 2 root root 4096 Feb  6 16:34 abc
lrwxrwxrwx 1 root root    3 Feb  6 16:34 def -> abc
lrwxrwxrwx 1 root root    7 Feb  6 16:33 dst.txt -> src.txt
-rw-r--r-- 1 root root   12 Feb  6 16:32 src.txt

5) Inode number for all the files/directories (source and destination are different)

shashi@linuxtechi ~}$  ls -li
total 8
794585 drwxr-xr-x 2 root root 4096 Feb  6 16:34 abc
794586 lrwxrwxrwx 1 root root    3 Feb  6 16:34 def -> abc
794584 lrwxrwxrwx 1 root root    7 Feb  6 16:33 dst.txt -> src.txt
794583 -rw-r--r-- 1 root root   12 Feb  6 16:32 src.txt

6) Symbolic links can be created for directories as explained earlier. Once these directories with symbolic link are created, inside these directories files can be created. This will make it more interesting to know how this behaves now.  When the files are created in the source directory, the same is reflected in destination directory as well. The following steps explains this clearly.

shashi@linuxtechi ~}$ $ cd abc
shashi@linuxtechi ~}$  touch 123.txt
shashi@linuxtechi ~}$  vi 123.txt
Hello
:wq!
shashi@linuxtechi ~}$  touch 456.txt
shashi@linuxtechi ~}$  cd ..
shashi@linuxtechi ~}$  ls -l
total 8
drwxr-xr-x 2 root root 4096 Feb  6 16:36 abc
lrwxrwxrwx 1 root root    3 Feb  6 16:34 def -> abc
lrwxrwxrwx 1 root root    7 Feb  6 16:33 dst.txt -> src.txt
-rw-r--r-- 1 root root   12 Feb  6 16:32 src.txt
shashi@linuxtechi ~}$ cd def
shashi@linuxtechi ~}$ ls -l
total 4
-rw-r--r-- 1 root root 6 Feb  6 16:37 123.txt
-rw-r--r-- 1 root root 0 Feb  6 16:36 456.txt
shashi@linuxtechi ~}$ cat 123.txt
Hello

Note 3:- We can have any number of nested links. But the user/admin who creates these symbolic  links should be aware of the fact that these will lead to confusion. Sometimes may be  forgotten and may create unwanted results. So they have to be careful.

Note 4:-  There are some possibilities of “symbolic” or “soft” link pointing to a non-existing link. This is called as “Dangling link”. This will be pointing to no-where.

Note 5:- there are system calls used in linux/unix to create symbolic OR hard links at programming level (using system level C/C++ programs). These are 1) symlink 2) symlinkat.

These should not be confused with command line utilities that I have described above.

Removing Soft Link / Symbolic Links

Soft or symbolic links are removed using the ‘rm’ and unlink commands.

Syntax:

# rm <soft-link-filename>

# unlink <soft-link-filename>

Removing Soft Link directories

Syntax:

# rm <soft-link-directory>

# unlink <soft-link-directory>

Conclusions:

Creating links whether hard or soft will be very helpful for admins and developers. The above reference material will come handy while understanding what type of link we are creating and how it is helpful. Also this article will help in understanding the differences and utilization of links.