Essential Linux Commands Every DevOps Engineer Should Know – Part 2

This content originally appeared on DEV Community and was authored by Ahmed Radwan

Previously, we discussed several categories of Linux commands, including system information, file and directory management, system maintenance, user and groups management, package management, and network management. In this article, we will continue with a list of additional important Linux commands that are commonly used by DevOps/Linux engineers and other technical professionals.

Before we proceed with the list of commands, let's address some common questions about Linux and DevOps. Understanding these questions and the answers will provide a better understanding of the topic in general.

Do DevOps Engineers use Linux?

Yes, DevOps engineers use Linux extensively in their work. Linux is a popular operating system for hosting applications, managing infrastructure, and automating processes, which are all key responsibilities of a DevOps engineer. Linux is also widely used for cloud computing, which is becoming increasingly important in the world of DevOps.

DevOps engineers use Linux for a variety of tasks, including managing servers and other infrastructure components, configuring network settings, deploying applications, monitoring performance and availability, and automating processes. They also use Linux-based tools for source code management, continuous integration and delivery, and collaboration.

In addition to Linux, DevOps engineers may also work with other operating systems such as Windows and macOS, depending on the needs of their organization. However, given the popularity of Linux in the DevOps world and its extensive support for various tools and technologies, it's critical for DevOps engineers to have a good understanding of Linux and its commands.

What is Shell Scripting?

Shell scripting is a way to write computer programs using commands that you would normally type in a command line or terminal. Think of it like giving your computer a set of instructions to follow, just like you would give a friend a list of things to do.

For example, imagine you want to create a script to copy all of the photos in a folder to a new folder. Instead of manually copying and pasting each file, you can write a shell script to do it for you.

Example of what the shell script might look like:

#!/bin/bash
mkdir new_folder
cp *.jpg new_folder

This script first creates a new folder called "new_folder" using the "mkdir" command. Then, it uses the "cp" command to copy all files with the ".jpg" extension from the current folder to the new folder.

Shell scripting can be really helpful when you have a repetitive task to do or need to automate a process.

What are Environment Variables?

Environment variables are like secret codes that your computer uses to keep track of important information about your system and the programs you're running. They're like little pieces of information that your computer can remember and use later, so you don't have to keep typing them in over and over again.

For example, imagine you want to run a program that requires your username and password. Instead of typing them in every time you run the program, you can use environment variables to store them.

Example of how environment variables might be used:

export USERNAME=johndoe
export PASSWORD=MySecurePassword123

In this example, we're setting two environment variables - "USERNAME" and "PASSWORD". We're assigning the value "johndoe" to the "USERNAME" variable, and "MySecurePassword123" to the "PASSWORD" variable.

Now, when we run the program that needs our username and password, we can use these environment variables instead of typing them in each time:

./my_program -u $USERNAME -p $PASSWORD

The "$" symbol tells the computer to look for the value of the environment variable, instead of treating it like a normal word.

Environment variables can be really helpful for keeping track of important information, like your username and password, or the location of important files.

What is container development?

Container development is a process of building, testing, and deploying applications using containerization technology. Containers are lightweight, portable, and self-contained environments that can run anywhere, from developer laptops to production servers.

In container development, developers package their applications and dependencies into containers, which can then be deployed to various environments, such as development, testing, and production. Containers allow developers to build applications that are isolated from the underlying infrastructure, making it easier to manage dependencies, ensure consistency across different environments, and deploy applications quickly.

Container development often involves the use of container orchestration platforms, such as Kubernetes and Docker Swarm, which help manage the deployment and scaling of containerized applications. These platforms can automatically provision and manage containers across a cluster of servers, ensuring high availability and scalability.

Container development has become increasingly popular in recent years due to its many benefits, including faster development and deployment cycles, improved scalability and portability, and better resource utilization. It also helps enable a DevOps approach, as developers and operations teams can work together more closely, using the same tools and processes, to ensure the smooth and efficient operation of containerized applications.

Now let us dive in with more Linux commands...

System Logs

System logs are a crucial tool for diagnosing and troubleshooting issues on a system. Here are three commonly used commands for managing and analyzing system logs:

• journalctl: This command is used to query and display logs from the systemd journal, which is a centralized logging system used in many modern Linux distributions. journalctl can be used to view logs from all available log sources, or from a specific log source such as the kernel, a service, or a user session. Take a look at this example:

$ journalctl -u sshd.service

This command will display the logs for the sshd.service system service.

• dmesg: This command is used to display kernel ring buffer messages, which contain information about the status of the kernel and the system hardware. dmesg can be used to view the most recent kernel messages or all kernel messages since the last boot. Example:

$ dmesg | tail

This command will display the last few lines of the kernel ring buffer.

• tail: This command is used to show the last few lines of a file. It can be used to monitor log files in real-time, by using the -f option to follow the file as new entries are added. Example:

$ tail -f /var/log/syslog

This command will display the latest entries in the syslog file as they are added.

File Compression and Archiving

file compression and archiving are essential tools for managing large amounts of data. Here are a few commonly used commands for file compression and archiving:

• tar: This command is used to create and extract tar archives, which are used to bundle files and directories together. tar can be used to compress and decompress files using the gzip or bzip2 algorithms. Here are some examples:

$ tar -cvzf archive.tar.gz /path/to/directory

This command will create a new tar archive called archive.tar.gz of the /path/to/directory directory, using the gzip compression algorithm.

$ tar -xvzf archive.tar.gz

This command will extract the contents of the archive.tar.gz archive to the current directory.

• gzip and bzip2: These commands are used to compress and decompress files using the gzip and bzip2 algorithms, respectively. Here are some examples:

$ gzip file.txt

This command will compress the file.txt file using the gzip algorithm, creating a new file called file.txt.gz.

$ gzip -d file.txt.gz

This command will decompress the file.txt.gz file, creating a new file called file.txt.

$ bzip2 file.txt

This command will compress the file.txt file using the bzip2 algorithm, creating a new file called file.txt.bz2.

$ bzip2 -d file.txt.bz2

This command will decompress the file.txt.bz2 file, creating a new file called file.txt.

• zip and unzip: These commands are used to create and extract zip archives, which are a popular file format for compressing and archiving files. Here are some examples:

$ zip archive.zip file1.txt file2.txt

This command will create a new zip archive called archive.zip containing the file1.txt and file2.txt files.

$ unzip archive.zip

This command will extract the contents of the archive.zip file to the current directory.

• 7z: This command is used to create and extract 7z archives, which are another popular file format for compressing and archiving files. Here are some examples:

$ 7z a archive.7z /path/to/directory

This command will create a new 7z archive called archive.7z of the /path/to/directory directory.

$ 7z x archive.7z

This command will extract the contents of the archive.7z archive to the current directory.

System Security

Secure remote access and network security are essential for protecting sensitive data and systems. Here are a few commonly used commands for secure remote access and network security:

• ssh: This command is used to securely connect to a remote system over the network. ssh encrypts all traffic between the local and remote systems, providing a secure and private connection. Example:

$ ssh user@remote_host

This command will securely connect to the remote_host system as the user user.

• scp: This command is used to securely copy files to or from a remote system over the network. scp uses the same encryption as ssh to protect the data in transit. Example:

$ scp file.txt user@remote_host:/path/to/destination/

This command will securely copy the file.txt file to the remote_host system at the /path/to/destination/ directory.

• openssl: This command is a versatile cryptographic library that can be used for a wide range of security tasks, such as SSL/TLS connections, certificate management, and more. Here are some examples:

$ openssl s_client -connect example.com:443

This command will establish an SSL/TLS connection to the example.com web server on port 443, and display the server's SSL/TLS certificate.

$ openssl x509 -in cert.pem -text

This command will display the details of the SSL/TLS certificate stored in the cert.pem file.

• iptables: This command-line tool is used to configure the Linux kernel's firewall, providing network security by controlling the incoming and outgoing network traffic. Example:

$ sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

This command will allow incoming TCP traffic on port 22 (the default SSH port) through the firewall.

Process Management

Process management is essential for controlling system resources and optimizing system performance. Here are a few commonly used commands for process management:

• pgrep: This command is used to search for processes by name or other attributes. pgrep returns the process ID (PID) of matching processes, which can be used with other commands such as kill or renice. Example:

$ pgrep -u user_name -f process_name

This command will search for processes with the name process_name that are being run by the user_name user.

• pkill: This command is used to send a signal to one or more processes to terminate them. pkill can be used with the same search criteria as pgrep to select which processes to terminate. Example:

$ pkill -u user_name -f process_name

This command will terminate all processes with the name process_name that are being run by the user_name user.

• nice: This command is used to set the priority of a process. The priority value ranges from -20 (highest priority) to 19 (lowest priority), with 0 being the default priority. Example:

$ nice -n 10 command

This command will run the command with a lower priority (higher nice value) than the default priority.

• renice: This command is used to change the priority of a running process. The command takes the PID of the process as an argument, as well as the new priority value. Example:

$ renice -n 10 1234

This command will change the priority of the process with PID 1234 to a lower priority (higher nice value) than the current priority.

• screen: This command is a terminal multiplexer that allows you to run multiple terminal sessions within a single screen session. This is particularly useful when working with remote systems, as it allows you to detach and re-attach to the terminal sessions as needed. Example:

$ screen

This command will start a new screen session. To detach from the session, press Ctrl-A, then d. To re-attach to the session, use the screen -r command.

System Monitoring

system monitoring is essential for managing system resources and optimizing system performance. Here are a few commonly used commands for system monitoring:

• htop: This command is an interactive process viewer that shows CPU and memory usage, and allows you to manage processes. htop provides a real-time view of the system's process usage and system resources, making it a powerful tool for managing system performance. Here's an example:

$ htop

This command will start the htop process viewer.

• nmon: This command is a system performance monitor that shows CPU, memory, disk, and network usage in real time. nmon provides a detailed view of the system's performance, allowing you to identify performance bottlenecks and optimize system performance. Here's an example:

$ nmon

This command will start the nmon performance monitor.

1. iostat: This command is used to display CPU, disk, and network I/O statistics. iostat provides a detailed view of the system's I/O usage, allowing you to identify I/O bottlenecks and optimize system performance. Here's an example:

$ iostat -d -x 1

This command will display the disk I/O statistics every second.

• vmstat: This command is used to display virtual memory statistics. vmstat provides a detailed view of the system's memory usage, allowing you to identify memory bottlenecks and optimize system performance. Here's an example:

$ vmstat 1

This command will display the memory statistics every second.

• free: This command is used to display memory usage statistics. free provides a simple view of the system's memory usage, allowing you to identify memory usage trends and optimize system performance. Here's an example:

$ free -h

This command will display the memory usage statistics in a human-readable format.

• perf: This command is a profiling tool for Linux that can be used to analyze system performance. perf can be used to identify performance bottlenecks, analyze system call and interrupt activity, and optimize program behavior. Here's an example:

$ perf record ./myprogram

This command will record performance data for myprogram and generate a report using perf report.

Text Processing

text processing is an essential task that involves manipulating text data in various ways. Here are a few commonly used commands for text processing:

• grep: This command is used to search for a pattern in a file or input stream. grep can be used with regular expressions to search for complex patterns in text data. Here's an example:

$ grep 'pattern' file.txt

This command will search for lines in file.txt that contain the word pattern.

• sed: This command is a stream editor that can perform various text transformations on an input stream. sed can be used to find and replace text, insert or delete lines, and perform other text transformations. Here's an example:

$ sed 's/pattern/replacement/g' file.txt

This command will replace all occurrences of pattern with replacement in file.txt.

• awk: This command is a powerful programming language for text processing. awk can be used to manipulate text data in many ways, such as selecting columns, performing calculations, and formatting output. Here's an example:

$ awk '{ print $1, $3 }' file.txt

This command will print the first and third columns of file.txt.

• cut: This command is used to select portions of each line of a file. cut can be used to extract specific columns or ranges of characters from text data. Here's an example:

$ cut -d ',' -f 1,3 file.csv

This command will extract the first and third columns of file.csv, assuming the columns are delimited by commas.

• tr: This command is used to translate or delete characters in a stream of text. tr can be used to convert uppercase characters to lowercase, remove whitespace, or perform other character transformations. Here's an example:

$ tr '[:upper:]' '[:lower:]' < file.txt

This command will convert all uppercase characters in file.txt to lowercase.

• sort: This command is used to sort the lines of a file or input stream. sort can be used to sort text data in various ways, such as numerically, alphabetically, or based on field positions. Here's an example:

$ sort file.txt

This command will sort the lines of file.txt in alphabetical order.

• uniq: This command is used to remove duplicate lines from a file or input stream. uniq can be used to remove consecutive or non-consecutive duplicate lines. Here's an example:

$ uniq file.txt

This command will remove all consecutive duplicate lines from file.txt.

• wc: This command is used to count the number of lines, words, and characters in a file or input stream. wc can be used to generate word counts and perform other text statistics. Here's an example:

$ wc file.txt

This command will display the line, word, and character count of file.txt.

Debugging Tools

Debugging is an important part of software development, and Linux provides a variety of powerful debugging tools for developers. Here are a few commonly used commands for debugging in Linux:

• strace: This command is used to trace system calls and signals of a process. strace can be used to identify system call errors, diagnose application problems, and analyze system performance. Here's an example:

$ strace ./myprogram

$ strace ./myprogram

This command will trace the system calls and signals of myprogram.

• ltrace: This command is used to trace library calls of a process. ltrace can be used to identify library call errors, diagnose application problems, and analyze system performance. Here's an example:

$ ltrace ./myprogram

$ ltrace ./myprogram

This command will trace the library calls of myprogram.

• gdb: This command is the GNU Debugger, used to debug and analyze C/C++ programs. gdb can be used to identify errors, analyze program flow, and modify program behavior. Here's an example:

$ gdb ./myprogram

$ gdb ./myprogram

This command will start the GNU Debugger for myprogram.

• valgrind: This command is a tool for memory debugging, memory leak detection, and profiling. valgrind can be used to identify memory errors, diagnose application problems, and analyze system performance. Here's an example:

$ valgrind ./myprogram

$ valgrind ./myprogram

This command will run myprogram under valgrind to detect memory errors.

Miscellaneous

• find: This command is used to search for files in a directory hierarchy. Here's an example:

$ find /home/user/Documents -name "*.txt"

This command will search for all files with the extension .txt in the Documents directory of the user user.

• cron: This command is a tool for scheduling periodic tasks. It can be used to schedule commands to run at specific intervals, such as daily, weekly, or monthly. Here's an example:

$ crontab -e

This command will open the user's crontab file in an editor, where they can add entries to schedule commands to run at specific times.

• watch: This command is used to execute a command periodically and display the output. It can be used to monitor the output of a command in real-time. Here's an example:

$ watch -n 1 "ls -l"

This command will run the ls -l command every 1 second and display the output in the terminal.

• history: This command is used to display a list of recently executed commands. It can be used to recall previous commands and to repeat them. Here's an example:

$ history

This command will display a list of the user's recently executed commands.

• sudo: This command is used to execute a command with superuser privileges. It can be used to perform tasks that require root-level access, such as installing packages or modifying system files. Here's an example:

$ sudo apt-get update

This command will update the package list using the apt-get command with root-level access.

• diff: This command is used to compare two files or directories and display the differences between them. It can be used to identify changes, additions, and deletions between two versions of a file or directory. Here's an example:

$ diff file1.txt file2.txt

This command will compare the contents of file1.txt and file2.txt and display the differences between them.

• rsync: This command is used to synchronize files and directories between two locations, such as a local machine and a remote server. It can be used to transfer files securely and efficiently and to perform incremental backups. Here's an example:

$ rsync -avz /path/to/local/dir user@remote:/path/to/remote/dir

This command will synchronize the contents of the local directory /path/to/local/dir with the remote directory /path/to/remote/dir on the server remote, using the rsync protocol.

• curl: This command is used to transfer data from or to a server, using various protocols (such as HTTP, FTP, or SMTP). curl can be used to test API endpoints, download files, and perform other network-related tasks. Here's an example:

$ curl http://example.com

This command will fetch the contents of http://example.com and display them in the terminal.

• wget: This command is used to download files from the Internet. wget can be used to download files in the background, resume interrupted downloads and mirror entire websites. Here's an example:

$ wget http://example.com/file.zip

This command will download file.zip from http://example.com.

• ssh-keygen: This command is used to generate SSH key pairs for secure authentication. ssh-keygen can be used to create new keys, import or export keys, and manage key authentication settings. Here's an example:

$ ssh-keygen -t rsa

This command will generate a new RSA key pair for SSH authentication.

Conclusion

DevOps engineers are responsible for the development, deployment, and management of software applications and systems. They need to be proficient in a wide range of tools and technologies, including various Linux commands, to be successful in their role. Here are some reasons why it's critical for DevOps engineers to be aware of all these commands :

1. Streamline workflows: By mastering these Linux commands, DevOps engineers can streamline their workflows, automate repetitive tasks, and become more efficient in their daily work.

2. Improve productivity: Knowing these commands can help DevOps engineers troubleshoot issues faster, perform tasks with greater accuracy, and respond to incidents more quickly, ultimately improving productivity.

3. Manage infrastructure: DevOps engineers need to be able to manage servers, containers, and other infrastructure components, and many of these commands are essential for that.

4. Troubleshoot issues: When issues arise in production, DevOps engineers need to be able to quickly diagnose and fix the problem. Knowing these Linux commands can help them identify the root cause of issues and resolve them quickly.

Hers is a good assignment for you to practice more with the commands above and what's in part 1 as well. Creating a cheatsheet of these Linux commands can help you as a DevOps engineer to quickly access and reference these tools when needed. It can also serve as a reference for new team members or as a reminder for infrequently used commands. By being proficient in these commands, DevOps engineers can be more effective in their roles, ultimately contributing to the success of the organization.

This content originally appeared on DEV Community and was authored by Ahmed Radwan