5 log parsing commands

Have you ever tried to find something in the server log file? While downloading and opening the file in an editor might seem straightforward, it’s often time-consuming and unproductive. Instead, using command-line tools can be more efficient and effect…


This content originally appeared on DEV Community and was authored by cuongnp

Have you ever tried to find something in the server log file? While downloading and opening the file in an editor might seem straightforward, it's often time-consuming and unproductive. Instead, using command-line tools can be more efficient and effective. Here are some common commands you should try.

The practice file today is system.log

2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped

1. Display the Contents of the Log File

cat Command

  • Purpose: used to display the content of files.
  • Usage: cat filename
  • Example: cat server.log 
$ cat system.log
2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped

2. Search for lines

grep

  • Purpose: powerful command for searching text using patterns, and filtering log entries based on specific criteria.
  • Usage: cat filename | grep “filter-condition” or grep condition filename
  • Example: grep "ERROR" server.log 
$ grep "ERROR" system.log

2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied

3. Display Lines with Customize Condition

awk

  • Purpose: Introduce awk as a powerful text processing tool, ideal for manipulating data and generating reports.
  • Usage: awk condition filename
  • Example 1: Display lines with timestamps between 13:50:00 and 14:00:00: 
$ awk '/13:5[0-9]:[0-9][0-9]/ || /14:00:00/' system.log

2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
  • Example 2: Extract and print the date and time of each entry 
$ awk '{print $1, $2}' system.log

2024-06-12 13:39:30
2024-06-12 13:40:12
2024-06-12 13:41:05
2024-06-12 13:42:16
2024-06-12 13:43:27
2024-06-12 13:44:38
2024-06-12 13:45:49
2024-06-12 13:46:50
2024-06-12 13:47:51
2024-06-12 13:48:52
2024-06-12 13:49:53
2024-06-12 13:50:54
2024-06-12 13:51:55
2024-06-12 13:52:56
2024-06-12 13:53:57
2024-06-12 13:54:58
2024-06-12 13:55:59
2024-06-12 13:56:00
2024-06-12 13:57:01
2024-06-12 13:58:02
2024-06-12 13:59:03
2024-06-12 14:00:04
2024-06-12 14:01:05
2024-06-12 14:02:06
2024-06-12 14:03:07
2024-06-12 14:04:08
2024-06-12 14:05:09
2024-06-12 14:06:10
2024-06-12 14:07:11
2024-06-12 14:08:12
2024-06-12 14:09:13
2024-06-12 14:10:14
2024-06-12 14:11:15
2024-06-12 14:12:16
2024-06-12 14:13:17
2024-06-12 14:14:18

4. Sort Log Entries

sort

  • Purpose: Sort lines in text files.
  • Usage: Sort log entries by date, time, or any other field.
  • Example: cat system.log | awk '{print $1, $2, $3}' | sort 
$ cat system.log | awk '{print $1, $2, $3}' | sort
2024-06-12 13:39:30 [INFO]
2024-06-12 13:40:12 [ERROR]
2024-06-12 13:41:05 [INFO]
2024-06-12 13:42:16 [WARNING]
2024-06-12 13:43:27 [INFO]
2024-06-12 13:44:38 [ERROR]
2024-06-12 13:45:49 [INFO]
2024-06-12 13:46:50 [INFO]
2024-06-12 13:47:51 [INFO]
2024-06-12 13:48:52 [INFO]
2024-06-12 13:49:53 [INFO]
2024-06-12 13:50:54 [ERROR]
2024-06-12 13:51:55 [WARNING]
2024-06-12 13:52:56 [INFO]
2024-06-12 13:53:57 [ERROR]
2024-06-12 13:54:58 [INFO]
2024-06-12 13:55:59 [INFO]
2024-06-12 13:56:00 [INFO]
2024-06-12 13:57:01 [INFO]
2024-06-12 13:58:02 [ERROR]
2024-06-12 13:59:03 [INFO]
2024-06-12 14:00:04 [WARNING]
2024-06-12 14:01:05 [INFO]
2024-06-12 14:02:06 [ERROR]
2024-06-12 14:03:07 [INFO]
2024-06-12 14:04:08 [INFO]
2024-06-12 14:05:09 [INFO]
2024-06-12 14:06:10 [INFO]
2024-06-12 14:07:11 [INFO]
2024-06-12 14:08:12 [ERROR]
2024-06-12 14:09:13 [WARNING]
2024-06-12 14:10:14 [INFO]
2024-06-12 14:11:15 [ERROR]
2024-06-12 14:12:16 [INFO]
2024-06-12 14:13:17 [INFO]
2024-06-12 14:14:18 [INFO]

5. Unique the display result

uniq

  • Purpose: Describe how uniq removes or counts duplicate lines.
  • Usage: cat filename | uniq -c
  • Example: cat server.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c 
cat system.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c
   2 Cleanup
   2 Could
   4 Failed

Final thought

Analyzing information files is crucial for system administration, troubleshooting, and monitoring. Using a combination of command-line tools like cat, grep, awk, sort, and uniq, you can effectively manage and extract valuable insights from your log files.


This content originally appeared on DEV Community and was authored by cuongnp


Print Share Comment Cite Upload Translate Updates
APA

cuongnp | Sciencx (2024-06-23T06:00:05+00:00) 5 log parsing commands. Retrieved from https://www.scien.cx/2024/06/23/5-log-parsing-commands/

MLA
" » 5 log parsing commands." cuongnp | Sciencx - Sunday June 23, 2024, https://www.scien.cx/2024/06/23/5-log-parsing-commands/
HARVARD
cuongnp | Sciencx Sunday June 23, 2024 » 5 log parsing commands., viewed ,<https://www.scien.cx/2024/06/23/5-log-parsing-commands/>
VANCOUVER
cuongnp | Sciencx - » 5 log parsing commands. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/06/23/5-log-parsing-commands/
CHICAGO
" » 5 log parsing commands." cuongnp | Sciencx - Accessed . https://www.scien.cx/2024/06/23/5-log-parsing-commands/
IEEE
" » 5 log parsing commands." cuongnp | Sciencx [Online]. Available: https://www.scien.cx/2024/06/23/5-log-parsing-commands/. [Accessed: ]
rf:citation
» 5 log parsing commands | cuongnp | Sciencx | https://www.scien.cx/2024/06/23/5-log-parsing-commands/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts