Cheatsheet & Examples: awk

Basic Text Processing and Pattern Matching
Example Usage:
awk '{ print }' file.txt

What it does:
Prints every line of the input file.

Command-line Arguments Explained:

• { print }: The default awk script that outputs each line.
• file.txt: The input file to process.

Filtering Lines Based on Patterns
Example Usage:
awk '/error/ { print }' log.txt

What it does:
Prints lines containing the word "error" from the input file.

Command-line Arguments Explained:

• /error/: A pattern to match lines containing "error".
• { print }: The action to execute for matching lines.
• log.txt: The input file to search within.

Print Specific Fields from a File
Example Usage:
awk '{ print $1, $3 }' data.txt

What it does:
Displays the first and third fields of each line in the input file.

Command-line Arguments Explained:

• { print $1, $3 }: Outputs the first and third fields of each line.
• data.txt: The input file to extract fields from.

Using Field Separators with -F
Example Usage:
awk -F, '{ print $2 }' data.csv

What it does:
Prints the second field of each line in a CSV file (using comma as the separator).

Command-line Arguments Explained:

• -F,: Sets the field separator to a comma.
• { print $2 }: Outputs the second field.
• data.csv: The input file to process.

Summing Values in a Column
Example Usage:
awk '{ sum += $1 } END { print sum }' numbers.txt

What it does:
Calculates the sum of values in the first column of the input file and prints it.

Command-line Arguments Explained:

• { sum += $1 }: Accumulates the value of the first field.
• END { print sum }: Outputs the final sum after processing all lines.
• numbers.txt: The input file containing numerical data.

Printing Line Numbers
Example Usage:
awk '{ print NR, $0 }' file.txt

What it does:
Displays each line of the file prepended with its line number.

Command-line Arguments Explained:

• NR: Built-in variable representing the current line number.
• $0: Represents the entire line.
• file.txt: The input file to process.

Using BEGIN and END Blocks
Example Usage:
awk 'BEGIN { print "Start" } END { print "End" }' file.txt

What it does:
Prints "Start" before processing the file and "End" after processing completes.

Command-line Arguments Explained:

• BEGIN { print "Start" }: Executes once at the beginning.
• END { print "End" }: Executes once at the end.
• file.txt: The input file being processed.

Formatting Output with printf
Example Usage:
awk '{ printf "%-10s %5d\n", $1, $2 }' data.txt

What it does:
Formats and prints the first field as a string, the second as a number, with specific spacing.

Command-line Arguments Explained:

• printf: Formats output according to the specified pattern.
• %-10s: Left-justifies the first field in a 10-character width.
• %5d: Right-justifies the second field in a 5-character width.
• data.txt: The input file to process.

Handling Multiple Files
Example Usage:
awk 'NR == 1 { print FILENAME }' file1.txt file2.txt

What it does:
Prints the filename of each input file at the first line of processing.

Command-line Arguments Explained:

• NR == 1: Condition to check for the first line of each file.
• FILENAME: Built-in variable containing the current input file name.
• file1.txt file2.txt: Multiple input files to process.

Using Variables with -v
Example Usage:
awk -v threshold=100 '$1 > threshold { print $0 }' data.txt

What it does:
Filters and prints lines where the first field exceeds the value of threshold.

Command-line Arguments Explained:

• -v threshold=100: Defines a user variable threshold with the value 100.
• $1 > threshold: Condition to check if the first field exceeds threshold.
• data.txt: The input file to process.

Calculating Field Counts
Example Usage:
awk '{ print NF }' file.txt

What it does:
Prints the number of fields in each line of the input file.

Command-line Arguments Explained:

• NF: Built-in variable indicating the number of fields in the current line.
• file.txt: The input file to analyze.

Using Regular Expressions in Patterns
Example Usage:
awk '/^[A-Z]/ { print }' file.txt

What it does:
Prints lines that start with an uppercase letter.

Command-line Arguments Explained:

• /^[A-Z]/: A regular expression pattern matching lines starting with an uppercase letter.
• { print }: Action to output matching lines.
• file.txt: The input file to process.

Substituting Text in a File
Example Usage:
awk '{ gsub(/old/, "new"); print }' file.txt

What it does:
Replaces all occurrences of "old" with "new" in each line of the input file.

Command-line Arguments Explained:

• gsub(/old/, "new"): Substitutes "old" with "new" globally in the line.
• print: Outputs the modified line.
• file.txt: The input file to process.

Counting Occurrences of a Pattern
Example Usage:
awk '/error/ { count++ } END { print "Errors:", count }' log.txt

What it does:
Tallies the number of lines matching "error" and prints the total.

Command-line Arguments Explained:

• /error/: Pattern to match lines containing "error".
• count++: Increments a variable for each match.
• END { print "Errors:", count }: Outputs the final count after processing.
• log.txt: The input file to analyze.

Printing Lines with Specific Field Lengths
Example Usage:
awk 'length($0) > 80 { print }' file.txt

What it does:
Prints lines in the file that are longer than 80 characters.

Command-line Arguments Explained:

• length($0) > 80: Condition checks if the line length exceeds 80.
• $0: Represents the entire line.
• file.txt: The input file to process.

Using for Loops to Iterate Fields
Example Usage:
awk '{ for(i=1; i<=NF; i++) print $i }' file.txt

What it does:
Prints each field of every line on a separate line.

Command-line Arguments Explained:

• for(i=1; i<=NF; i++): Iterates over all fields in the line.
• print $i: Outputs each field individually.
• file.txt: The input file to process.

Using if Conditions for Filtering
Example Usage:
awk '$2 > 50 { print $1 }' data.txt

What it does:
Prints the first field of lines where the second field exceeds 50.

Command-line Arguments Explained:

• $2 > 50: Condition checks if the second field's value is greater than 50.
• print $1: Outputs the first field of matching lines.
• data.txt: The input file to process.

Changing Output Field Separator with OFS
Example Usage:
awk '{ OFS=","; print $1, $2 }' data.txt

What it does:
Prints the first and second fields separated by a comma.

Command-line Arguments Explained:

• OFS=",": Sets the output field separator to a comma.
• print $1, $2: Outputs the first and second fields with the new separator.
• data.txt: The input file to process.

Using NR to Process Specific Lines
Example Usage:
awk 'NR == 5 { print }' file.txt

What it does:
Prints the fifth line of the input file.

Command-line Arguments Explained:

• NR == 5: Condition checks if the current line number is 5.
• print: Outputs the matching line.
• file.txt: The input file to process.

Using FILENAME in END Block
Example Usage:
awk 'END { print FILENAME }' file.txt

What it does:
Prints the name of the last processed file after finishing.

Command-line Arguments Explained:

• END { print FILENAME }: Executes after all lines are processed and outputs the filename.
• file.txt: The input file being processed.

Printing Fields with Custom Separator
Example Usage:
awk 'BEGIN { FS=":"; OFS="," } { print $1, $2 }' data.txt

What it does:
Processes fields separated by colons and outputs them with commas.

Command-line Arguments Explained:

• BEGIN { FS=":"; OFS="," }: Sets input/output field separators before processing.
• print $1, $2: Outputs the first and second fields with the new separator.
• data.txt: The input file to process.