Bash (Scripting) - join lines in a variable

by
Jeremy Canfield |
Updated: March 08 2020
| Bash (Scripting) articles
Let's consider a scenario where you need to merges lines of data in a variable. In this example, the $foo variable contains two lines of data that will be merged into one line of data.
foo="
Employee,name=John Doe,occupation=Engineer
Employee,name=John Doe,department=IT Infrastructure
Employee,name=Jane Doe,occupation=Sales Manager
Employee,name=Jane Doe,department=Sales Department
"
The following script demonstrates the logic.
#!/bin/bash
foo="
Employee,name=John Doe,occupation=Engineer
Employee,name=John Doe,department=IT Infrastructure
Employee,name=Jane Doe,occupation=Sales Manager
Employee,name=Jane Doe,department=Sales Department
"
# Tell bash to split at newline
IFS=$'\n'
# Create an empty variable that will contain the merged data
merged=""
# Loop through the variable that contains the lines that need to be merged ($foo in this example)
for line in $foo
do
# Create unique variable for each field of data from each line in the $foo variable
field2=$(echo $line | awk -F',' '{print $2}') # name
field3=$(echo $line | awk -F',' '{print $3}') # occupation or department
echo "Evaluating '$field2, $field3'"
# if $merged does not yet contain a line with the common field (name in this example), then execute the logic inside of the if statement
if [[ $merged != *"$field2"* ]]; then
echo "\$merged does not contain *$field2* - Adding '$field2 $field3' to the \$merged variable"
# Append the data to the $merged variable
merged+="$field2, $field3"
# Add a new line
merged+=$'\n'
# if $merged contain a line with the common field (name in this example) but the line does not yet contain the data in the $field3 variable, then execute the logic inside this elif statement
elif [[ $merged = *"$field2"* && $merged != *"$field3"* ]]; then
echo "\$merged already contains '*$field2*' - Appending '$field3' to the line that contains '$field2' in \$merged"
# Store the data that already exists in $merged in a variable
for data in $merged
do
existingData=$(echo "$data" | grep "$field2")
done
# Append the data to the $merged variable
merged="${merged/$existingData/$existingData, $field3}"
# if $merged already contains the data, do nothing
else [[ $merged = *"$field2"* && $merged = *"$field3"* ]]
echo "\$merged already contains '$field2' and '$field3' - skipping this record and moving on"
fi
echo
done
echo
echo "-----FINAL RESULTS-----"
# Loop through each line
for line in $merged
do
echo $line
done
# Cleanup
unset IFS
Running this script will produce a variable that contains the merged data.
name=John Doe, occupation=Engineer, department=IT Infrastructure
name=Jane Doe, occupation=Sales Manager, department=Sales Department
Did you find this article helpful?
If so, consider buying me a coffee over at