Combine merge or join lines in a variable in BASH

Home > Search
  by

Let's consider a scenario where you need to merges lines of data in a variable. In this example, the $foo variable contains two lines of data that will be merged into one line of data.

foo="
Employee,name=John Doe,occupation=Engineer
Employee,name=John Doe,department=IT Infrastructure
Employee,name=Jane Doe,occupation=Sales Manager
Employee,name=Jane Doe,department=Sales Department
"

 

The following script demonstrates the logic.

#!/bin/bash

foo="
Employee,name=John Doe,occupation=Engineer
Employee,name=John Doe,department=IT Infrastructure
Employee,name=Jane Doe,occupation=Sales Manager
Employee,name=Jane Doe,department=Sales Department
"

# Tell bash to split at newline
IFS=$'\n'

# Create an empty variable that will contain the merged data
merged=""

# Loop through the variable that contains the lines that need to be merged ($foo in this example)
for line in $foo
do

  # Create unique variable for each field of data from each line in the $foo variable
  field2=$(echo $line | awk -F',' '{print $2}') # name
  field3=$(echo $line | awk -F',' '{print $3}') # occupation or department

  echo "Evaluating '$field2, $field3'"

  # if $merged does not yet contain a line with the common field (name in this example), then execute the logic inside of the if statement
  if [[ $merged != *"$field2"* ]]; then

    echo "\$merged does not contain *$field2* - Adding '$field2 $field3' to the \$merged variable"

    # Append the data to the $merged variable
    merged+="$field2, $field3"

    # Add a new line
    merged+=$'\n'

  # if $merged contain a line with the common field (name in this example) but the line does not yet contain the data in the $field3 variable, then execute the logic inside this elif statement
  elif [[ $merged = *"$field2"* && $merged != *"$field3"* ]]; then

    echo "\$merged already contains '*$field2*' - Appending '$field3' to the line that contains '$field2' in \$merged"

    # Store the data that already exists in $merged in a variable
    for data in $merged
    do
      existingData=$(echo "$data" | grep "$field2")
    done

    # Append the data to the $merged variable
    merged="${merged/$existingData/$existingData, $field3}"


  # if $merged already contains the data, do nothing
  else [[ $merged = *"$field2"* && $merged = *"$field3"* ]]

    echo "\$merged already contains '$field2' and '$field3' - skipping this record and moving on"

  fi

  echo

done 

echo
echo "-----FINAL RESULTS-----"

# Loop through each line
for line in $merged
do
  echo $line
done

# Cleanup
unset IFS

 

Running this script will produce a variable that contains the merged data.

name=John Doe, occupation=Engineer, department=IT Infrastructure
name=Jane Doe, occupation=Sales Manager, department=Sales Department

 



Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.




Please enter in the box below so that we can be sure you are a human.




Comments