Here is an example of how to use a try catch block finally in PowerShell. The only thing I don't like about this is that the finally block will always be invoked so I like to use return in the catch block to avoid moving into the finally block.

try {
    Remove-Item "C:\Path\To\File"
}
catch {
    Write-Warning "caught the following error"
    foreach ($line in $Error) {
      Write-Warning $line
    }
    return
}
finally {
    Write-Host "Finally do this"
}

If the exception being caught is not critical you might want to go with Continue instead of return.

try {
    Remove-Item "C:\Path\To\File"
}
catch {
    Write-Warning "caught the following error"
    Write-Warning $Error[0]
    Continue
}