Bootstrap FreeKB - Amazon Web Services (AWS) - Send log data to Kinesis Data Stream using Kinesis Agent
Amazon Web Services (AWS) - Send log data to Kinesis Data Stream using Kinesis Agent


This assumes you are familiar with Kinesis. If not, check out my article Getting Started with Kinesis.

Let's say you have a Kinesis Data Stream named my-kinesis-data-stream.

 

There are a few different ways to install and setup the Amazon Kinesis Java Agent.

Regardless of which Amazon Kinesis Java Agent you use, each will use agent.json to configure files that should be put onto your Kinesis Data Stream. For example, let's say your agent.json file contains something like this. In this example, /tmp/app.log files will be put onto my-kinesis-data-stream.

There are much better options rather that having your AWS Access Key and Secret Key in agent.json, but again, I'm just trying to keep this simple just to get Kinesis up and working.

]$ cat agent.json
{
  "awsAccessKeyId": "ABK349fJFM499FMDFD23",
  "awsSecretAccessKey": "Ab587Cd234Ef98gG9834Aa345908bB23409234Cc",
  "flows": [
    {
      "filePattern": "/tmp/app.log*",
      "kinesisStream": "my-kinesis-data-stream"
    }
  ]
}

 

Let's create /tmp/app.log and append Hello and World to app.log.

~]$ touch /tmp/app.log
~]$ echo "Hello" >> /tmp/app.log
~]$ echo "World" >> /tmp/app.log

 

/var/log/aws-kinesis-agent/aws-kinesis-agent.log should contain something like this, which shows /tmp/app.log were put onto my-kinesis-data-stream.

2024-05-17 01:03:22.323+0000  (FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.

2024-05-17 01:03:22.327+0000  (FileTailer[fh:yourdeliverystream:/tmp/app.log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:yourdeliverystream:/tmp/app.log*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.

2024-05-17 01:03:22.329+0000  (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 420046ms

2024-05-17 01:03:27.041+0000  (FileTailer[fh:yourdeliverystream:/tmp/app.log*]) com.amazon.kinesis.streaming.agent.tailing.FirehoseParser [INFO] FirehoseParser[fh:yourdeliverystream:/tmp/app.log*]: Opening /tmp/app.log for parsing.
2024-05-17 01:03:27.041+0000  (FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*]) com.amazon.kinesis.streaming.agent.tailing.KinesisParser [INFO] KinesisParser[kinesis:my-kinesis-data-stream:/tmp/app.log*]: Opening /tmp/app.log for parsing.

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee



Comments


Add a Comment


Please enter b071ca in the box below so that we can be sure you are a human.