
This assumes you are familiar with Kinesis. If not, check out my article Getting Started with Kinesis.
Let's say you have a Kinesis Data Stream named my-kinesis-data-stream.
There are a few different ways to install and setup the Amazon Kinesis Java Agent.
- Install and setup Amazon Kinesis Java Agent from GitHub
- Install and setup Amazon Kinesis Java Agent using yum or dnf
- Install and setup Amazon Kinesis Java Agent using Docker
Regardless of which Amazon Kinesis Java Agent you use, each will use agent.json to configure files that should be put onto your Kinesis Data Stream. For example, let's say your agent.json file contains something like this. In this example, /tmp/app.log files will be put onto my-kinesis-data-stream.
There are much better options rather that having your AWS Access Key and Secret Key in agent.json, but again, I'm just trying to keep this simple just to get Kinesis up and working.
]$ cat agent.json
{
"awsAccessKeyId": "ABK349fJFM499FMDFD23",
"awsSecretAccessKey": "Ab587Cd234Ef98gG9834Aa345908bB23409234Cc",
"flows": [
{
"filePattern": "/tmp/app.log*",
"kinesisStream": "my-kinesis-data-stream"
}
]
}
Let's create /tmp/app.log and append Hello and World to app.log.
~]$ touch /tmp/app.log
~]$ echo "Hello" >> /tmp/app.log
~]$ echo "World" >> /tmp/app.log
/var/log/aws-kinesis-agent/aws-kinesis-agent.log should contain something like this, which shows /tmp/app.log were put onto my-kinesis-data-stream.
2024-05-17 01:03:22.323+0000 (FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2024-05-17 01:03:22.327+0000 (FileTailer[fh:yourdeliverystream:/tmp/app.log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:yourdeliverystream:/tmp/app.log*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2024-05-17 01:03:22.329+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 420046ms
2024-05-17 01:03:27.041+0000 (FileTailer[fh:yourdeliverystream:/tmp/app.log*]) com.amazon.kinesis.streaming.agent.tailing.FirehoseParser [INFO] FirehoseParser[fh:yourdeliverystream:/tmp/app.log*]: Opening /tmp/app.log for parsing.
2024-05-17 01:03:27.041+0000 (FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*]) com.amazon.kinesis.streaming.agent.tailing.KinesisParser [INFO] KinesisParser[kinesis:my-kinesis-data-stream:/tmp/app.log*]: Opening /tmp/app.log for parsing.
Did you find this article helpful?
If so, consider buying me a coffee over at