
This assumes you are familiar with Kinesis. If not, check out my article Getting Started with Kinesis.
There are a few different Kinesis Agents. For example, in the AWS console, after selecting your Kinesis Data Stream, the Applications tab lists a few Kinesis Agents. These are known as the Producers.
There are a few different ways to install and setup the Amazon Kinesis Java Agent.
- Install and setup Amazon Kinesis Java Agent from GitHub
- Install and setup Amazon Kinesis Java Agent using yum or dnf (this article)
- Install and setup Amazon Kinesis Java Agent using Docker
dnf install or yum install can be used to install the Kinesis Agent.
dnf install aws-kinesis-agent
The /etc/aws-kinesis/agent.json file should have something like this. Let's update "kinesisStream" to have the name of your Kinesis Data Stream.
There are much better options rather that having your AWS Access Key and Secret Key in agent.json, but again, I'm just trying to keep this simple just to get Kinesis up and working.
~]$ cat /etc/aws-kinesis/agent.json
{
"awsAccessKeyId": "ABK349fJFM499FMDFD23",
"awsSecretAccessKey": "Ab587Cd234Ef98gG9834Aa345908bB23409234Cc",
"flows": [
{
"filePattern": "/tmp/app.log*",
"kinesisStream": "my-kinesis-data-stream"
}
]
}
Start and enable aws-kinesis-agent.
sudo systemctl start aws-kinesis-agent
sudo systemctl enable aws-kinesis-agent
And verify that aws-kinesis-agent is running.
~]$ sudo systemctl status aws-kinesis-agent
● aws-kinesis-agent.service - LSB: Daemon for Amazon Kinesis Agent.
Loaded: loaded (/etc/rc.d/init.d/aws-kinesis-agent; generated)
Active: active (running) since Fri 2024-05-17 00:56:20 UTC; 8s ago
Assuming there are no issues, /var/log/aws-kinesis-agent/aws-kinesis-agent.log should contain something like this.
~]$ cat /var/log/aws-kinesis-agent/aws-kinesis-agent.log
2024-05-17 01:00:22.323+0000 (FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:my-kinesis-data-stream:/tmp/app.log*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2024-05-17 01:00:22.329+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 240046ms
2024-05-17 01:00:22.327+0000 (FileTailer[fh:yourdeliverystream:/tmp/app.log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:yourdeliverystream:/tmp/app.log*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
Next you probably will want to Send log data to Kinesis Data Stream using Kinesis Agent.
Did you find this article helpful?
If so, consider buying me a coffee over at