This assumes you are familiar with Kinesis. If not, check out my article Getting Started with Kinesis.

There are a few different Kinesis Agents. For example, in the AWS console, after selecting your Kinesis Data Stream, the Applications tab lists a few Kinesis Agents. These are known as the Producers.

There are a few different ways to install and setup the Amazon Kinesis Java Agent.

• Install and setup Amazon Kinesis Java Agent from GitHub (this article)
• Install and setup Amazon Kinesis Java Agent using yum or dnf
• Install and setup Amazon Kinesis Java Agent using Docker

At the GitHub page, there is a link to Download Zip, which should point to https://github.com/awslabs/amazon-kinesis-agent/archive/refs/heads/master.zip.

So on a Linux system, such as one of your EC2 Instances, wget can be used to download master.zip. If possible, I would do this on an EC2 Instance just to minimize issues that may occur, especially since this is just supposed to be a simple Getting Started Article.

~]$ wget https://github.com/awslabs/amazon-kinesis-agent/archive/refs/heads/master.zip
--2024-05-15 01:24:34--  https://github.com/awslabs/amazon-kinesis-agent/archive/refs/heads/master.zip
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/awslabs/amazon-kinesis-agent/zip/refs/heads/master [following]
--2024-05-15 01:24:34--  https://codeload.github.com/awslabs/amazon-kinesis-agent/zip/refs/heads/master
Resolving codeload.github.com (codeload.github.com)... 140.82.113.9
Connecting to codeload.github.com (codeload.github.com)|140.82.113.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘master.zip’

master.zip                                                      [ <=>                                                                                                                                     ] 391.60K  --.-KB/s    in 0.05s

2024-05-15 01:24:34 (8.43 MB/s) - ‘master.zip’ saved [400995]

And then the unzip command can be used to unzip master.zip into your present working directory.

unzip master.zip

You should now have both master.zip and a directory named amazon-kinesis-agent-master in your present working directory.

~]$ ls -l
drwxr-xr-x. 8 ec2-user ec2-user  16384 Jun  2  2023 amazon-kinesis-agent-master
-rw-r--r--. 1 ec2-user ec2-user 400995 May 15 01:24 master.zip

The master.zip file can be removed.

rm master.zip

And in the amazon-kinesis-agent-master directory there should be a file named setup.

~]$ ls -l amazon-kinesis-agent-master/
total 72
-rw-r--r--. 1 ec2-user ec2-user   311 Jun  2  2023 CODE_OF_CONDUCT.md
-rw-r--r--. 1 ec2-user ec2-user  3607 Jun  2  2023 CONTRIBUTING.md
-rw-r--r--. 1 ec2-user ec2-user  4572 Jun  2  2023 LICENSE.txt
-rw-r--r--. 1 ec2-user ec2-user   106 Jun  2  2023 NOTICE.txt
-rw-r--r--. 1 ec2-user ec2-user  6404 Jun  2  2023 README.md
drwxr-xr-x. 2 ec2-user ec2-user 16384 Jun  2  2023 bin
-rw-r--r--. 1 ec2-user ec2-user  2948 Jun  2  2023 build.xml
drwxr-xr-x. 4 ec2-user ec2-user    36 Jun  2  2023 configuration
-rw-r--r--. 1 ec2-user ec2-user  8839 Jun  2  2023 pom.xml
-rwxr-xr-x. 1 ec2-user ec2-user  8217 Jun  2  2023 setup
drwxr-xr-x. 3 ec2-user ec2-user    17 Jun  2  2023 src
drwxr-xr-x. 2 ec2-user ec2-user    88 Jun  2  2023 support
drwxr-xr-x. 3 ec2-user ec2-user    40 Jun  2  2023 tst

Let's move into the amazon-kinesis-agent-master directory and display the setup CLI help. The output say this must be run as root.

~]# cd amazon-kinesis-agent-master

~]$ ./setup --help
Detected OS Distro: RedHat
This script must be run as root

You can either su to root of if you have sudo permission, use sudo.

~]$ sudo ./setup --help
Detected OS Distro: RedHat
Usage:
To install Kinesis Agent, type:
  sudo <script-path>/setup --install
To uninstall Kinesis Agent, type:
  sudo <script-path>/setup --uninstall

Let's install the Kinesis Agent. There should be a good amount of output, and if all goes well, the last line should be "Your installation has completed!".

]$ sudo ./setup --install
. . .
Your installation has completed!

I try to start the agent and I get the following.

~]$ /home/ec2-user/amazon-kinesis-agent-master/bin/start-aws-kinesis-agent
/home/ec2-user/amazon-kinesis-agent-master/bin/start-aws-kinesis-agent: line 125: cd: /home/ec2-user/amazon-kinesis-agent-master/bin/../share/aws-kinesis-agent/lib: No such file or directory

The lib (library files) used by the Agent are located at /usr/share/aws-kinesis-agent/lib/.

~]$ ls -l /usr/share/aws-kinesis-agent/lib/
total 30192
-rw-r--r--. 1 root root  241471 May 15 01:40 AWSKinesisStreamingDataAgent-1.1.jar
-rw-r--r--. 1 root root  533121 May 15 01:40 aws-java-sdk-cloudwatch-1.12.390.jar
-rw-r--r--. 1 root root 1050520 May 15 01:40 aws-java-sdk-core-1.12.390.jar
-rw-r--r--. 1 root root 8431026 May 15 01:40 aws-java-sdk-ec2-1.12.390.jar
-rw-r--r--. 1 root root 1551241 May 15 01:40 aws-java-sdk-kinesis-1.12.390.jar
-rw-r--r--. 1 root root  150756 May 15 01:40 aws-java-sdk-sts-1.12.390.jar
-rw-r--r--. 1 root root   41123 May 15 01:40 commons-cli-1.2.jar
-rw-r--r--. 1 root root  232771 May 15 01:40 commons-codec-1.6.jar
-rw-r--r--. 1 root root  434678 May 15 01:40 commons-lang3-3.4.jar
-rw-r--r--. 1 root root   20937 May 15 01:40 commons-logging-adapters-1.1.jar
-rw-r--r--. 1 root root   44598 May 15 01:40 commons-logging-api-1.1.jar
-rw-r--r--. 1 root root 2792264 May 15 01:40 guava-29.0-jre.jar
-rw-r--r--. 1 root root  780321 May 15 01:40 httpclient-4.5.13.jar
-rw-r--r--. 1 root root  158982 May 15 01:40 httpclient-cache-4.5.1.jar
-rw-r--r--. 1 root root  326594 May 15 01:40 httpcore-4.4.3.jar
-rw-r--r--. 1 root root   31379 May 15 01:40 httpcore-ab-4.4.3.jar
-rw-r--r--. 1 root root  355663 May 15 01:40 httpcore-nio-4.4.3.jar
-rw-r--r--. 1 root root   40698 May 15 01:40 httpmime-4.5.1.jar
-rw-r--r--. 1 root root   68079 May 15 01:40 jackson-annotations-2.10.3.jar
-rw-r--r--. 1 root root  349108 May 15 01:40 jackson-core-2.10.3.jar
-rw-r--r--. 1 root root 1405163 May 15 01:40 jackson-databind-2.10.5.1.jar
-rw-r--r--. 1 root root   58567 May 15 01:40 jackson-dataformat-cbor-2.10.3.jar
-rw-r--r--. 1 root root  108591 May 15 01:40 jackson-dataformat-xml-2.10.3.jar
-rw-r--r--. 1 root root   63504 May 15 01:40 jcommander-1.48.jar
-rw-r--r--. 1 root root  621992 May 15 01:40 joda-time-2.8.2.jar
-rw-r--r--. 1 root root   19943 May 15 01:40 jsr305-3.0.1.jar
-rw-r--r--. 1 root root  208006 May 15 01:40 log4j-1.2-api-2.17.1.jar
-rw-r--r--. 1 root root  301872 May 15 01:40 log4j-api-2.17.1.jar
-rw-r--r--. 1 root root 1790452 May 15 01:40 log4j-core-2.17.1.jar
-rw-r--r--. 1 root root   24279 May 15 01:40 log4j-slf4j-impl-2.17.1.jar
-rw-r--r--. 1 root root 1950607 May 15 01:40 lombok-1.18.22.jar
-rw-r--r--. 1 root root   32127 May 15 01:40 slf4j-api-1.7.12.jar
-rw-r--r--. 1 root root 6637455 May 15 01:40 sqlite-jdbc-3.20.1.jar

Just to get around this issue the quick and dirty way, I create the /home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib directory.

mkdir --parents /home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib

And then I copy the JAR files in /usr/share/aws-kinesis-agent/lib to /home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib.

cp /usr/share/aws-kinesis-agent/lib/* /home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib

The /etc/aws-kinesis/agent.json file should have something like this.

There are much better options rather that having your AWS Access Key and Secret Key in agent.json, but again, I'm just trying to keep this simple just to get Kinesis up and working.

~]$ cat /etc/aws-kinesis/agent.json
{
  "awsAccessKeyId": "ABK349fJFM499FMDFD23",
  "awsSecretAccessKey": "Ab587Cd234Ef98gG9834Aa345908bB23409234Cc",
  "flows": [
    {
      "filePattern": "/tmp/app.log*",
      "kinesisStream": "my-kinesis-data-stream"
    }
  ]
}

The Kinesis Agent requires Java version 1.7 or higher.

~]$ java --version
openjdk 17.0.8 2023-07-18 LTS
OpenJDK Runtime Environment Corretto-17.0.8.7.1 (build 17.0.8+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.8.7.1 (build 17.0.8+7-LTS, mixed mode, sharing)

Let's try to start the Kinesis Agent. Be aware that by default, this will run in the foreground, not in the background, which is why I used & so that this runs in the background.

sudo /home/ec2-user/amazon-kinesis-agent-master/bin/start-aws-kinesis-agent &

The ps command should show that the Kinesis Java Virtual Machine (JVM) is running. Nice!

[ec2-user@ip-172-31-86-107 ~]$ ps -ef | grep -i kinesis
root       93532   93530  3 01:02 pts/0    00:00:02 /usr/lib/jvm/java-17-amazon-corretto/bin/java -server -Xms32m -Xmx512m -XX:OnOutOfMemoryError="/bin/kill -9 %p" -cp /home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/AWSKinesisStreamingDataAgent-1.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/aws-java-sdk-cloudwatch-1.12.390.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/aws-java-sdk-core-1.12.390.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/aws-java-sdk-ec2-1.12.390.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/aws-java-sdk-kinesis-1.12.390.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/aws-java-sdk-sts-1.12.390.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/commons-cli-1.2.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/commons-codec-1.6.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/commons-lang3-3.4.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/commons-logging-adapters-1.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/commons-logging-api-1.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/guava-29.0-jre.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/httpclient-4.5.13.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/httpclient-cache-4.5.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/httpcore-4.4.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/httpcore-ab-4.4.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/httpcore-nio-4.4.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/httpmime-4.5.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jackson-annotations-2.10.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jackson-core-2.10.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jackson-databind-2.10.5.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jackson-dataformat-cbor-2.10.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jackson-dataformat-xml-2.10.3.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jcommander-1.48.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/joda-time-2.8.2.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/jsr305-3.0.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/log4j-1.2-api-2.17.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/log4j-api-2.17.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/log4j-core-2.17.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/log4j-slf4j-impl-2.17.1.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/lombok-1.18.22.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/slf4j-api-1.7.12.jar:/home/ec2-user/amazon-kinesis-agent-master/share/aws-kinesis-agent/lib/sqlite-jdbc-3.20.1.jar:/usr/share/java/*: com.amazon.kinesis.streaming.agent.Agent

Next you probably will want to Send log data to Kinesis Data Stream using Kinesis Agent.