AWS Kinesis

Amazon Kinesis Data Streams are used to collect and process large streams of data records in real time. AWS Kinesis Connector consumes data records from Kinesis Data Streams. For more information on AWS Kinesis Click Here.

Prerequisites

• Your own application that pushes data directly into a Kinesis data stream. The type of data pushed can include IT infrastructure log data, application logs, etc.

• Required credentials:
  • AWS_SECRET_ACCESS_KEY
  • AWS_ACCESS_KEY_ID

Creating IAM users with programmatic access

• In the navigation pane, choose Users and then choose Add user.
  

• Enter the user name for the new user.

• On the same screen, under Select AWS access type, check the checkbox to enable programmatic access.

• Choose Next: Permissions.

• On the Set permissions page, specify how you want to assign permissions to this set of new users. Choose Attach existing policies directly > Create Policy
  

• Create policy using json method

• Following is the required policy in JSON format

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:CreateStream",
                "kinesis:DeleteStream",
                "kinesis:Put*",
                "kinesis:Get*",
                "kinesis:DescribeStreamSummary",
                "kinesis:DescribeStream"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

• You can find the newly created policy in Attach existing policies directly

• Apply the policy to the IAM user.

• Click Next: Tags button
  

• Add user tags for identification purposes and click on Next:Review button.

• Review all the configurations and click Create user button.

• Click on the Download .csv button to download the key pair or click on Show link under the Secret access key and then copy/save both the Access key ID and Secret access key.

In case of an already existing IAM user, create access keys using the steps mentioned here: AWS Account and Access Keys

Configurations

The following are the configurations to forward AWS Kinesis stream records to DNIF.‌

• Click Save after entering all the required details and click Test Connection, to test the configuration.

• A Connection successful message will be displayed on screen along with the time stamp.

• If the connection is not successful an error message will be displayed. Refer Troubleshooting Connector Validations for more details on the error message.