awsListObjects Documentation

Purpose

The node is designed to interact with AWS S3 (Simple Storage Service) to retrieve a list of objects stored within a specified S3 bucket. It allows users to define parameters such as the bucket name and optional prefix for filtering results, enabling efficient management and retrieval of stored data.

Settings

The node has several configurable settings that dictate its behavior and the data it retrieves. Below is a detailed explanation of each setting:

1. bucket_name

• Input Type: String
• Description: This setting specifies the name of the S3 bucket that contains the objects to be listed. It is a mandatory field; if left empty, an error will be thrown.
• Default Value: None (it must be provided).

2. prefix

• Input Type: String
• Description: The prefix setting determines a substring that the object keys must begin with in order to be included in the results. This allows for filtering the list of objects to only those that fit a certain naming convention or path structure.
• Default Value: An empty string (''), which means no prefix filtering.

3. max_keys

• Input Type: Numeric
• Description: This setting specifies the maximum number of objects to return in a single response. It helps to manage the amount of data transferred in a single request and can help with pagination.
• Default Value: 1000, which is the maximum limit by default but can be adjusted to a smaller number depending on user needs.

4. continuation_token

• Input Type: String or Null
• Description: When listing a large number of objects, this token helps in pagination. If a previous list operation has returned a continuation token, it can be specified here to fetch the next set of objects.
• Default Value: null indicates that the request will begin from the first object.

How It Works

1. Connection Initialization: The function begins by creating an integration with AWS S3 using the the integration connection method. It requires the context for establishing the connection.

2. Parameter Construction: A parameters object is created, gathering input values for bucket_name, prefix, max_keys, and continuation_token, prioritizing user inputs over configuration defaults.

3. Validation: The function checks whether the bucket_name is provided. If not, it throws an error indicating this is a required parameter.

4. Data Retrieval: It invokes the listObjects method of the S3 integration with the constructed parameters to retrieve a list of objects from the specified bucket.

5. Output Structure: The function formats the response into an object containing the retrieved objects, metadata about the query, and pagination information.

6. Error Handling: Any errors during the process are caught and logged, with a user-friendly error message returned.

Data It Expects

The node expects the following input data:

• Inputs: An object containing bucket_name, prefix, max_keys, and continuation_token.
• Config: A configuration object that can provide default values for the parameters not provided by inputs.
• Context: An object required for establishing a connection to AWS services.

AI Integrations & Billing Impacts

• AI Integrations: Currently, there are no specific AI integrations noted within this logic function. It operates purely as a data retrieval tool.

• Billing Impacts: Billing will be incurred based on AWS S3 usage, specifically on the number of API requests made and the data retrieval. The max_keys setting can impact the frequency of requests and thus potential costs.

Use Cases & Examples

Use Case 1: Data Management for a Large E-commerce Application

A large online retail platform needs to manage product images stored in S3 buckets effectively. They require the ability to list objects and filter results based on specific categories.

Use Case 2: Content Delivery for a Media Streaming Service

A media streaming service uses S3 to store video and audio files. They need to manage the retrieval of media assets efficiently to provide seamless streaming experiences.

Use Case 3: Automated Backup System

An organization implements an automated backup solution that uploads files to S3. They require a reliable way to list and verify the contents of backup buckets periodically.

Example Configuration for Use Case 1

For the e-commerce application that wants to retrieve all product images starting with 'products/2023/', the configuration might look as follows:

{
  "inputs": {
    "bucket_name": "ecommerce-product-images",
    "prefix": "products/2023/",
    "max_keys": 500
  },
  "config": {
    "bucket_name": "default-bucket", // Default bucket if not specified
    "prefix": ""
  },
  "context": {
    "clientId": "your-client-id-here"
  }
}

In this configuration:

• bucket_name is set to "ecommerce-product-images" to specify the relevant bucket.
• prefix filters for images related to the year 2023.
• max_keys limits the response to 500 objects, reducing load and managing costs effectively.