Enrichment Logic Documentation
Overview
The Enrichment logic is a powerful feature of Vantage that leverages AI capabilities to enhance data records. It sends each row of input data to a large language model (LLM) for tasks such as classification, sentiment analysis, entity extraction, or summarization. The purpose of this functionality is to derive valuable insights from raw data automatically, allowing organizations to make data-driven decisions efficiently and effectively.
Configuration
Settings
| Setting Name | Input Type | Description | Default Value |
|---|---|---|---|
promptTemplate | String | Specifies the template used for generating prompts for the LLM. You can reference columns in the data using placeholders like {{column_name}}. Altering this affects how the model interprets the data, resulting in different outputs based on the task defined. | "" (empty string) |
outputColumn | String | Defines the name of the column in the output data where the AI model's results will be stored. Changing this will direct the output of the AI enrichment task to a different column in the resulting dataset. | "ai_result" |
batchSize | Numeric | Controls the number of rows sent to the AI model in a single batch for processing. Reducing this value would lead to more frequent interactions with the model but can increase overall processing time. The maximum allowed size is 50. | 50 |
task | String | Specifies the type of task to be performed by the LLM. Acceptable values include “classify”, “sentiment”, “extract”, and “summarize.” Changing this affects the prompt used and, subsequently, the nature of the insights generated from the data. | "classify" |
Functionality Flow
-
Input Data Handling: The component processes input data by extracting rows and validating them. If the input does not conform to the expected format, it returns an empty dataset.
-
Prompt Template Resolution: The component evaluates the provided
promptTemplate. If it is not specified, a default based on the selectedtaskis used. -
AI Integration Retrieval: The component attempts to fetch the preferred AI integration for the user, which is then used to send requests to the LLM.
-
Batch Processing: Data is processed in manageable batches based on the
batchSize, with an adjustment during the workflow builder mode to limit processing to 100 rows only. -
Prompt Interpolation: Each row’s data is incorporated into the prompt template using the
interpolatePromptfunction, which replaces placeholders with actual row values. -
Sending Requests to the AI: The concatenated prompts for each batch are sent to the selected AI integration, and responses are parsed and returned to output.
-
Error Handling: If an error occurs during the AI call, the function captures the exception and appends an error message to the appropriate output rows.
-
Final Output Compilation: All processed data is compiled, including any rows that were not processed due to preview mode limitations.
Expected Data Input
The component expects the input to be structured as rows of data, where each row can contain multiple columns. Supported formats include:
- An array of objects: Each object should represent a row with key-value pairs corresponding to column names and values.
- An object with a
datakey containing an array of rows.
Use Cases & Examples
Use Cases
-
Customer Sentiment Analysis: A retail company can use the enrichment logic to analyze customer feedback from a dataset, extracting sentiments to gauge customer satisfaction levels.
-
Entity Extraction for CRM: A business may leverage the enrichment logic to identify and extract entities such as customer names, organizations, and dates from a dataset of unstructured customer interaction notes.
-
Industry Classification: A tech firm could classify various company records into defined industry categories, improving their targeting for marketing campaigns.
Example Configuration
Use Case: Customer Sentiment Analysis
To perform sentiment analysis on customer feedback records, the configuration for the enrichment logic can be set as follows:
{
"promptTemplate": "Analyze the sentiment of the following customer feedback: {{feedback}}",
"outputColumn": "sentiment_result",
"batchSize": 20,
"task": "sentiment"
}- Input Data: The following example input data may be provided to the enrichment node:
[
{"feedback": "I love the product, it's amazing!"},
{"feedback": "I'm not happy with the service."},
{"feedback": "The experience was okay, nothing special."}
]- Expected Output: Running this configuration will analyze each customer feedback record, and the output will look like this:
[
{"feedback": "I love the product, it's amazing!", "sentiment_result": "positive"},
{"feedback": "I'm not happy with the service.", "sentiment_result": "negative"},
{"feedback": "The experience was okay, nothing special.", "sentiment_result": "neutral"}
]AI Integration and Billing Implications
The enrichment logic involves interactions with AI services, where costs may vary based on the number of rows processed and the complexity of the tasks being performed. Each batch sent to the AI model counts towards billing, thus determining the economic impact. Users should monitor their batch sizes and the frequency of data enrichment operations to optimize their costs effectively. Additionally, utilizing preview mode helps to control expenses during development and testing phases.