workflowInput Documentation

Overview

The workflowInput node in the Vantage analytics and data platform is used to execute a separate workflow by its ID and return its output as the output of the current node. This capability allows users to chain multiple workflows together, enhancing modularity and reusability in data processing pipelines.

Purpose

The primary purpose of the workflowInput logic is to facilitate the chaining of outputs from one workflow as the inputs for another. This allows for complex data transformations and analytics processes to be crafted through a series of smaller, manageable workflows instead of handling all processes in a single workflow.

How It Works

The workflowInput node operates by accepting configurations that specify which workflow to execute and which output node, if any, to target. It manages input from both configuration and runtime parameters, retrieves and processes the output, and returns structured data along with metadata.

Workflow Execution Flow:

1. Input Validation: It checks that a valid source workflow ID has been provided.
2. Workflow Invocation: It successfully runs the corresponding workflow.
3. Output Handling: If specified, it retrieves data from a specific output node, leveraging Redis for caching and performance optimization.
4. Returns Data: Finally, it formats the output data into a structured reply including metadata.

Settings

Configuration Settings

1. sourceWorkflowId

   • Input Type: String
   • Description: This is the identifier for the source workflow that the node will execute. It is required for the node to function. If this value is not provided, an error is returned.
   • Default Value: None (must be provided)

2. sourceOutputNodeId

   • Input Type: String (optional)
   • Description: When specified, this setting targets a specific output node from the source workflow to retrieve its output. This is beneficial for workflows that produce multiple output nodes, allowing users to precise which output they wish to utilize.
   • Default Value: None (optional; will target the primary output if not provided)

Additional Runtime Context

• context
  • Input Type: Object (optional)
  • Description: This parameter can be used to pass additional contextual information or settings to the workflow being executed. It may contain configurations relevant to the execution environment.
  • Default Value: An empty object ({}).

Expected Data

Input Data

1. The workflowInput expects a configuration object that includes the sourceWorkflowId, with an optional sourceOutputNodeId and context.

2. The sourceWorkflowId must point to a valid workflow ID in the system.

Output Data

The output of workflowInput will include:

• An object with a property output1 that contains:
  • data: The result obtained from executing the specified workflow, which can be structured data, an array, or raw output as determined during processing.
  • metadata: An object containing:
    • sourceWorkflowId: The ID of the workflow that was executed.
    • rowCount: Number of rows in the returned dataset, if applicable.
    • fetchedAt: Timestamp of when the data was fetched.

Use Cases & Examples

Use Cases

1. Data Preprocessing:

   • A business needs to pull sales data from multiple regions, process it for missing values, and chain the results for further analytics. workflowInput can run the preprocessing workflow and obtain cleaned data for an analytics workflow.

2. Dynamic Reporting:

   • A reporting tool that generates various reports based on user queries. Each report can be constructed with various workflows, where the output from one workflow feeds into another via workflowInput, enabling dynamic report creation.

3. Data Pipeline Automation:

   • A company uses multiple workflows for ETL (Extract, Transform, Load) processes. By chaining these workflows using workflowInput, they can automate the entire pipeline smoothly, reducing manual effort and errors.

Detailed Example

Use Case: A retail analytics company wants to analyze sales performance across multiple stores.

Configuration Example:

{
  "inputs": {},
  "config": {
    "sourceWorkflowId": "2023-sales-data-extraction",
    "sourceOutputNodeId": "cleanedDataNode"
  },
  "context": {}
}

Explanation of Example Configuration

• sourceWorkflowId: This is set to "2023-sales-data-extraction", pointing to the ID of the workflow responsible for data extraction.
• sourceOutputNodeId: This uses the value "cleanedDataNode", indicating that the workflow output should be taken from a specific node that contains the cleaned sales data.
• context: Since no specific context is required, an empty object is passed.

Using this setup, when the workflowInput node is executed, it runs the 2023-sales-data-extraction workflow, retrieves the cleaned sales data from the specified output node, and returns it along with metadata about the data retrieval process.

AI Integrations & Billing Impact

AI Integrations

While the workflowInput logic itself does not include direct AI integrations, it can be employed in conjunction with workflows that utilize AI models for advanced analytics. For instance, one might chain a predictive analytics workflow that uses machine learning to analyze the data retrieved via workflowInput.

Billing Impact

The use of workflowInput may incur costs associated with executing workflows, particularly regarding compute resources during runtime. Each executed workflow could count toward usage limits or billing calculations in Vantage, depending on the pricing model utilized. Furthermore, caching strategies via Redis may mitigate some costs, optimizing the data reuse and reducing recurrent executions.

This documentation provides a comprehensive overview of the workflowInput, detailing its function, configuration settings, use cases, and impact on operations within the Vantage platform.