ContextPreview Documentation

Overview

ContextPreview is a feature within the Vantage analytics and data platform that serves as a debugging tool for users to see how AI understands a particular query. It allows users to input a sample question and observe which snippets of data would be selected by the AI, along with their relevance scores and the actual context that will be included in the AI's prompt.

Purpose

The primary purpose of ContextPreview is to provide transparency into the AI's decision-making process by displaying the selected data snippets and their scores when users submit a query. This helps users refine their queries for better outcomes and understand what information the AI will consider when forming responses.

Settings

The following are the settings available in the ContextPreview component:

1. Query Input

• Name: Query
• Type: String
• Description: This is a free-text input where users can type a sample query. The input is essential to execute the context preview, as it serves as the basis for fetching relevant snippets.
• Default Value: Empty string.
• Behavior: If the input is empty, the "Preview" button is disabled, and the context cannot be submitted. When the user types in the input box and presses "Enter" or clicks "Preview," the input is sent to the backend for processing.

2. Page Context Input

• Name: Page Context
• Type: String
• Description: An optional field that allows users to specify the context for the page where the query is initiated (e.g., "dashboard," "workflow editor"). This context may influence snippet selection to improve relevance.
• Default Value: Empty string.
• Behavior: This field is hidden by default and can be revealed by clicking "Advanced Options." It enhances the AI's understanding of the query's environment when provided.

3. Loading State

• Name: Loading
• Type: Boolean
• Description: A state indicating whether the query is currently processing. It is used to show a loading animation while fetching results.
• Default Value: false
• Behavior: When set to true, disables the "Preview" button and shows a loading indicator. The button becomes active again when loading completes.

4. Show Raw Context

• Name: Show Raw Context
• Type: Boolean
• Description: This setting determines if the raw context string should be displayed.
• Default Value: false
• Behavior: When toggled on, it reveals the exact context string sent to the AI, offering insight into what data snippets were selected.

5. Show Advanced Options

• Name: Show Advanced
• Type: Boolean
• Description: Indicates whether advanced settings, including the Page Context input, are visible.
• Default Value: false
• Behavior: When activated, it expands the interface to show the Page Context input field, allowing users to specify additional context for the AI.

6. Result Data

• Name: Result
• Type: Object
• Description: Holds the data returned as a response from the AI, encapsulating selected snippets, their scores, and a summary of the findings.
• Default Value: null
• Behavior: This object is populated only after a successful query execution. It includes:
  • selectedCount: Number of snippets selected.
  • totalSnippets: Total snippets available for selection.
  • estimatedTokens: Estimated number of tokens resulting from the selected context.
  • snippets: An array containing detailed information about each selected snippet (id, name, score, type, etc.).

How It Works

When a user inputs a sample query and clicks the "Preview" button or presses "Enter," the following sequence occurs:

1. The component validates the query to ensure it is not empty.
2. The loading state is activated while the component fetches data from the endpoint /api/ai/snippets/preview using a POST request, which includes the query and optional page context as JSON.
3. Upon receiving a response, the component updates the Result state with the data returned, which includes a number of insights on how snippets were selected.
4. The component displays the relevant snippets along with their scores and context strings, and allows users to toggle the advanced settings.
5. Users can hide or show raw context details, providing insight into how data is injected into the AI prompt.

Data Expectations

The component expects the following data from the backend when the user runs a query:

• A JSON response containing:
  • selectedCount: Numeric value representing the number of snippets selected.
  • totalSnippets: Numeric value representing the total snippets available.
  • estimatedTokens: Numeric estimate of tokens based on context.
  • snippets: An array containing objects with the following fields:
    • id: Unique identifier for the snippet.
    • name: Display name of the snippet.
    • score: Relevance score of the snippet.
    • type: Type of the snippet (e.g., text, image).
    • alwaysInclude: Boolean indicating if the snippet is always included.
    • isOrgLevel: Boolean indicating if the snippet is at the organizational level.
    • reasons: Array containing explanations for the selection.
    • entryStats: An object containing stats (approved, pending, rejected).

Use Cases & Examples

Use Cases

1. Debugging AI Responses in Customer Support: A support agent can use ContextPreview to refine their queries to ensure the AI retrieves the most relevant snippets to assist customers effectively.

2. Optimizing Query Performance: Data analysts can leverage ContextPreview to understand which snippets are being selected for queries run frequently, thus optimizing the AI's responses and reducing response times by ensuring high-relevance snippets are used.

3. Training the AI Models: AI developers can utilize this feature to experiment with different queries and page contexts to assess model performance and adjust training datasets accordingly.

Example Configuration

Use Case: Optimizing Query Performance

An analyst needs to optimize queries for a dashboard showing sales data. They want to ensure that queries about "total sales" and "monthly growth" return the most relevant data snippets.

Configuration Example:

• Query: "What were the total sales last month?"
• Page Context: "dashboard"

Using this configuration in the ContextPreview:

1. The analyst enters the query in the Query input box.
2. They expand the advanced options and enter "dashboard" in the Page Context input.
3. They click "Preview" to fetch the results.

Expected Results

• The response might show:
  • Selected Count: 5 snippets selected
  • Total Snippets: 10 available
  • Estimated Tokens: 30 tokens
  • Snippets:
    • Snippet 1: { id: 1, name: "Total Sales", score: 950, type: "financial", alwaysInclude: true, reasons: ["Highly relevant"], entryStats: { approved: 100, pending: 5, rejected: 0 }}
    • Snippet 2: { id: 2, name: "Monthly Growth", score: 850, type: "financial", alwaysInclude: false, reasons: ["Relevant based on query"], entryStats: { approved: 90, pending: 10, rejected: 2 }}

This configuration not only helps the analyst understand the context of queried information but also aids in refining future queries to ensure optimal performance.