LLM Integration¶

Overview¶

NebulaFlow integrates with Large Language Models (LLMs) via the pi SDK and pi OpenRouter provider. This guide covers configuration, usage, and best practices for LLM nodes.

Providers and Authentication¶

NebulaFlow uses pi's shared ModelRuntime for built-in providers, custom providers, OAuth, API keys, and model catalogs.

Configure authentication through pi /login, ~/.pi/agent/auth.json, or the selected provider's standard environment variable. Examples include OPENAI_API_KEY, ANTHROPIC_API_KEY, and OPENROUTER_API_KEY.

Pi configuration locations:

~/.pi/agent/settings.json: global model defaults.
<workspace>/.pi/settings.json: project model-default overrides.
~/.pi/agent/auth.json: API keys and OAuth credentials.
~/.pi/agent/models.json: custom providers and models.

Example pi settings:

{
  "defaultProvider": "openrouter",
  "defaultModel": "anthropic/claude-sonnet-4"
}

A model selected directly on an LLM node takes priority over pi's configured default. The model selector shows models authenticated and available through ModelRuntime.

LLM Node Configuration¶

Basic Configuration¶

interface LLMNode {
    type: NodeType.LLM
    data: {
        title: string
        content: string  // Prompt template
        active: boolean
        model: { id: string; title?: string }
        reasoningEffort?: 'minimal' | 'low' | 'medium' | 'high'
        systemPromptTemplate?: string
        disabledTools?: string[]
        timeoutSec?: number
        dangerouslyAllowAll?: boolean
        attachments?: AttachmentRef[]
    }
}

Example Configurations¶

Basic LLM Node¶

{
    type: NodeType.LLM,
    data: {
        title: 'Simple Query',
        content: 'What is the capital of France?',
        active: true,
        model: { id: 'openrouter/anthropic/claude-3-5-sonnet' }
    }
}

LLM with Variable Input¶

{
    type: NodeType.LLM,
    data: {
        title: 'Process User Input',
        content: 'Analyze the following text and provide a summary: ${1}',
        active: true,
        model: { id: 'openrouter/openai/gpt-4o' },
        reasoningEffort: 'medium'
    }
}

LLM with System Prompt¶

{
    type: NodeType.LLM,
    data: {
        title: 'Code Review Agent',
        content: 'Review this code: ${1}',
        active: true,
        model: { id: 'openrouter/anthropic/claude-3-5-sonnet' },
        systemPromptTemplate: 'You are a senior software engineer with 10+ years of experience. Provide detailed, constructive feedback on code quality, security, and best practices.'
    }
}

LLM with Tool Restrictions¶

{
    type: NodeType.LLM,
    data: {
        title: 'Safe Assistant',
        content: 'Answer this question: ${1}',
        active: true,
        model: { id: 'openrouter/anthropic/claude-3-5-sonnet' },
        disabledTools: ['bash', 'filesystem', 'network'],
        dangerouslyAllowAll: false
    }
}

LLM with Attachments¶

{
    type: NodeType.LLM,
    data: {
        title: 'Image Analysis',
        content: 'Describe what you see in this image',
        active: true,
        model: { id: 'openrouter/openai/gpt-4o' },
        attachments: [
            {
                id: 'image1',
                kind: 'image',
                source: 'file',
                path: '/path/to/diagram.png',
                altText: 'Architecture diagram'
            }
        ]
    }
}

Model Selection¶

Available Models¶

Models are loaded from two sources:

pi SDK models - Built-in models from Amp
OpenRouter models - Configured in workspace settings

Viewing Available Models¶

Open workflow editor
Add LLM node
Click model dropdown
Select from available models

Model ID Format¶

pi SDK models: - gpt-4o - gpt-4o-mini - claude-3-5-sonnet

OpenRouter models: - openrouter/anthropic/claude-3-5-sonnet - openrouter/openai/gpt-4o - openrouter/google/gemini-pro

Model Resolution¶

The selected model ID is normalized via the pi SDK's resolveModel function. If resolution fails, the raw ID is used.

Prompt Engineering¶

Prompt Templates¶

Use template variables to reference upstream node outputs:

# Basic variable reference
"Process this: ${1}"

# Multiple variables
"Analyze ${1} and summarize: ${2}"

# Named variables (if using Variable nodes)
"User query: ${userQuery}"

System Prompts¶

Set a system prompt to guide model behavior:

{
    data: {
        systemPromptTemplate: 'You are a helpful assistant specialized in technical documentation.'
    }
}

Best Practices¶

Be specific - Clear instructions yield better results
Provide examples - Show the model what you want
Use variables - Reference upstream data
Set constraints - Guide the model's behavior
Test iteratively - Refine prompts based on results

Reasoning Effort¶

What is Reasoning Effort?¶

Reasoning effort controls how much computational effort the model uses to generate responses.

Levels¶

minimal - Fast, less thorough
low - Quick analysis
medium - Balanced (default)
high - Deep analysis, more tokens

Configuration¶

{
    data: {
        reasoningEffort: 'high'  // Use for complex tasks
    }
}

When to Use Each Level¶

minimal: - Simple factual questions - Quick responses - Low-cost operations

low: - General queries - Standard analysis - Balanced performance

medium: - Complex reasoning - Detailed analysis - Default for most tasks

high: - Critical decisions - Deep analysis - Maximum accuracy

Tool Calling¶

What is Tool Calling?¶

Tool calling allows LLMs to invoke external functions or tools.

Tool Name Resolution¶

Tools are automatically resolved via the pi SDK:

// Tools are normalized to official names
disabledTools: ['bash', 'filesystem']  // Normalized to official tool names

Disabling Tools¶

Prevent certain capabilities:

{
    data: {
        disabledTools: ['bash', 'filesystem', 'network']
    }
}

Available Tools¶

Common tools: - bash - Shell command execution - filesystem - File system access - network - Network requests - code_execution - Code execution

Attachments¶

What are Attachments?¶

Attachments allow LLMs to process images and other media.

Attachment Types¶

Images:

{
    attachments: [
        {
            id: 'image1',
            kind: 'image',
            source: 'file',
            path: '/path/to/image.png',
            altText: 'Description of image'
        }
    ]
}

URL-based images:

{
    attachments: [
        {
            id: 'image2',
            kind: 'image',
            source: 'url',
            url: 'https://example.com/image.png',
            altText: 'Description of image'
        }
    ]
}

Using Attachments¶

Add attachment to LLM node configuration
Reference in prompt if needed
Model processes attachment automatically

Conversation History¶

What is Conversation History?¶

LLM nodes maintain conversation history for multi-turn interactions.

How it Works¶

Thread ID is generated for each conversation
History is stored in workflow state
Context is preserved across multiple executions

Using Conversation History¶

// LLM node automatically maintains history
// No special configuration needed

Thread Management¶

Thread IDs are stored per node: - Unique thread per LLM node - Persisted in workflow state - Reused across executions

Example workflow:

LLM Node (Thread A) → LLM Node (Thread A) → LLM Node (Thread B)

Execution Flow¶

LLM Node Execution¶

Input collection - Gather inputs from upstream nodes
Prompt building - Construct prompt from template
Model selection - Choose model from configuration
API call - Send request to LLM provider
Streaming response - Receive tokens as they're generated
Result collection - Assemble complete response
History update - Store conversation context

Streaming¶

LLM nodes stream responses in real-time:

// Events received during execution
{
    type: 'node_assistant_content',
    data: {
        nodeId: 'llm-node-1',
        threadID: 'thread-abc-123',
        content: [
            { type: 'text', text: 'Hello' },
            { type: 'thinking', thinking: 'I need to respond...' }
        ]
    }
}

Error Handling¶

Common errors: - No authenticated pi model - configure pi /login, auth.json, or the provider environment variable - Model not found - Check model ID - Rate limit exceeded - Wait and retry - Network error - Check connectivity

Performance Optimization¶

Token Usage¶

Monitor token counts:

// Token count events
{
    type: 'token_count',
    data: {
        count: 150,
        nodeId: 'llm-node-1'
    }
}

Optimization tips: - Use appropriate reasoning effort - Keep prompts concise - Use system prompts to guide behavior - Cache repeated queries

Model Selection¶

Choose the right model: - Simple tasks - Use smaller, faster models - Complex reasoning - Use larger models with high reasoning effort - Cost-sensitive - Use cheaper models when possible

Caching¶

Implement caching: - Cache repeated queries - Store results in workflow state - Use Variable nodes for reuse

Security Considerations¶

API Keys¶

Never commit API keys:

# Good: Environment variables
export OPENAI_API_KEY="your_key"

# Bad: Hardcoded in workflows
# content: "Use key: sk-..."

Safety Controls¶

Use Amp safety features:

{
    data: {
        dangerouslyAllowAll: false,  // Keep false for safety
        disabledTools: ['bash', 'filesystem']  // Disable dangerous tools
    }
}

Data Privacy¶

Be aware of: - What data is sent to LLM providers - Provider data retention policies - Compliance requirements

Troubleshooting¶

Common Issues¶

"pi SDK not available"¶

Cause: SDK not properly linked

Solution:

npm install

"No authenticated pi model is available"¶

Cause: Environment variable missing

Solution:

export OPENAI_API_KEY="your_openai_api_key_here"

"Model not found"¶

Cause: Model ID incorrect or not available

Solution: 1. Check available models in UI 2. Verify model ID in workspace settings 3. Ensure API key is valid

"Rate limit exceeded"¶

Cause: Too many API requests

Solution: 1. Wait and retry 2. Use different model 3. Implement retry logic

"Timeout"¶

Cause: Request taking too long

Solution: 1. Increase timeout setting 2. Use faster model 3. Simplify prompt

Debugging Tips¶

Enable debug logging:

// Extension logs show:
// - API requests
// - Model resolution
// - Error details

Check events: - node_assistant_content - Streaming responses - token_count - Token usage - node_execution_status - Execution status

Integration Patterns¶

Pattern 1: Simple Query¶

Text Node (query)
    └── LLM Node (answer query)
    └── Preview Node (display result)

Pattern 2: Multi-Step Processing¶

Text Node (input)
    └── LLM Node (analyze)
    └── LLM Node (summarize)
    └── Preview Node (final result)

Pattern 3: Conditional Processing¶

Text Node (input)
    └── IF Node (check condition)
        ├── True: LLM Node (process A)
        └── False: LLM Node (process B)
    └── Preview Node (combine results)

Pattern 4: Loop Processing¶

Loop Start (iterations=5)
    └── LLM Node (process item ${i})
Loop End
    └── Accumulator Node (collect results)

Pattern 5: Multi-Model Comparison¶

Text Node (query)
    ├── LLM Node (Model A)
    ├── LLM Node (Model B)
    └── Accumulator Node (compare responses)

Best Practices¶

General¶

Configure provider authentication through pi /login, auth.json, or a provider environment variable.
Choose appropriate models - Balance cost vs capability
Use reasoning effort - Optimize for your use case
Test with small prompts - Verify behavior before scaling
Monitor token usage - Track costs
Use system prompts - Guide model behavior
Handle errors gracefully - LLM calls can fail

Prompt Design¶

Be specific - Clear instructions yield better results
Provide examples - Show the model what you want
Use variables - Reference upstream data
Set constraints - Guide the model's behavior
Test iteratively - Refine prompts based on results

Security¶

Use environment variables - For API keys
Enable safety controls - Keep dangerouslyAllowAll false
Disable dangerous tools - Prevent unwanted capabilities
Monitor data sent - Be aware of privacy implications

Performance¶

Choose right model - Match model to task complexity
Use appropriate reasoning - Balance speed and quality
Implement caching - Avoid redundant calls
Monitor token usage - Track and optimize costs

Advanced Configuration¶

Custom System Prompts¶

{
    data: {
        systemPromptTemplate: `You are a specialized assistant with the following capabilities:
1. Code analysis and review
2. Documentation generation
3. Technical writing
4. Bug identification

Please provide detailed, actionable feedback.`
    }
}

Timeout Configuration¶

{
    data: {
        timeoutSec: 300  // 5 minutes
    }
}

Safety Override (Use with Caution)¶

{
    data: {
        dangerouslyAllowAll: true,  // Bypasses safety checks
        disabledTools: []  // Enable all tools
    }
}

Warning: Only use in trusted environments.

Node Types Reference - LLM node details
Protocol Reference - Message protocol
Events Reference - Event system
Integrations Overview - All integrations
Advanced Workflows - Advanced patterns

LLM Integration¶

Overview¶

Providers and Authentication¶

LLM Node Configuration¶

Basic Configuration¶

Example Configurations¶

Basic LLM Node¶

LLM with Variable Input¶

LLM with System Prompt¶

LLM with Tool Restrictions¶

LLM with Attachments¶

Model Selection¶

Available Models¶

Viewing Available Models¶

Model ID Format¶

Model Resolution¶

Prompt Engineering¶

Prompt Templates¶

System Prompts¶

Best Practices¶

Reasoning Effort¶

What is Reasoning Effort?¶

Levels¶

Configuration¶

When to Use Each Level¶

Tool Calling¶

What is Tool Calling?¶

Tool Name Resolution¶

Disabling Tools¶

Available Tools¶

Attachments¶

What are Attachments?¶

Attachment Types¶

Using Attachments¶

Conversation History¶

What is Conversation History?¶

How it Works¶

Using Conversation History¶

Thread Management¶

Execution Flow¶

LLM Node Execution¶

Streaming¶

Error Handling¶

Performance Optimization¶

Token Usage¶

Model Selection¶

Caching¶

Security Considerations¶

API Keys¶

Safety Controls¶

Data Privacy¶

Troubleshooting¶

Common Issues¶

"pi SDK not available"¶

"No authenticated pi model is available"¶

"Model not found"¶

"Rate limit exceeded"¶

"Timeout"¶

Debugging Tips¶

Integration Patterns¶

Pattern 1: Simple Query¶

Pattern 2: Multi-Step Processing¶

Pattern 3: Conditional Processing¶

Pattern 4: Loop Processing¶

Pattern 5: Multi-Model Comparison¶

Best Practices¶

General¶

Prompt Design¶

Security¶

Performance¶

Advanced Configuration¶

Custom System Prompts¶

Timeout Configuration¶

Safety Override (Use with Caution)¶

Related Documentation¶