nebulaflow

External API Integration

Overview

NebulaFlow enables seamless integration with external APIs through CLI nodes using tools like curl, wget, or custom scripts. This guide covers connecting to REST, GraphQL, and webhook endpoints, handling authentication, managing errors, and optimizing performance.

Key capabilities:

REST APIs - Full HTTP method support (GET, POST, PUT, DELETE, etc.)
GraphQL APIs - Query execution via POST requests
Webhook processing - Receive and process HTTP callbacks
Authentication - API keys, Bearer tokens, OAuth (via scripts)
Error handling - Retry logic, rate limit detection, status code validation

CLI Node as API Client

All API interactions are performed using CLI nodes that execute HTTP requests via curl or similar command-line tools.

Basic CLI Node Configuration

interface CLINode {
    type: NodeType.CLI
    data: {
        title: string
        content: string  // Command to execute (e.g., curl)
        mode?: 'command' | 'script'
        shell?: 'bash' | 'sh' | 'zsh' | 'pwsh' | 'cmd'
        safetyLevel?: 'safe' | 'advanced'
        streamOutput?: boolean
        stdin?: { ... }
        env?: { ... }
        flags?: { ... }
    }
}

Important: For API calls, set safetyLevel: 'advanced' only if you trust the API endpoint and need to bypass approval for automated workflows.

REST API Integration

GET Request

Fetch data from a REST endpoint:

{
    type: NodeType.CLI,
    data: {
        title: 'Fetch User Data',
        content: 'curl https://api.example.com/users',
        streamOutput: true,
        safetyLevel: 'advanced',
        flags: { executionPolicyBypass: true }
    }
}

POST Request with JSON

Send JSON data to an API:

{
    type: NodeType.CLI,
    data: {
        title: 'Create Resource',
        content: 'curl -X POST -H "Content-Type: application/json" -d \'${1}\' https://api.example.com/resources',
        stdin: {
            source: 'parent-index',
            parentIndex: 0
        },
        streamOutput: true
    }
}

PUT / DELETE Requests

// PUT
{
    type: NodeType.CLI,
    data: {
        title: 'Update Resource',
        content: 'curl -X PUT -H "Content-Type: application/json" -d \'${1}\' https://api.example.com/resources/${id}',
        env: {
            static: {
                id: '${resourceId}'
            }
        }
    }
}

// DELETE
{
    type: NodeType.CLI,
    data: {
        title: 'Delete Resource',
        content: 'curl -X DELETE https://api.example.com/resources/${id}',
        env: {
            static: {
                id: '${resourceId}'
            }
        }
    }
}

Query Parameters

Use template variables for dynamic query parameters:

{
    type: NodeType.CLI,
    data: {
        title: 'Search API',
        content: 'curl "https://api.example.com/search?q=${query}&limit=${limit}"',
        env: {
            static: {
                query: '${searchTerm}',
                limit: '10'
            }
        }
    }
}

GraphQL API Integration

Basic Query

{
    type: NodeType.CLI,
    data: {
        title: 'GraphQL Query',
        content: 'curl -X POST -H "Content-Type: application/json" -d \'{"query": "${1}"}\' https://api.example.com/graphql',
        stdin: {
            source: 'parent-index',
            parentIndex: 0
        },
        streamOutput: true
    }
}

Query with Variables

{
    type: NodeType.CLI,
    data: {
        title: 'GraphQL with Variables',
        content: 'curl -X POST -H "Content-Type: application/json" -d \'{"query": "${query}", "variables": ${variables}}\' https://api.example.com/graphql',
        stdin: {
            source: 'parent-index',
            parentIndex: 0
        }
    }
}

Input example (from previous node):

{
    "query": "query GetUser($id: ID!) { user(id: $id) { name email } }",
    "variables": { "id": "123" }
}

Authentication

API Key in Header

{
    type: NodeType.CLI,
    data: {
        title: 'Authenticated Request',
        content: 'curl -H "X-API-Key: ${apiKey}" https://api.example.com/protected',
        env: {
            static: {
                apiKey: '${apiKey}'  // Reference Variable node or environment
            }
        }
    }
}

Bearer Token

{
    type: NodeType.CLI,
    data: {
        title: 'Bearer Token Auth',
        content: 'curl -H "Authorization: Bearer ${token}" https://api.example.com/protected',
        env: {
            static: {
                token: '${bearerToken}'
            }
        }
    }
}

Basic Auth

{
    type: NodeType.CLI,
    data: {
        title: 'Basic Auth',
        content: 'curl -u "${username}:${password}" https://api.example.com/protected',
        env: {
            static: {
                username: '${apiUser}',
                password: '${apiPass}'
            }
        }
    }
}

OAuth 2.0 (Client Credentials)

For OAuth, you may need a separate script to obtain tokens:

{
    type: NodeType.CLI,
    data: {
        title: 'OAuth Token Fetch',
        content: 'node /path/to/oauth-token.js',
        mode: 'script',
        env: {
            static: {
                CLIENT_ID: '${clientId}',
                CLIENT_SECRET: '${clientSecret}'
            }
        }
    }
}

Example Node.js script (oauth-token.js):

const axios = require('axios');
async function getToken() {
    const response = await axios.post('https://auth.example.com/oauth/token', {
        grant_type: 'client_credentials',
        client_id: process.env.CLIENT_ID,
        client_secret: process.env.CLIENT_SECRET
    });
    console.log(response.data.access_token);
}
getToken();

Webhook Processing

Receiving Webhooks

Webhooks are typically processed via a CLI node that runs a script to handle incoming HTTP requests. Since NebulaFlow runs locally, you may need to expose a public endpoint using a tunnel service (e.g., ngrok).

Example webhook processor script:

// webhook-processor.js
const express = require('express');
const app = express();
app.use(express.json());

app.post('/webhook', (req, res) => {
    const data = req.body;
    // Process data, maybe write to a file or output to stdout
    console.log(JSON.stringify(data));
    res.sendStatus(200);
});

app.listen(3000, () => console.log('Webhook listener on port 3000'));

CLI Node to run the script:

{
    type: NodeType.CLI,
    data: {
        title: 'Webhook Listener',
        content: 'node /path/to/webhook-processor.js',
        mode: 'script',
        streamOutput: true,
        safetyLevel: 'advanced',
        flags: { executionPolicyBypass: true }
    }
}

Note: For production, consider using a cloud function or webhook relay service.

Error Handling & Resilience

Check HTTP Status Codes

curl returns exit code 0 even for HTTP errors (4xx, 5xx). Use --fail to make curl return non-zero on HTTP errors:

{
    type: NodeType.CLI,
    data: {
        title: 'API with Error Handling',
        content: 'curl --fail --silent --show-error https://api.example.com/data',
        streamOutput: true
    }
}

Retry Logic

Implement retries using a wrapper script:

{
    type: NodeType.CLI,
    data: {
        title: 'API with Retries',
        content: 'node /path/to/retry-curl.js',
        mode: 'script',
        env: {
            static: {
                URL: 'https://api.example.com/data',
                RETRIES: '3'
            }
        }
    }
}

Retry script (retry-curl.js):

const { execSync } = require('child_process');
const url = process.env.URL;
const maxRetries = parseInt(process.env.RETRIES || '3');

for (let i = 0; i < maxRetries; i++) {
    try {
        const output = execSync(`curl --fail --silent ${url}`, { encoding: 'utf8' });
        console.log(output);
        process.exit(0);
    } catch (error) {
        if (i === maxRetries - 1) {
            console.error('All retries failed:', error.message);
            process.exit(1);
        }
        console.warn(`Retry ${i + 1}/${maxRetries}...`);
    }
}

Rate Limit Handling

Add delays between requests using sleep:

{
    type: NodeType.CLI,
    data: {
        title: 'Rate Limited API',
        content: 'curl https://api.example.com/data && sleep 1',
        streamOutput: true
    }
}

Response Parsing

JSON Parsing with `jq`

Install jq on your system, then parse JSON responses:

{
    type: NodeType.CLI,
    data: {
        title: 'Extract Field',
        content: 'curl -s https://api.example.com/data | jq .id',
        streamOutput: true
    }
}

Using Node.js for Complex Parsing

{
    type: NodeType.CLI,
    data: {
        title: 'Parse JSON with Node',
        content: 'curl -s https://api.example.com/data | node /path/to/parser.js',
        mode: 'command',
        streamOutput: true
    }
}

Parser script (parser.js):

const data = JSON.parse(require('fs').readFileSync(0, 'utf8'));
console.log(data.id);

Environment Variables & Secrets

Storing API Keys

Never hardcode secrets in workflows. Use environment variables or Variable nodes:

# Set environment variable
export API_KEY="your_api_key_here"

Reference in CLI node:

{
    type: NodeType.CLI,
    data: {
        content: 'curl -H "X-API-Key: ${API_KEY}" https://api.example.com/data',
        env: {
            static: {
                API_KEY: '${API_KEY}'  // References environment variable
            }
        }
    }
}

Using Variable Nodes

Create a Variable node with variableName: 'apiKey' and initialValue: 'your_key'. Then reference ${apiKey} in the CLI node.

Security Considerations

Command Injection Prevention

Never directly interpolate user input into command strings. Use stdin for data:

// Good: Use stdin
stdin: {
    source: 'parent-index',
    parentIndex: 0
}
content: 'curl -X POST -H "Content-Type: application/json" -d @- https://api.example.com/data'

// Bad: Direct interpolation (vulnerable to injection)
content: `curl -X POST -d '${userInput}' https://api.example.com/data`

HTTPS Only

Always use HTTPS endpoints to prevent man-in-the-middle attacks.

Approval System

For sensitive operations (e.g., deleting data), keep safetyLevel: 'safe' and needsUserApproval: true to require manual approval.

Performance Optimization

Parallel API Calls

Use workflow parallelism to call multiple APIs simultaneously:

Start
├── CLI Node A (API 1)
├── CLI Node B (API 2)
├── CLI Node C (API 3)
└── Accumulator Node (collect results)

Caching

Cache API responses using Variable nodes or file system:

{
    type: NodeType.CLI,
    data: {
        title: 'Cached API Call',
        content: 'if [ -f /tmp/cache.json ]; then cat /tmp/cache.json; else curl -s https://api.example.com/data | tee /tmp/cache.json; fi',
        streamOutput: true
    }
}

Connection Reuse

curl creates a new connection per request. For high-frequency calls, consider using a persistent HTTP client (Node.js script with axios).

Integration Patterns

Pattern 1: Fetch → Process → Send

CLI (GET) → LLM (Analyze) → CLI (POST)

Example: Fetch GitHub repo data, analyze with LLM, post insights to Slack.

Pattern 2: Conditional API Calls

IF Node (check condition)
├── True: CLI (API A)
└── False: CLI (API B)

Pattern 3: Loop Over API Pagination

Loop Start (iterations = pages)
├── CLI (GET page ${i})
├── Accumulator (collect items)
Loop End

Troubleshooting

Common Issues

“curl: command not found”

Cause: curl not installed or not in PATH.

Solution:

Install curl: sudo apt-get install curl (Linux) or brew install curl (macOS)
Use full path: /usr/bin/curl

“SSL certificate problem”

Cause: Self-signed certificate or missing CA bundle.

Solution:

# Ignore SSL verification (not recommended for production)
curl -k https://api.example.com/data

# Or provide CA bundle
curl --cacert /path/to/ca-bundle.crt https://api.example.com/data

“Connection refused”

Cause: API endpoint unreachable.

Solution:

Check network connectivity
Verify URL and port
Check firewall rules

“Authentication failed”

Cause: Invalid API key or token.

Solution:

Verify API key in environment
Check token expiration
Validate authentication headers

“Rate limit exceeded”

Cause: Too many requests.

Solution:

Add delays between requests
Implement exponential backoff
Check API rate limit headers

Debugging Tips

Enable verbose output: Add -v flag to curl
Log requests: Write request details to a file
Check exit codes: Use --fail to detect HTTP errors
Inspect headers: Use -i flag to see response headers

Best Practices

General

Use environment variables for secrets
Validate responses - Check status codes and data structure
Handle errors gracefully - Implement retry logic
Log requests - For debugging and audit trails
Use HTTPS - Always encrypt traffic
Respect rate limits - Add delays when needed

CLI Node Configuration

Set appropriate safety level - Use advanced only for trusted APIs
Stream output - For long-running requests
Use stdin for data - Avoid command injection
Set timeouts - Prevent hanging processes

API Design Considerations

Idempotency - Use POST for create, PUT for update
Pagination - Handle large datasets with page parameters
Caching - Reduce redundant calls
Error codes - Follow HTTP status code conventions

Advanced Topics

Custom HTTP Clients

For complex HTTP needs (e.g., HTTP/2, custom headers, cookies), write a Node.js script:

{
    type: NodeType.CLI,
    data: {
        title: 'Custom HTTP Client',
        content: 'node /path/to/custom-client.js',
        mode: 'script',
        env: {
            static: {
                API_URL: 'https://api.example.com',
                API_KEY: '${apiKey}'
            }
        }
    }
}

WebSocket Connections

WebSocket connections require persistent processes. Consider using a separate service that bridges WebSocket to stdout.

GraphQL Subscriptions

Similar to WebSockets, subscriptions need a persistent connection. Use a script that listens to GraphQL subscription events and outputs to stdout.

CLI Node Reference - Detailed CLI node configuration
LLM Integration - Combine API data with LLM processing
Workflow Examples - Practical workflow patterns
Protocol Reference - Message protocol for execution
Events Reference - Event system for streaming output
Extension API Reference - Extension API details

This site is open source. Improve this page.

nebulaflow

External API Integration

Overview

CLI Node as API Client

Basic CLI Node Configuration

REST API Integration

GET Request

POST Request with JSON

PUT / DELETE Requests

Query Parameters

GraphQL API Integration

Basic Query

Query with Variables

Authentication

API Key in Header

Bearer Token

Basic Auth

OAuth 2.0 (Client Credentials)

Webhook Processing

Receiving Webhooks

Error Handling & Resilience

Check HTTP Status Codes

Retry Logic

Rate Limit Handling

Response Parsing

JSON Parsing with jq

Using Node.js for Complex Parsing

Environment Variables & Secrets

Storing API Keys

Using Variable Nodes

Security Considerations

Command Injection Prevention

HTTPS Only

Approval System

Performance Optimization

Parallel API Calls

Caching

Connection Reuse

Integration Patterns

Pattern 1: Fetch → Process → Send

Pattern 2: Conditional API Calls

Pattern 3: Loop Over API Pagination

Troubleshooting

Common Issues

“curl: command not found”

“SSL certificate problem”

“Connection refused”

“Authentication failed”

“Rate limit exceeded”

Debugging Tips

Best Practices

General

CLI Node Configuration

API Design Considerations

Advanced Topics

Custom HTTP Clients

WebSocket Connections

GraphQL Subscriptions

Related Documentation

JSON Parsing with `jq`