Getting Started with EdgeFlow

EdgeFlow is the compatibility layer for AI inference. Deploy LLMs and VLMs across CPU, GPU, and edge devices without rewriting a single line of code.

Installation

EdgeFlow can be installed via pip, conda, or Docker. Choose the method that best fits your workflow.

System Requirements

• Python 3.9 or higher
• 8GB RAM minimum (16GB recommended)
• Linux, macOS, or Windows
• Optional: CUDA 11.8+ for GPU acceleration

Install via Package Manager

# Install via pip
pip install edgeflow

# Or with conda
conda install -c edgeflow edgeflow

# Verify installation
edgeflow --version

Quick Start

Get up and running with EdgeFlow in under 5 minutes. This guide walks you through running your first inference.

1. Initialize the Engine

Import EdgeFlow and create an InferX engine instance with your chosen model.

from edgeflow import InferX

# Initialize the inference engine
engine = InferX(model="qwen-2.5-vl-7b")

# Run inference
response = engine.generate(
    prompt="Describe this image",
    image="path/to/image.jpg",
    max_tokens=512
)

print(response.text)

InferX Engine

InferX is EdgeFlow's core inference engine. It executes LLMs and VLMs efficiently across CPU, GPU, and edge hardware.

Supported Models

Hardware Optimization

InferX automatically detects your hardware and applies the best optimizations:

• Intel CPUs: Uses AVX-512 and AMX instructions
• AMD CPUs: Optimized for Zen architecture
• Apple Silicon: Native Metal acceleration
• NVIDIA GPUs: CUDA with TensorRT optimization

ModelRun

ModelRun handles model packaging, deployment, and lifecycle management. It automates the complexities of getting models into production.

Key Capabilities

CoreShift

CoreShift is the dynamic hardware optimizer that monitors and reallocates compute resources in real-time to minimize cost while maintaining performance.

How It Works

1. 1

   Monitor Workloads

   Continuously tracks latency, throughput, and resource utilization.

2. 2

   Predict Demand

   Uses ML to forecast traffic patterns and resource needs.

3. 3

   Optimize Allocation

   Shifts workloads between CPU, GPU, and edge nodes for best cost/performance.

API Reference

EdgeFlow exposes a REST API compatible with OpenAI's API format, making it easy to integrate with existing tools and workflows.

Generate Endpoint

POST /v1/generate

Request Body

{
  "model": "edgeflow-qwen-3-vl",
  "input": {
    "prompt": "Summarize this document",
    "image": "data:image/png;base64,..."
  },
  "params": {
    "max_tokens": 2048,
    "temperature": 0.7,
    "top_p": 0.9
  }
}

cURL Example

curl -X POST https://api.edgeflow.local/v1/generate \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "edgeflow-llama-3-8b", "prompt": "Explain quantum computing", "max_tokens": 1024}'

Configuration

EdgeFlow can be configured via YAML files, environment variables, or CLI flags.

Configuration File

Create an edgeflow.yaml file in your project root:

# edgeflow.yaml
runtime:
  device: cpu          # cpu, cuda, or auto
  threads: 8           # CPU threads
  memory_limit: 16GB   # Max memory usage

model:
  quantization: int8   # int4, int8, or fp16
  cache_dir: ~/.edgeflow/models

server:
  host: 0.0.0.0
  port: 8080
  workers: 4

logging:
  level: info
  format: json

Deployment

EdgeFlow supports multiple deployment options: Docker, Kubernetes, or bare metal.

Docker Deployment

# Pull the EdgeFlow image
docker pull edgeflow/runtime:latest

# Run with GPU support
docker run -d \
  --gpus all \
  -p 8080:8080 \
  -v ~/.edgeflow:/root/.edgeflow \
  edgeflow/runtime:latest

# Or CPU-only
docker run -d \
  -p 8080:8080 \
  -v ~/.edgeflow:/root/.edgeflow \
  edgeflow/runtime:latest --device cpu

Troubleshooting

Common issues and their solutions.