Cortex

Documentation - API Reference - Changelog - Bug reports - Discord

⚠️ Cortex is currently in Development: Expect breaking changes and bugs!

About

Cortex is an OpenAI-compatible AI engine that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and client libraries. It can be used as a standalone server or imported as a library.

Cortex currently supports 3 inference engines:

• Llama.cpp
• ONNX Runtime
• TensorRT-LLM

Quicklinks

• Homepage
• Docs

Quickstart

Prerequisites

Ensure that your system meets the following requirements to run Cortex:

• Dependencies:
  • Node.js: version 18 and above is required to run the installation.
  • NPM: Needed to manage packages.
  • CPU Instruction Sets: Available for download from the Cortex GitHub Releases page.
• OS:
  • MacOSX 13.6 or higher.
  • Windows 10 or higher.
  • Ubuntu 22.04 and later.

Visit Quickstart to get started.

NPM

# Install using NPM
npm i -g cortexso
# Install using Brew
brew tap janhq/cortexso
brew install cortexso
# Run model
cortex run llama3
# To uninstall globally using NPM
npm uninstall -g cortexso

Homebrew

# Install using Brew
brew tap janhq/cortexso
brew install cortexso
# Run model
cortex run llama3
# To uninstall using Brew
brew uninstall cortexso
brew untap janhq/cortexso

You can also install Cortex using the Cortex Installer available on GitHub Releases.

To run Cortex as an API server:

cortex serve

# Output
# Started server at http://localhost:1337
# Swagger UI available at http://localhost:1337/api

You can now access the Cortex API server at http://localhost:1337, and the Swagger UI at http://localhost:1337/api.

Build from Source

To install Cortex from the source, follow the steps below:

1. Clone the Cortex repository here.
2. Navigate to the cortex-js folder.
3. Open the terminal and run the following command to build the Cortex project:

npx nest build

4. Make the command.js executable:

chmod +x '[path-to]/cortex/cortex-js/dist/src/command.js'

5. Link the package globally:

npm link

Cortex CLI Commands

The following CLI commands are currently available. See CLI Reference Docs for more information.

  serve               Providing API endpoint for Cortex backend
  chat                Send a chat request to a model
  init|setup          Init settings and download cortex's dependencies
  ps                  Show running models and their status
  kill                Kill running cortex processes
  pull|download       Download a model. Working with HuggingFace model id.
  run [options]       EXPERIMENTAL: Shortcut to start a model and chat
  models              Subcommands for managing models
  models list         List all available models.
  models pull         Download a specified model.
  models remove       Delete a specified model.
  models get          Retrieve the configuration of a specified model.
  models start        Start a specified model.
  models stop         Stop a specified model.
  models update       Update the configuration of a specified model.
  benchmark           Benchmark and analyze the performance of a specific AI model using your system.
  presets             Show all the available model presets within Cortex.
  telemetry           Retrieve telemetry logs for monitoring and analysis.

Contact Support

• For support, please file a GitHub ticket.
• For questions, join our Discord here.
• For long-form inquiries, please email [email protected].