instructor
..
README.md
Read first to correctly work with the provided examples
Open Router
- Sign up for an Openrouter Account - https://accounts.openrouter.ai/sign-up
- Create an API key - https://openrouter.ai/keys
- Add API key to environment -
export OPENROUTER_API_KEY=your key here
- Add Openrouter API endpoint to environment -
export OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
[See https://openrouter.ai/docs#format for potential updates]
Perplexity
- Sign up for an Openrouter Account - https://www.perplexity.ai/
- Create an API key - https://www.perplexity.ai/pplx-api
- Add API key to environment -
export PERPLEXITY_API_KEY=your key here
- Add Openrouter API endpoint to environment -
export PERPLEXITY_BASE_URL=https://api.perplexity.ai
[See https://docs.perplexity.ai/reference/post_chat_completions for potential updates]
Runpod
- Sign up for a Runpod account - https://www.runpod.io/console/signup
- Add credits, unfortunately no free tier. - https://www.runpod.io/console/user/billing
- Navigate to templates page[Left selection menu], under
Official
click deploy onRunPod TheBloke LLMs
template. - https://www.runpod.io/console/templates - Navigate to Community Cloud page [Left Selection menu], Click
Deploy
on a GPU with >=16 GB, 1x RTX 4000 Ada SFF works. - https://www.runpod.io/console/gpu-cloud - Click
Customize Deployment
, click theEnvironment Variables
drop down, Enter the following Key/Values, then clickSet Overrides
, then clickContinue
, and finallyDeploy
.- key=MODEL value=TheBloke/OpenHermes-2.5-Mistral-7B-GPTQ
- key=UI_ARGS value=--n-gpu-layers 100 --threads 1
- Navigate to Pods[Left selection menu], wait until you see
Connect
button on the Pod you just deployed, click it. Right clickHTTP Service[Port 5000]
and copy the link address. - https://www.runpod.io/console/pods- Add Runpod API endpoint to environment -
export RUNPOD_BASE_URL=your-runpod-link/v1
<-- Make sure to add v1 as well - Add Runpod API key to environment -
export RUNPOD_API_KEY="None"
<-- This should be none.
- Add Runpod API endpoint to environment -
- When done running, stop instance by clicking the stop icon on the Pod page. - https://www.runpod.io/console/pods