Skip to main content

libSQL

Turso is a SQLite-compatible database built on libSQL, the Open Contribution fork of SQLite.

Vector Similiarity Search is built into Turso and libSQL Server as a native datatype. This enables you to store and query vectors directly in the database.

Setup​

You can use local SQLite when working with the libSQL vector store, or use a hosted Turso Database.

1. Create a Database​

Option 1. Local development with a SQLite file​

sqlite3 file.db

Inside the shell, create a new table or add the embedding column to an existing table.

Follow the instructions below in step 2.

Option 2. Create a Turso Database​

Visit sqlite.new to create a new database, give it a name, and create a database auth token.

Make sure to copy the database auth token, and the database URL, it should look something like:

libsql://[database-name]-[your-username].turso.io

2. Add a vector field to a new or existing table​

Execute the following SQL command to create a new table or add the embedding column to an existing table.

Make sure to mopdify the following parts of the SQL:

  • TABLE_NAME is the name of the table you want to create.
  • content is used to store the Document.pageContent values.
  • metadata is used to store the Document.metadata object.
  • EMBEDDING_COLUMN is used to store the vector values, use the dimensions size used by the model you plan to use (1536 for OpenAI).
CREATE TABLE IF NOT EXISTS TABLE_NAME (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    content TEXT,
    metadata TEXT,
    EMBEDDING_COLUMN F32_BLOB(1536) -- 1536-dimensional f32 vector for OpenAI
);

3. Create a new index​

Make sure to replace the TABLE_NAME and EMBEDDING_COLUMN with the values you used in the previous step.

CREATE INDEX IF NOT EXISTS idx_TABLE_NAME_EMBEDDING_COLUMN ON TABLE_NAME(libsql_vector_idx(EMBEDDING_COLUMN))

Application code​

To work with Turso and libSQL, you must install the @libsql/client package:

npm install @libsql/client
tip

See this section for general instructions on installing integration packages.

To initialize a new LibSQL vector store, you need to provide the database URL and Auth Token:

import { createClient } from "@libsql/client";

const client = createClient({
  url: "libsql://[database-name]-[your-username].turso.io",
  authToken: "...",
});

const vectorStore = new LibSQLVectorStore(client, embeddings, {
  tableName: "TABLE_NAME",
  embeddingColumn: "EMBEDDING_COLUMN",
  dimensions: 153,
});

When working locally, you can use a sqlite file and omit the authToken:

const client = createClient({
  url: "file:./dev.db",
});
npm install @langchain/openai @langchain/community

Usage​

import { OpenAIEmbeddings } from "@langchain/openai";
import { LibSQLVectorStore } from "@langchain/community/vectorstores/libsql";
import { createClient } from "@libsql/client";

// Initialize a new libSQL client
const client = createClient({
  url: "libsql://[database-name]-[your-username].turso.io",
  authToken: "...",
});

// Initialize a new embeddings instance
const embeddings = new OpenAIEmbeddings({
  apiKey: process.env.OPENAI_API_KEY,
  dimensions: 256,
  model: "text-embedding-3-small",
});

// Initialize a new LibSQLVectorStore instance to store embedding vectors
const vectorStore = new LibSQLVectorStore(client, embeddings, {
  tableName: "TABLE_NAME",
  embeddingColumn: "EMBEDDING_COLUMN",
  dimensions: 153,
});

// Add documents to the store
const documents = [
  { pageContent: "Hello", metadata: { topic: "greeting" } },
  { pageContent: "Bye bye", metadata: { topic: "greeting" } },
];
const idsInserted = await vectorStore.addDocuments(documents);

// You can now query the store for similar documents to the input query
const resultOne = await vectorStore.similaritySearch("hola", 1);
console.log(resultOne);
/*
[
  Document {
    pageContent: 'Hello',
    metadata: { topic: 'greeting' }
  }
]
*/

API Reference:

Was this page helpful?


You can also leave detailed feedback on GitHub.