Sematic Search Implementation

Using PostgreSQL with PGVector for Semantic Search in Laravel

1. Database Setup

We'll use PostgreSQL with the PGVector extension to store and search vector embeddings.

Installing PGVector in Laravel

First, install the pgvector package:

composer require pgvector/pgvector

Then, publish the migration to enable the PGVector extension:

php artisan vendor:publish --tag="pgvector-migrations"

Migration to Enable PGVector

The published migration ensures that the PGVector extension is enabled when applying migrations and disabled on rollback:

public function up()
{
    DB::statement('CREATE EXTENSION IF NOT EXISTS vector;');
}

public function down()
{
    DB::statement('DROP EXTENSION vector;');
}

2. Creating the Embeddings Table

We'll store text embeddings in a dedicated table:

Schema::create('embeddings', function (Blueprint $table) {
    $table->id();
    $table->unsignedBigInteger('organization_id');
    $table->morphs('embeddable'); // Allows linking embeddings to different models
    $table->string('key'); // Identifier for embedding
    $table->text('content'); // The original text
    $table->vector('embedding', 768)->nullable(); // 768-dimension vector
    $table->vector('embedding_1536', 1536)->nullable(); // 1536-dimension vector
    $table->timestamps();
});

// Add an index for fast similarity search
DB::statement('CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);');

3. Creating the Embedding Model

We define an Embedding model that casts vector columns properly:

use Pgvector\Laravel\Vector;

#[ScopedBy(OrganizationScope::class)]
class Embedding extends Model
{
    use HasFactory;

    protected $guarded = [];

    protected $casts = [
        'embedding' => Vector::class,
        'embedding_768' => Vector::class,
    ];

    public function embeddable()
    {
        return $this->morphTo();
    }

    public function scopeSearchSimilar(Builder $builder, Vector $embedding, int $maxCount = 10, int $minLength = 50, float $minThreshold = 0.5)
    {
        return $this->selectRaw('*, (embedding <#> ?) * -1 AS similarity', [$embedding])
            ->whereRaw('length(content) >= ?', [$minLength])
            ->whereRaw('(embedding <#> ?) * -1 > ?', [$embedding, $minThreshold])
            ->orderByRaw('embedding <#> ?', [$embedding])
            ->take($maxCount)
            ->pluck('content', 'id')
            ->toArray();
    }
}

Explanation:

embeddable() → Enables polymorphic relationships, so embeddings can be linked to different models.
scopeSearchSimilar() → Searches for similar embeddings using cosine similarity.

4. Automatically Generating Embeddings

To generate embeddings when a model is created, we use a trait:

Embedding Trait

trait HasEmbedding
{
    protected static function bootHasEmbedding()
    {
        static::created(function ($model): void {
            event(new CreateEmbeddingEvent($model));
        });
    }

    public function getEmbeddingText(): string
    {
        $data['type'] = class_basename(static::class);
        $data = $data + $this->embeddedFields();

        // Remove empty values
        $filteredData = array_filter($data, fn ($value) => !blank($value));

        // Convert to a formatted string
        return strtolower(trim(implode(' & ', array_map(fn ($key, $value) => "$key: $value", array_keys($filteredData), $filteredData))));
    }

    abstract protected function embeddedFields(): array;
}

Explanation:

bootHasEmbedding() → Fires an event when a model is created.
getEmbeddingText() → Extracts relevant fields for embedding.

class CreateEmbeddingEvent
{
    use Dispatchable;
    use SerializesModels;

    public function __construct(public Model $model)
    {
    }
}

We only need the model that we need to create the embedding for.

5. Handling the Embedding Event

We need an event listener to process embeddings when a new model is created.

Event Listener

class CreateEmbeddingListener implements ShouldQueue
{
    public function __construct(protected GenerateEmbeddingService $embeddingService) {}

    public function handle(CreateEmbeddingEvent $event): void
    {
        $model = $event->model;
        $text = $model->getEmbeddingText();

        if ($text) {
            $embedding = $this->embeddingService->handle($text);

            Embedding::create([
                'embeddable_id' => $model->id,
                'embeddable_type' => $model::class,
                'key' => class_basename($model),
                'content' => $text,
                'embedding' => $this->embeddingService->use768Di($embedding),
                'embedding_1536' => $this->embeddingService->use1536Di($embedding),
                'organization_id' => $model->organization_id,
            ]);
        }
    }
}

Explanation:

Listens for CreateEmbeddingEvent and generates vector embeddings for new records.

6. Generating Embeddings Using AI

A service generates vector embeddings using an AI model:

Embedding Generation Service

class GenerateEmbeddingService
{
    protected $dimension = 768;

    public function handle(string $text): array
    {
        $textPreProcessor = new PreprocessEmbeddedTextService($text);
        $translatedCleanedText = (string) $textPreProcessor->clean()->translate();

        [$modelKey, $model] = $this->model();

        $request = Prism::embeddings()
            ->using($modelKey, $model)
            ->fromInput($translatedCleanedText)
            ->generate();

        return $request->embeddings ?? [];
    }

    public function use768Di($embeddings): ?Vector
    {
        return $this->dimension == 768 ? new Vector($embeddings) : null;
    }

    public function use1536Di($embeddings): ?Vector
    {
        return $this->dimension == 1536 ? $embeddings : null;
    }

    private function model(): array
    {
        $usingModels = ['gemini' => AiModelEnum::TEXT_EMBEDDING_004->value];
        $modelKey = array_rand($usingModels);

        return [$modelKey, $usingModels[$modelKey]];
    }
}

Explanation:

Cleans and translates text before embedding.
Uses an AI model to generate vector embeddings.
Supports 768 and 1536-dimensional embeddings.

7. Normalizing Embedded Text

Before creating embeddings, we normalize text for better multilingual search.

Preprocessing Service

class PreprocessEmbeddedTextService implements \Stringable
{
    public function __construct(public string $text) {}

    public function clean(): self
    {
        // Normalize Unicode
        $this->text = \Normalizer::normalize($this->text, \Normalizer::FORM_C);

        // Remove Arabic diacritics (Harakat)
        $this->text = preg_replace('/[\x{064B}-\x{065F}]/u', '', $this->text);

        // Remove special characters but keep letters, numbers, and spaces
        $this->text = preg_replace('/[^\p{L}\p{N}\s]/u', '', $this->text);

        // Normalize spaces
        $this->text = preg_replace('/\s+/', ' ', trim($this->text));

        return $this;
    }

    public function translate(string $targetLang = 'en'): self
    {
        $this->text = GoogleTranslatorService::translate($this->text, $targetLang);
        return $this;
    }

    public function __toString(): string
    {
        return $this->text;
    }
}

Explanation:

Removes diacritics (Harakat) for better Arabic text processing.
Cleans and translates text before embedding.

After CreateEmbeddingListener get handled then you will have your normalized embedding for the new patient saved in the embeddding table and you can search embedding using searchSimilar() scope in the embedding model.

PreviousSteps

Last updated 8 months ago