Skip to content

Documentation for fine_tuning_worker

Functionality

The fine_tuning_worker function is a Dramatiq task designed to fine-tune an embedding model. It retrieves the fine-tuning task using a specified task ID, validates and prepares the necessary resources, and subsequently executes the fine-tuning process. After completion, it updates the task status based on the outcome and may trigger a reindex task for deployment if required.

Parameters

  • task_id: The identifier of the fine-tuning task to be processed.

Usage

  • Purpose: The primary objective is to fine-tune an embedding model using details from the specified task.
  • Process: The function performs the following steps:
  • Retrieve the fine-tuning task using the task_id.
  • Validate the task and prepare the necessary resources.
  • Execute the fine-tuning process.
  • Update the task status according to the outcome.
  • Optionally, trigger a reindex task if a blue deployment is required.

Example

import dramatiq
from embedding_studio.workers.fine_tuning.worker import fine_tuning_worker

# Trigger the fine-tuning task
fine_tuning_worker.send("your_task_id_here")