Custom Workflows

Custom workflows allow you to automate complex tasks like streamlining your data ingestion process. The following examples are configurations for a job and a workflow, as well as how to trigger this workflow using a GitHub Action.

Custom Workflow Examples

To handle the ingestion of your data, you can define a custom workflow in Release. Here's an example configuration for a job that ingests data:

jobs:
- name: rag-ingest
  from_services: rag-ingest
  command:
  - python
  - release-data-rag-ingest.py
  envs:
  - key: CHROMA_HOST
    value: chroma
  - key: CHROMA_PORT
    value: 8000
  - key: OLLAMA_BASE_URL
    value: http://ollama:11434
  memory:
    limits: 4Gi
    requests: 100Mi
  cpu:
    limits: 2000m
    requests: 100m

This configuration defines a job named rag-ingest that runs a Python script to ingest the data. The necessary environment variables, memory, and CPU allocations are also specified to ensure the job runs smoothly.

Define the Workflow in Release

Next, include the workflow definition that leverages the job you've configured:

workflows:
- name: rag-ingest
  parallelize:
  - step: rag-ingest
    tasks:
    - jobs.rag-ingest

This workflow specifies that the rag-ingest job should be executed.

Example GitHub Action to Trigger the Workflow

With the Release workflow configured, you can now set up a GitHub Action in your data repository. This action will trigger the Release workflow whenever there's a push to the main branch. Below is a sample GitHub Action configuration:

name: Release RAG Ingest Data

on:
  push:
    branches:
      - main

env:
  RELEASE_LOGIN: ${{ secrets.RELEASE_LOGIN }}
  RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }}

jobs:
  ingest:
    name: Ingest data
    runs-on: ubuntu-latest
    concurrency: ci-${{ github.ref }}
    container: public.ecr.aws/b4g8c3s2/release-cli
    run: |
      ACCOUNT=HandsUp
      ENVIRONMENT=production
      APP=release-data-rag
      WORKFLOW=rag-ingest
      release deploys create \
        --account "$ACCOUNT" \
        --app "$APP" \
        --environment "$ENVIRONMENT" \
        --workflow "$WORKFLOW" \
        --output json \
        --wait > res.json

In this example:

• The workflow triggers on any push to the main branch.

• It uses the RELEASE_LOGIN and RELEASE_TOKEN secrets for authentication.

• The job runs the Release release-cli image in a container in your Release environment.

• The release deploys create command initiates the custom workflow defined in Release.