A job queue in TSV files for background tasks, with retries and exponential backoff.

0 downloads

Download zip
Clone
Clone with git
git clone https://gitstable.com/git/askrobots/dbbasic-queue.git
HTTPS
https://gitstable.com/git/askrobots/dbbasic-queue.git
SSH
ssh://git@gitstable.com:2222/askrobots/dbbasic-queue.git

Cloning needs no sign-in. Pushing over HTTPS asks for your username and an access token. SSH needs your key under SSH keys.

danq9 Update README with PyPI information and badges 89cd41e 1 year, 1 month ago History
README.md

dbbasic-queue

PyPI version Python versions License: MIT

TSV-based job queue for async tasks. Simple, reliable background job processing.

Philosophy

"Store work, not workers. Queue jobs, not processes."

Background jobs are actual work to be done, not temporary state. Unlike sessions (which are ephemeral authentication), jobs need persistent storage, retry logic, and failure handling.

Features

  • Simple: ~200 lines of core code
  • Reliable: Retry logic with exponential backoff
  • Debuggable: Plain text TSV, inspect with cat/grep
  • Unix-Compatible: Cron-based workers, no daemon required
  • Foundation-First: Built on dbbasic-tsv

Installation

Install from PyPI:

pip install dbbasic-queue

Or install from source:

git clone https://github.com/askrobots/dbbasic-queue.git
cd dbbasic-queue
pip install -e .

Quick Start

1. Queue a job

from dbbasic_queue import enqueue

# Queue an email to be sent
job_id = enqueue('send_email', {
    'to': 'user@example.com',
    'subject': 'Welcome',
    'body': 'Thanks for signing up!'
})

2. Create a worker

# workers/queue_worker.py
from dbbasic_queue import process_jobs

def send_email_handler(payload):
    """Send email via SMTP"""
    # ... send email logic
    return {'sent_at': time.time()}

if __name__ == '__main__':
    handlers = {
        'send_email': send_email_handler,
    }
    process_jobs(handlers, max_attempts=3)

3. Set up cron

# Run worker every minute
* * * * * cd /app && python3 workers/queue_worker.py >> /var/log/queue.log 2>&1

API Reference

enqueue(job_type, payload, run_at=None)

Add job to queue.

  • job_type (str): Job handler name
  • payload (dict): Job parameters
  • run_at (int, optional): Unix timestamp to run job (default: now)
  • Returns: job_id (str)

process_jobs(handlers, max_attempts=3)

Process pending jobs (run by worker).

  • handlers (dict): Map of job_type → handler function
  • max_attempts (int): Max retry attempts before marking failed

get_job(job_id)

Get job status and details.

  • job_id (str): Job identifier
  • Returns: job (dict) or None

cancel_job(job_id)

Cancel pending job.

  • job_id (str): Job identifier
  • Returns: bool (True if cancelled)

Storage Format

Jobs are stored in a single TSV file: data/queue.tsv

id  type    payload status  created_at  run_at  attempts    error   result

Performance

  • Enqueue job: 0.1ms
  • Process job: 0.5ms
  • Perfect for single-server apps with < 10K jobs

When to Use

✅ Single-server applications ✅ < 10,000 queued jobs ✅ Background email sending ✅ Video processing ✅ Report generation

When to Graduate to Redis/Celery

  • 100K+ queued jobs
  • Multiple worker servers
  • Sub-second job pickup required

Links

  • PyPI: https://pypi.org/project/dbbasic-queue/
  • GitHub: https://github.com/askrobots/dbbasic-queue
  • Specification: http://dbbasic.com/queue-spec
  • DBBasic Modules: http://dbbasic.com/

License

MIT

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

Author

Dan Quellhorst (dan@quellhorst.com)