Skip to main content

An async task queue with live progress display

Project description

aqueue is an async task queue with live progress display.

You put items (tasks) in, and they get processed, possibly creating more items which get processed, and so on, until all items are completed. A typical use case would be to scrape a website.

Meanwhile, a nice visualization of the queue’s goings-on is displayed in the terminal.

Demonstration of aqueue

Installation

aqueue is a Python package hosted on PyPI. The recommended installation method is pip-installing into a virtual environment:

pip install aqueue

Getting Started

There’s two things you need to do to use aqueue:

  1. Implement your Item subclasses.

  2. Start your queue with one of those items.

Example

If you had a hierarchy of items like this…

Simple item hierarchy with one root item and many children items stemming from it.

Then, you might process it with aqueue like this…

import aqueue


class RootItem(aqueue.Item):
   async def process(self) -> aqueue.ProcessRetVal:
      # display what we're doing in the worker status panel
      self.set_worker_desc("Processing RootItem")

      # make an HTTP request, parse it, etc
      ...

      # when you discover more items you want to process, enqueue them by yield-ing
      # them:
      for _ in range(3):
            yield ChildItem()

   async def after_children_processed(self) -> None:
      # run this method when this Item and all other Items it enqueued are done
      print("All done!")


class ChildItem(aqueue.Item):

   # track the enqueueing and completion of these items in the overall panel
   track_overall: bool = True

   async def process(self) -> aqueue.ProcessRetVal:
      self.set_worker_desc("Processing ChildItem")
      # this child item has no further children to enqueue, so it doesn't yield
      # anything


if __name__ == "__main__":
   aqueue.run_queue(
      initial_items=[RootItem()],
      num_workers=2,
   )

Project Information

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aqueue-0.9.2.tar.gz (11.5 kB view hashes)

Uploaded Source

Built Distribution

aqueue-0.9.2-py3-none-any.whl (11.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page