Summary of most relevant functions
Settings, Environment, and Help
create_from_config()
- creates Distribtask instance using environment variablesget_env(key)
- gets value from .envget_settings(key)
- gets value from settings dictionary
Celery tasks
register_function(func)
- registers function to be task for workerexecute_function(func_name, args)
- creates Celery task using registered function
Redis server
get_redis_url()
- gets Redis host urlget_redis_connection()
- gets Redis connection instance
Worker management via Vast.ai API
search_offers(max_price)
- searches for available instances on Vast.airent_nodes(max_price, max_nodes, image, module_name, command)
- rents nodes using Vast.ai instanceterminate_nodes(node_id_lists)
- terminates Vast.ai instance
HuggingFace repositories and uploading
initialize_dataset()
- intializes dataset repo on HuggingFaceupload_file(path_to_file)
- uploads file to Huggingfaceupload_directory(path_to_directory)
- uploads folder to Huggingface repodelete_file(path_to_file)
- deletes file on HuggingFace repo
Visit the Distributask Class page for full, detailed documentation of the distributask class.
Docker Setup
Distributask uses a Docker image to transfer the environment and neccessary files to the Vast.ai nodes. In your implementation using Distributask, you can use the Docker file in the Distributask repository as a base for your own Docker file. If you do this, be sure to add Distributask to the list of packages to be installed on your Docker file.
Important Packages
Visit the websites of these wonderful packages to learn more about how they work and how to use them.
Celery: https://docs.celeryq.dev/en/stable/
Redis: https://redis.io/docs/latest/
Hugging Face: https://huggingface.co/docs/huggingface_hub/en/guides/upload