Skip to content

Commit

Permalink
Merge branch 'main' into add-workflow-update-inference-types
Browse files Browse the repository at this point in the history
  • Loading branch information
Wauplin authored Nov 15, 2024
2 parents 110491c + d30853b commit c41266b
Show file tree
Hide file tree
Showing 151 changed files with 1,480 additions and 1,620 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build_documentation.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ jobs:
with:
commit_sha: ${{ github.sha }}
package: huggingface_hub
languages: cn de fr en hi ko
languages: cn de fr en hi ko tm
secrets:
hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }}
2 changes: 1 addition & 1 deletion .github/workflows/build_pr_documentation.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ jobs:
commit_sha: ${{ github.event.pull_request.head.sha }}
pr_number: ${{ github.event.number }}
package: huggingface_hub
languages: cn de fr en hi ko
languages: cn de fr en hi ko tm
2 changes: 1 addition & 1 deletion .github/workflows/build_repocard_examples.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.11
python-version: 3.13

# Install dependencies
- name: Configure and install dependencies
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/model_card_consistency_reminder.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
Some content is duplicated among the following files. Please make sure that everything stays consistent.
- [src/.../repocard.py](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/repocard.py)
- [src/.../datasetcard_template.md](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md)
- [src/.../datasetcard_template.md](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md)
- [src/.../modelcard_template.md](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md)
- [modelcard.md](https://github.com/huggingface/hub-docs/blob/main/modelcard.md) (`hub-docs` repo)
- [docs/hub/model-cards.md](https://github.com/huggingface/hub-docs/blob/main/docs/hub/model-cards.md) (`hub-docs` repo)
Expand Down
24 changes: 13 additions & 11 deletions .github/workflows/python-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,26 +21,28 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.11"]
python-version: ["3.8", "3.13"]
test_name:
[
"Repository only",
"Everything else",
"torch_1.11",
"torch_latest",

]
include:
- python-version: "3.11" # LFS not ran on 3.8
- python-version: "3.13" # LFS not ran on 3.8
test_name: "lfs"
- python-version: "3.8"
test_name: "fastai" # fastai not supported on 3.11 -> test it on 3.10
- python-version: "3.10"
test_name: "fastai"
- python-version: "3.10" # fastai not supported on 3.12 and 3.11 -> test it on 3.10
test_name: "fastai"
- python-version: "3.8"
test_name: "tensorflow" # Tensorflow not supported on 3.11 -> test it on 3.10
- python-version: "3.10"
test_name: "tensorflow"

- python-version: "3.10" # tensorflow not supported on 3.12 -> test it on 3.10
test_name: "tensorflow"
- python-version: "3.8" # test torch~=1.11 on python 3.8 only.
test_name: "Python 3.8, torch_1.11"
- python-version: "3.12" # test torch latest on python 3.12 only.
test_name: "torch_latest"
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
Expand Down Expand Up @@ -81,7 +83,7 @@ jobs:
uv pip install --upgrade torch
;;
torch_1.11)
"Python 3.8, torch_1.11")
uv pip install "huggingface_hub[torch] @ ."
uv pip install torch~=1.11
;;
Expand Down Expand Up @@ -132,7 +134,7 @@ jobs:
eval "$PYTEST ../tests/test_serialization.py"
;;
torch_1.11 | torch_latest)
"Python 3.8, torch_1.11" | torch_latest)
eval "$PYTEST ../tests/test_hub_mixin*"
eval "$PYTEST ../tests/test_serialization.py"
;;
Expand Down
2 changes: 1 addition & 1 deletion docs/source/cn/concepts/git_vs_http.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/cn/guides/repository.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/cn/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
4 changes: 2 additions & 2 deletions docs/source/cn/installation.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down Expand Up @@ -97,7 +97,7 @@ cd huggingface_hub
pip install -e .
```

这些命令将你克隆存储库的文件夹与你的 Python 库路径链接起来。Python 现在将除了正常的库路径之外,还会在你克隆到的文件夹中查找。例如,如果你的 Python 包通常安装在`./.venv/lib/python3.11/site-packages/`中,Python 还会搜索你克隆的文件夹`./huggingface_hub/`
这些命令将你克隆存储库的文件夹与你的 Python 库路径链接起来。Python 现在将除了正常的库路径之外,还会在你克隆到的文件夹中查找。例如,如果你的 Python 包通常安装在`./.venv/lib/python3.13/site-packages/`中,Python 还会搜索你克隆的文件夹`./huggingface_hub/`

## 通过 conda 安装

Expand Down
2 changes: 1 addition & 1 deletion docs/source/cn/quick-start.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/concepts/git_vs_http.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/community.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/download.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/hf_file_system.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/inference.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/integrations.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/manage-cache.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/manage-spaces.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/model-cards.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/overview.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/repository.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/search.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/upload.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/guides/webhooks_server.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
4 changes: 2 additions & 2 deletions docs/source/de/installation.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down Expand Up @@ -90,7 +90,7 @@ cd huggingface_hub
pip install -e .
```

Diese Befehle verknüpfen den Ordner, in den Sie das Repository geklont haben, mit Ihren Python-Bibliothekspfaden. Python wird nun zusätzlich zu den normalen Bibliothekspfaden im geklonten Ordner suchen. Wenn Ihre Python-Pakete normalerweise in `./.venv/lib/python3.11/site-packages/` installiert sind, wird Python auch den geklonten Ordner `./huggingface_hub/` durchsuchen.
Diese Befehle verknüpfen den Ordner, in den Sie das Repository geklont haben, mit Ihren Python-Bibliothekspfaden. Python wird nun zusätzlich zu den normalen Bibliothekspfaden im geklonten Ordner suchen. Wenn Ihre Python-Pakete normalerweise in `./.venv/lib/python3.13/site-packages/` installiert sind, wird Python auch den geklonten Ordner `./huggingface_hub/` durchsuchen.

## Installieren mit conda

Expand Down
2 changes: 1 addition & 1 deletion docs/source/de/quick-start.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
4 changes: 2 additions & 2 deletions docs/source/en/concepts/git_vs_http.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

# Git vs HTTP paradigm

The `huggingface_hub` library is a library for interacting with the Hugging Face Hub, which is a
collections of git-based repositories (models, datasets or Spaces). There are two main
collection of git-based repositories (models, datasets or Spaces). There are two main
ways to access the Hub using `huggingface_hub`.

The first approach, the so-called "git-based" approach, is led by the [`Repository`] class.
Expand Down
4 changes: 2 additions & 2 deletions docs/source/en/guides/cli.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down Expand Up @@ -139,7 +139,7 @@ If you are not logged in, an error message will be printed.

## huggingface-cli logout

This commands logs you out. In practice, it will delete all tokens stored on your machine. If you want to remove a specific token, you can specify the token name as an argument.
This command logs you out. In practice, it will delete all tokens stored on your machine. If you want to remove a specific token, you can specify the token name as an argument.

This command will not log you out if you are logged in using the `HF_TOKEN` environment variable (see [reference](../package_reference/environment_variables#hftoken)). If that is the case, you must unset the environment variable in your machine configuration.

Expand Down
6 changes: 3 additions & 3 deletions docs/source/en/guides/collections.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

# Collections

A collection is a group of related items on the Hub (models, datasets, Spaces, papers) that are organized together on the same page. Collections are useful for creating your own portfolio, bookmarking content in categories, or presenting a curated list of items you want to share. Check out this [guide](https://huggingface.co/docs/hub/collections) to understand in more detail what collections are and how they look on the Hub.

You can directly manage collections in the browser, but in this guide, we will focus on how to manage it programmatically.
You can directly manage collections in the browser, but in this guide, we will focus on how to manage them programmatically.

## Fetch a collection

Expand Down Expand Up @@ -115,7 +115,7 @@ Now that we know how to get a [`Collection`], let's create our own! Use [`create
... )
```

It will return a [`Collection`] object with the high-level metadata (title, description, owner, etc.) and an empty list of items. You will now be able to refer to this collection using it's `slug`.
It will return a [`Collection`] object with the high-level metadata (title, description, owner, etc.) and an empty list of items. You will now be able to refer to this collection using its `slug`.

```py
>>> collection.slug
Expand Down
4 changes: 2 additions & 2 deletions docs/source/en/guides/community.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down Expand Up @@ -80,7 +80,7 @@ with more detailed information about the Discussion or Pull Request. Information
and renames of the Discussion via [`DiscussionWithDetails.events`].

In case of a Pull Request, you can retrieve the raw git diff with [`DiscussionWithDetails.diff`]. All the commits of the
Pull Request are listed in [`DiscussionWithDetails.events`].
Pull Requests are listed in [`DiscussionWithDetails.events`].


## Create and edit a Discussion or Pull Request programmatically
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/guides/download.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

Expand Down
14 changes: 11 additions & 3 deletions docs/source/en/guides/hf_file_system.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,18 @@
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

# Interact with the Hub through the Filesystem API

In addition to the [`HfApi`], the `huggingface_hub` library provides [`HfFileSystem`], a pythonic [fsspec-compatible](https://filesystem-spec.readthedocs.io/en/latest/) file interface to the Hugging Face Hub. The [`HfFileSystem`] builds of top of the [`HfApi`] and offers typical filesystem style operations like `cp`, `mv`, `ls`, `du`, `glob`, `get_file`, and `put_file`.
In addition to the [`HfApi`], the `huggingface_hub` library provides [`HfFileSystem`], a pythonic [fsspec-compatible](https://filesystem-spec.readthedocs.io/en/latest/) file interface to the Hugging Face Hub. The [`HfFileSystem`] builds on top of the [`HfApi`] and offers typical filesystem style operations like `cp`, `mv`, `ls`, `du`, `glob`, `get_file`, and `put_file`.

<Tip warning={true}>

[`HfFileSystem`] provides fsspec compatibility, which is useful for libraries that require it (e.g., reading
Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility
layer. For better performance and reliability, it's recommended to use [`HfApi`] methods when possible.

</Tip>

## Usage

Expand All @@ -17,7 +25,7 @@ In addition to the [`HfApi`], the `huggingface_hub` library provides [`HfFileSys
['datasets/my-username/my-dataset-repo/data/train.csv', 'datasets/my-username/my-dataset-repo/data/test.csv']

>>> # List all ".csv" files in a repo
>>> fs.glob("datasets/my-username/my-dataset-repo/**.csv")
>>> fs.glob("datasets/my-username/my-dataset-repo/**/*.csv")
['datasets/my-username/my-dataset-repo/data/train.csv', 'datasets/my-username/my-dataset-repo/data/test.csv']

>>> # Read a remote file
Expand Down
Loading

0 comments on commit c41266b

Please sign in to comment.