Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cargo package registry #21888

Merged
merged 25 commits into from
Feb 5, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
5f5f1bb
Add Cargo registry.
KN4CK3R Nov 21, 2022
0db12a7
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Nov 21, 2022
9da9d58
Add setting entry.
KN4CK3R Nov 21, 2022
ef668a7
Resolve import cycle.
KN4CK3R Nov 22, 2022
a805e10
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Nov 22, 2022
f216281
Fix test.
KN4CK3R Nov 22, 2022
0405d2a
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Dec 8, 2022
16a3128
Adapt to merged methods.
KN4CK3R Dec 8, 2022
6b46a48
lint
KN4CK3R Dec 9, 2022
c53718b
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Dec 9, 2022
7085cdf
Merge branch 'main' into feature-cargo
KN4CK3R Dec 14, 2022
8b287db
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Dec 19, 2022
7805de6
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jan 14, 2023
2cde9ac
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jan 15, 2023
f9326c6
Check content size.
KN4CK3R Jan 15, 2023
2c9a04f
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jan 25, 2023
b5fcf44
Fix dependency json format.
KN4CK3R Jan 25, 2023
d391bdf
Use pointers.
KN4CK3R Jan 26, 2023
ecf5378
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jan 26, 2023
d026426
Force non-nil values in json.
KN4CK3R Jan 26, 2023
cc5848b
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Jan 26, 2023
c43d84e
Field is allowed to be missing.
KN4CK3R Jan 26, 2023
51cd55d
Merge branch 'main' into feature-cargo
lunny Jan 28, 2023
1f65da1
Merge branch 'main' of https://github.com/go-gitea/gitea into feature…
KN4CK3R Feb 4, 2023
002d86b
Merge branch 'main' into feature-cargo
lunny Feb 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions custom/conf/app.example.ini
Original file line number Diff line number Diff line change
Expand Up @@ -2379,6 +2379,8 @@ ROUTER = console
;LIMIT_TOTAL_OWNER_COUNT = -1
;; Maxmimum size of packages a single owner can use (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
;LIMIT_TOTAL_OWNER_SIZE = -1
;; Maxmimum size of a Cargo upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
;LIMIT_SIZE_CARGO = -1
;; Maxmimum size of a Composer upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
;LIMIT_SIZE_COMPOSER = -1
;; Maxmimum size of a Conan upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
Expand Down
1 change: 1 addition & 0 deletions docs/content/doc/advanced/config-cheat-sheet.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -1180,6 +1180,7 @@ Task queue configuration has been moved to `queue.task`. However, the below conf
- `CHUNKED_UPLOAD_PATH`: **tmp/package-upload**: Path for chunked uploads. Defaults to `APP_DATA_PATH` + `tmp/package-upload`
- `LIMIT_TOTAL_OWNER_COUNT`: **-1**: Maxmimum count of package versions a single owner can have (`-1` means no limits)
- `LIMIT_TOTAL_OWNER_SIZE`: **-1**: Maxmimum size of packages a single owner can use (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_CARGO`: **-1**: Maxmimum size of a Cargo upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_COMPOSER`: **-1**: Maxmimum size of a Composer upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_CONAN`: **-1**: Maxmimum size of a Conan upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_CONTAINER`: **-1**: Maxmimum size of a Container upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
Expand Down
109 changes: 109 additions & 0 deletions docs/content/doc/packages/cargo.en-us.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
date: "2022-11-20T00:00:00+00:00"
title: "Cargo Packages Repository"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slug: "packages/cargo"
draft: false
toc: false
menu:
sidebar:
parent: "packages"
name: "Cargo"
weight: 5
identifier: "cargo"
---

# Cargo Packages Repository

Publish [Cargo](https://doc.rust-lang.org/stable/cargo/) packages for your user or organization.

**Table of Contents**

{{< toc >}}

## Requirements

To work with the Cargo package registry, you need [Rust and Cargo](https://www.rust-lang.org/tools/install).

Cargo stores informations about the available packages in a package index stored in a git repository.
This repository is needed to work with the registry.
The following section describes how to create it.

## Index Repository

Cargo stores informations about the available packages in a package index stored in a git repository.
In Gitea this repository has the special name `_cargo-index`.
After a package was uploaded, its metadata is automatically written to the index.
The content of this repository should not be manually modified.

The user or organization package settings page allows to create the index repository along with the configuration file.
If needed this action will rewrite the configuration file.
This can be useful if for example the Gitea instance domain was changed.

If the case arises where the packages stored in Gitea and the information in the index repository are out of sync, the settings page allows to rebuild the index repository.
This action iterates all packages in the registry and writes their information to the index.
If there are lot of packages this process may take some time.

## Configuring the package registry

To register the package registry the Cargo configuration must be updated.
Add the following text to the configuration file located in the current users home directory (for example `~/.cargo/config.toml`):

```
[registry]
default = "gitea"

[registries.gitea]
index = "https://gitea.example.com/{owner}/_cargo-index.git"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name maybe conflicted with exist repository.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have chosen the underscore to signal that this is something special, managed. I'm uncertain how to call it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... Does this repository ever have to accept pushes? Would it make sense for a user to ever want to push to this repository? Does the repo need to have history?

If no pushes we could do something like mount it on {owner}/packages/cargo.git and only allow read access externally.

If there's no need for history at all we can do something even cleverer like actually just emulate git.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cargo client does a pull on that repo to get the latest updates. There is no need for a user to push to that repo. I thought about new routes too but decided otherwise because lots of our code wants to see a repo model and there would be none.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No push, no UI view, http pull only(no ssh) and if the contents of the index file are human-readable? If it's machine-readable, it's no necessary to display them in UI.

Copy link
Member

@lunny lunny Dec 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if one owner have many cargo packages, will all the indexes be stored in the {owner}/_cargo-index.git repository? Looks like Github have no _cargo-index.git repository for packages owner.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if one owner have many cargo packages, will all the indexes be stored in the {owner}/_cargo-index.git repository?

Yes, it's an index for all packages available in the registry. Here is the official crates.io index: https://github.com/rust-lang/crates.io-index

Looks like Github have no _cargo-index.git repository for packages owner.

Yep, the reason is Github does not support Cargo packages. 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/rust-lang/crates.io-index

Looks like every index file is a JSON text file and that repository also contains a workflow file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they are build in index.go.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zeripath Do you see a realistic chance to extract the git routes to work without the need for a valid models.Repository reference? Another question is where the files for the index repo should be stored.


[net]
git-fetch-with-cli = true
```

| Parameter | Description |
| --------- | ----------- |
| `owner` | The owner of the package. |

If the registry is private or you want to publish new packages, you have to configure your credentials.
Add the credentials section to the credentials file located in the current users home directory (for example `~/.cargo/credentials.toml`):

```
[registries.gitea]
token = "Bearer {token}"
```

| Parameter | Description |
| --------- | ----------- |
| `token` | Your [personal access token]({{< relref "doc/developers/api-usage.en-us.md#authentication" >}}) |

## Publish a package

Publish a package by running the following command in your project:

```shell
cargo publish
```

You cannot publish a package if a package of the same name and version already exists. You must delete the existing package first.

## Install a package

To install a package from the package registry, execute the following command:

```shell
cargo add {package_name}
```

| Parameter | Description |
| -------------- | ----------- |
| `package_name` | The package name. |

## Supported commands

```
cargo publish
cargo add
cargo install
cargo yank
cargo unyank
cargo search
```
1 change: 1 addition & 0 deletions docs/content/doc/packages/overview.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ The following package managers are currently supported:

| Name | Language | Package client |
| ---- | -------- | -------------- |
| [Cargo]({{< relref "doc/packages/cargo.en-us.md" >}}) | Rust | `cargo` |
| [Composer]({{< relref "doc/packages/composer.en-us.md" >}}) | PHP | `composer` |
| [Conan]({{< relref "doc/packages/conan.en-us.md" >}}) | C++ | `conan` |
| [Container]({{< relref "doc/packages/container.en-us.md" >}}) | - | any OCI compliant client |
Expand Down
3 changes: 3 additions & 0 deletions models/packages/descriptor.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
repo_model "code.gitea.io/gitea/models/repo"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/json"
"code.gitea.io/gitea/modules/packages/cargo"
"code.gitea.io/gitea/modules/packages/composer"
"code.gitea.io/gitea/modules/packages/conan"
"code.gitea.io/gitea/modules/packages/container"
Expand Down Expand Up @@ -129,6 +130,8 @@ func GetPackageDescriptor(ctx context.Context, pv *PackageVersion) (*PackageDesc

var metadata interface{}
switch p.Type {
case TypeCargo:
metadata = &cargo.Metadata{}
case TypeComposer:
metadata = &composer.Metadata{}
case TypeConan:
Expand Down
6 changes: 6 additions & 0 deletions models/packages/package.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ type Type string

// List of supported packages
const (
TypeCargo Type = "cargo"
TypeComposer Type = "composer"
TypeConan Type = "conan"
TypeContainer Type = "container"
Expand All @@ -46,6 +47,7 @@ const (
)

var TypeList = []Type{
TypeCargo,
TypeComposer,
TypeConan,
TypeContainer,
Expand All @@ -63,6 +65,8 @@ var TypeList = []Type{
// Name gets the name of the package type
func (pt Type) Name() string {
switch pt {
case TypeCargo:
return "Cargo"
case TypeComposer:
return "Composer"
case TypeConan:
Expand Down Expand Up @@ -94,6 +98,8 @@ func (pt Type) Name() string {
// SVGName gets the name of the package type svg image
func (pt Type) SVGName() string {
switch pt {
case TypeCargo:
return "gitea-cargo"
case TypeComposer:
return "gitea-composer"
case TypeConan:
Expand Down
6 changes: 6 additions & 0 deletions models/packages/package_property.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@ func GetPropertiesByName(ctx context.Context, refType PropertyType, refID int64,
return pps, db.GetEngine(ctx).Where("ref_type = ? AND ref_id = ? AND name = ?", refType, refID, name).Find(&pps)
}

// UpdateProperty updates a property
func UpdateProperty(ctx context.Context, pp *PackageProperty) error {
_, err := db.GetEngine(ctx).ID(pp.ID).Update(pp)
return err
}

// DeleteAllProperties deletes all properties of a ref
func DeleteAllProperties(ctx context.Context, refType PropertyType, refID int64) error {
_, err := db.GetEngine(ctx).Where("ref_type = ? AND ref_id = ?", refType, refID).Delete(&PackageProperty{})
Expand Down
144 changes: 144 additions & 0 deletions modules/packages/cargo/parser.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package cargo

import (
"encoding/binary"
"errors"
"io"
"regexp"

"code.gitea.io/gitea/modules/json"
"code.gitea.io/gitea/modules/validation"

"github.com/hashicorp/go-version"
)

const PropertyYanked = "cargo.yanked"

var (
ErrInvalidName = errors.New("package name is invalid")
ErrInvalidVersion = errors.New("package version is invalid")
)

// Package represents a Cargo package
type Package struct {
Name string
Version string
Metadata *Metadata
Content io.Reader
}

// Metadata represents the metadata of a Cargo package
type Metadata struct {
Dependencies []*Dependency `json:"dependencies,omitempty"`
Features map[string][]string `json:"features,omitempty"`
Authors []string `json:"authors,omitempty"`
Description string `json:"description,omitempty"`
DocumentationURL string `json:"documentation_url,omitempty"`
ProjectURL string `json:"project_url,omitempty"`
Readme string `json:"readme,omitempty"`
Keywords []string `json:"keywords,omitempty"`
Categories []string `json:"categories,omitempty"`
License string `json:"license,omitempty"`
RepositoryURL string `json:"repository_url,omitempty"`
Links string `json:"links,omitempty"`
}

type Dependency struct {
Name string `json:"name"`
VersionReq string `json:"version_req"`
Features []string `json:"features"`
Optional bool `json:"optional"`
DefaultFeatures bool `json:"default_features"`
Target string `json:"target"`
Kind string `json:"kind"`
Registry string `json:"registry"`
ExplicitNameInToml string `json:"explicit_name_in_toml"`
}

var nameMatch = regexp.MustCompile(`\A[a-zA-Z][a-zA-Z0-9-_]{0,63}\z`)

// ParsePackage reads the metadata and content of a package
func ParsePackage(r io.Reader) (*Package, error) {
var size uint32
if err := binary.Read(r, binary.LittleEndian, &size); err != nil {
return nil, err
}

p, err := parsePackage(io.LimitReader(r, int64(size)))
if err != nil {
return nil, err
}

if err := binary.Read(r, binary.LittleEndian, &size); err != nil {
return nil, err
}

p.Content = io.LimitReader(r, int64(size))
KN4CK3R marked this conversation as resolved.
Show resolved Hide resolved

return p, nil
}

func parsePackage(r io.Reader) (*Package, error) {
var meta struct {
Name string `json:"name"`
Vers string `json:"vers"`
Deps []*Dependency `json:"deps"`
Features map[string][]string `json:"features"`
Authors []string `json:"authors"`
Description string `json:"description"`
Documentation string `json:"documentation"`
Homepage string `json:"homepage"`
Readme string `json:"readme"`
ReadmeFile string `json:"readme_file"`
Keywords []string `json:"keywords"`
Categories []string `json:"categories"`
License string `json:"license"`
LicenseFile string `json:"license_file"`
Repository string `json:"repository"`
Links string `json:"links"`
}
if err := json.NewDecoder(r).Decode(&meta); err != nil {
return nil, err
}

if !nameMatch.MatchString(meta.Name) {
return nil, ErrInvalidName
}

if _, err := version.NewSemver(meta.Vers); err != nil {
return nil, ErrInvalidVersion
}

if !validation.IsValidURL(meta.Homepage) {
meta.Homepage = ""
}
if !validation.IsValidURL(meta.Documentation) {
meta.Documentation = ""
}
if !validation.IsValidURL(meta.Repository) {
meta.Repository = ""
}

return &Package{
Name: meta.Name,
Version: meta.Vers,
Metadata: &Metadata{
Dependencies: meta.Deps,
Features: meta.Features,
Authors: meta.Authors,
Description: meta.Description,
DocumentationURL: meta.Documentation,
ProjectURL: meta.Homepage,
Readme: meta.Readme,
Keywords: meta.Keywords,
Categories: meta.Categories,
License: meta.License,
RepositoryURL: meta.Repository,
Links: meta.Links,
},
}, nil
}
Loading