Skip to content

Commit

Permalink
Add Cargo package registry (#21888)
Browse files Browse the repository at this point in the history
This PR implements a [Cargo registry](https://doc.rust-lang.org/cargo/)
to manage Rust packages. This package type was a little bit more
complicated because Cargo needs an additional Git repository to store
its package index.

Screenshots:

![grafik](https://user-images.githubusercontent.com/1666336/203102004-08d812ac-c066-4969-9bda-2fed818554eb.png)

![grafik](https://user-images.githubusercontent.com/1666336/203102141-d9970f14-dca6-4174-b17a-50ba1bd79087.png)

![grafik](https://user-images.githubusercontent.com/1666336/203102244-dc05743b-78b6-4d97-998e-ef76341a978f.png)

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
  • Loading branch information
KN4CK3R and lunny authored Feb 5, 2023
1 parent 7baeb9c commit df789d9
Show file tree
Hide file tree
Showing 35 changed files with 1,660 additions and 125 deletions.
2 changes: 2 additions & 0 deletions custom/conf/app.example.ini
Original file line number Diff line number Diff line change
Expand Up @@ -2458,6 +2458,8 @@ ROUTER = console
;LIMIT_TOTAL_OWNER_COUNT = -1
;; Maximum size of packages a single owner can use (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
;LIMIT_TOTAL_OWNER_SIZE = -1
;; Maximum size of a Cargo upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
;LIMIT_SIZE_CARGO = -1
;; Maximum size of a Composer upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
;LIMIT_SIZE_COMPOSER = -1
;; Maximum size of a Conan upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
Expand Down
1 change: 1 addition & 0 deletions docs/content/doc/advanced/config-cheat-sheet.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -1213,6 +1213,7 @@ Task queue configuration has been moved to `queue.task`. However, the below conf
- `CHUNKED_UPLOAD_PATH`: **tmp/package-upload**: Path for chunked uploads. Defaults to `APP_DATA_PATH` + `tmp/package-upload`
- `LIMIT_TOTAL_OWNER_COUNT`: **-1**: Maximum count of package versions a single owner can have (`-1` means no limits)
- `LIMIT_TOTAL_OWNER_SIZE`: **-1**: Maximum size of packages a single owner can use (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_CARGO`: **-1**: Maximum size of a Cargo upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_COMPOSER`: **-1**: Maximum size of a Composer upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_CONAN`: **-1**: Maximum size of a Conan upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
- `LIMIT_SIZE_CONDA`: **-1**: Maximum size of a Conda upload (`-1` means no limits, format `1000`, `1 MB`, `1 GiB`)
Expand Down
109 changes: 109 additions & 0 deletions docs/content/doc/packages/cargo.en-us.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
date: "2022-11-20T00:00:00+00:00"
title: "Cargo Packages Repository"
slug: "packages/cargo"
draft: false
toc: false
menu:
sidebar:
parent: "packages"
name: "Cargo"
weight: 5
identifier: "cargo"
---

# Cargo Packages Repository

Publish [Cargo](https://doc.rust-lang.org/stable/cargo/) packages for your user or organization.

**Table of Contents**

{{< toc >}}

## Requirements

To work with the Cargo package registry, you need [Rust and Cargo](https://www.rust-lang.org/tools/install).

Cargo stores informations about the available packages in a package index stored in a git repository.
This repository is needed to work with the registry.
The following section describes how to create it.

## Index Repository

Cargo stores informations about the available packages in a package index stored in a git repository.
In Gitea this repository has the special name `_cargo-index`.
After a package was uploaded, its metadata is automatically written to the index.
The content of this repository should not be manually modified.

The user or organization package settings page allows to create the index repository along with the configuration file.
If needed this action will rewrite the configuration file.
This can be useful if for example the Gitea instance domain was changed.

If the case arises where the packages stored in Gitea and the information in the index repository are out of sync, the settings page allows to rebuild the index repository.
This action iterates all packages in the registry and writes their information to the index.
If there are lot of packages this process may take some time.

## Configuring the package registry

To register the package registry the Cargo configuration must be updated.
Add the following text to the configuration file located in the current users home directory (for example `~/.cargo/config.toml`):

```
[registry]
default = "gitea"
[registries.gitea]
index = "https://gitea.example.com/{owner}/_cargo-index.git"
[net]
git-fetch-with-cli = true
```

| Parameter | Description |
| --------- | ----------- |
| `owner` | The owner of the package. |

If the registry is private or you want to publish new packages, you have to configure your credentials.
Add the credentials section to the credentials file located in the current users home directory (for example `~/.cargo/credentials.toml`):

```
[registries.gitea]
token = "Bearer {token}"
```

| Parameter | Description |
| --------- | ----------- |
| `token` | Your [personal access token]({{< relref "doc/developers/api-usage.en-us.md#authentication" >}}) |

## Publish a package

Publish a package by running the following command in your project:

```shell
cargo publish
```

You cannot publish a package if a package of the same name and version already exists. You must delete the existing package first.

## Install a package

To install a package from the package registry, execute the following command:

```shell
cargo add {package_name}
```

| Parameter | Description |
| -------------- | ----------- |
| `package_name` | The package name. |

## Supported commands

```
cargo publish
cargo add
cargo install
cargo yank
cargo unyank
cargo search
```
1 change: 1 addition & 0 deletions docs/content/doc/packages/overview.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ The following package managers are currently supported:

| Name | Language | Package client |
| ---- | -------- | -------------- |
| [Cargo]({{< relref "doc/packages/cargo.en-us.md" >}}) | Rust | `cargo` |
| [Composer]({{< relref "doc/packages/composer.en-us.md" >}}) | PHP | `composer` |
| [Conan]({{< relref "doc/packages/conan.en-us.md" >}}) | C++ | `conan` |
| [Conda]({{< relref "doc/packages/conda.en-us.md" >}}) | - | `conda` |
Expand Down
3 changes: 3 additions & 0 deletions models/packages/descriptor.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
repo_model "code.gitea.io/gitea/models/repo"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/json"
"code.gitea.io/gitea/modules/packages/cargo"
"code.gitea.io/gitea/modules/packages/composer"
"code.gitea.io/gitea/modules/packages/conan"
"code.gitea.io/gitea/modules/packages/conda"
Expand Down Expand Up @@ -129,6 +130,8 @@ func GetPackageDescriptor(ctx context.Context, pv *PackageVersion) (*PackageDesc

var metadata interface{}
switch p.Type {
case TypeCargo:
metadata = &cargo.Metadata{}
case TypeComposer:
metadata = &composer.Metadata{}
case TypeConan:
Expand Down
6 changes: 6 additions & 0 deletions models/packages/package.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ type Type string

// List of supported packages
const (
TypeCargo Type = "cargo"
TypeComposer Type = "composer"
TypeConan Type = "conan"
TypeConda Type = "conda"
Expand All @@ -46,6 +47,7 @@ const (
)

var TypeList = []Type{
TypeCargo,
TypeComposer,
TypeConan,
TypeConda,
Expand All @@ -64,6 +66,8 @@ var TypeList = []Type{
// Name gets the name of the package type
func (pt Type) Name() string {
switch pt {
case TypeCargo:
return "Cargo"
case TypeComposer:
return "Composer"
case TypeConan:
Expand Down Expand Up @@ -97,6 +101,8 @@ func (pt Type) Name() string {
// SVGName gets the name of the package type svg image
func (pt Type) SVGName() string {
switch pt {
case TypeCargo:
return "gitea-cargo"
case TypeComposer:
return "gitea-composer"
case TypeConan:
Expand Down
6 changes: 6 additions & 0 deletions models/packages/package_property.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ func GetPropertiesByName(ctx context.Context, refType PropertyType, refID int64,
return pps, db.GetEngine(ctx).Where("ref_type = ? AND ref_id = ? AND name = ?", refType, refID, name).Find(&pps)
}

// UpdateProperty updates a property
func UpdateProperty(ctx context.Context, pp *PackageProperty) error {
_, err := db.GetEngine(ctx).ID(pp.ID).Update(pp)
return err
}

// DeleteAllProperties deletes all properties of a ref
func DeleteAllProperties(ctx context.Context, refType PropertyType, refID int64) error {
_, err := db.GetEngine(ctx).Where("ref_type = ? AND ref_id = ?", refType, refID).Delete(&PackageProperty{})
Expand Down
169 changes: 169 additions & 0 deletions modules/packages/cargo/parser.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT

package cargo

import (
"encoding/binary"
"errors"
"io"
"regexp"

"code.gitea.io/gitea/modules/json"
"code.gitea.io/gitea/modules/validation"

"github.com/hashicorp/go-version"
)

const PropertyYanked = "cargo.yanked"

var (
ErrInvalidName = errors.New("package name is invalid")
ErrInvalidVersion = errors.New("package version is invalid")
)

// Package represents a Cargo package
type Package struct {
Name string
Version string
Metadata *Metadata
Content io.Reader
ContentSize int64
}

// Metadata represents the metadata of a Cargo package
type Metadata struct {
Dependencies []*Dependency `json:"dependencies,omitempty"`
Features map[string][]string `json:"features,omitempty"`
Authors []string `json:"authors,omitempty"`
Description string `json:"description,omitempty"`
DocumentationURL string `json:"documentation_url,omitempty"`
ProjectURL string `json:"project_url,omitempty"`
Readme string `json:"readme,omitempty"`
Keywords []string `json:"keywords,omitempty"`
Categories []string `json:"categories,omitempty"`
License string `json:"license,omitempty"`
RepositoryURL string `json:"repository_url,omitempty"`
Links string `json:"links,omitempty"`
}

type Dependency struct {
Name string `json:"name"`
Req string `json:"req"`
Features []string `json:"features"`
Optional bool `json:"optional"`
DefaultFeatures bool `json:"default_features"`
Target *string `json:"target"`
Kind string `json:"kind"`
Registry *string `json:"registry"`
Package *string `json:"package"`
}

var nameMatch = regexp.MustCompile(`\A[a-zA-Z][a-zA-Z0-9-_]{0,63}\z`)

// ParsePackage reads the metadata and content of a package
func ParsePackage(r io.Reader) (*Package, error) {
var size uint32
if err := binary.Read(r, binary.LittleEndian, &size); err != nil {
return nil, err
}

p, err := parsePackage(io.LimitReader(r, int64(size)))
if err != nil {
return nil, err
}

if err := binary.Read(r, binary.LittleEndian, &size); err != nil {
return nil, err
}

p.Content = io.LimitReader(r, int64(size))
p.ContentSize = int64(size)

return p, nil
}

func parsePackage(r io.Reader) (*Package, error) {
var meta struct {
Name string `json:"name"`
Vers string `json:"vers"`
Deps []struct {
Name string `json:"name"`
VersionReq string `json:"version_req"`
Features []string `json:"features"`
Optional bool `json:"optional"`
DefaultFeatures bool `json:"default_features"`
Target *string `json:"target"`
Kind string `json:"kind"`
Registry *string `json:"registry"`
ExplicitNameInToml string `json:"explicit_name_in_toml"`
} `json:"deps"`
Features map[string][]string `json:"features"`
Authors []string `json:"authors"`
Description string `json:"description"`
Documentation string `json:"documentation"`
Homepage string `json:"homepage"`
Readme string `json:"readme"`
ReadmeFile string `json:"readme_file"`
Keywords []string `json:"keywords"`
Categories []string `json:"categories"`
License string `json:"license"`
LicenseFile string `json:"license_file"`
Repository string `json:"repository"`
Links string `json:"links"`
}
if err := json.NewDecoder(r).Decode(&meta); err != nil {
return nil, err
}

if !nameMatch.MatchString(meta.Name) {
return nil, ErrInvalidName
}

if _, err := version.NewSemver(meta.Vers); err != nil {
return nil, ErrInvalidVersion
}

if !validation.IsValidURL(meta.Homepage) {
meta.Homepage = ""
}
if !validation.IsValidURL(meta.Documentation) {
meta.Documentation = ""
}
if !validation.IsValidURL(meta.Repository) {
meta.Repository = ""
}

dependencies := make([]*Dependency, 0, len(meta.Deps))
for _, dep := range meta.Deps {
dependencies = append(dependencies, &Dependency{
Name: dep.Name,
Req: dep.VersionReq,
Features: dep.Features,
Optional: dep.Optional,
DefaultFeatures: dep.DefaultFeatures,
Target: dep.Target,
Kind: dep.Kind,
Registry: dep.Registry,
})
}

return &Package{
Name: meta.Name,
Version: meta.Vers,
Metadata: &Metadata{
Dependencies: dependencies,
Features: meta.Features,
Authors: meta.Authors,
Description: meta.Description,
DocumentationURL: meta.Documentation,
ProjectURL: meta.Homepage,
Readme: meta.Readme,
Keywords: meta.Keywords,
Categories: meta.Categories,
License: meta.License,
RepositoryURL: meta.Repository,
Links: meta.Links,
},
}, nil
}
Loading

0 comments on commit df789d9

Please sign in to comment.