Fixes #7292 - API File Contents bug #7301

richmahn · 2019-06-26T10:12:36Z

A few of bugs fixed with the File Content API response, one from issue #7292, but also the COMPLETE functionality that GitHub has for the GET contents API endpoint. Here are bug fixes:

BUG: html_url field had blob/<ref>/<file> in the URL. This is how GitHub does it, but Gitea does src/<ref type>/<ref>/<file>, so now the code determines the ref type give and gives the proper html_url (as noted in API Repo Get File Content Endpoint returns non-working "blob" HTML_URL #7292)
FIX: The content of the file in base64 encoding was not being returned as GitHub does. Added the encoding and content field to the response (see GitHub's file content response: https://developer.github.com/v3/repos/contents/#response-if-content-is-a-file)
FIX: Before the Get File Contents API endpoint only took a branch as a ref, when it should take a tag and commit ID as well. This is now added and ?ref=[branch|tag|commit] will work, and the URLs given in the response will properly preserve this.
FIX: Before it only worked with files as the path in the URL. Now works for directories (returns an array of FileContentsResponse objects as the response), symlinks (will give the target of the symlink in the response), and submodules (will give the submodules's git URL in the response). The symlink target, the submodule git URL, and file contents & encoding are null if they do not apply to the type of entry. Also files return in a directory list of entries will not have their content and encoding (will be null) as that would take too long to return all file contents.

Again, please see how this is closer to what GitHub does: https://developer.github.com/v3/repos/contents/#response-if-content-is-a-file, https://developer.github.com/v3/repos/contents/#response-if-content-is-a-directory, https://developer.github.com/v3/repos/contents/#response-if-content-is-a-symlink and https://developer.github.com/v3/repos/contents/#response-if-content-is-a-submodule
(only difference is that I leave the target:, submodule_git_url:, content: and encoding: properties still in the response as null even if not used as having a different response objects for each entry type would have been a headache and not work with swagger.)

Also note, these changes to this API endpoint shouldn't affect anyone as they haven't been released yet (see my PR for the original endpoint code: #6314 - milestone 1.9.0)

Below in another comment I give some examples. Compare with the old response: https://try.gitea.io/api/v1/repos/richmahn/test/contents/README

richmahn · 2019-06-26T10:23:05Z

Here are some example JSON Responses with the GET URL:

No ref: http://localhost:3000/api/v1/repos/root/test/contents/README.md

{
  "name": "README.md",
  "path": "README.md",
  "sha": "f3b60aa0df362906d8b64a16aa2eed6c3ec48906",
  "type": "file",
  "size": 28,
  "encoding": "base64",
  "content": "IyB0ZXN0CiMjIHRlc3QgdjIKCnRlc3QgZGVzYw==",
  "target": null,
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=master",
  "html_url": "http://localhost:3000/root/test/src/branch/master/README.md",
  "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f3b60aa0df362906d8b64a16aa2eed6c3ec48906",
  "download_url": "http://localhost:3000/root/test/raw/branch/master/README.md",
  "submodule_git_url": null,
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=master",
    "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f3b60aa0df362906d8b64a16aa2eed6c3ec48906",
    "html": "http://localhost:3000/root/test/src/branch/master/README.md"
  }
}

With branch: http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=develop

{
  "name": "README.md",
  "path": "README.md",
  "sha": "f3b60aa0df362906d8b64a16aa2eed6c3ec48906",
  "type": "file",
  "size": 28,
  "encoding": "base64",
  "content": "IyB0ZXN0CiMjIHRlc3QgdjIKCnRlc3QgZGVzYw==",
  "target": null,
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=develop",
  "html_url": "http://localhost:3000/root/test/src/branch/develop/README.md",
  "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f3b60aa0df362906d8b64a16aa2eed6c3ec48906",
  "download_url": "http://localhost:3000/root/test/raw/branch/develop/README.md",
  "submodule_git_url": null,
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=develop",
    "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f3b60aa0df362906d8b64a16aa2eed6c3ec48906",
    "html": "http://localhost:3000/root/test/src/branch/develop/README.md"
  }
}

Short commit ID: http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=b7b2

{
  "name": "README.md",
  "path": "README.md",
  "sha": "311fbf6da73cdd85bcc29f4f3809cd4dc2f5cb83",
  "type": "file",
  "size": 17,
  "encoding": "base64",
  "content": "IyB0ZXN0Cgp0ZXN0IGRlc2M=",
  "target": null,
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=b7b2",
  "html_url": "http://localhost:3000/root/test/src/commit/b7b2828f27a39a1967e7762b0d4799fe69343652/README.md",
  "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/311fbf6da73cdd85bcc29f4f3809cd4dc2f5cb83",
  "download_url": "http://localhost:3000/root/test/raw/commit/b7b2828f27a39a1967e7762b0d4799fe69343652/README.md",
  "submodule_git_url": null,
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=b7b2",
    "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/311fbf6da73cdd85bcc29f4f3809cd4dc2f5cb83",
    "html": "http://localhost:3000/root/test/src/commit/b7b2828f27a39a1967e7762b0d4799fe69343652/README.md"
  }
}

Long Commit: http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=b7b2828f27a39a1967e7762b0d4799fe69343652

{
  "name": "README.md",
  "path": "README.md",
  "sha": "311fbf6da73cdd85bcc29f4f3809cd4dc2f5cb83",
  "type": "file",
  "size": 17,
  "encoding": "base64",
  "content": "IyB0ZXN0Cgp0ZXN0IGRlc2M=",
  "target": null,
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=b7b2828f27a39a1967e7762b0d4799fe69343652",
  "html_url": "http://localhost:3000/root/test/src/commit/b7b2828f27a39a1967e7762b0d4799fe69343652/README.md",
  "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/311fbf6da73cdd85bcc29f4f3809cd4dc2f5cb83",
  "download_url": "http://localhost:3000/root/test/raw/commit/b7b2828f27a39a1967e7762b0d4799fe69343652/README.md",
  "submodule_git_url": null,
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=b7b2828f27a39a1967e7762b0d4799fe69343652",
    "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/311fbf6da73cdd85bcc29f4f3809cd4dc2f5cb83",
    "html": "http://localhost:3000/root/test/src/commit/b7b2828f27a39a1967e7762b0d4799fe69343652/README.md"
  }
}

Tag: http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=v1

{
  "name": "README.md",
  "path": "README.md",
  "sha": "f03dd716384d52b861c36f88521576870a8cd9fc",
  "type": "file",
  "size": 25,
  "encoding": "base64",
  "content": "IyB0ZXN0CiMjIHRlc3QKCnRlc3QgZGVzYw==",
  "target": null,
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=v1",
  "html_url": "http://localhost:3000/root/test/src/tag/v1/README.md",
  "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f03dd716384d52b861c36f88521576870a8cd9fc",
  "download_url": "http://localhost:3000/root/test/raw/tag/v1/README.md",
  "submodule_git_url": null,
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/README.md?ref=v1",
    "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f03dd716384d52b861c36f88521576870a8cd9fc",
    "html": "http://localhost:3000/root/test/src/tag/v1/README.md"
  }
}

A directory listing using the {dirpath} GET API Endpoint: http://localhost:3000/api/v1/repos/root/test/contents/test3

[
  {
    "name": "dir1",
    "path": "test3/dir1",
    "sha": "f05af273ba36fe5176e5eaab349661a56b3d27a0",
    "type": "dir",
    "size": 0,
    "encoding": null,
    "content": null,
    "target": null,
    "url": "http://localhost:3000/api/v1/repos/root/test/contents/test3/dir1?ref=master",
    "html_url": "http://localhost:3000/root/test/src/branch/master/test3/dir1",
    "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f05af273ba36fe5176e5eaab349661a56b3d27a0",
    "download_url": null,
    "submodule_git_url": null,
    "_links": {
      "self": "http://localhost:3000/api/v1/repos/root/test/contents/test3/dir1?ref=master",
      "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f05af273ba36fe5176e5eaab349661a56b3d27a0",
      "html": "http://localhost:3000/root/test/src/branch/master/test3/dir1"
    }
  },
  {
    "name": "test1lnk.txt",
    "path": "test3/test1lnk.txt",
    "sha": "f075d99da92271d466b7447eb9d5c7d5aef2b59a",
    "type": "symlink",
    "size": 18,
    "encoding": null,
    "content": null,
    "target": "../test2/test1.txt",
    "url": "http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk.txt?ref=master",
    "html_url": "http://localhost:3000/root/test/src/branch/master/test3/test1lnk.txt",
    "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f075d99da92271d466b7447eb9d5c7d5aef2b59a",
    "download_url": "http://localhost:3000/root/test/raw/branch/master/test3/test1lnk.txt",
    "submodule_git_url": null,
    "_links": {
      "self": "http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk.txt?ref=master",
      "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f075d99da92271d466b7447eb9d5c7d5aef2b59a",
      "html": "http://localhost:3000/root/test/src/branch/master/test3/test1lnk.txt"
    }
  },
  {
    "name": "test1lnk2.txt",
    "path": "test3/test1lnk2.txt",
    "sha": "6493b67ba9a37b7b93dd1e3f26321da3ee63ddff",
    "type": "file",
    "size": 6,
    "encoding": null,
    "content": null,
    "target": null,
    "url": "http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk2.txt?ref=master",
    "html_url": "http://localhost:3000/root/test/src/branch/master/test3/test1lnk2.txt",
    "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/6493b67ba9a37b7b93dd1e3f26321da3ee63ddff",
    "download_url": "http://localhost:3000/root/test/raw/branch/master/test3/test1lnk2.txt",
    "submodule_git_url": null,
    "_links": {
      "self": "http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk2.txt?ref=master",
      "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/6493b67ba9a37b7b93dd1e3f26321da3ee63ddff",
      "html": "http://localhost:3000/root/test/src/branch/master/test3/test1lnk2.txt"
    }
  }
]

Entry is a symlink: http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk.txt?ref=master

{
  "name": "test1lnk.txt",
  "path": "test3/test1lnk.txt",
  "sha": "f075d99da92271d466b7447eb9d5c7d5aef2b59a",
  "type": "symlink",
  "size": 18,
  "encoding": null,
  "content": null,
  "target": "../test2/test1.txt",
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk.txt?ref=master",
  "html_url": "http://localhost:3000/root/test/src/branch/master/test3/test1lnk.txt",
  "git_url": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f075d99da92271d466b7447eb9d5c7d5aef2b59a",
  "download_url": "http://localhost:3000/root/test/raw/branch/master/test3/test1lnk.txt",
  "submodule_git_url": null,
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/test3/test1lnk.txt?ref=master",
    "git": "http://localhost:3000/api/v1/repos/root/test/git/blobs/f075d99da92271d466b7447eb9d5c7d5aef2b59a",
    "html": "http://localhost:3000/root/test/src/branch/master/test3/test1lnk.txt"
  }
}

Entry is a submodule: http://localhost:3000/api/v1/repos/root/test/contents/lib/mymodule?ref=master

{
  "name": "mymodule",
  "path": "lib/mymodule",
  "sha": "10911ad59caa64736e30cc1ac310b9a3c1c992c6",
  "type": "submodule",
  "size": 0,
  "encoding": null,
  "content": null,
  "target": null,
  "url": "http://localhost:3000/api/v1/repos/root/test/contents/lib/mymodule?ref=master",
  "html_url": null,
  "git_url": null,
  "download_url": null,
  "submodule_git_url": "git@github.com:richmahn/test2",
  "_links": {
    "self": "http://localhost:3000/api/v1/repos/root/test/contents/lib/mymodule?ref=master",
    "git": null,
    "html": null
  }
}

lunny · 2019-06-26T11:19:30Z

modules/git/repo.go

+}
+
+// GetRefType gets the type of the ref based on the string
+func (repo *Repository) GetRefType(ref string) RepoRefType {


Could we return ObjectType on https://github.com/go-gitea/gitea/blob/master/modules/git/repo_object.go#L14 ?

Looking into it. I actually realized I did not have this at all responding like GitHub, especially when it comes to being 4 different types of objects (dir, file, symlink, submodule) that could be returned, as well well as a list of entries. So this is being redone.

Added ObjectBranch and now use those as a way to get the string of a ref. Also moved my GetRefType() into this file.

…d submodules

…-content-bug

codecov-io · 2019-06-28T02:17:56Z

Codecov Report

❗ No coverage uploaded for pull request base (master@738285a). Click here to learn what that means.
The diff coverage is 64.02%.

@@            Coverage Diff            @@
##             master    #7301   +/-   ##
=========================================
  Coverage          ?   41.25%           
=========================================
  Files             ?      464           
  Lines             ?    63107           
  Branches          ?        0           
=========================================
  Hits              ?    26035           
  Misses            ?    33666           
  Partials          ?     3406

Impacted Files	Coverage Δ
modules/git/blob.go	`52.5% <0%> (ø)`
routers/api/v1/api.go	`71.36% <100%> (ø)`
modules/repofiles/file.go	`83.33% <100%> (ø)`
modules/repofiles/content.go	`61.03% <57.93%> (ø)`
modules/git/repo_object.go	`25.92% <63.63%> (ø)`
routers/api/v1/repo/file.go	`71.31% <97.36%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 738285a...23498be. Read the comment docs.

…-content-bug

Fixes go-gitea#7292 - API File Contents bug

3dfafaf

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 26, 2019

richmahn added 3 commits June 26, 2019 06:31

Fix to linting

c66f026

Fixes lint issues

74564a2

Fixes integreation test

989f4fb

lunny reviewed Jun 26, 2019

View reviewed changes

lunny added the type/bug label Jun 26, 2019

lunny added this to the 1.9.0 milestone Jun 26, 2019

richmahn added 10 commits June 27, 2019 17:10

API contents endpoint full functionality for directories, symlinks an…

f676db5

…d submodules

API contents endpoint full functionality for directories, symlinks an…

35dd33d

…d submodules

Lint fixes

ac16348

Fixes to content and swagger

6dc209b

Fixes for swagger

459efd4

Fixes to content and swagger

c941709

Swagger update

ae0b30b

Swagger fix

4078622

Fix to test

b436486

Merge remote-tracking branch 'upstream/master' into fix-7292-api-file…

5931946

…-content-bug

richmahn added 9 commits June 28, 2019 05:16

ContentsResponse struct update: better name and some fields can be nil

e0d98b0

ContentsResponse struct update: better name and some fields can be nil

cc4212b

Update comments

ff2b1c4

Fixes to integration tests

c3e496d

Swagger update

4c8b11d

Merge remote-tracking branch 'upstream/master' into fix-7292-api-file…

19ac334

…-content-bug

Fixes to integration tests

0d65643

Adds comment

3d2c6ad

Adds comment

dd963a1

zeripath approved these changes Jun 28, 2019

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jun 28, 2019

Merge remote-tracking branch 'upstream/master' into fix-7292-api-file…

23498be

…-content-bug

lafriks approved these changes Jun 29, 2019

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jun 29, 2019

techknowlogick merged commit cd96dee into go-gitea:master Jun 29, 2019

jeffliu27 pushed a commit to jeffliu27/gitea that referenced this pull request Jul 18, 2019

Fixes go-gitea#7292 - API File Contents bug (go-gitea#7301)

a15d9e7

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #7292 - API File Contents bug #7301

Fixes #7292 - API File Contents bug #7301

richmahn commented Jun 26, 2019 •

edited

Loading

richmahn commented Jun 26, 2019 •

edited

Loading

lunny Jun 26, 2019

richmahn Jun 27, 2019

richmahn Jun 27, 2019

codecov-io commented Jun 28, 2019 •

edited

Loading

Fixes #7292 - API File Contents bug #7301

Fixes #7292 - API File Contents bug #7301

Conversation

richmahn commented Jun 26, 2019 • edited Loading

richmahn commented Jun 26, 2019 • edited Loading

lunny Jun 26, 2019

Choose a reason for hiding this comment

richmahn Jun 27, 2019

Choose a reason for hiding this comment

richmahn Jun 27, 2019

Choose a reason for hiding this comment

codecov-io commented Jun 28, 2019 • edited Loading

Codecov Report

richmahn commented Jun 26, 2019 •

edited

Loading

richmahn commented Jun 26, 2019 •

edited

Loading

codecov-io commented Jun 28, 2019 •

edited

Loading