Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't update Data when body is empty #125

Merged
merged 1 commit into from
Jun 18, 2023
Merged

fix: don't update Data when body is empty #125

merged 1 commit into from
Jun 18, 2023

Conversation

llimllib
Copy link
Contributor

@llimllib llimllib commented Jun 18, 2023

When a request to /anything has an empty body, the data field of the output should be the empty string. Currently, go-httpbin is returning data:application/octet-stream;base64,`.

  • Should the output actually be null? That would match files, form, and json, but also would require deeper changes since Data is stored as a string not a pointer, and I didn't want to mess with things any more than I had to. An empty string works fine for my purposes
  • how can I test this PR? I tried but I got confused by the TestAnything method
  • all the current tests pass under this change

This PR closes #124. Before:

$ curl -s 'http://0.0.0.0:8080/anything?one=two'
{
  "args": {
    "one": [
      "two"
    ]
  },
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Host": [
      "0.0.0.0:8080"
    ],
    "User-Agent": [
      "curl/7.88.1"
    ]
  },
  "method": "GET",
  "origin": "127.0.0.1:60402",
  "url": "http://0.0.0.0:8080/anything?one=two",
  "data": "data:application/octet-stream;base64,",
  "files": null,
  "form": null,
  "json": null
}

After:

$ curl -s 'http://0.0.0.0:8080/anything?one=two'
{
  "args": {
    "one": [
      "two"
    ]
  },
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Host": [
      "0.0.0.0:8080"
    ],
    "User-Agent": [
      "curl/7.88.1"
    ]
  },
  "method": "GET",
  "origin": "127.0.0.1:60595",
  "url": "http://0.0.0.0:8080/anything?one=two",
  "data": "",
  "files": null,
  "form": null,
  "json": null
}

@mccutchen mccutchen added the compat Tracking issues around compatibility with original httpbin implementation label Jun 18, 2023
@codecov
Copy link

codecov bot commented Jun 18, 2023

Codecov Report

Merging #125 (2cec1a1) into main (7e81a93) will decrease coverage by 0.21%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##             main     #125      +/-   ##
==========================================
- Coverage   98.41%   98.20%   -0.21%     
==========================================
  Files           8        8              
  Lines        1448     1450       +2     
==========================================
- Hits         1425     1424       -1     
- Misses         17       19       +2     
- Partials        6        7       +1     
Impacted Files Coverage Δ
httpbin/helpers.go 96.26% <25.00%> (-1.23%) ⬇️

@mccutchen
Copy link
Owner

+1, I agree that this change makes sense.

  • Should the output actually be null? That would match files, form, and json, but also would require deeper changes since Data is stored as a string not a pointer, and I didn't want to mess with things any more than I had to. An empty string works fine for my purposes

That's a good question, let me think about it a little bit this morning. My first instinct is that an empty string is a reasonable enough compromise that, even if null would be more consistent with other optional elements of these responses, it's probably not worth the effort it would take to coerce the golang type system & json serializer into handling this for us.

I might take a quick pass locally just to see how feasible something like a type OptionalData string w/ custom marshal/unmarshal methods might be, but I also feel like every time I try to do this a) it takes me forever to figure out and b) it doesn't end up working quite the way I want it to.

  • how can I test this PR? I tried but I got confused by the TestAnything method

Also a good question, and I'm sorry for any confusion! I know the test suite is a bit of a mess, and I have no idea off the top of my head how best to test this, but lemme take a look.

@mccutchen
Copy link
Owner

That's a good question, let me think about it a little bit this morning. My first instinct is that an empty string is a reasonable enough compromise that, even if null would be more consistent with other optional elements of these responses, it's probably not worth the effort it would take to coerce the golang type system & json serializer into handling this for us.

Okay, took a quick pass at this, and I don't think it's worth all of the additional overhead that would be required to marshal empty bodies as null in the response.

The custom type approach feels like a whole lot of complexity (including a particularly heavyweight custom MarshalJSON method that would have to allocate an extra temporary buffer in order to use a json.Encoder in order to disable HTML escaping in order to maintain compatibility with this other compatibility-related change), and for now I don't particularly want to get into the business of working with string pointers everywhere.

@mccutchen
Copy link
Owner

  • how can I test this PR? I tried but I got confused by the TestAnything method

Also a good question, and I'm sorry for any confusion! I know the test suite is a bit of a mess, and I have no idea off the top of my head how best to test this, but lemme take a look.

Oh boy, I just spent wayyyy too long trying to figure out why the existing test cases for requests with empty bodies were showing a different behavior from the curl test (either the tests should have failed, or the curl should have returned an empty string for the "data" field).

Turns out it's down to a couple of things:

  • Most/all of the tests in the test suite use httptest.NewRecorder and just call the ServeHTTP method directly, rather than spinning up a live HTTP server using httptest.NewServer and making an actual HTTP request, like so:
    r, _ := http.NewRequest(verb, path, nil)
    r.Header.Set("Content-Type", test.contentType)
    w := httptest.NewRecorder()
    app.ServeHTTP(w, r)
  • After adding some logging, it became clear that this conditional is true when using the test recorder and false when making a real HTTP request:
    func parseBody(w http.ResponseWriter, r *http.Request, resp *bodyResponse) error {
    if r.Body == nil {
    return nil
    }

So, weirdly, we already have unit tests in place that describe the behavior we're actually implementing in this pull request, they just weren't testing the actual live behavior in the face of an actual HTTP request.

My inclination is to land this change as-is and then I'll follow up with an update to the test suite to stop using the recorder approach. I've considered that a number of times in the past for vague notions of correctness, but this is the first time I can remember running into an actual behavior difference that mattered quite so much.

@mccutchen mccutchen merged commit a70a847 into mccutchen:main Jun 18, 2023
@mccutchen
Copy link
Owner

FYI, this was released in v2.9.0

@llimllib
Copy link
Contributor Author

Thanks so much! I want to add that I've found this project really useful, thanks for all the work

mccutchen added a commit that referenced this pull request Jun 28, 2023
In the course of validating #125, we discovered that using the stdlib's
[`httptest.ResponseRecorder`][0] mechanism to drive the vast majority of
our unit tests led to some slight brokenness due to subtle differences
in the way those "simulated" requests are handled vs "real" requests to
a live HTTP server, as [explained in this comment][1].

That prompted me to do a big ass refactor of the entire test suite,
swapping httptest.ResponseRecorder for interacting with a live server
instance via [`httptest.Server`][2].

This should make the test suite more accurate and reliable in the long
run by ensuring that the vast majority of tests are making actual HTTP
requests and reading responses from the wire.

Note that updating these tests also uncovered a few minor bugs in
existing handler code, fixed in a separate commit for visibility.

P.S. I'm awfully sorry to anyone who tries to merge or rebase local test
changes after this refactor lands, that is goign to be a nightmare. If
you run into issues resolving conflicts, feel free to ping me and I can
try to help!

[0]: https://pkg.go.dev/net/http/httptest#ResponseRecorder
[1]: #125 (comment)
[2]: https://pkg.go.dev/net/http/httptest#Server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compat Tracking issues around compatibility with original httpbin implementation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

data property should be empty on a GET to /anything
2 participants