Add API functional tests, describe the testing process

[m-zajac]

No description provided.

[mwarzynski]

nit: I guess, it should be evolve over time.

https://dictionary.cambridge.org/dictionary/english/evolve

The company has evolved over the years into a multi-million dollar organization.

[mwarzynski]

nit: Actually, I also don't parse the next sentence in terms of grammar: "having the code working on the API" (the structure is: "having it doing X").

My proposal:

Therefore, writing tests based on the API will reduce the maintenance burden.
Whenever you will change the internal workings of the application, API level tests should work fine without any changes.
What is even more important, such tests would be (and inevitably will be) a **huge** help in case of some heavy refactoring in the future.

[m-zajac]

Thank you for these comments. You were right, it didn't make sense :) I changed it almost as you proposed.

[mwarzynski]

nit: I would argue tests should be scenarios related to particular functionality of the application. In majority of cases, functionality tests would require usage of more API endpoints than just one.

[m-zajac]

That would be the case for e2e tests, but I don't think it is for functional API testing. In this case we don't care about how for example the frontend app is using our API. I think that maybe naming is a bit confusing here? But I still think naming those test "functional" makes sense.

[mwarzynski]

Firstly, let's cite the definitions.

(Assumption: e2e == system level.)

System testing is testing conducted on a complete integrated system to evaluate the system's compliance with its specified requirements.

Source: https://en.wikipedia.org/wiki/System_testing

Functional testing is a quality assurance (QA) process and a type of black-box testing that bases its test cases on the specifications of the software component under test.

Source: https://en.wikipedia.org/wiki/Functional_testing

---

I believe I used term "functionality tests" properly: we would want to focus on testing functionalities as defined: "if the software component satisfies the specifications". I didn't mean to mention e2e, since this architecture has only one component, therefore e2e == integration tests (i.e. there is no frontend).

Regarding my original comment, I would just argue that guidelines for API level tests shouldn't focus on particular "API method", but instead on the tested functionality. You also outline the importance of focus on "functional testing". It is just a wording of the point in the guidelines that my original comment relates to.

Currently, at least I understood it this way, the guidelines propose to "write a test for an API method" which literally suggests to create separate test case per each endpoint (e.g. focus on validation of the request/response format and not outline the importance of testing the actual logic). In my opinion, we would rather want to outline the focus on testing functionalities and reflect such need in the guidelines. Even more, I believe your intent is to test the functionality as well (e.g. TestUpdateBike focuses on testing the functionality in order to check if overall logic of "Update" works properly). However, it doesn't use multiple endpoints which I will try to cover below.

In majority of cases, functionality tests would require usage of more API endpoints than just one.

In my original comment, I assumed we would want to use multiple endpoints to test "scenarios". Why? In order to test the "Update"/"Delete" operation on Bike object, we would have to firstly create one. In your tests you use a database adapter to create Bikes (createSpecificBike). I personally would rather implement such tests by calling v1.CreateBike endpoint, but your approach is also fine given the current complexity of the system. However, if the system would grow, and we would add more logic to objects creation, then injecting the data directly to database from the test code (which would skip business logic) might lead to a state in the database which doesn't fully express what would happen if we have used a real endpoint. Therefore, for more complex systems (or ones which have the "real" potential to become one), I would rather use endpoints instead of database adapters to populate the data. In this particular case of Bike service, I guess it doesn't matter.

To be honest, I might have been overly strict, because it sounds like in the end we would like to see (more or less) the same implementation of functional tests (which is actually expressed as a source code pretty clearly). Feel free to disregard this comment.

[m-zajac]

Hey @mwarzynski, sorry for the late reply.

I argue that for functional testing you don't need to know the context of the app (how it is used in a system, or by an external client), so it doesn't really make sense to write usage scenarios. You are not the consumer of the API, so by writing such scenarios you can fool yourself that you know exactly how your client is using the API and falsely claim that you have proper API usage tests. And in a REST API, each endpoint defines specific functionality almost by definition (they usually expose atomic operations on resources, they usually are independent), so I think in this example it makes sense to focus functional tests on endpoints. But I agree I have to restate the guidelines, because that doesn't always have to be the case.

I agree that I created too much isolation of the endpoints in tests, I'll improve that.

I'll make a few changes and get back to you. I'm having a break from coding recently, but I hope I'll get back to it soon :)

Merry Christmas! :)