After reading into all the ways one can test rust code, I have concluded that there are four types of test which one can write, and they are unit tests, integration tests, doc tests and examples. This post is to put forward a mental model of these four options, and see how useful a metaphor it is.
First, I regard all tests as a form of documentation. The only variance is in the intended audience. It is only an appreciated side-effect that they can be executed automatically to check if the conditions still hold after source code changes. This makes me treat both documentation and tests as a first-class citizen along with the source code. The only difference is the problem they are designed to solve. Your genius in optimising your library means nothing if a newcomer doesn't know what it does, or if a veteran cannot find out quickly if a bug they have is your fault or theirs.
For a given feature, there is the way one expects it to be used most of the time, and there are all the edge cases on which it can succeed or fail. The edge cases define the boundary of the functionality, and in my mind, contribute to defining it fully in the first place. Identifying the edge cases is part of the work to complete the feature. Documenting these edge cases is the job of unit tests. Doc tests, on the other hand, provide a common usage example of a single function which does not test the feature in some unique way. Ideally, if the doc test fails, then one of my unit tests is failing too.
Integration tests and Example tests are the equivalent but for the public API of the crate, rather than the internal plumbing. Integration tests find the boundaries of the API, whereas the Examples directory offers ways of using my library in a standard sort of way, which again, should fail only if one of the integration tests fails.
In conclusion, I think of Unit/Integration tests as the super user tests, which a developer/user can consult if there is unexpected behaviour of the feature/API, whereas Doc tests and Examples are for newcomers to have an introduction to the library and to decide quickly whether it is appropriate for their use case.
Option | Audience | Scope |
---|---|---|
Unit | Experienced users | Single functions |
Integration | Experienced users | Entire library |
Doc | Newcomers | Single Functions |
Example | Newcomers | Entire library |
Thoughts?