Testing on High: Bottom-up versus Top-down Test-driven Development 40

Posted by ryan Mon, 19 Nov 2007 02:13:21 GMT

I recently talked to a number of Rails developers about their general approach to testing some new functionality they're about to code. I asked these developers if they found it to be more useful to start testing from the bottom-up or top-down. I suggested to them that, since Rails uses the MVC pattern, it's easy to think of the view, or user interface, as the "top", and the model as the "bottom". Surprisingly, nearly every developer that I asked this question of answered that they prefer to start from the bottom, or model, and test upwards. Nearly every one! I expected that I'd get a much more mixed response than I have. In fact, I think that the correct place to start testing is precisely at the highest level possible, to reduce the risk of building software based on incorrect assumptions of how best to solve a user requirement.

Bottom-up Testing

Bottom-up testing implies bottom-up design in TDD. In bottom-up design, a developer would probably consider the high-level objectives and break them up into manageable components that interact with each other to provide the desired functionality. The developer thinks about how each component will be used by its client components, and tests accordingly.

The problem with the bottom-up approach is that it's difficult to really know how a component needs to be used by its clients until the clients are implemented. To consider how the clients will be implemented, the developer must also think about how those clients will be used by their clients. This thought process continues until we reach the summit of our mighty design! Hopefully, when the developer is done pondering, they can write a suite of tests for a component which directly solves the needs of its client components. In my experience, however, this is rarely the case. What really happens is that the lower-level components tend either to do too much, too little, or the right amount in a way that is awkward or complicated to make use of.

The advantage of bottom-up testing is that, since we're starting with the most basic, fundamental components, we guarantee that we'll have some working software fairly quickly. However, since the software being written may not be closely associated with the high-level user requirements, it may not produce results that are necessarily valuable to the user. A simple client could quickly be written which demonstrates how the components work to the user, but that's besides the point unless the application being developed is a simple application. In such a case, the bottom-level of components are probably close enough to the top-level ones that there is little risk involved in choosing either the bottom-up or top-down approach.

Unless you're writing a small application, the code is probably going to have to support unforeseen use cases. When this comes as a result of ungrounded assumptions about the software that's already been written, this can mean a lot of rework. I can tell you from experience, once you realize that your lower-level components don't fit the bill for the higher levels in the system, it can be quite a chore to go back and fix, remove, or replace all of that unnecessary or incorrect code.

Top-down Testing

Top-down testing implies top-down design in TDD. Following the top-down approach, the developer will pick the highest level of the system to be tested; that is to say, the part of the system that has the closest correlation to the user requirements. This approach is sometimes referred to as Behavior Driven Development. Whatever it's called, the point is that you test the most critical parts of the application first.

Since software is often written for human users, the most critical parts usually involve the front-end as it relates to the value being provided by the system being developed. When testing from the top-down, the effort is the inverse of bottom-up testing: Instead of spending a lot of time thinking about how the components to be developed will be used by other components to be developed, the focus is on how the user needs to interact with the system. Testing involves proving that the system supports the required usability. For an application with a graphical front-end, this might involve testing for a minimal version of that front-end.

The disadvantage of top-down testing is that you can end up with a lot of stubbed or mocked code that you then have to go back and implement. This means it might take longer before you have software that actually does something besides pass tests. However, there are ways that you can minimize this sort of recursive development problem.

One way to minimize the time between starting development of a feature and demonstrating functionality that is valuable to the user is to focus on a thin slice of the overall architectural pie of the application. For example, there may be a number of views that need to be implemented before the system provides some major piece of functionality. However, the developer can focus on one view at a time, or one part of the view. That way, the number of components that need to be implemented before the system does something useful is small; ideally, one component in each architectural layer that I need build out, and often times only a part of the overall functionality of each component.

Another way to minimize the amount of time before the system does something useful is to code a small bit of functionality without worrying about breaking the problem up into classes until you have some tested, working code to analyze. You can then use established methods for refactoring to bring the code to an acceptable level of quality.

The advantage of top-down testing is that you write functionality that solves the most critical functionality first. This generally means starting development at a high level. When the system eventually does something besides pass tests, what it does will provide value to its users. Additionally, because development starts at a high level, the code that is written is based on the current understanding of the problem, and not on assumptions. This guarantees that the tests and code that are written are not superfluous.

Conclusion

The challenge with top-down testing is that you must be highly disciplined to ensure that the code you write is being refactored and is properly evolving into a cohesive domain model for the application. This is compared with bottom-up testing, where you start with the domain model and build your system around it. Either way, you're going to be refactoring code. The difference is in where the time in refactoring is spent. In my experience, when doing bottom-up testing, more time is spent correcting incorrect assumptions about how the domain model will be used than on actually improving code that already works to solve the user requirements. In order to avoid making assumptions about the code being written, it must be written at the level that is closest to providing actual value to the end-user. In so doing, the developer focuses on continuous refinement of code that already provides value, as opposed to speculative design and development.

  1. David Chelimsky about 11 hours later:

    Great article. Questions:

    Did you ask the developers who prefer bottom-up if they're doing TDD/BDD and relentless refactoring?

    The disadvantage of top-down testing is that you can end up with a lot of stubbed or mocked code that you then have to go back and implement.

    When you say "go back and implement," are you going back to your tests and replacing mocks with the real objects?

    Do you use tools like RSpec or ZenTest to separate your tests for each component layer?

  2. Dave Hoover about 12 hours later:

    I agree with your thoughts on this issue. Rather than "Top-Down Testing", I call it "Outside-In Testing".

  3. Ryan Kinderman about 12 hours later:

    They all said they did TDD. As for relentless refactoring, let's just say that I was trying to ask leading questions in that direction, to see if they'd go there themselves. I did, however, generally end up talking about refactoring in some capacity with each dev.

    When you say "go back and implement," are you going back to your tests and replacing mocks with the real objects?

    No, I mean that you end up with mocks in your tests that represent yet-to-be-developed components in your application. These yet-to-be-developed components have to be implemented before something like an integration test of the functionality passes, which would prove out a particular aspect of a user feature from end-to-end. In other words, the feature being developed just will not actually work until the functionality that is mocked or stubbed is implemented. Does that make sense? Perhaps I should clarify the original statement.

    Do you use tools like RSpec or ZenTest to separate your tests for each component layer?

    I definitely use and love RSpec. As for ZenTest, I used it in its early stages, when most of what it did was automate running tests, and sometimes had problems doing that. It does a lot more these days, and it might be worth a try.

    As for separating my tests for each component layer: with Rails development, I find that I usually have my unit tests broken out into controllers, helpers, libs, mailers, models, services, and such. I don't necessarily clearly define the layers that these categories of components fall into, since the idea of layers, while useful at a high level (model/view/controller), can become transient when a developer tries to define them in more detail. I prefer to keep them rough, and then simply pay attention to having proper interactions between components.

    I've been using a combination of Rails integration tests and Selenium to perform integration testing for the systems I build.

  4. David Chelimsky about 13 hours later:

    @Dave: did you know that Outside-In is "official" BDD terminology these days?

  5. Yi Wen about 18 hours later:

    Good writing, Ryan, I think the reason why people tend to write tests bottom-up is that they are reluctant to use mocks/stubs in their tests, which I disagree. So it is simply impossible working top-down.

  6. Ryan Kinderman 1 day later:

    @Yi: I wouldn't say it's impossible. I think I mention in the article that it's possible to code a solution directly using top-down development. I started doing something similar for an XML diffing algorithm I wrote not too long ago. I started with the most basic test that would demonstrate some diffing functionality, such as:

    it "reports no difference between two empty strings"

    This didn't get me too far, but it set things up like the top-level diffing method, its initial signature, and so forth. It wasn't until I had 20-something passing examples that I started recognizing a pattern and actually started pulling out classes. And even then, I didn't have to mock.

    I ultimately did end up mocking to simplify the tests, but one can get away with not mocking using top-down development as well as they can with bottom-up. It's just a matter of how isolated the tests are going to be. Obviously, in either case, fully-isolated tests are the best scenario.

  7. James Herdman 3 days later:

    Good article. I thoroughly enjoyed it.

    I'm wondering if sometimes the nature of the project demands a particular testing approach? Data-centric projects, and (perhaps) projects wherein one ports an application from one language to the next, seem to demand a bottom-up approach. Green-field projects are much more suitable for outside-in.

    Another factor I can imagine coming into play is the nature of the development team. For instance, are the developers the UI designers, and are they comfortable and competent in this domain? If not, they'll probably opt for (or be forced to) the bottom-up approach.

  8. Liz 3 days later:

    Fantastic post, Ryan. I also love Outside-In.

  9. Merlyn Albery-Speyer 6 days later:

    Good post, Ryan. I do find the difference in outside-in and inside-out philosophies interesting. I recommend reading from xUnit Test Patterns on the topic if you haven't already.

    TDD from the outside-in is my preference, but I can't quite express why it feels 'the right way' to me.

    I would argue that nothing is working until it's working end-to-end. Is outside-in really slower as you suggest?

    Unfortunately, I don't always start at the highest level of abstraction. I'm limited to the highest level that can provide me with rapid feedback otherwise development quickly stops being TDD in my opinion. So, I pretty much always start just below the user interface layer. Does that make me some sort of mutant middle-in TDDer then?

  10. Ryan Kinderman 7 days later:

    @Merlyn: I don't think that the outside-in testing approach is necessarily slower in producing a result that has value to the user. That being said, it can be slower at producing a component at any level that is fully-functional. To understand what I'm getting at, imagine that you start by writing a little HTML for a web app feature you're building. Once you get something that looks to the user like it's going to work, you write some tests around it, mocking or stubbing out component behavior that the views delegate to. You then start testing/implementing the functionality that the views invoke, which may result in more stubbing or mocking. It may take a while before you can stop mocking or stubbing components and get to a level where the component is sufficiently small that it doesn't need to call off to other components, and you can fully test and implement its functionality.

    If you were to use the inside-out approach, the idea is that you start at the component that is sufficiently small to fully-implement. You then write components that use that component's functionality. Since you start with inside-out precisely where you stop using outside-in, it's quicker to fully-implement a component or set of components using inside-out, at least initially. Does that make sense?

    As for whether testing just below the UI layer is not purely outside-in:

    I think a decision always has to be made regarding where the "outer-most" point at which to start testing is. This doesn't necessarily have to be at the UI layer. In any such decision, the correct choice is always the point that is the most critical to the user. Often, the user doesn't know where that point is, because they're not used to thinking of solutions in terms of software interfaces. In such cases, the most critical point is probably at the UI layer, since helping the user visualize possibilities in that area provides them with a framework for communicating their own needs, which they need to be able to do before a developer spends too much time writing software for them.

    On the other hand, sometimes the user needs are already well-defined. In such cases, the most critical starting point may move down from the UI. This will also happen if the points at which the UI will interact with the lower layers are fairly well-defined, such as in Ruby on Rails, which provides a development process that is more-or-less followed consistently when development begins on each new feature in a system. For instance, in your typical Rails application, there is little question how a resource's CRUD interfaces should be implemented, at least initially, to provide some basic functionality to the user.

    Yet another scenario where testing below the UI layer is appropriate is when the user is more concerned about whether the underlying functionality is implemented correctly. I once worked for a client who only wanted a basic UI initially - just enough to show them that some of the underlying system functionality could actually be implemented, given its perceived complexity. In that case, the most critical aspects of the system were, in fact, below the UI.

  11. Cumshot over 2 years later:
  12. best2222 over 2 years later:

    Ultra-cheap brand store: ◆ Business philosophy: integrity, pragmatic and innovative business philosophy to provide customers with satisfactory service management products and services for the target! ◆service management products and services for the target! ◆ brand: Louis Vuitton, Chanel, Gucci, Hermes, Prada, Bvlgari, fendi Mulberry and so on ◆ Operating principles: quality, low price, discount and more, with goods, there are different styles ◆store on a regular basis will launch various new products, welcome your patronage ◆ E-mail: sale@best2222.com ◆ Website: http://www.best2222.com

  13. rayban sunglasses over 2 years later:

    If you love your life ,please take care of your eyes,will you go out without a <a href=" http://www.eyewear-

    rayban.com/">rayban sunglasses?

  14. watchestoo over 2 years later:

    very good article,thank you for sharing

    Your sharing information is very useful for me,thank you! Now i share you with my favorite watches http:// www.watchestoo.com

  15. dofollow blogs list over 2 years later:

    Are the developers the UI designers, and are they comfortable and competent in this domain? If not, they'll probably opt for (or be forced to) the bottom-up approach.

  16. 事发当时 over 2 years later:

    士大夫倒萨

  17. step up converter over 2 years later:

    I agree with your thoughts on this issue. Rather than "Top-Down Testing", I call it "Outside-In Testing".

  18. cheap p90x outlet over 2 years later:

    I love sport,I bought p90x form a store named cheap p90x outlet .This store sele discount p90x is very very cheap.I love p90x wholesale very much.

  19. Johnsmithlucky2012 over 2 years later:

    Good post, Ryan. I do find the difference in outside-in and inside-out philosophies interesting. I recommend reading from xUnit Test Patterns on the topic if you haven't already.

    TDD from the outside-in is my preference, but I can't quite express why it feels 'the right way' to me.

    I would argue that nothing is working until it's working end-to-end. Is outside-in really slower as you suggest?

    Unfortunately, I don't always start at the highest level of abstraction. I'm limited to the highest level that can provide me with rapid feedback otherwise development quickly stops being TDD in my opinion. So, I pretty much always start just below the user interface layer. Does that make me some sort of mutant middle-in TDDer then?

  20. bet365link over 2 years later:

    I'd like to learn more about you all talked about. Thanks for sharing!

  21. Alexander Wang over 2 years later:

    www.christian-louboutinboot.com Recently,he has made a line of footwear known as Espadrilles christian louboutin online.They are sold in partner with One & Only and at their resorts in the Bahamas,Dubai,the Maldives,Los Cabos and Muaritius.They are also sold at his boutiques in London and New York.The line was inspired by his numerous stays at the Maldives resort.He was inspired by the colors there and considers this line a great shoe for looking good while on vacation.The colors are meant to be shown off under the summer sun,when relaxing.

  22. Termination Letter for Employee Dismissal over 2 years later:

    I completely agree and I just wanted to say that I really like your blog. Is it Wordpress based?Termination Letter for Employee Dismissal

  23. belly button jewelry almost 3 years later:

    gold belly rings It’s really great post. I would like to appreciate your work and would like to tell to my friends. Thanks for sharing

  24. CATT almost 3 years later:

    Recently,he has made a line of footwear known as Espadrilles christian louboutin online.They are sold in partner with One & Only and at their resorts in the Bahamas,Dubai,the Maldives,Los Cabos and Muaritius.They are also sold at his boutiques in London christian louboutin sale and New York.The line was inspired by his numerous stays at the Maldives resort.He was inspired by the colors there and considers this line a great shoe for looking good while on vacation.The colors are meant to be shown off under the summer sun,when relaxing.

  25. football uniforms almost 3 years later:

    We are a professional & reliable supplier of football uniforms , we have offered cheap jerseys for 6 years, Our product has the great reputation for fine works, and elegant materials.
    We can provide the reliable quality football jerseys with the competitive price.

  26. cheap sports jerseys almost 3 years later:

    cheap sprots jerseys are accord with many sports fans demands.If you want to buy any football uniforms,don't forget our best-seller NFL jerseys on sale ,quality and favorable price is our credit standing guarantee.Thanks for your visiting.

  27. tom almost 3 years later:

    lose weight fast  <a

    href="http://www.39yd.com" target="_blank">lose weight  <a

    href="http://www.39yd.com" target="_blank">lose weight product

  28. women bags about 3 years later:

    women bags If you like is

    super popular bags, and how to choose to have unlimited encumbrance?

    If your height in 165 centimeters of above, should choose as far as

    possible about 60 cm in length, vertical dress magazine into size

    bags, If your height in the following words, then 158 cm should choose

    sword-boat about 50 centimeters, can cross dressing into magazine size

    bags, spin figure scale.

  29. Short Break Up Poems about 3 years later:

    Interesting post and thanks for sharing. Some things in here I have not thought about before.

  30. women bags over 3 years later:

    women bags Pocket and exquisite hand bag as the perfect accessories banquets always lets a person star-studded. In daily life, economic fund hand bag also absolutely let you star flavour is dye-in-the-wood. Different material, women bagsmultiple color style hand bag with you hand in hand, less aglet of bondage, show your beautiful hand curve, perfect show your strong momentum.

  31. women bags over 3 years later:

    women bagsNo women can

    withstand the temptation of exquisite bags, especially and suit

    collocation, women bags it has become a classic tide

    over the following bag is dressed up, not only the atmosphere and

    joker, actually the basic principles of the bag is simple and

    practical. women bags

  32. Prada outlet over 3 years later:

    Moret: the volunteer Andrew Pink: the expat Olympics: cycling guide Olympics: venue guide Olympics: volunteer guide Olympics: tickets

  33. adas over 3 years later:

    adas

  34. givenchy hobo handbag over 3 years later:

    givenchyhobo handbagis one classic style of the brand,givenchy hobo handbagwhich belongs to cheap givenchy hobo handbag,is also one of the latest designer givenchy hobo handbag.It has the same look and materials with the original.

  35. menkw109 over 3 years later:

    beautiful friend, or his enviable long legs.

  36. http://www.shunkycrusher.com almost 4 years later:

    I think this post is very useful, and I like it very much. http://www.jaw-breaker.org

  37. Snygga cyklar about 5 years later:

    There is one thing to be said about the UAE, and that is that the warm weather @what and cool water make it a fantastic place to go sailing, as the waters are often very calm, clear and exceptionally peaceful for most holidaymakers.

  38. Cyklar about 5 years later:

    I just wanna say thank you for the information that you have been shared to us readers. Thanks for posting this kind of theme.

  39. Nuvalift Reviews over 5 years later:

    Amazing! Its really remarkable paragraph, I have got much clear idea about from this article.

  40. natural remedies over 5 years later:

    I always used to study post in news papers but now as I am a user of net so from now I am using net for content, thanks to web.