In one of the previous sections of this course, you wrote a web scraper script to fetch information from the CodingNomads recipe collection. You'll now practice writing tests for your code by adding a unittest test suite to your web scraper code and refactoring your original code in the process of doing this.
Web scrapers are a great example of where having a solid test suite comes in handy. Content on the web--and website structure--changes frequently. With your test suite in place, you'll always know when something stops working, and you'll have a good idea of where to go in order to fix the issue.
Test Small Pieces
When running your test suite previously, you saw that it'll point you to the exact test case that failed. If each test case corresponds with one specific function in your production code, that'll give you a good idea of where to look for the bug in your code.
Photo by alevision.co https://unsplash.com/@alevisionco
Of course, this requires that you wrote your production code in a way that makes it possible to test it in small pieces. The solution for this is to compartmentalize each piece of functionality into its own function.
Tasks
- Revisit your web scraper script and think about which pieces represent fundamental functionality that you could compartmentalize into their own function.
- Take notes in your notebook!
You could go ahead and refactor the web scraper code you wrote and add functions so that you can later write tests to confirm these functions work as expected. However, you'll take another approach and write your tests first, then refactor your production code to make the tests pass.
This process is called test-driven development (TDD), and you'll learn more about it in the next lesson.
Summary: Python Testing for Your Web Scraper
- Tests are important for web scrapers since web content changes frequently
- It is best to test small pieces of code at a time
- TDD is about writing your tests before writing your code