Manually testing an application is time-consuming, costly, and difficult to scale as your application grows: as you add more features to your application, you have to add more functional tests. And getting those additional tests done usually means adding headcount.
Automated functional testing can speed up the testing process, provide more consistent results, and give one person the ability to manage the testing workload of five or more manual testers.
But automated functional testing comes with its own challenges:
- Most tools and frameworks don’t actually test what the user sees—they evaluate the application’s code—so they miss more bugs than manual testers.
- Most tools require programming skills, which leaves room for human error.
- For automated tests that evaluate an application’s code, changes to the application’s code can break these tests and require time-consuming maintenance.
In this guide to automated functional testing, we’ll cover everything you need to know to get started and avoid as many of these challenges as possible. We’ll discuss:
- Functional testing basics
- The evolution of functional testing tools
- How modern tools like Rainforest QA are changing functional testing
- When it makes sense to add functional software testing to your release process
Ready to automate your functional testing? Sign up for Rainforest QA—you can run up to five hours of no-code automated tests for free, every month. It’s only $5/hour after that.
Contents
Functional testing basics
Functional testing is the process of confirming that all the elements of an application work together as a whole and appear correctly in the user interface (UI) so that a user can perform desired actions.
Different types of functional testing include:
- Integration testing. Testing combinations of modules that have already individually passed unit testing to make sure they work together.
- Smoke testing. A small set of UI tests performed to ensure the basic functionalities of a new feature are working.
- Regression testing. A test suite executed right before releasing a new feature or product to ensure that the new changes didn’t break any existing critical functionalities of the app.
- API testing. A series of tests to make sure APIs return expected results with the desired level of performance.
Unit testing is sometimes considered functional testing because it evaluates whether a unit of code behaves as intended. However, for the purposes of this article, we won’t include it as a type of functional testing because it doesn’t touch the user interface.
For example, say you’re building a calculator app. With unit testing, you can confirm the accuracy of the “sum” function. When given the input of “2+2,” the function returns the answer “4.” But to be fully confident that the calculator will work for users, you need to test how it works from the UI.
That’s where functional testing comes in. Functional testing would verify that the correct numbers and signs are actually appearing in the UI. Even if the “sum” function is working (i.e. the unit tests passed), a mistake in integrating other functionalities could cause a ‘2’ to appear on the visual layer when the backend of the app sends a ‘3’, causing the function to return the answer ‘5.’
Non-functional testing evaluates the app’s performance in terms of speed, security, and reliability. For example, a functional test would verify that a user can successfully complete a signup process. A non-functional test might test how long the sign-up process would take if 3,000 people were trying to access the web application at the same time.
A few types of tests that fall under non-functional testing include:
- Security testing: Tests used to verify security mechanisms are working properly (e.g., that credit card numbers are encrypted on the backend).
- Load testing: Tests used to verify the application still responds under additional demands (e.g., 3,000 users access the site).
- Performance testing: Tests used to measure variables such as speed and resolution.
- Usability testing: Tests that evaluate how intuitive and easy-to-use the software will be for the end-user.
The evolution of functional testing tools
Before automated software testing, the only way to test your application was manually. It was common practice to jot down a few notes about the features that needed to be tested or create a spreadsheet of user paths (sometimes called data-driven testing). Then, someone would spend a considerable amount of time clicking through the application looking for bugs.
Even with the many test automation tools available today, a lot of teams still start with manual testing because it can be done without any additional training by whoever is available. Manual testing is a good option for test cases that frequently change or that require subjective judgment (“Do the images look clear?”). However, it’s not a long term, scalable solution for all your functional testing.
Teams run into these difficulties with manual testing:
- Test execution is inconsistent and prone to human error, thanks to inattentional blindness resulting from rote, repetitive testing.
- Test execution is very time-consuming.
- Bugs found during manual testing can be hard for developers to reproduce because of variations between the tester’s and the developer’s testing environments.
- Testing is typically more expensive to execute on a per-hour basis because it relies on human resources instead of cheap computer processing power.
Eventually, teams usually find they need more test coverage than is feasible to build and maintain with manual testing. That’s where automated testing comes in.
In 2004, Selenium was introduced as the first open-source web browser automation tool for software testing.
Selenium is a testing framework that helps developers write lines of code to make a browser mimic end-user interactions with a web page or application. The Selenium code searches for specific lines of code (i.e. locator IDs) that represent the presence of an element (such as a button) in the user interface. If the locator ID is found, the test assumes the element is visible and the test passes.
As Selenium evolved, developers were able to run tests in parallel, author tests for any scripting language (such as Java or Python), use record-and-playback to write simple tests with Selenium IDE, and perform cross-browser testing with the Selenium Webdriver. This changed the face of software testing because suddenly:
- Tests could be repeated the exact same way every time with more control over what was being tested and with more consistent results.
- Developers could quickly execute multiple tests with a click of a button rather than sit and wait for someone to manually complete test steps.
As use of Selenium became more common, other tools appeared that made it easier to write and organize automated test scripts and to run even more tests simultaneously. Some of the tools also made it easier to understand why tests failed by capturing screenshots of the UI or snapshots of the underlying code at the point of the failure.
But even with these improvements and add-ons, doing automated functional testing with Selenium and its offshoots still presents several challenges:
- It’s very time-consuming to write tests because you have to write code for every assertion (e.g., Is the button shape visible and correct? Does the button read ‘Sign up’? Is it located in the top right corner?).
- Tests need to be constantly updated. Because Selenium works by interacting with the underlying code of an application (also called the DOM), Selenium test scripts rely on locator IDs to find elements (buttons, forms, etc.) and apply actions to them. These locators often change with app updates, which means the tests have to constantly get updated.
- It requires programming skills, which means your developers have to spend some of their time writing and maintaining UI tests rather than building new features, or you have to hire dedicated (and often expensive) QA engineers.
In response to this, commercial tools started appearing that offered “no-code” testing solutions. However, most of these tools simply generate the code for you. You will still run into the problems of:
- Having to update tests nearly every time the underlying code changes.
- Needing an engineer to understand why a test failed and keep tests up-to-date.
Further, all testing solutions that test the underlying code—without interacting with the visual layer—may end up missing a lot of bugs that the majority of users would notice immediately.
The only way to get around these difficulties is to stop using code to test code.
Rainforest QA is a no-code automated functional testing tool that mimics real user behavior by interacting with the UI on the visual layer rather than with the underlying code. This means:
- You’re testing what the user actually sees, so you catch more bugs that actually affect users.
- From start to finish, you don’t have to write a single line of code.
The next section takes a detailed look at how Rainforest QA differs from all other tools that test the underlying code.
How modern tools like Rainforest QA are changing functional testing
Rainforest QA uses pixel-matching to locate elements and verify all aspects of the element (i.e. color, text, shape, position, etc.). With code-based tools, you would need to write a separate test step (or line of code) for each aspect of the element. Rainforest tests can verify all of that with just one screenshot.
To write or maintain a test step in Rainforest QA:
- Select from the dropdown menu of actions (click, fill, wait, observe, etc.).
- Click and drag to take a screenshot of the element you want to apply the action to, or specify how to apply the action (i.e. ‘wait’ for ‘3 seconds’).
You can also choose to embed one test into another test to speed up test creation (and test maintenance).
For example, if you have a test for a signup flow, you can embed that test into every other test that starts with a signup flow.
Once you’ve written or embedded every test step, you can playback the test to verify it will run as intended. Rainforest QA also offers an API, CLI, and a CircleCI Orb so developers can kick off a suite of Rainforest tests along with their unit tests and integration tests.
Finally, Rainforest allows for multiple tests to be executed simultaneously on our cloud of virtual machines. You can run tests on the latest and older versions of Chrome, Firefox, Safari, and Microsoft Edge.
Tests Don’t Break with Minor Changes to the Underlying Code
All code-based automation tools look for locators in the underlying code to verify if an element is visible. For example, let’s say you want to verify that a message saying ‘signup was successful’ appears at the end of a signup process. Code-based tests will search through the underlying code for the element locator of the message. If the locator is present, the test will pass.
The problem with this is that even the slightest change in the element locator can break your test—even if the end results look identical to the user. Maybe your team renamed the locator during an update, or accidentally missed a letter in the locator when creating the test. Either way, a code based test would fail because it couldn’t find an exact match, even though the visual element would look and act the same way for the user.
Because Rainforest tests use pixel-matching to verify elements, they are less susceptible to minor changes in the underlying code. In fact, a Rainforest test will only fail if there is a visual change in the UI—which is more likely to represent a real bug.
Test Actions Outside of the Browser
Most automation solutions can only generate scripts that stay inside a single browser tab. With Rainforest’s proprietary automation, you can create tests that open new tabs and new windows, open and close browsers, and even interact with desktop apps.
For example, the video walkthrough shown above covers how to create a test that saves a file to the desktop from one location, and then uploads the file to Google Drive.
This feature helps test how your app interacts with desktop apps, how it interacts with browsers with ad-blockers, and other complex functions.
Quickly Understand and Report Every Failure with Video Playbacks and Integrations
Rainforest records a video of every test (whether the test passes or fails) to make it easier to debug issues. Test results also include HTTP logs and JSON browser logs to help with debugging.
Rainforest QA also integrates with tools like Slack and Microsoft Teams, so you can get instant notifications for any test failure. If you integrate with Jira, every time you need a bug to be fixed, you can automatically create a ticket for the software development team that includes a video recording of the test and relevant logs.
This dramatically reduces the amount of time it takes to triage bugs.
Start for Free and Only Pay for What You Need
Our free-forever Startup plan makes software test automation accessible to anyone. This plan has everything you need for test automation—parallel testing on virtual machines and unlimited team members—without any hidden costs. You get up to 5 hours of free testing every month and it’s only $5/hr after that.
How and when to add functional test automation to your release process
The most important factor to consider when adding automated functional testing to your release process is the stability of your product.
If you have a consistent series of functional tests that don’t change very often and get run frequently, it’s likely that automation could save you a lot of time and money.
However, if your product or feature is still early-on in the development cycle, with major code changes occurring on a regular basis, it’s not time to automate yet. You’re better off waiting until the application is more stable.
Some test cases will never be a good fit for automation, which is why the Rainforest platform also offers on-demand manual testing by testers in our worldwide community of QA specialists. With one platform, you can scale up manual and automated testing without adding headcount.
Get started with automated functional testing with Rainforest QA
Rainforest automation enables your team to move faster—without breaking things. It’s easy for teams to get started with automated UI testing and build as you go in an agile environment.
You don’t need to learn a new programming language, you don’t need to buy additional services from other vendors to run or manage your tests, and anyone on your team can create, run, maintain, debug, and interpret tests.
It’s a fast and scalable all-in-one solution that’s appropriate for teams that are just dipping their toes into automated testing, as well as QA-mature teams doing continuous testing with a suite of 500+ software tests.
Get started with Rainforest QA for free. You can run up to five hours of no-code automated tests for free, every month. It’s only $5/hour after that.
