Write and debug tests easily, with full action history, time travel, screenshots, rapid re-execution, video records, retryability, interactivity, isolation and more.
pump
, wait or retry anymore. Just write down what you want, and the framework will retry and wait.And also...
integration_test
still there: Freely use everything from integration_test
, mockito
, test
, etc.NOTE: The video only contains features up to 2022.05. New features not included in the video: The 5x speedup, the enhanced golden, etc.
https://user-images.githubusercontent.com/5236035/167066810-d0aa36ba-0113-4140-92f9-cec0a9e77ed1.mov
It has been used extensively in my own 200kloc Flutter app in production environment, and I integrate it with my CI infra so it is run everyday.
P.S. Surely, this package - especially the doc - is not yet perfect. If having problems, just create an issue and I usually reply quite quickly.
I have been using these components internally for a long time, but have not found time to make cleanup (e.g. remove internal dependencies) in order to open-source it. If you like it, feel free to tell me and I will prioritize it!
In addition to the standard way of integration tests, i.e. run them on a real mobile device / a simulator, this package supports running the exact same testing code on your host computer (Windows/MacOS/Linux) with similar experience.
This leads to a big speedup and reduction of needed resources. As we all know, running code on simulators / real devices are slow, and hard to parallelize as well. Running on host computer, on the other hand, does not have this problem.
Remark: In my own app in production, it is >10x faster, but without creating public and reproducible tests, I do not want to claim such a big improvement - so test by youself and see how fast it provides :)
How to use it: Just write code normally, and execute your code as if it is a widget test. If you see weird fonts, you may want loadAppFonts.
See actions/taps/assertions taken in the tests, with friendly descriptions (in the left panel)
Tap an action to see its screenshots
P.S. Use mouse wheels and dragging to zoom and pan the screenshots.
P.S. The gif is outdated - new version has (1) an overview of all screenshots (2) a ruler for each screenshot.
Edit code, save, run - within seconds
Watch what has happened in full detail (in right panel)
P.S. Highlighted action is in sync with the playing video. Tap action to seek video.
Temporarily play with the app, interactively. (Flutter originally cannot interact with app in test mode)
No manual pump
, wait or retry anymore. Just write down what you want, and the framework will retry and wait.
Example:
await t.get(find.byTooltip('Fetch Data From Backend')).tap();
// OK even if "fetch data" needs undeterministic time interval. Will automatically pump, wait and retry.
await t.get(find.text('New Data')).should(findsOneWidget);
More in quickstart and tutorials below.
One test will no longer destroy environment of other tests - now you can run run each with full isolation.
This is especially helpful in big projects (such as mine), when an end-to-end setup is not easy to tear down, and one failed test can cause all subsequent tests to have damaged execution environment and thus fail.
Technical details: If this mode is enabled, a hot restart will be performed after each attempt of each test.
When something is not working, it tries to print out as much information as possible to help you quickly debug.
For example, suppose you provide a finder
that accidentally matches two widgets. Then, convenient_test will tell you the location of each widget, as well as a chain of render objects. Together with the screenshots/videos it provides, the finder problem can be fixed easily.
Remark: The GIFs are outdated and does not show this feature.
This is a standalone feature, i.e. you can use it without using convenient_test.
It is a simple GUI to list and check all golden differences, with interactivity like pan and zoom. I personally use it daily as follows: After changing a feature, I run all regression tests. Then, if some tests report a golden change, I look at this page to check whether the change is intended or a bug.
To enter this page, tap the "golden" button at the top of homepage.
GoldenConfig.maxToleration
, GoldenConfig.greaterThanToleration
)Overlay
. Then, if you golden by find.byKey(yourButton)
, you will not screen the hint bubble as well. By using this cropBbox
feature, you can take a screenshot of the whole screen, and crop the parts (the button) you want. (GoldenConfig.cropBbox
)--update-goldens
. (GoldenConfig.allowUpdate
)goldenFileComparator = EnhancedLocalFileComparator.configFromCurrent(); // setup
// just like the old way, except that we call `EnhancedLocalFileComparator.createUri` with extra configurations
final config = GoldenConfig(...);
await expectLater(whatever, matchesGoldenFile(EnhancedLocalFileComparator.createUri('something.png', config)));
enterTextWithoutReplace
): The standard enterText
replaces the old content, making it inconvenient if we want to append some text - which is the normal behavior of humans.runAsyncAndPumpUntil
, pumpAndSettleWithRunAsync
, etc): If your code does something that needs "real" async, a simple pumpAndSettle
will not work, so this function is created.integration_test
is still thereYou can still use everything from integration_test
, mockito
, test
, etc. This package is not a reinvented wheel, and has exposed the underlying integration_test
to you.
If you want to migrate to this package from raw integration_test
, have a look at Getting Started section below.
Flaky is flaky, and we are aware of it. It is neither failed nor success, so you will not be alarmed by false positives, and not completely ignore it.
This tool can be run without GUI and only produce log data and exit code, making it suitable to be run in a CI. If you want to examine the details with GUI, just open the generated artifact in the GUI using Load Report
button.
// TODO: Add more doc about how to use it (informal doc here: https://github.com/fzyzcjy/flutter_convenient_test/issues/240)
Tap "Run" icon button after each test or group to re-run only that test/group, without running anything else. All within seconds - no need to recompile like what the original test package did.
Useful when replaying videos and screenshots
Mark
s have colored bordersTap "Raw Logs" in the right panel to see raw outputs of a test.
Following is a brief description of the functionalities of each button in the header panel.
SaveReport
toggle is enabled or run within CI/headless modeFollowing are about the toggles:
--update-goldens
flag in a traditional Flutter testpackages/convenient_test/example
folder.flutter run /path/to/flutter_convenient_test/packages/convenient_test/example/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=/path/to/this/app
. Can also be run via VSCode or Android Studio with similar commands.packages/convenient_test_manager
. (See details in section below)pubspec.yaml
, add convenient_test: ^1.0.0
to the dependencies
section, and convenient_test_dev: ^1.0.0
to the dev_dependencies
section. As normal, we need to flutter pub get
.integration_test/main_test.dart
file in your app. Fill it like void main() => convenientTestMain(MyConvenientTestSlot(), () { ... the normal test code you write });
. See the example package for demonstration.flutter run /path/to/your/app/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=/path/to/this/app
. Can also be run via VSCode or Android Studio with similar commands.packages/convenient_test_manager
. (See details in section below)manager
There are precompiled binaries for each commit, at the "artifacts" section in the CI page. For example, download manager_macos
artifact, unzip it, and open it.
Currently only MacOS and Linux has CI for this, and I am willing to accept a PR for Windows.
flutter run
/ flutter profile
/ flutter build
packages/convenient_test_manager
is nothing but a normal Flutter Windows/MacOS/Linux app, so run it following the Flutter official doc.
Surely, it can also be run via flutter profile
mode to speed up. Or use flutter build
and the standard approaches to generate a release version binary and use it.
There are a few ways to configure the manager:
$HOME/.config/convenient_test.json
.convenient_test_manager_dart
command line)For all options that are configurable and all environment variable names, please see consts.dart
. (By doing so you can always see the up-to-date information and there is no possiblility to see outdated doc.)
If you are trying to use this package with Android Virtual Emulator (AVD), you may need to set kConvenientTestManagerHost
to 10.0.2.2
because of its network topology. Please see issue #253 for more details.
Video replaying
feature is not yet implemented in Android (but feel free to PR!).Author: fzyzcjy
Source: https://github.com/fzyzcjy/flutter_convenient_test
License: MIT license