wTest

We publish the wTest tool and the evaluation results in this page.

Subjects

Open-source apps (download)

The following table lists the 44 open-source Android apps used in our evaluation.

Index	Package	Index	Package
1	ch.admin.bag.covidcertificate.wallet	23	com.foobnix.pro.pdf.reader
2	info.plateaukao.einkbro	24	org.quantumbadger.redreader
3	ch.rmy.android.http_shortcuts	25	com.gelakinetic.mtgfam
4	io.github.wulkanowy	26	org.schabi.newpipe
5	com.activitymanager	27	com.gianlu.aria2app
6	me.ccrama.redditslide	28	org.secuso.privacyfriendlyweather
7	com.amaze.filemanager	29	com.github.doomsdayrs.apps.shosetsu.debug
8	ml.docilealligator.infinityforreddit	30	org.softeg.slartus.forpdaplus
9	com.boardgamegeek	31	com.github.k1rakishou.chan
10	net.bible.android.activity	32	org.wikipedia
11	com.cookiegames.smartcookie	33	com.ichi2.anki
12	net.cyclestreets	34	org.woheller69.weather
13	com.dar.nclientv2	35	com.joshuacerdenia.android.nicefeed
14	org.andstatus.app	36	com.manimarank.spell4wiki
15	com.ds.avare	37	com.nutomic.syncthingandroid
16	org.catrobat.paintroid	38	com.perflyst.twire
17	com.duckduckgo.mobile.android	39	com.tachibana.downloader
18	org.commonvoice.saverio	40	cz.martykan.forecastie
19	com.farmerbb.notepad	41	de.blau.android
20	org.koitharu.kotatsu	42	de.danoeh.antennapod
21	com.flxrs.dankchat	43	de.freehamburger
22	org.openhab.habdroid	44	de.westnordost.streetcomplete

Closed-source apps (download)

The following table lists the 30 closed-source Android apps used in our evaluation.

Index	Package	Index	Package
1	com.anydesk.anydeskandroid	16	com.til.timesnews
2	com.atomczak.notepat	17	com.trovit.android.apps.cars
3	com.bagatrix.mathway.android	18	com.usps
4	com.bestweatherfor.bibleoffline_pt_ra	19	com.waveline.nabd
5	com.cricbuzz.android	20	com.xb.topnews
6	com.cricbuzz.android.vernacular	21	com.xiaomi.midrop
7	com.indeed.android.jobsearch	22	com.yahoo.mobile.client.android.weather
8	com.marvel.comics	23	de.mobile.android.app
9	com.mobilefootie.wc2010	24	es.roid.and.trovit
10	com.naver.linewebtoon	25	jp.co.jorudan.nrkj
11	com.ne.hdv	26	jp.hotpepper.android.beauty.hair
12	com.playit.videoplayer	27	livio.pack.lang.en_US
13	com.pulselive.bcci.android	28	ru.litres.android
14	com.sec.app.samsungprintservice	29	teamDoppelGanger.SmarterSubway
15	com.shopping.limeroad	30	com.pl.premierleague

Coverage results

WebView-specific property coverage, WebView API call site coverage, and code coverage for open-source apps (link)
WebView-specific property coverage, WebView API call site coverage, and code coverage for closed-source apps (link)

Detected bugs

os-bug1 (link)
os-bug2 (link)
os-bug3 (link)
os-bug4 (link)
os-bug5 (link)
os-bug6 (link)
os-bug7 (link)
os-bug8 (link)
os-bug9 (link)
os-bug10 (link)
os-bug11 (link)
os-bug12 (link)
os-bug13 (link)
os-bug14 (link)
os-bug15 (link)
os-bug16 (link)
os-bug17 (link)
os-bug18 (link)
os-bug19 (link)
os-bug20 (link)
os-bug21 (link)
os-bug22 (link)
os-bug23 (link)
os-bug24 (org.catrobat.paintroid)
cs-bug1 (com.atomczak.notepat)
cs-bug2 (com.atomczak.notepat)
cs-bug3 (com.usps)
cs-bug4 (com.usps)
cs-bug5 (com.bestweatherfor.bibleoffline_pt_ra)
cs-bug6 (com.pl.premierleague)
cs-bug7 (com.pl.premierleague)
cs-bug8 (com.sec.app.samsungprintservice)
cs-bug9 (com.marvel.comics)
cs-bug10 (com.anydesk.anydeskandroid)
cs-bug11 (ru.litres.android)
cs-bug12 (ru.litres.android)

Tool

Our tool is composed of 2 parts. The first part instruments an Android app. The second part generates tests for the instrumented app. But before testing an app, a customized Android OS needs to be loaded by Android emulators or real devices.

Customized Android OS

The instrumented app needs to run on a customized Android system because we add one more ID field in java.lang.Object in order to ease the variable tracking. It is very simple to use our customized Android system image, you just need to follow the following 2 steps

Download our customized system.img, ramdisk.img, VerifiedBootParams.textproto (They are for Android 10)
Replace the original system.img, ramdisk.img, VerifiedBootParams.textproto in your Android SDK with the downloaded files

Usually, these files are placed under <Android_SDK_Root>/system-images/android-29/google_apis/x86

Once you have finished the above two steps, you can try to start the Android emulator as usual.

Instrumentation tool (download)

Requirements

Java 1.8 or above
Python 2 or 3 (We have tested on Python 2.7.16 and Python 3.6.8)
Node.js (We have tested on v16.13.2, v12.16.2, and v10.16.0. You may find these versions here. If you are using a Linux-based OS, you may install node by following the steps here)

Steps to use

Create a folder for an app under test. The folder name should be the app’s package name (We have created a folder for Wikipedia app under <path_to_Instrumentation>/apps).
Run sh instrument.sh APP_FOLDER APP_PACKAGE_NAME PORT_NUM APP_TYPE NON_LIB_PKGS to instrument an app. APP_TYPE can be OPEN_SOURCE or CLOSE_SOURCE. PORT_NUM is a port used by the instrumented app to communicate with a nodejs server to instrument dynamically-loaded JavaScript code. It is recommended to be set as 3016, 3018, 3020, …, etc. NON_LIB_PKGS is an array of package names separated by “,” that are used to filter out library code in an open-source app (e.g., if an app’s application code is in a.b.c and a.b.d, NON_LIB_PKGS should be a.b.c,a.b.d. you may use NA if the app is a closed-source app). An example command can be sh instrument.sh apps/org.wikipedia org.wikipedia 3016 OPEN_SOURCE org.wikipedia).
When instrumentation finishes, you can see an output folder under the APP_FOLDER. The apk file whose name ends with -aligned-debugSigned.apk is the instrumented apk.

Test generation (wTest) (download)

requirements

export JAVA_HOME=YOUR_JAVA_HOME (example: export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/)
export ANDROID_HOME=YOUR_ANDROID_HOME (example: export ANDROID_HOME=/Library/Android/sdk)
export PATH=$PATH:${ANDROID_HOME}
export PATH=$PATH:<path_to_wTest>/node_modules/.bin (make sure you have downloaded wTest)
export PATH=$PATH:${ANDROID_HOME}/emulator
export PATH=$PATH:${ANDROID_HOME}/platform-tools
build-tools 27.0.3 (for ComboDroid) and 29.0.3 are required (they are usually at ${ANDROID_HOME}/build-tools)

Steps to use

Enter <path_to_Instrumentation>/js and run sh launchServer.sh PORT_NUM. This nodejs server is registered on a PORT_NUM in order to recevice JavaScript code that is dynamically constructed in the app under test. The server is responsible for instrumenting the js code and sending the instrumented code back to the app. PORT_NUM should be the same as the one used when instrumenting the app. An example command can be sh launchServer.sh 3016
Launch Appium by appium -p APPIUM_PORT (example: appium -p 4723)
Launch an Android emulator
Enter <path_to_wTest> and run sh test.sh STRATEGY PATH_TO_APP_FOLDER TIME_LIMIT ANDROID_SDK_PATH EMULATOR_ID APPIUM_PORT APP_TYPE AVD_NAME BUILD_TOOLS_VERSION SYSTEM_PORT. STRATEGY can be wVar (stands for wTest), or wDroid, or api (stands for wTest-API), or QTesting, or ComboDroid, or Fastbot (stands for Fastbot2). AVD_NAME is the name of the android emulator. BUILD_TOOLS_VERSION must be set as 29.0.3. SYSTEM_PORT is a port used by appium if you test multiple apps in parallel. It is recommended to be set as 8200, 8201, 8202, etc. An example command can be sh test.sh wVar <path_to_Instrumentation>/apps/org.wikipedia 3600 ${ANDROID_HOME} emulator-5554 4723 OPEN_SOURCE emulator0 29.0.3 8200
When reaching the time limit, you can see the output under <path_to_wTest>/output/<STRATEGY>/<APP_PACKAGE_NAME> (e.g., <path_to_wTest>/output/wVar/org.wikipedia). Under this folder, coverage.txt contains the covered WebView-specific properties and WebView API call sites over 60 minutes (time0, time1, …, time59 means 1 minute, 2 minute, …, 60 minute). Code coverage is also reported.