Skip to content

rvsmart: fix exploration bugs and improve static analysis usage #35

Description

@phtcosta

Problem Statement

Deep trace analysis of the comparison experiment (100 APKs, 600s, 3 reps) revealed 17 issues in rvsmart's exploration algorithm that collectively explain the -4.39pp method coverage gap vs APE (p<0.001).

The most critical finding: BACK action is never executed (0 of 251,081 actions) due to a hash domain mismatch in ActionSelector. Additionally, 24.5% of all iterations are wasted on ping-pong cycles with no detection, and static analysis data is parsed at activity-level when widget-level data is available in the JSON.

Affected Domains

  • Tools (rvsmart Java agent)
  • Agent
  • Platform
  • Experiment
  • Core
  • Instrumentation
  • Analysis

Scope

Bugs (6)

  • BUG-01 (CRITICAL): BACK permanently disabled — getParents(contentHash) vs record(structHash) mismatch in ActionSelector
  • BUG-02 (CRITICAL): No cycle detection — 2-state ping-pong wastes 24.5% of iterations (61,470 total)
  • BUG-03 (HIGH): getSaturationRate() returns 1.0 when totalActions=0 → infinite RESTART loops
  • BUG-04 (HIGH): Stuck recovery always RESTARTs to same screen (no escape for single-activity apps)
  • BUG-05 (MEDIUM): Preference screen content hash explosion traps Phase 1
  • BUG-06 (MEDIUM): SystemDialogDetector only recognizes 4 system packages

Algorithm Gaps (4)

  • GAP-1 (HIGH): isClusterForced() is dead code (never called by ActionSelector)
  • GAP-2 (MEDIUM): NavigationMap limited to observed transitions
  • GAP-3 (LOW): 3-failure filter excludes transient failures permanently from Phase 1
  • GAP-4 (MEDIUM): Saturation based on execution count, not success count

Saturation/Tracking (3)

  • SAT-1: recordAction() called BEFORE executeAction() — action counted even on silent failure
  • SAT-2: Widget marked saturated with 4 failed attempts (success ignored)
  • SAT-3: isInteractive() excludes longClickable — invisible to LLM prompt

Static Analysis Data Loss (4)

  • SA-1: Widget-level MOP scoring (handlers per-widget available but discarded)
  • SA-2: WTG-guided widget targeting (which widget opens which activity)
  • SA-3: Static inputType for InputValueGenerator
  • SA-4: Handler-level navigation hints for LLM

Acceptance Criteria

  • BUG-01: BACK action is executed in >50% of APKs (currently 1/100)
  • BUG-02: Ping-pong cycles detected and broken within 10 iterations
  • BUG-03: No APKs with 100% RESTART rate due to saturation bug
  • BUG-04: Single-activity apps use exhaustive interaction instead of RESTART loop
  • BUG-05: Time in Preferences activities limited
  • BUG-06: SystemDialogDetector handles Google Play Services, permissioncontroller
  • GAP-1: isClusterForced() wired into ActionSelector
  • GAP-2: Periodic random exploration of unobserved transitions
  • GAP-3: Failure filter uses time-decayed count instead of absolute
  • GAP-4: Saturation uses success count, not execution count
  • SAT-1: recordAction() moved to after execution
  • SAT-2: Saturation threshold considers success rate
  • SAT-3: isInteractive() includes longClickable
  • SA-1: MopScorer uses widget-level handler data
  • SA-2: WtgScorer targets specific widgets for activity transitions
  • SA-3: InputValueGenerator falls back to static inputType
  • SA-4: LLM prompt includes handler-level hints
  • Docker image rebuilt and tested on 15 worst APKs
  • Method coverage gap vs APE reduced by ≥3pp

Priority

High — blocks further calibration experiments.

Track

Quick Path (bug fixes + mechanical improvements, no design decisions)

References

  • Analysis: docs/20260310_comparacao_resultados.md (Sections 8-18)
  • Source: rvsec/rvsec-android/rvsmart/src/main/java/br/unb/cic/rvsmart/
  • Experiment data: data/results/cmp01-cmp08/

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions