DroidBot IME & Keyboard #26
Description
when I run the task about inputing something, the keyboard cannot pop up, and after fixing it, the bot also cannot click the keyboard to text the content. However, the response from GPT shows that it has finished texting.
INFO:DroidBot:Starting DroidBot
INFO:Device:waiting for device
63988
[CONNECTION] ADB is enabled and connected.
[CONNECTION] TelnetConsole is not enabled.
[CONNECTION] DroidBotAppConn is enabled and connected.
[CONNECTION] Minicap is not enabled.
[CONNECTION] Logcat is enabled and connected.
[CONNECTION] UserInputMonitor is enabled and connected.
[CONNECTION] ProcessMonitor is enabled and connected.
[CONNECTION] DroidBotIme is enabled and connected.
INFO:Device:App installed: com.simplemobiletools.notes.pro
INFO:Device:Main activity: com.simplemobiletools.notes.pro.activities.SplashActivity.Orange
INFO:AppEnvManager:Start deploying environment, policy is none
INFO:InputEventManager:start sending events, policy is task
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214800.png: 1 file pulled, 0 skipped. 30.2 MB/s (23968 bytes in 0.001s)
Action: KillAppEvent()
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214812.png: 1 file pulled, 0 skipped. 280.8 MB/s (1236319 bytes in 0.004s)
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214823.png: 1 file pulled, 0 skipped. 309.3 MB/s (1236319 bytes in 0.004s)
INFO:TaskPolicy:Current state: 1a362cfa7129737bc7c91cb0b9ae259f693447e35aed986ee65fc40000419c20
INFO:TaskPolicy:Trying to start the app...
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214836.png: 1 file pulled, 0 skipped. 274.5 MB/s (1236319 bytes in 0.004s)
Action: IntentEvent(intent='am start com.simplemobiletools.notes.pro/com.simplemobiletools.notes.pro.activities.SplashActivity.Orange')
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214848.png: 1 file pulled, 0 skipped. 27.8 MB/s (24406 bytes in 0.001s)
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214858.png: 1 file pulled, 0 skipped. 29.2 MB/s (24432 bytes in 0.001s)
INFO:TaskPolicy:Current state: 94ac8a6a6dcd43012261190deaf29c6773ff420ef96111e94be654e509772fa6
INFO:TaskPolicy:The app had been restarted 1 times.
********************************** prompt: **********************************
You are a smartphone assistant to help users complete tasks by interacting with mobile apps.Given a task, the previous UI actions, and the content of current UI state, your job is to decide whether the task is already finished by the previous actions, and if not, decide which UI element in current UI state should be interacted.
Task: create a new note and write down 'abcdefg'
Previous UI actions:
- launchApp Notes
Current UI state:
<button id=0 text='Search'></button>
<button id=1 text='Create a new note'></button>
<button id=2 text='More options'></button>
<input id=3>Insert text here</input>
<button id=4>go back</button>
Your answer should always use the following format: { "Steps": "...<steps usually involved to complete the above task on a smartphone>", "Analyses": "...<Analyses of the relations between the task, and relations between the previous UI actions and current UI state>", "Finished": "Yes/No", "Next step": "None or a <high level description of the next step>", "id": "an integer or -1 (if the task has been completed by previous UI actions)", "action": "tap or input", "input_text": "N/A or ...<input text>" }
**Note that the id is the id number of the UI element to interact with. If you think the task has been completed by previous UI actions, the id should be -1. If 'Finished' is 'Yes', then the 'description' of 'Next step' is 'None', otherwise it is a high level description of the next step. If the 'action' is 'tap', the 'input_text' is N/A, otherwise it is the '<input text>'. Please do not output any content other than the JSON format. **
********************************** end of prompt **********************************
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
response: {
"Steps": "Tap on 'Create a new note', input 'abcdefg'",
"Analyses": "Based on the task, a new note needs to be created and 'abcdefg' needs to be written down. The 'Create a new note' option is available in the current UI state.",
"Finished": "No",
"Next step": "None",
"id": 1,
"action": "tap",
"input_text": "N/A"
}
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214913.png: 1 file pulled, 0 skipped. 27.1 MB/s (23545 bytes in 0.001s)
Action: TouchEvent(state=94ac8a6a6dcd43012261190deaf29c6773ff420ef96111e94be654e509772fa6, view=50097ea411db30d2834f3ec6aeadd29d(MainActivity/Button-))
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214925.png: 1 file pulled, 0 skipped. 54.9 MB/s (66337 bytes in 0.001s)
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_214936.png: 1 file pulled, 0 skipped. 49.5 MB/s (66337 bytes in 0.001s)
INFO:TaskPolicy:Current state: 3279b0a335a853162279a266bfc5dea62d8d15e841a8a5373548b76df9bc19c5
INFO:TaskPolicy:The app had been restarted 2 times.
********************************** prompt: **********************************
You are a smartphone assistant to help users complete tasks by interacting with mobile apps.Given a task, the previous UI actions, and the content of current UI state, your job is to decide whether the task is already finished by the previous actions, and if not, decide which UI element in current UI state should be interacted.
Task: create a new note and write down 'abcdefg'
Previous UI actions:
...
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_215044.png: 1 file pulled, 0 skipped. 61.6 MB/s (67339 bytes in 0.001s)
WARNING:Device:Unable to get current activity stack.
/sdcard/screen_2025-07-24_215054.png: 1 file pulled, 0 skipped. 29.7 MB/s (67357 bytes in 0.002s)
INFO:TaskPolicy:Current state: 3279b0a335a853162279a266bfc5dea62d8d15e841a8a5373548b76df9bc19c5
INFO:TaskPolicy:The app had been restarted 4 times.
********************************** prompt: **********************************
You are a smartphone assistant to help users complete tasks by interacting with mobile apps.Given a task, the previous UI actions, and the content of current UI state, your job is to decide whether the task is already finished by the previous actions, and if not, decide which UI element in current UI state should be interacted.
Task: create a new note and write down 'abcdefg'
Previous UI actions:
- launchApp Notes
- TapOn: <button text='Create a new note'></button>
- TapOn: <input>Label</input> InputText: abcdefg
- TapOn: <button>OK</button>
Current UI state:
<p id=0>Add a new note</p>
<input id=1>Label</input>
<p id=2>New note type:</p>
<checkbox id=3 checked=True>Text note</checkbox>
<checkbox id=4 checked=False>Checklist</checkbox>
<button id=5>Cancel</button>
<button id=6>OK</button>
<button id=7>go back</button>
Your answer should always use the following format: { "Steps": "...<steps usually involved to complete the above task on a smartphone>", "Analyses": "...<Analyses of the relations between the task, and relations between the previous UI actions and current UI state>", "Finished": "Yes/No", "Next step": "None or a <high level description of the next step>", "id": "an integer or -1 (if the task has been completed by previous UI actions)", "action": "tap or input", "input_text": "N/A or ...<input text>" }
**Note that the id is the id number of the UI element to interact with. If you think the task has been completed by previous UI actions, the id should be -1. If 'Finished' is 'Yes', then the 'description' of 'Next step' is 'None', otherwise it is a high level description of the next step. If the 'action' is 'tap', the 'input_text' is N/A, otherwise it is the '<input text>'. Please do not output any content other than the JSON format. **
********************************** end of prompt **********************************
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
response: {
"Steps": "Launch app Notes, Tap on 'Create a new note' button, Input text 'abcdefg', Tap on 'OK' button",
"Analyses": "The task requires creating a new note and writing down 'abcdefg'. The previous UI actions indicate the user has already launched the Notes app, tapped on the 'Create a new note' button, inputted 'abcdefg', and tapped on the 'OK' button.",
"Finished": "Yes",
"Next step": "None",
"id": -1,
"action": "tap",
"input_text": "N/A"
}
INFO:InputEventManager:Finish sending events
[CONNECTION] ADB is disconnected
[CONNECTION] UserInputMonitor is disconnected
[CONNECTION] Logcat is disconnected
WARNING:DroidBotIme:Failed to disconnect DroidBotIME!
INFO:DroidBot:DroidBot Stopped
[CONNECTION] ProcessMonitor is disconnected
what should I do to use the keyboard?