Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meterpreter Migrate #12390

Closed
akefallonitis opened this issue Oct 3, 2019 · 28 comments · Fixed by #12502
Closed

Meterpreter Migrate #12390

akefallonitis opened this issue Oct 3, 2019 · 28 comments · Fixed by #12502
Assignees
@bwatters-r7
Labels
bug confirmed Issues confirmed by a committer

Comments

@akefallonitis
Copy link

akefallonitis commented Oct 3, 2019 •
edited
Loading

When i try migrate on metepreter the operation timeouts and then my shell breaks.
Running in high integrate context and as system.

MSF latest version
meterpreter x64 running shellcode on Windows 10 latest build
No AV No Firewall and No EDR running

Anyone having the same issue ?
@timwr
Copy link
Contributor

timwr commented Oct 3, 2019

I think this is a known "issue"

@akefallonitis
Copy link
Author

I cannot find any new issue regarding the latest 5.0.50-dev and migrate please assist

@bwatters-r7 bwatters-r7 self-assigned this Oct 8, 2019
@bwatters-r7
Copy link
Contributor

After bisecting this, it looks like a regression introduced in 47afa21

Looking at that briefly, I don't understand why? I need to dig some more, though, so I assigned this to myself.

@bwatters-r7 bwatters-r7 added confirmed Issues confirmed by a committer bug labels Oct 8, 2019
@OJ
Copy link
Contributor

OJ commented Oct 8, 2019 via email

@bwatters-r7
Copy link
Contributor

I mean..... I'm hopping between commits 540e269 and 47afa21. The the results are consistent....

540e269 migrates successfully

47afa21 times out, but occasionally (10%-20%) calls back after a timeout, and it is running in the migrated process.

https://media.giphy.com/media/BcpsfmGhjAwcE/giphy.gif

@OJ
Copy link
Contributor

OJ commented Oct 8, 2019 via email

@bwatters-r7
Copy link
Contributor

OK... I feel a little better; it turns out that this might be a intermittent problem, rather than a definitive problem, which explains why bisect might have landed on this commit (though I tested it three times manually on each side, and it was repeatable manually)

I set up automated testing and got these results (commit hashes listed at the top)

Commit before 47afa21

image

Commit 47afa21

image

I am pretty sure this is a regression, as all tests fail on master...

Commit a6ec513 (Current Master):

image

@bwatters-r7
Copy link
Contributor

OK.... after running a ton of tests, I think I may have found the real slim shady.

Commit 760defd AKA: #11184

image

Commit 880e597 AKA: #11176 (Preceding land)

image

I also tested the other two lands in either direction. Everything after 760defd had complete failures, everything before had partial success. Now to figure out why the Ruby bump made the difference?

@OJ
Copy link
Contributor

OJ commented Oct 11, 2019 via email

@bwatters-r7
Copy link
Contributor

bwatters-r7 commented Oct 11, 2019 •
edited
Loading

Here's the log:

[10/11/2019 15:10:18] [e(0)] meterpreter: Error running command migrate: Rex::TimeoutError Operation timed out.
[10/11/2019 15:10:18] [d(0)] meterpreter: Call stack:
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/packet_dispatcher.rb:177:in `send_request'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:747:in `negotiate_tlv_encryption'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/client_core.rb:691:in `migrate'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console/command_dispatcher/core.rb:1222:in `cmd_migrate'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:105:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in `block in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `each'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:68:in `block in interact'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:146:in `run'
/home/tmoose/rapid7/metasploit-framework/lib/rex/post/meterpreter/ui/console.rb:66:in `interact'
/home/tmoose/rapid7/metasploit-framework/lib/msf/base/sessions/meterpreter.rb:576:in `_interact'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/interactive.rb:51:in `interact'
/home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1364:in `cmd_sessions'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in `block in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `each'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `run_single'
/home/tmoose/rapid7/metasploit-framework/lib/msf/ui/console/command_dispatcher/exploit.rb:215:in `cmd_exploit'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:523:in `run_command'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:474:in `block in run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `each'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:468:in `run_single'
/home/tmoose/rapid7/metasploit-framework/lib/rex/ui/text/shell.rb:151:in `run'
/home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
/home/tmoose/rapid7/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
./msfconsole:49:in `<main>'

Though not particularly helpful, TBH.

@OJ
Copy link
Contributor

OJ commented Oct 11, 2019 via email

@OJ
Copy link
Contributor

OJ commented Oct 11, 2019 via email

@bwatters-r7
Copy link
Contributor

The target process has never crashed.

@OJ
Copy link
Contributor

OJ commented Oct 11, 2019 via email

@an1mehacker
Copy link

an1mehacker commented Oct 13, 2019 •
edited
Loading

I had the EXITFUNC set to thread and this error happened. When I changed it to process it's fine. Not sure if this helps in any way tho.

@jeffmcjunkin
Copy link
Contributor

This still affects Windows 10 Enterprise (version 1903) and Metasploit 5.0.55-dev:

meterpreter > run migrate -n spoolsv.exe

[!] Meterpreter scripts are deprecated. Try post/windows/manage/migrate.
[!] Example: run post/windows/manage/migrate OPTION=value [...]
[*] Current server process: Icecast2.exe (1264)
[+] Migrating to 1940

[-] Could not migrate in to process.
[-] Operation timed out.
meterpreter > ps
[-] Error running command ps: Rex::TimeoutError Operation timed out.
meterpreter > exit
[*] Shutting down Meterpreter...

[*] 10.10.10.20 - Meterpreter session 1 closed.  Reason: User exit

msf5 exploit(windows/http/icecast_header) > version
Framework: 5.0.55-dev-
Console  : 5.0.55-dev-

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019

I'll try to fix this soon!

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019

Quick investigation shows that http-based transports are fine. Migration works as expected.

reverse_tcp is busted. The migration happens behind the scenes, and the first packet to go out over the wire once the socket has been taken over by the new process is the TLV encryption packet. The new session in the target process correctly creates an AES key and writes the response to the socket, however on the MSF side, no data is received. MSF just sits there and waits for data on the socket, and nothing happens.

Not sure why yet.

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019 •
edited
Loading

Definitely something strange going on with the Ruby side. I honestly don't know what it is. This is a pic of the Windows box. We can see that the TLV encryption response packet is actually going out on the wire and we get an ACK from MSF:

wut

But on the MSF side, I can see that select is being called on the socket as part of the typical wait/dispatch loop, but no data is coming through on it.

Going to have to call on some Ruby gurus to help out with this one @wvu-r7 @bcook-r7 and whoever else is a master of the Rubies. I honestly don't know what's going on. The socket is still open, data is sent to it from Meterp, and it has been ACK'd. But as far as MSF is concerned the select call doesn't return with any data and so the wait just times out.

Does anyone know of anything that may have changed in Ruby 2.6+ that could cause this kind of behaviour?

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019 

rv = Rex::ThreadSafe.select([ self.sock.fd ], nil, nil, PING_TIME)

This is the line, in packet_dispatcher.rb which times out when it shouldn't. I haven't dived into Rex::ThreadSafe.select, but perhaps there's something in there that's causing the problem?

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019

I changed the above line to use ::IO.select instead, and it didn't help. So something else is up.

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019

Also, FWIW, if you have a handler open and listening, exiting the current session will result in Meterp reconnecting and from there everything works fine.

@OJ
Copy link
Contributor

OJ commented Oct 22, 2019

Confirmed that reverting to Ruby 2.5.1 results in everything working perfectly.

Stop the world I want to get off

@OJ
Copy link
Contributor

OJ commented Oct 23, 2019

Confirmed broken in 2.5.7 ! Doing the painful binary search to see which version killed it.

@OJ
Copy link
Contributor

OJ commented Oct 23, 2019

OK after a painful binary search, I can say that:

  • Ruby 2.5.4 works on my machine
  • Ruby 2.5.5 does not work on my machine

So I'm going to say that changes introduced in Ruby 2.5.5 are causing the problem. The thing is, it doesn't really look like anything changed in the source could have caused it. The Release Notes don't really imply much either.

So I'm at a bit of a loss. HALP PLZ!

@OJ
Copy link
Contributor

OJ commented Oct 24, 2019

It is interesting that the issue manifests in a case where multiple entities are working on the socket, indicating that it could be related to threading. I'd be keen to see if patching out that line in the latest ruby would result in things working.

But yeah, not sure where to go from here.

@bwatters-r7
Copy link
Contributor

Yeah. I took a look into it the other day and I asked a few rubier people than me, and there's nothing that stuck out as obvious.

@busterb
Copy link
Member

busterb commented Oct 28, 2019

I started taking a look at this last night, and have a working diff on Ruby 2.6.5. Looks like some assumptions about thread lifetimes need stronger guarantees; MSF might have been relying on undefined behavior for a while.

@busterb busterb mentioned this issue Oct 28, 2019
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bwatters-r7 bwatters-r7

Labels
bug confirmed Issues confirmed by a committer
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix process migration on reverse_tcp meterpreter sessions w/ newer Ruby busterb/metasploit-framework
7 participants
@OJ @timwr @jeffmcjunkin @busterb @bwatters-r7 @akefallonitis @an1mehacker