ct0 not in cookies (most likely ip ban)

[HazardCodeBolt]

Lately,
The following error has been appearing to me when I try to log in using twscrape login_accounts.

2024-08-14 12:15:33.159 | ERROR | twscrape.accounts_pool:login:165 - Failed to login 'XAccount': ct0 not in cookies (most likely ip ban)

When I logged in to the account I found out it wasn't blocked.

I have solved that issue for some accounts by logging in manually and then removing the accounts from the database using twscrape del_accounts XAccount .... Then log in to the account manually. Then add the account using twscrape add_accounts ..., then try re-login again using twscrape relogin XAccount ....

It solved the issue for me, but I have to do that manually whenever I face it again. Can we consider automating that?

I have found out that sometimes Twitter requests the email in the middle of your login even if you typed the account name.
Maybe that's something new to the Twitter login process.

I don't know exactly if that's an issue with me or with the login process. But I am here to learn.

[HazardCodeBolt]

A note: it takes time from logging in manually to twscrape accepting the login, about 15 minutes.

[frameartist]

Guys I found the solution

Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id>
The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.

Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

[frameartist]

Guys I found the solution

Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id> The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.

Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

Forgot to mention I added a few print statements for debugging purposes. These print statements shouldn't affect the functionality and can be ignored

[caterpillar1219]

@frameartist Thanks for sharing your finding! I also found https://antibot.blog/twitter/ where someone investigates this mystic "x-client-transaction-id" header. This is also consistent with @Jwom's finding in #175 (comment). FWIW this header is very likely to be related to account bans. There are also some cloned repos e.g. https://github.com/yeyuchen198/twitter-tid-generator. Highly appreciated if someone can create a python version.

[Jwom]

收到

[bachelow]

Guys I found the solution

Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id> The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.

Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

@frameartist Thanks you very much. Based on your answer I modified the login function this way and it worked for me

  async` def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
      log_id = f"{acc.username} - {acc.email}"
      if acc.active:
          logger.info(f"account already active {log_id}")
          return acc
    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        while True:
            rep = await next_login_task(ctx, rep)
            if not rep:
                break

        # assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = "your ct0 string"
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

I have a dozen of accounts and I took one ct0 string from one random logged account using F12 too. It worked for all accounts.

[imperatrona]

@caterpillar1219 someone on twitter have read this article and changed hash salt from "bird" to "obfiowerehiring" (obfio is article author username on github), the script doesn't work anymore. they changed last byte from 1 to 3, some of the inputs like targetTime, and something else cause i can't manage to fix it

[takabinance]

So @frameartist is hard-coding x-client-transaction-id and @bachelow is hard-coding x-csrf-token.
Are both of these still working - same code over multiple accounts, still alive after a few days?

[bachelow]

So @frameartist is hard-coding x-client-transaction-id and @bachelow is hard-coding x-csrf-token. Are both of these still working - same code over multiple accounts, still alive after a few days?

For me yes, but it become unmanageable with more than 4/5 accounts.

[takabinance]

Why unmanageable? I read that you could use the same x-csrf-token on multiple accounts. Did I read that wrong? On Aug 23, 2024, at 5:14 AM, bachelow ***@***.***> wrote: So @frameartist is hard-coding x-client-transaction-id and @bachelow is hard-coding x-csrf-token. Are both of these still working - same code over multiple accounts, still alive after a few days? For me yes, but it become unmanageable with more than 4/5 accounts. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[sangeenrapidlabs]

Guys I found the solution

Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id> The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.

Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

Your provided solution works great, but I am receiving another issue now Session expired or banned 403 - -1/-1 - SAliukum31074 - OK. The error is in the file queue_client. Has anyone solved this issue or come accross it?

[bachelow]

Your provided solution works great, but I am receiving another issue now Session expired or banned 403 - -1/-1 - SAliukum31074 - OK. The error is in the file queue_client. Has anyone solved this issue or come accross it?

Yeah that's why I said earlier it was tedious with more than 4/5 accounts. To answer your remark and the one of @takabinance I had to relog using twscrape relogin account several time at the start of my scraping (usually 2 or 3 times) and it ends up working.

[E-geek]

Ok, I found the solution, but I'm not sure about the quality.
Anyway: yes, header x-client-transaction-id is required. And this header should generate every request.
Good news: function for generation incomes only 2 args: URI and Method. The function returns base64 string without ends '=' letters.
Bad news: the internal part of the function is very strange and are applied many tricks to avoid deobfuscation. This is possible BUT(!)
The easiest way is using node.js (in CLI mode) and just requesting a URI and method before every request by the client and adding the header. If you want, I can write the simplest node CLI to generate the header.

[Jwom]

收到

[kqvanity]

Guys I found the solution

Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id> The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.

Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

It's working for me as well. maybe a PR?

[frameartist]

Guys I found the solution
Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id> The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.
Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

It's working for me as well. maybe a PR?

The problem is we need to hardcode the id, which apparently won't last forever. So I'd prefer filing a PR once we reverse engineer the whole mechanism.

[malone6]

Guys I found the solution

Go to <your venv or python installation>/site-packages/twscrape/login.py around line 264 in the function async def login(acc: Account, cfg: LoginConfig | None = None) -> Account: after client.headers["x-guest-token"] = guest_token, add a new line client.headers["x-client-transaction-id"] = <your client transaction id> The complete function looks like this:

async def login(acc: Account, cfg: LoginConfig | None = None) -> Account:
    log_id = f"{acc.username} - {acc.email}"
    if acc.active:
        logger.info(f"account already active {log_id}")
        return acc

    cfg, imap = cfg or LoginConfig(), None
    if cfg.email_first and not cfg.manual:
        imap = await imap_login(acc.email, acc.email_password)

    async with acc.make_client() as client:
        guest_token = await get_guest_token(client)
        client.headers["x-guest-token"] = guest_token
        client.headers["x-client-transaction-id"] = "<a base64 string>"

        rep = await login_initiate(client)
        ctx = TaskCtx(client, acc, cfg, None, imap)
        print("-------------------")
        print(rep.content)
        print()
        while True:
            rep = await next_login_task(ctx, rep)
            print(rep.content if rep else "No response")
            print()
            if not rep:
                break

        assert "ct0" in client.cookies, "ct0 not in cookies (most likely ip ban)"
        client.headers["x-csrf-token"] = client.cookies["ct0"]
        client.headers["x-twitter-auth-type"] = "OAuth2Session"

        acc.active = True
        acc.headers = {k: v for k, v in client.headers.items()}
        acc.cookies = {k: v for k, v in client.cookies.items()}
        return acc

To obtain a client transaction id, use chrome to login any twitter account then use the F12 developer tools to randomly grab one from the network records. It seems like once you have a valid one you'll be able to login all accounts, so it's quite convenient once you setup everything.

Read this stackoverflow for more about how this x-client-transaction-id is generated. I don't have time to reverse engineer this but I hope someone could invent a way to generate these ids automatically in Python

I tried this method, this base64 string was grab from netword record in chrome（same account）, but it still didn't work.
And I found "x-client-transaction-id" is diffrent between network records, is the extra verification already enabled on this id?

[mattpilleul]

Ok, I found the solution, but I'm not sure about the quality. Anyway: yes, header x-client-transaction-id is required. And this header should generate every request. Good news: function for generation incomes only 2 args: URI and Method. The function returns base64 string without ends '=' letters. Bad news: the internal part of the function is very strange and are applied many tricks to avoid deobfuscation. This is possible BUT(!) The easiest way is using node.js (in CLI mode) and just requesting a URI and method before every request by the client and adding the header. If you want, I can write the simplest node CLI to generate the header.

Hi! Any chance to share your solution ?