Unable to get the full amount of comments

[Magicen0722]

Hi,I can't seem to get all the comment info and all the postings using this? I didn't find all the comments inside full_comment, here is the example: for post in get_posts('nintendo', pages=1, options={"comments": True, "reactors": True ,},extra_info=True) :
Within this posting (https://www.facebook.com/nintendo/posts/5417215251696217)It has 331 comments but only catches 30 or less, I've added a cookie.
set_cookies('facebook.com_cookies.txt')

[neon-ninja]

I'm unable to reproduce this issue. The code:

set_cookies("cookies.txt")
post = next(get_posts(post_urls=["https://www.facebook.com/nintendo/posts/5417215251696217"], options={"comments": True}))
print(f"Comments: {post['comments']}, Top level comments: {len(post['comments_full'])}, Replies: {sum(len(c['replies']) for c in post['comments_full'])}")

outputs:

Comments: 336, Top level comments: 113, Replies: 129

for me. Try update to version 0.2.57 with pip install -U facebook-scraper.

[Magicen0722]

I tested the code you provided and my results were consistent with yours, then after I upgraded the library to version 0.2.57, I retried the following code:

for post in get_posts('Nintendo', pages=2, options={"comments": True, "progress": True, "reactors": True},cookies=file):
    print(f "Post url:{post['post_url']}, Comments: {post['comments']}, Top level comments: {len(post['comments_full'])}, Replies: {sum(len(c[' replies']) for c in post['comments_full'])}")
    posts.append(post)

I got this, I tried to update the cookies and change the VM in a different region to execute it, and I got the same result as:

Post url:https://facebook.com/Nintendo/posts/5423676927716716, Comments: 20, Top level comments: 12, Replies: 4
Post url:https://facebook.com/Nintendo/posts/5420672481350494, Comments: 61, Top level comments: 0, Replies: 0
Post url:https://facebook.com/Nintendo/posts/5417215251696217, Comments: 338, Top level comments: 0, Replies: 0
Post url:https://facebook.com/Nintendo/posts/5408649879219421, Comments: 25, Top level comments: 0, Replies: 0
Post url:https://facebook.com/Nintendo/posts/5397266660357743, Comments: 77, Top level comments: 0, Replies: 0
Post url:https://facebook.com/Nintendo/posts/5391258890958520, Comments: 183, Top level comments: 0, Replies: 0

Do you have any solutions?Thanks for your help!

[neon-ninja]

I think it would be best to decouple these processes anyway, scrape posts first, then come back later for the comments using post_urls.

[Magicen0722]

I tried using
options={"comments": True}
the results were consistent, but strangely I executed each posting url separately
post = next(get_posts(post_urls["https://facebook.com/Nintendo/posts/5397266660357743"],options={"comments": True}))
it worked... Anyway thanks for the help!

[neon-ninja]

I think c0a01e6 would fix this issue. It seems I inadvertently caused this bug in 5882a5d. Give it a try and see how you go

[Magicen0722]

It still does not work under this use，There are even times when the comments_full field is missing.
I'm guessing maybe it's because I'm using a VPN and other proxy servers? But it still doesn't work on the servers in Western Europe.It still does not work under this use.

for post in get_posts('Nintendo', pages=2, options={"comments": True, "progress": True, "reactors": True},cookies=file):
            print(f"Post url:{post['post_url']}, Comments: {post['comments']}, Top level comments: {len(post['comments_full'])}, Replies: {sum(len(c[' replies']) for c in post['comments_full'])}")
            posts.append(post)

I now maintain two arrays to keep this function working,I extracted the url of the postings of a single account and then called the get_posts function again to use the url for individual crawling.

posts_url = []
posts_sup = []

for post in get_posts(user, options={"progress": True,}, cookies=file, pages=5):
  posts_url.append(post['post_url'])

print(len(posts_url))

def get_post(post_url):
  url = post_url
  for post in get_posts(post_urls=[url], cookies=file, options={"comments": True,
                                                                "progress": True,
                                                                "reactors": True, }, extra_info=True):
      print(post["comments"], len(post["comments_full"]))
      return post

for i in posts_url:
  print(i)
  posts_sup.append(get_post(i))