Skip to content

feat(jobs): add twitter-scraper#13

Merged
mudler merged 10 commits intomainfrom
feat/twitter-scraper
Dec 5, 2024
Merged

feat(jobs): add twitter-scraper#13
mudler merged 10 commits intomainfrom
feat/twitter-scraper

Conversation

@mudler
Copy link
Contributor

@mudler mudler commented Nov 26, 2024

Signed-off-by: mudler <mudler@localai.io>
Signed-off-by: mudler <mudler@localai.io>
Signed-off-by: mudler <mudler@localai.io>
"github.com/sirupsen/logrus"
)

type TweetResult struct {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is extracted/adapted from the masa-oracle implementation

return tweets, nil
}

// End of adapted code from masa-oracle (commit: bf277c646d44c49cc387bc5219c900e96b06dc02)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this file,adaptation ends here

@@ -0,0 +1,45 @@
package twitter
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this package was entirely copied as-is. no changes introduced

@mudler
Copy link
Contributor Author

mudler commented Nov 26, 2024

Adapted on masa-finance/masa-oracle#638

Signed-off-by: mudler <mudler@localai.io>
@mudler mudler force-pushed the feat/twitter-scraper branch from c0072a2 to a490356 Compare November 26, 2024 17:59
Signed-off-by: mudler <mudler@localai.io>
@mudler
Copy link
Contributor Author

mudler commented Nov 26, 2024

TODO: add twitter tests

@codecov
Copy link

codecov bot commented Nov 26, 2024

Codecov Report

Attention: Patch coverage is 13.47518% with 122 lines in your changes missing coverage. Please review.

Project coverage is 43.90%. Comparing base (8af6478) to head (102d635).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
internal/jobs/twitter.go 0.00% 121 Missing ⚠️
internal/jobserver/worker.go 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main      #13       +/-   ##
===========================================
- Coverage   57.35%   43.90%   -13.45%     
===========================================
  Files           6        7        +1     
  Lines         340      476      +136     
===========================================
+ Hits          195      209       +14     
- Misses        124      245      +121     
- Partials       21       22        +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: mudler <mudler@localai.io>
@mudler mudler force-pushed the feat/twitter-scraper branch from edf48e5 to 446f675 Compare November 27, 2024 08:55
Signed-off-by: mudler <mudler@localai.io>
Signed-off-by: mudler <mudler@localai.io>
@mudler mudler self-assigned this Dec 2, 2024
Signed-off-by: mudler <mudler@localai.io>
Signed-off-by: mudler <mudler@localai.io>
@mudler mudler marked this pull request as ready for review December 5, 2024 17:36
@mudler mudler merged commit 72a24df into main Dec 5, 2024
@mudler mudler deleted the feat/twitter-scraper branch December 5, 2024 18:02
teslashibe pushed a commit that referenced this pull request Jul 14, 2025
* feat(jobs): add twitter scraper job type

Signed-off-by: mudler <mudler@localai.io>

* chore: wire-up twitter config

Signed-off-by: mudler <mudler@localai.io>

* chore: do not re-create scrapers each time

Signed-off-by: mudler <mudler@localai.io>

* Adapt twitter code to latest changes

Signed-off-by: mudler <mudler@localai.io>

* chore(fix): populate jobWorkers

Signed-off-by: mudler <mudler@localai.io>

* chore(tests): add simple twitter test

Signed-off-by: mudler <mudler@localai.io>

* chore(tests): increase test area

Signed-off-by: mudler <mudler@localai.io>

* chore(tests): store cookies to cache

Signed-off-by: mudler <mudler@localai.io>

* correctly map cookie dir

Signed-off-by: mudler <mudler@localai.io>

* Skip Twitter tests

Signed-off-by: mudler <mudler@localai.io>

---------

Signed-off-by: mudler <mudler@localai.io>
grantdfoster added a commit that referenced this pull request Oct 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move scraper code to tee-worker repository

1 participant