URLs rejected from file

[Jkuzz]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

"Clear bug in Crawlee"

As far as I understand any of "x.com" URLs rejected when uploaded from file: https://github.com/apify-projects/store-twitter/issues/95 for right now I guess only matters for twitter but please be aware just in case (edited)

Reported in slack

Code sample

No response

Package version

???

Node.js version

???

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

This regex fails:

https://github.com/apify/crawlee/blob/master/packages/utils/src/internals/general.ts#L8

[janbuchar]

This regex fails:

https://github.com/apify/crawlee/blob/master/packages/utils/src/internals/general.ts#L8

Will it slow us down a lot if we actually parse the URLs instead of half-assing it with regex?

[B4nan]

that regexp is used to parse list of URLs, not a single URL. i would guess using custom regexp will be actually slower in general, especially nowadays with the new ada URL parser in node, but the problem is harder on its own - the input might not be just a simple URL list, it could be a JSON too i think. IIRC we use the regexp to find any URL in any string.

[B4nan]

the regexp apparently expects at least 2 characters in the hostname