Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,35 @@ const prompt = 'Best restaurants near me';
})();
```

#### Time Range Filter

Use `timeRange` to filter search results by date range:

```javascript
import { searchScraper } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const prompt = 'Latest news about AI';

(async () => {
try {
const response = await searchScraper(apiKey, prompt, 5, null, null, {
timeRange: 'past_week' // Only results from the past week
});
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```

Available time range options:
- `past_hour` - Results from the past hour
- `past_24_hours` - Results from the past 24 hours
- `past_week` - Results from the past week
- `past_month` - Results from the past month
- `past_year` - Results from the past year

### Crawl API

Start a crawl job to extract structured data from a website and its linked pages, using a custom schema.
Expand Down Expand Up @@ -1145,6 +1174,7 @@ Searches and extracts information from multiple web sources using AI.
- `extractionMode` (boolean): Whether to use AI extraction
- `renderHeavyJs` (boolean): Whether to render heavy JavaScript
- `locationGeoCode` (string): The geo code of the location to search in (e.g., "us", "gb", "de")
- `timeRange` (string): The date range to filter results. Options: "past_hour", "past_24_hours", "past_week", "past_month", "past_year"
- `mock` (boolean): Override mock mode for this request

**Returns:** Promise that resolves to an object containing:
Expand Down
7 changes: 6 additions & 1 deletion src/searchScraper.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,12 @@ import { getMockResponse } from './utils/mockResponse.js';
* AI extraction costs 10 credits per page, markdown conversion costs 2 credits per page.
* @param {boolean} [options.stealth=false] - Enable stealth mode to avoid bot detection
* @param {string} [options.locationGeoCode=null] - The geo code of the location to search in (e.g., "us", "gb", "de")
* @param {string} [options.timeRange=null] - The date range to search in. Valid values: "past_hour", "past_24_hours", "past_week", "past_month", "past_year"
* @returns {Promise<string>} Extracted data in JSON format matching the provided schema
* @throws - Will throw an error in case of an HTTP failure.
*/
export async function searchScraper(apiKey, prompt, numResults = 3, schema = null, userAgent = null, options = {}) {
const { mock = null, renderHeavyJs = false, extractionMode = true, stealth = false, locationGeoCode = null } = options;
const { mock = null, renderHeavyJs = false, extractionMode = true, stealth = false, locationGeoCode = null, timeRange = null } = options;

// Check if mock mode is enabled
const useMock = mock !== null ? mock : isMockEnabled();
Expand Down Expand Up @@ -66,6 +67,10 @@ export async function searchScraper(apiKey, prompt, numResults = 3, schema = nul
payload.location_geo_code = locationGeoCode;
}

if (timeRange) {
payload.time_range = timeRange;
}

if (schema) {
if (schema instanceof ZodType) {
payload.output_schema = zodToJsonSchema(schema);
Expand Down