Skip to content
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.

Commit

Permalink
rename the package
Browse files Browse the repository at this point in the history
  • Loading branch information
jeffmcaffer committed Mar 20, 2017
1 parent d76a9f2 commit e74890f
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 19 deletions.
34 changes: 19 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,36 @@
# Crawler command line

[GHCrawler](https://github.com/Microsoft/ospo-ghcrawler.git) is utility for walking GitHub APIs and tracking GitHub events. This command line app allows you to control various
aspects of a crawler's behavior. There is some overlap in function with the [Crawler Dashboard](https://github.com/Microsoft/crawler-dashboard.git). This project also has a simple Node client library for talking to a crawler.
[GHCrawler](https://github.com/Microsoft/ghcrawler.git) is utility for walking GitHub APIs and tracking GitHub events. This command line app allows you to control various aspects of a crawler's behavior. There is some overlap in function with the [Crawler Dashboard](https://github.com/Microsoft/ghcrawler-dashboard.git). This project also has a simple Node client library for talking to a crawler.

# Controlling a crawler

The ```cc``` utility is in the ```bin``` directory of this repo. It can be run interactively or as a single command processor. The general format of using the command line is
The `cc` utility is in the `bin` directory of this repo. It can be run interactively or as a single command processor. The general format of using the command line is

```node cc [options] [command]```
```
node cc [options] [command]
```

where the available options are:

```-i``` -- Run in interactive mode
`-i` -- Run in interactive mode

```-s <url>``` -- Control the crawler service running at the given url. Defaults to http://localhost:3000. You can also set the CRAWLER_SERVICE_URL environment variable.
`-s <url>` -- Control the crawler service running at the given url. Defaults to http://localhost:3000. You can also set the CRAWLER_SERVICE_URL environment variable.

```-t``` -- The crawler service API token to use. This can also be supplied via the ```CRAWLER_SERVICE_AUTH_TOKEN``` environment variable. If not defined in either place, the default ```"secret"``` value is used.
`-t` -- The crawler service API token to use. This can also be supplied via the `CRAWLER_SERVICE_AUTH_TOKEN` environment variable. If not defined in either place, the default value of `secret` is used.

and the available commands are:
The available commands are:

```start [count]``` -- Start the crawler processing with count concurrent operations. If count is not specified, 1 is the default. On a reasonably fast network a count of 10 to 15 should be sufficient. This also depends on how many tokens you are using.
`start [count]` -- Start the crawler processing with count concurrent operations. If count is not specified, 1 is the default. On a reasonably fast network a count of 10 to 15 should be sufficient. This also depends on how many tokens you are using.

```stop``` -- Stop crawler processing. The crawler service is left running but it stops pulling requests off the queue.
`stop` -- Stop crawler processing. The crawler service is left running but it stops pulling requests off the queue. This is the same as `start 0`.

```queue <requests...>``` -- Queues the given requests for processing. The requests parameter is a list of GitHub "org" and/or "org/repo" names.
`queue <requests...>` -- Queues the given requests for processing. The requests parameter is a list of GitHub "org" and/or "org/repo" names.

```orgs <org orgs...>``` -- Set the crawler's to traverse only the GitHub orgs named in the given list.
`orgs <org orgs...>` -- Set the crawler's to traverse only the GitHub orgs named in the given list.

```config``` -- Dumps the crawler service's configuration to the console.
`config` -- Dumps the crawler service's configuration to the console.

```tokens <spec...>``` -- Set the GitHub tokens to be used by the crawler when calling GitHub APIs. The spec value is a list of token specs. Each spec has the form <token>#<trait>,<trait>... where the token is the GitHub OAuth or Personal Access token and the comma-separated list of traits identify what permissions the token has. The available traits are: ```public```, ```admin```, ```private```. You can list as many tokens and traits as you like. Note that you can also configure the GitHub tokens the CRAWLER_GITHUB_TOKENS environment variable instead **before starting the crawler**. For example, ```export CRAWLER_GITHUB_TOKENS="<token1>#public <token2>#admin"```.
`tokens <spec...>` -- Set the GitHub tokens to be used by the crawler when calling GitHub APIs. The spec value is a list of token specs. Each spec has the form `<token>#<trait>,<trait>...` where the token is the GitHub OAuth or Personal Access token and the comma-separated list of traits identify what permissions the token has. The available traits are: `public`, `admin`, `private`. You can list as many tokens and traits as you like. Note that you can also configure the GitHub tokens the `CRAWLER_GITHUB_TOKENS` environment variable instead **before starting the crawler**. For example, `export CRAWLER_GITHUB_TOKENS="<token1>#public;<token2>#admin"`.

A typical sequence shown in the snippet below configures the crawler with a set of tokens, configures the org filter set and then queues and starts the processing of the org.

Expand All @@ -40,12 +41,15 @@ http://localhost:3000> orgs contoso-d
http://localhost:3000> queue contoso-d
http://localhost:3000> start 5
http://localhost:3000> exit
>
```

New commands are being added all the time. Check the help using `node bin/cc -help`.

# API
API doc is coming.

# Contributing

Currently we are adding commands and API as they are needed. We are very keen on adding things that users find valuable and would happily take pull requests. Adding a command is quite simple, just cut and paste some existing command and add your logic. Adding useful API is great too though it may involve adding function to the service (so a bit more work).

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
10 changes: 6 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
{
"name": "crawler-cli",
"name": "ghcrawler-cli",
"version": "0.1.3",
"description": "A simple command line app for controlling a GitHub Crawler",
"main": "./lib/crawlerClient.js",
"scripts": {
"start": "node ./bin/cc.js"
},
"keywords": [
"cli",
"command line",
"GitHub",
"API",
"crawler"
Expand All @@ -15,11 +17,11 @@
"license": "MIT",
"repository": {
"type": "git",
"url": "git+https://github.com/microsoft/crawler-cli.git"
"url": "git+https://github.com/microsoft/ghcrawler-cli.git"
},
"homepage": "https://github.com/microsoft/crawler-cli#readme",
"homepage": "https://github.com/microsoft/ghcrawler-cli#readme",
"bugs": {
"url": "https://github.com/microsoft/crawler-cli/issues"
"url": "https://github.com/microsoft/ghcrawler-cli/issues"
},
"files": [
"bin/cc",
Expand Down

0 comments on commit e74890f

Please sign in to comment.