Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tool to grep for ips that overlap #185

Merged

Conversation

talhahwahla
Copy link
Contributor

No description provided.

Copy link

linear bot commented Nov 8, 2023

BE-2457 CLI tool to grep for IPs that overlap

We have some ideas to add tooling into grepip or similar to grep for CIDRs and IP ranges specifically.

An additional idea could be to grep for all CIDR/Ranges that have a specific IP/CIDR/Range overlap with those in the file.

Could be called matchip as a new binary but if we can come up with a nice UX for this to fit into grepip it'll make that one thing more powerful and build upon it's already built-up attention / usage.

@talhahwahla
Copy link
Contributor Author

talhahwahla commented Nov 13, 2023

@UmanShahzad this is the current implementation

$ cat list1.txt 
8.8.8.8
100.0.0.0
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5
33.0.112.175/24

$ cat list2.txt 
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5/100
33.0.112.175/12
9.9.9.9/24
100.0.0.0

$ ipinfo matchip --filter list1.txt --criteria list2.txt 
33.0.112.175/24
100.0.0.0
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5

$ ipinfo matchip --filter=list1.txt,list2.txt --criteria list2.txt 
33.0.112.175/24
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5/100
33.0.112.175/12
9.9.9.9/24
100.0.0.0
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5
100.0.0.0

$ cat list1.txt | ipinfo matchip --filter - --criteria list2.txt 
33.0.112.175/24
100.0.0.0
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5

$ echo "8.8.8.8-9.9.9.9" | ipinfo matchip --filter - --criteria <(echo "8.8.8.8/12")
8.8.8.8-9.9.9.9
$ ipinfo matchip --help
Usage: ipinfo matchip --filter <file(s) | stdin> --criteria <file(s) | stdin>
Description:
  Prints the overlapping IPs and subnets.

Examples:
  # Match from a file
  $ ipinfo matchip --filter /path/to/list1.txt --criteria /path/to/list2.txt

  # Match from multiple files
  $ ipinfo matchip --filter=/path/to/list.txt,/path/to/list1.txt --criteria=/path/to/list2.txt,/path/to/list3.txt

  # Match from stdin
  $ cat /path/to/list1.txt | ipinfo matchip --filter - --criteria /path/to/list2.txt

Options:
  General:
    --filter, -f
      IPs, CIDRs, and/or Ranges to be filtered.
    --criteria, -c
      CIDRs and/or Ranges to check overlap with.
    --help
      show help.

@talhahwahla talhahwahla marked this pull request as ready for review November 13, 2023 16:05
@talhahwahla
Copy link
Contributor Author

The next round of work related to this is to make matchip work within or alongside the grepip -- for that we would want to have the grepip print subnets too.

Currently it gets the IPs only:

$ ipinfo grepip list2.txt --only-matching
33.0.112.175
100.0.0.0
8.8.8.8
10.0.0.0

Perhaps we can add a new flag (--include-subnet?) that shows the subnets too:

$ ipinfo grepip list2.txt --only-matching --include-subnet
33.0.112.175/12
100.0.0.0
8.8.8.8-10.0.0.0

Then we can pipe it to matchip:

$ cat list3.txt
8.8.8.8/24

$ ipinfo grepip list2.txt --only-matching --include-subnet | ipinfo matchip --filter - --criteria list3.txt 
8.8.8.8-10.0.0.0

Or maybe we can just have the equivalent of above as another flag in grepip:

$ ipinfo grepip list2.txt --match list3.txt 
8.8.8.8-10.0.0.0

Copy link
Contributor

@UmanShahzad UmanShahzad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have not checked actual algorithm yet

ipinfo/main.go Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Also add completions to main tool
  • Also add its doc in main ipinfo CLI doc
  • Also make it available as a separate tool like grepip (copy/paste, change name & version)
  • Ensure anywhere else we see grepip being used in e.g. build tools / etc, this is added there


func printHelpMatchIP() {
fmt.Printf(
`Usage: %s matchip --filter <file(s) | stdin> --criteria <file(s) | stdin>
Copy link
Contributor

@UmanShahzad UmanShahzad Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A more ideal form is one that matches grep's solution to this:

# single expression + single file
matchip 127.0.0.1 file1.txt

# single expression + multiple files
matchip 127.0.0.1 file1.txt file2.txt file3.txt

# multi expression + any files
cat expression-list1.txt | matchip -e 127.0.0.1 -e 8.8.8.8 -e - -e expression-list2.txt file1.txt file2.txt file3.txt

I.e., use an 'expression' or 'input' flag to label each input, equivalent to the --filter flag you currently have. However -f wouldn't be a great option since it commonly stands for file so could cause confusion. -e / --expression is what grep uses, so in the spirit of trying to re-use people's habits / knowledge of grep like we did in grepip, maybe that's ideal.

@talhahwahla
Copy link
Contributor Author

$ cat file1.txt 
8.8.8.8
100.0.0.0
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5
33.0.112.175/24

$ cat file2.txt 
c7b5:48cd:4b8b:27b8:75f7:b3ac:de09:82f5/100
33.0.112.175/12
9.9.9.9/24
100.0.0.0

$ ./matchip/build.sh

$ ./build/matchip file1.txt file2.txt  
33.0.112.175/12
100.0.0.0

$ cat file2.txt | ./build/matchip file1.txt - -e 1.1.1.1/1
33.0.112.175/24
33.0.112.175/12
9.9.9.9/24
8.8.8.8
100.0.0.0
100.0.0.0

matchip/main.go Outdated
show help.

Completions:
--completions-install
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't use real tab chars in this string

--expression, -e
CIDRs, ip-ranges to to check overlap with. Can be used multiple times.
--help
Show help.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs 2 less spaces


Flags:
--expression, -e
CIDRs, ip-ranges to to check overlap with. Can be used multiple times.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs 2 less spaces

@UmanShahzad UmanShahzad merged commit 805e4ec into master Nov 16, 2023
@UmanShahzad UmanShahzad deleted the talhahameed/be-2457-cli-tool-to-grep-for-ips-that-overlap branch November 16, 2023 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants