Skip to content

Conversation

jacobheun
Copy link
Contributor

@jacobheun jacobheun commented Jan 30, 2023

Summary

This PR contains a few dev utilities that I'm bundling together as a followup of doing retrieval testing of booster-bitswap

Docker Traffic Control

  • 364367e adds traffic control to the docker setup. It also adds a small latency to bitswap and http retrievals which mimics a reasonable local region latency of 10ms. The demo http server also now has a rate limited 1m connection and 100ms latency. This is probably excessive, but should also make it easier to test transfer components. This TC setup also has support for things like corrupted packets, but I've skipped including that for now.

Dense car file generator

  • b9653ae adds a generate-rand-car command to boostx, making it easy to create car files with very dense dag structures, which is helpful in retrieval testing. This would probably be better located in a separate dev/test utils command, but I added it to boostx for the time being as its intended for experimental commands anyway. Happy to switch this over to a test/dev util command if we have more things we'd like to live there. You can see an example usage of it in the script added in aa965b9:
# Generate a 5M random file. Chunk it into 512byte chunks, with a max if 8 links per level
$ boostx generate-rand-car -c=512 -l=8 -s=5120000 ./

Deal script

  • aa965b9 adds a new deal script that will automatically generate a random, dense car file and execute a deal. This makes it easy to seed a local devnet with data without going through the slower, tutorial style script. Future work here could turn this in an actual deal seed script with more variety in the car files being generated.
$ ./sample/random-deal.sh 
boost deal --verified=false --provider=t01000 --http-url=http://demo-http-server//app/public/bafykbzacec7kd4emrq66myjatybthq6eikczq2x2vloymro5uphvmkh6u2e6a.car --commp=baga6ea4seaqgudqjlkmvztlqarigpl65vloqs7v2qsamd3rvljewhgztqfb6kla --car-size=6159289 --piece-size=8388608 --payload-cid=bafykbzacec7kd4emrq66myjatybthq6eikczq2x2vloymro5uphvmkh6u2e6a --storage-price 20000000000

sent deal proposal
  deal uuid: a858d887-74d7-430c-8c07-d126442fd166
  storage provider: t01000
  client wallet: t3ql744hzi3cvvczmzbbcq72xse7qlqmoakjhukqrsn5opajqeayjpzwvd7kdill242c5ckqheofwdkx6fey5a
  payload cid: bafykbzacec7kd4emrq66myjatybthq6eikczq2x2vloymro5uphvmkh6u2e6a
  url: http://demo-http-server/bafykbzacec7kd4emrq66myjatybthq6eikczq2x2vloymro5uphvmkh6u2e6a.car
  commp: baga6ea4seaqgudqjlkmvztlqarigpl65vloqs7v2qsamd3rvljewhgztqfb6kla
  start epoch: 7485
  end epoch: 525885
  provider collateral: 0

@jacobheun jacobheun changed the title chore(docker): add traffic control to docker compose test: add traffic control to docker compose Feb 13, 2023
@jacobheun jacobheun changed the title test: add traffic control to docker compose test: add utilities and improvements for local testing Feb 13, 2023
update readme to ref deal script
@jacobheun jacobheun marked this pull request as ready for review February 13, 2023 17:55
@jacobheun jacobheun requested a review from dirkmc February 13, 2023 17:55
Copy link
Contributor

@dirkmc dirkmc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing 🙌

@dirkmc dirkmc merged commit f157694 into main Feb 14, 2023
@dirkmc dirkmc deleted the chore/docker-tc branch February 14, 2023 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants