Temp drive limits

[michaelc95]

Hiya,

Background.. I have 24x temp drives (10k sas HD's) and a bunch of destination drives. I would like to have 1 job that lists
all 24 temp and all my destination so the plotmanager can figure out the best place to put plots. However.. since I am using HDs for temp I dont want more then 1 plot on a single temp drive at a time based on IO. Each temp drive is 600 gb so I am concerned that Plot manager will attempt to stick 2 jobs on each given they have the space.

What I would like is a way to tell the plot manager for each temp directory I define how many plots it can handle concurrently.
sorta like this. Make it an optional parameter so it doesnt affect existing configs.
temporary_directory:

• Y:\Plotter1, 1
• Y:\Plotter2, 1

I thought about defining 1 job per temp directory to limit the concurrent jobs to 1 but listing all my destination directories in
that job means I could have many jobs decide to write to the same destination directory crushing it from an IO perspective.
I dont want to lock a temp directory to a specific destination directory because drives fill up, I'd like the system to be as hands off as possible and I dont have 24 destinations.

Something you could add? Or maybe provide me a pointer in the code where I could hard code each temp drive to 1 concurrent plot yet just have 1 overall job for the benefit of my destinations.

Thanks for releasing this to the world. Its a huge help!
Mike

[mbabak74]

Just put all the drives in a raid 0 configuration and the writes will be shared among the drives.
with 24 drives, you better make a few raid 0 drives. Maybe 6x4 drives. then make 6 jobs. (unless your cpu can't handle 6 jobs in fase 1)

[michaelc95]

Raid 0 is not a solution to the issue and will slow down my overall plotting process. The key is HD plotting is to dedicate a drive to a plot for the best performance. Thanks though.

[mbabak74]

Then I guess you set up 23 folders under temporary2_directory (the fist should go under temporary_directory)
and then setup max 24 concurent jobs.
with stagger you can make sure not all are starting on the same time.
If you have enough cpu cores, you can also set the max concurrent jobs to 48 or higher. If you set the stagger at 15 minutes, then after 6 hours it should start the second job on the first directory.
If your system is faster, you can lower the stagger, but I don't think your disks will be fast enough to finish a job in 6 hours or less.

[michaelc95]

Why would I use temp2?
I want to list 24 directories under temp

they are 600 gb disks which still means that the plot manager could randomly choose to start 2 on 1 disk given they have the space. All I am saying is that if you can define multiple temp directories in one job, you should be able to tell the product how many plots to put on each at a time. Given that disks or nvme might not be able to handle from an IO perspective X number of plots.

[iocon]

OK, just resize your temp drive partitions to ~400GB so only one plot fits?

[FlyingToto]

I dont' think the code took in consideration your use case of having more temp drives than final destination drives since it "doesn't make sense" because you have to use a temp drive first place in order to use a destination if you want.....
There are few things you can do:

• first use some of these drives for tmp, and some of them for 2nd tmp
• depending on the number of destinations, use some of your 24 drives as pseudo final , the script will automatically not plot if the final disk is full and setup a separate job using a tool like freefilesync which periodically move your completed plots off the "tmp-final" disks to your real final hdd... make this one sesquential to optimize the transfer...

so worst case scenario, let's say you have only 1 final drive and 24 hig speed small tiny "tmp" dirves... assuming it takes you 8h to plot, you could setup:
1 jop which will do 3 plots in parallel with 3 tmp , 3 tmp2 and your final disk (using 6 small drives and the final big disk)
1 job which will do 6 plots in parallel with 9 tmp, 9 tmp2&final on your small disks (using 18 small drives and 0 final disk)

• add a job using freeefilesync that automatically move completed plots from the final small disks to the final 1 big disk sequentially (it will still conflict with job 1 but at least it shouldn't impact the jobs in 2 since it will wait for the plot to be fully written to kick in)

then you can surely tweak more... :-)

[FlyingToto]

just one more thing....
even if you found these old drives in a garbage bin, you might not actually make any money... running 10k drives takes a "lot of power" for rather slow plotting against ssd and with the current rate of growth of chia, your time to get a plot will only keep increasing so you might never get your electricity costs back, that's one of the reason why your want ssd for plotting...