Add new compression scheduler implementation for improved simplicity and robustness; Remove old scheduler and consolidate the new schedulers

[wraymo]

References

Description

This PR refactors the schedulers by

• Removing the task consumer thread
• Refactoring task distribution by using group primitive for compression
• Removing the compression handler and delegating the work to the scheduler
• Removing the old search scheduler and executor
• Modifying associated package scripts
• Cleaning up the code

Validation performed

• Validated compression with the hive-24hrs dataset.
• Validated search with a query "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties"

[kirkrodrigues]

Can we move db_conn.commit() into fetch_new_jobs (to keep it modular if we ever use the function elsewhere) and then we can simplify the loop to:

Suggested change

	jobs = fetch_new_jobs(db_cursor)
	db_conn.commit()
	for job_row in jobs:
	for job_row in fetch_new_jobs(db_conn, db_cursor):

[wraymo]

But we need to pass db_conn to fetch_new_jobs while other database functions don't require this parameter and all of them commit outside the function

[kirkrodrigues]

Fair enough. I think the database ops need some refactoring to be less error prone but we can do that in another PR.

[kirkrodrigues]

Commit message: Add new compression scheduler implementation for improved simplicity and robustness; Remove old scheduler and consolidate the new schedulers. (#238)?

[wraymo]

Commit message: Add new compression scheduler implementation for improved simplicity and robustness; Remove old scheduler and consolidate the new schedulers. (#238)?

Sure, but let me resolve conflicts first