🎉 First off, thanks for using and contributing to IOHexperimenter!! 🎉
Note that this page is under construction. More detailed guidelines are being added.
What should I Know before Getting Started?
This section guides you through submitting a bug for IOHexperimenter. Following the guides will help the maintainers and the community understand your report.
The bug reports are tackled as Github issues. After determining which repository the bug is related to, create an issue on the repository and report following the template as below.
Description:
Versions:
Steps to Reproduce:
1.
2.
3.
Expected Behaviour:
Actual Behavior:
Additional Information:
Explain the issue with additional information that can help the maintainers. Please consider the following contents.
- Use a clear and descriptive title for the issue.
- Describe the steps of reproducing the issue in detail. Attachments of your implementation and screenshots are appreciated.
- Explain the behavior (or the result) that you expect.
- Are there additional packages required for reproducing the bug?
- What is your working environment, e.g., operating system, compiler version, etc.?
This section guides you through creating new benchmarking problems and suites for IOHexperimenter. Instead of testing additional problems using the proxy provided in IOHexperimenter, the goal is to integrate new problems as internal packages.
Follow the steps of contributing new problems:
The header files shall locate at the problem folder.
We require to create the new problem by subclassing the abstract problem class in IOHexperimenter, taking benefits of implementing more details, e.g., aforementioned transformations. This can be done by inheriting the corresponding problem registration class, which is
ioh::problem::IntegerSingleObjectivefor pseudo-Boolean problems, andioh::problem::RealSingleObjectivefor continuous problems.
In the following example, we show how to do this for pseudo-Boolean problems.
class NewBooleanProblem final : public ioh::problem::RegisteredProblem<NewBooleanProblem, ioh::problem::IntegerSingleObjective>
{
protected:
// [mandatory] The evaluate method is mandatory to implement
double evaluate(const std::vector<int> &x) override
{
// the function body goes here
return 0.0;
}
// [optional] If one wish to implement transformations on search variables
std::vector<int> transform_variables(std::vector<int> x) override
{
return x;
}
// [optional] If one wish to implement transformations on objective values
double transform_objectives(const double y) override
{
return y;
}
public:
/// [mandatory] This constructor always take `instance` as input even
/// if it is ineffective by default. `instance` would be effective if and only if
/// at least one of `transform_objectives` and `transform_objectives` is implemented
NewBooleanProblem(const int instance, const int n_variables) :
RegisteredProblem(
ioh::problem::MetaData(
1, // problem id, which will be overwritten when registering this class in all pseudo-Boolean problems
instance, // the instance id
"NewBooleanProblem", // problem name
n_variables // search dimensionality
)
)
{
}
};Please check the example of adding continuous problems in this manner.
A suite contains a set of problems that share common properties. Therefore, we suggest creating a base class and a CRTP class for the new problems (see the example of bbob problems), following the template as below:
namespace ioh::problem
{
//! new base class of continuous optimization problems, of which objective is minization.
class NewBase : public RealSingleObjective
{
public:
/**
* @brief Construct a new NewBase object
*
* @param problem_id The id of the problem
* @param instance The instance of the problem
* @param n_variables the dimension of the problem
* @param name the name of the problem
*/
NewBase(const int problem_id, const int instance, const int n_variables, const std::string &name) :
RealSingleObjective(MetaData(problem_id, instance, name, n_variables, common::OptimizationType::MIN))
{
}
};
/**
* @brief CRTP class for NewBase problems. Inherit from this class when defining new NewBase problems
*
* @tparam ProblemType The New NewBase problem class
*/
template <typename ProblemType>
class NewBaseProblem : public NewBase,
AutomaticProblemRegistration<ProblemType, NewBase>,
AutomaticProblemRegistration<ProblemType, RealSingleObjective>
{
public:
/**
* @brief Construct a new NewBaseProblem object
*
* @param problem_id The id of the problem
* @param instance The instance of the problem
* @param n_variables the dimension of the problem
* @param name the name of the problem
*/
NewBaseProblem(const int problem_id, const int instance, const int n_variables, const std::string &name) :
NewBase(problem_id, instance, n_variables, name)
{
}
};
}Inherit from the CRTP class when defining the new problems.
namespace ioh::problem
{
//! NewProblemere function problem id 42
class NewProblem final: public NewBaseProblem<NewProblem>
{
protected:
//! Evaluation method
double evaluate(const std::vector<double>&) override
{
auto result = 0.0;
return result;
}
public:
/**
* @brief Construct a new NewProblem object
*
* @param instance instance id
* @param n_variables the dimension of the problem
*/
NewProblem(const int instance, const int n_variables) :
NewBaseProblem(42, instance, n_variables, "NewProblem")
{
}
};
}After implementing the new problem classes, they can be created using either factory construction:
auto &problem_factory = ioh::problem::ProblemRegistry<ioh::problem::NewBase>::instance();
for (auto xi: problem_factory.names())
std::cout << xi << "\n";
auto problem = problem_factory.create(42, 1, 10);Or, we can immediately create a suite class, which only contains the problems that are defined using the same parent type (NewBase):
namespace ioh::suite {
struct NewProblems final : RealSuite<NewProblems>
{
/**
* @brief Construct a new BBOB object
*
* @param problem_ids List of problem ids
* @param instances List of problem instances (defaults to first instance)
* @param dimensions List of problem dimensions (defaults to 5D)
*/
NewProblems(const std::vector<int> &problem_ids = {}, const std::vector<int> &instances = {1},
const std::vector<int> &dimensions = {5}) :
RealSuite(problem_ids, instances, dimensions, "NewProblems", 100, 100,
reinterpret_cast<Factory &>(problem::ProblemFactoryType<problem::NewBase>::instance()))
{
}
};
}We can then see that the new suite (NewProblems) gets imeddiately added to the list of available real-valued suites:
auto& suitef = ioh::suite::SuiteRegistry<ioh::problem::RealSingleObjective>::instance();
for (auto xi: suitef.names())
std::cout << xi << "\n";Which we can use to easily loop over the problems:
auto suite = suitef.create("NewProblems", {42}, {1}, {2});
for (auto problem: *suite)
std::cout << *problem << "\n";After implementing the new problem classes, you may make a pull request. Check the list of making a pull request, and follow the template as below.
Problem Name(s):
Added file(s):
Definition(s) of the problem(s):
Description of the implementation:
Additional information (e.g., reference of the problems, additional required packages.)This section guides you through suggesting enhancements for IOHexperimenter, which includes providing additional features and improving existing functionalities. Before suggesting the enhancements, check the following list.
- Double check if the enhancements have been implemented by IOHexperimenter. Make sure that you are using the latest version.
- Check if the enhancements have been suggested in other issues. If so, please comment on that issue instead of creating a new one.
The suggestions are tackled as Github issues. After following the upper list and determining which repository you plan to suggest the enhancements for, create an issue using the template below.
Suggestions: ("New Features" or "Improving Existing functionalities")
Description in detail:
Name of the function and file (if the suggestion is about improving existing functionalities):
Additional information (e.g., any additional packages for implementing the enhancements):To make sure that your contributions are considered and understood, please follow the list:
- Follow the styleguides
- Follow the templates
- Verify that all status checks are passing after submitting the pull request.
- If you make a pull request for reporting bugs, follow the bug template.
- If you make a pull request for suggesting enchancements, follow the enhacement template
- If you make a pull request for adding new problems, follow the problem template.
- If you update documentations, follow the template as below.
Updated files:
Functionalities involved:
Old description (introduce briefly if the content is too long):
New description (introduce briefly if the content is too long):
- Use the present tense ("Add feature" not "Added feature")
- Use the imperative mood ("Improve the function..." not "Improves the function...")
- When only changing documentation, include "Update README" in the commit title
We follow the Bjarne Stroustrup's C++ Style.