Skip to content

[benchmark] Measure memory with rusage and a TON of gardening #18124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Jul 23, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
04894da
[benchmark][Gardening] Literate testing (markdown)
palimondo Jul 16, 2018
a74759b
[benchmark][docs] Re-formatting to markdown
palimondo Jul 16, 2018
43390e8
[benchmark] Combined `--list` tests, documentation
palimondo Jul 16, 2018
e3cbd18
[benchmark] Improved docs and tests
palimondo Jul 16, 2018
0efdd8d
[benchmark][Gardening] Nicer header composition
palimondo Jul 13, 2018
c7fc745
[benchmark] Removed bogus Totals stats
palimondo Jul 13, 2018
0836452
[benchmark][Gardening] BenchmarkResults formatting
palimondo Jul 16, 2018
8d0d25a
[benchmark] Verbose mode checks
palimondo Jul 17, 2018
fce7a45
[benchmark][Gardening] BenchmarkInfo replaces Test
palimondo Jul 17, 2018
0dbbc3d
[benchmark][Gardening] Indices are String
palimondo Jul 13, 2018
c198f44
[benchmark] Measure environment with rusage
palimondo Jul 17, 2018
377ee46
[benchmark] Test error handling parsing arguments
palimondo Jul 18, 2018
dd228d7
[benchmark][Gardening] guard benchArgs
palimondo Jul 13, 2018
371f155
[benchmark] Exit gracefully on argument errors
palimondo Jul 18, 2018
647376c
[benchmark][Gardening] Extract optionalArg func
palimondo Jul 19, 2018
7d19a03
[benchmark] Gracefully type-check attribute values
palimondo Jul 19, 2018
2e2848b
[benchmark][Gardening] Arguments w/ optional value
palimondo Jul 19, 2018
743c7ef
[benchmark][Gardening] Grouped argument parsing
palimondo Jul 19, 2018
f2c4262
[becnhmark][Gardening] Extracted checked function
palimondo Jul 20, 2018
b76c946
[benchmark][Gardening] `processArguments` is init
palimondo Jul 20, 2018
50575b1
[benchmark][Gardening] Introduce PartialTestConfig
palimondo Jul 20, 2018
bfa198b
[benchmark][Gardening] Immutable `TestConfig`
palimondo Jul 20, 2018
48d1558
[benchmark][Gardening] Nested test filter funcs
palimondo Jul 20, 2018
1841dde
[benchmark][Gardening] Unified argument parsing
palimondo Jul 20, 2018
9f90286
[benchmark][Gardening] Extracted `ArgumentParser`
palimondo Jul 20, 2018
e5cbfcc
[benchmark][Gardening] Declarative ArgumentParser
palimondo Jul 20, 2018
f674dd5
[benchmark][Gardening] Handle --help inside parser
palimondo Jul 21, 2018
50c79c5
[benchmark][Gardening] Local parser error handling
palimondo Jul 21, 2018
0c0ed3d
[benchmark][Gardening] Moved parseArgs into parser
palimondo Jul 21, 2018
f89d41a
[benchmark] Print detailed argument help
palimondo Jul 21, 2018
1961373
[benchmark] Log the MAX_RSS only w/ --memory flag
palimondo Jul 21, 2018
df5ccf3
[benchmark][Gardening] Better naming and comments
palimondo Jul 23, 2018
4ed3dcf
[benchmark][Gardening] `--sample-time` renaming
palimondo Jul 23, 2018
362f925
[benchmark][Gardening] Docs and error handling
palimondo Jul 23, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
266 changes: 213 additions & 53 deletions benchmark/utils/ArgParse.swift
Original file line number Diff line number Diff line change
Expand Up @@ -12,67 +12,227 @@

import Foundation

public struct Arguments {
public var progName: String
public var positionalArgs: [String]
public var optionalArgsMap: [String : String]

init(_ pName: String, _ posArgs: [String], _ optArgsMap: [String : String]) {
progName = pName
positionalArgs = posArgs
optionalArgsMap = optArgsMap
enum ArgumentError: Error {
case missingValue(String)
case invalidType(value: String, type: String, argument: String?)
case unsupportedArgument(String)
}

extension ArgumentError: CustomStringConvertible {
public var description: String {
switch self {
case let .missingValue(key):
return "missing value for '\(key)'"
case let .invalidType(value, type, argument):
return (argument == nil)
? "'\(value)' is not a valid '\(type)'"
: "'\(value)' is not a valid '\(type)' for '\(argument!)'"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could omit printing the type here. IMO it's sufficient to print "'(value)' is not a valid value for '(argument)'". Then you can also omit the (not-trivial) code to extract the type name. Printing the type name (e.g. "Int") is not very readable anyway.
Just a suggestion to make things simpler.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it was helpful user feedback. For example, when you use a wrong tag, the message will read: error: 'bogus' is not a valid 'BenchmarkCategory'. So you'll know what to look for.

The type-checking parser for BenchmarkCategory (func tags) is called from a nested context within the attribute parsers for tags and skip-tags, where it doesn't have access to the argument name. The best it could do, would be to say 'bogus' is not valid value'.

I'm not sure that's the right trade-off here. I think it's worth a little bit of code complexity for much better QoL.

case let .unsupportedArgument(argument):
return "unsupported argument '\(argument)'"
}
}
}

/// Using CommandLine.arguments, returns an Arguments struct describing
/// the arguments to this program. If we fail to parse arguments, we
/// return nil.
/// Type-checked parsing of the argument value.
///
/// We assume that optional switch args are of the form:
/// - Returns: Typed value of the argument converted using the `parse` function.
///
/// --opt-name[=opt-value]
/// -opt-name[=opt-value]
///
/// with opt-name and opt-value not containing any '=' signs. Any
/// other option passed in is assumed to be a positional argument.
public func parseArgs(_ validOptions: [String]? = nil)
-> Arguments? {
let progName = CommandLine.arguments[0]
var positionalArgs = [String]()
var optionalArgsMap = [String : String]()

// For each argument we are passed...
var passThroughArgs = false
for arg in CommandLine.arguments[1..<CommandLine.arguments.count] {
// If the argument doesn't match the optional argument pattern. Add
// it to the positional argument list and continue...
if passThroughArgs || !arg.starts(with: "-") {
positionalArgs.append(arg)
continue
/// - Throws: `ArgumentError.invalidType` when the conversion fails.
func checked<T>(
_ parse: (String) throws -> T?,
_ value: String,
argument: String? = nil
) throws -> T {
if let t = try parse(value) { return t }
var type = "\(T.self)"
if type.starts(with: "Optional<") {
let s = type.index(after: type.index(of:"<")!)
let e = type.index(before: type.endIndex) // ">"
type = String(type[s ..< e]) // strip Optional< >
}
throw ArgumentError.invalidType(
value: value, type: type, argument: argument)
}

/// Parser that converts the program's command line arguments to typed values
/// according to the parser's configuration, storing them in the provided
/// instance of a value-holding type.
class ArgumentParser<U> {
private var result: U
private var validOptions: [String] {
return arguments.compactMap { $0.name }
}
private var arguments: [Argument] = []
private let programName: String = {
// Strip full path from the program name.
let r = CommandLine.arguments[0].reversed()
let ss = r[r.startIndex ..< (r.index(of:"/") ?? r.endIndex)]
return String(ss.reversed())
}()
private var positionalArgs = [String]()
private var optionalArgsMap = [String : String]()

/// Argument holds the name of the command line parameter, its help
/// desciption and a rule that's applied to process it.
///
/// The the rule is typically a value processing closure used to convert it
/// into given type and storing it in the parsing result.
///
/// See also: addArgument, parseArgument
struct Argument {
let name: String?
let help: String?
let apply: () throws -> ()
}

/// ArgumentParser is initialized with an instance of a type that holds
/// the results of the parsing of the individual command line arguments.
init(into result: U) {
self.result = result
self.arguments += [
Argument(name: "--help", help: "show this help message and exit",
apply: printUsage)
]
}
if arg == "--" {
passThroughArgs = true
continue

private func printUsage() {
guard let _ = optionalArgsMap["--help"] else { return }
let space = " "
let maxLength = arguments.compactMap({ $0.name?.count }).max()!
let padded = { (s: String) in
" \(s)\(String(repeating:space, count: maxLength - s.count)) " }
let f: (String, String) -> String = {
"\(padded($0))\($1)"
.split(separator: "\n")
.joined(separator: "\n" + padded(""))
}
let positional = f("TEST", "name or number of the benchmark to measure")
let optional = arguments.filter { $0.name != nil }
.map { f($0.name!, $0.help ?? "") }
.joined(separator: "\n")
print(
"""
usage: \(programName) [--argument=VALUE] [TEST [TEST ...]]

positional arguments:
\(positional)

optional arguments:
\(optional)
""")
exit(0)
}
// Attempt to split it into two components separated by an equals sign.
let components = arg.split(separator: "=")
let optionName = String(components[0])
if validOptions != nil && !validOptions!.contains(optionName) {
print("Invalid option: \(arg)")
return nil

/// Parses the command line arguments, returning the result filled with
/// specified argument values or report errors and exit the program if
/// the parsing fails.
public func parse() -> U {
do {
try parseArgs() // parse the argument syntax
try arguments.forEach { try $0.apply() } // type-check and store values
return result
} catch let error as ArgumentError {
fputs("error: \(error)\n", stderr)
exit(1)
} catch {
fflush(stdout)
fatalError("\(error)")
}
}
var optionVal : String
switch components.count {
case 1: optionVal = ""
case 2: optionVal = String(components[1])
default:
// If we do not have two components at this point, we can not have
// an option switch. This is an invalid argument. Bail!
print("Invalid option: \(arg)")
return nil

/// Using CommandLine.arguments, parses the structure of optional and
/// positional arguments of this program.
///
/// We assume that optional switch args are of the form:
///
/// --opt-name[=opt-value]
/// -opt-name[=opt-value]
///
/// with `opt-name` and `opt-value` not containing any '=' signs. Any
/// other option passed in is assumed to be a positional argument.
///
/// - Throws: `ArgumentError.unsupportedArgument` on failure to parse
/// the supported argument syntax.
private func parseArgs() throws {

// For each argument we are passed...
for arg in CommandLine.arguments[1..<CommandLine.arguments.count] {
// If the argument doesn't match the optional argument pattern. Add
// it to the positional argument list and continue...
if !arg.starts(with: "-") {
positionalArgs.append(arg)
continue
}
// Attempt to split it into two components separated by an equals sign.
let components = arg.split(separator: "=")
let optionName = String(components[0])
guard validOptions.contains(optionName) else {
throw ArgumentError.unsupportedArgument(arg)
}
var optionVal : String
switch components.count {
case 1: optionVal = ""
case 2: optionVal = String(components[1])
default:
// If we do not have two components at this point, we can not have
// an option switch. This is an invalid argument. Bail!
throw ArgumentError.unsupportedArgument(arg)
}
optionalArgsMap[optionName] = optionVal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you just "apply" the argument here? Then you could get rid of optionalArgsMap.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could, but isn't the separation of the argument syntax parsing from the value type checking in the two separate methods worth keeping?

}
}

/// Add a rule for parsing the specified argument.
///
/// Stores the type-erased invocation of the `parseArgument` in `Argument`.
///
/// Parameters:
/// - name: Name of the command line argument. E.g.: `--opt-arg`.
/// `nil` denotes positional arguments.
/// - property: Property on the `result`, to store the value into.
/// - defaultValue: Value used when the command line argument doesn't
/// provide one.
/// - help: Argument's description used when printing usage with `--help`.
/// - parser: Function that converts the argument value to given type `T`.
public func addArgument<T>(
_ name: String?,
_ property: WritableKeyPath<U, T>,
defaultValue: T? = nil,
help: String? = nil,
parser: @escaping (String) throws -> T? = { _ in nil }
) {
arguments.append(Argument(name: name, help: help)
{ try self.parseArgument(name, property, defaultValue, parser) })
}
optionalArgsMap[optionName] = optionVal
}

return Arguments(progName, positionalArgs, optionalArgsMap)
/// Process the specified command line argument.
///
/// For optional arguments that have a value we attempt to convert it into
/// given type using the supplied parser, performing the type-checking with
/// the `checked` function.
/// If the value is empty the `defaultValue` is used instead.
/// The typed value is finally stored in the `result` into the specified
/// `property`.
///
/// For the optional positional arguments, the [String] is simply assigned
/// to the specified property without any conversion.
///
/// See `addArgument` for detailed parameter descriptions.
private func parseArgument<T>(
_ name: String?,
_ property: WritableKeyPath<U, T>,
_ defaultValue: T?,
_ parse: (String) throws -> T?
) throws {
if let name = name, let value = optionalArgsMap[name] {
guard !value.isEmpty || defaultValue != nil
else { throw ArgumentError.missingValue(name) }

result[keyPath: property] = (value.isEmpty)
? defaultValue!
: try checked(parse, value, argument: name)
} else if name == nil {
result[keyPath: property] = positionalArgs as! T
}
}
}
Loading