<< Documentation Home
PTS Test Specification

Phoronix Test Suite Test Specification

This document describes the XML tags for writing a test profile.

PhoronixTestSuite/TestProfile/

Tag: Version
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_PTSVERSION
Required: Yes
Description: The version of the test profile itself, not dependent upon the main Phoronix Test Suite version or the version of the software package itself.
Supported Values: ( Formatted: x.x.x )

Tag: TestType
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_HARDWARE_TYPE
Required: Recommended
Description: The hardware area of the system that this test is designed to stress.
Supported Values: System, Processor, Graphics, Disk, Memory, Motherboard

Tag: SoftwareType
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_SOFTWARE_TYPE
Required: Recommended
Description: The type of software that this test consists of.
Supported Values: Game, Utility, Simulator, Other

Tag: Maintainer
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_MAINTAINER
Required: Yes
Description: The name of this profile's maintainer.

Tag: License
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_LICENSE
Required: Yes
Description: The license of the software itself.
Supported Values: FREE, NON-FREE, RESTRICTED (Will only work with certain hardware configurations), UNKNOWN

Tag: Status
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_STATUS
Required: Yes
Description: The status of the test profile itself.
Supported Values: VERIFIED, UNVERIFIED, BROKEN

Tag: SupportedArchitectures
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_SUPPORTEDARCHS
Required: No - Defaults to assume all architectures are supported
Description: If the test software is limited to running on select system architectures, list them in this tag with each architecture delimited by a comma.
Supported Values: ( Examples: x86, x86_64 )

Tag: SupportedPlatforms
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_SUPPORTEDPLATFORMS
Required: Recommended - Defaults to assume all operating systems are supported
Description: If the test software is limited to running on select operating systems, list them in this tag with each OS being delimited by a comma.
Supported Values: Linux, Solaris, FreeBSD, BSD

Tag: UnsupportedPlatforms
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_UNSUPPORTEDPLATFORMS
Required: No - Defaults to assume all operating systems are supported
Description: If the test software is limited to running on select operating systems, list the ones NOT supported by this test. This tag is the opposite of SupportedPlatforms. Operating systems not listed here are assumed to be supported. This option will be dropped in Phoronix Test Suite 2.4 and later, using the SupportedPlatforms tag is the preferred way of reporting OS compatibility.
Supported Values: Linux, Solaris, FreeBSD, BSD

Tag: ExternalDependencies
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_EXDEP
Required: No
Description: The external dependencies needed for the test. Multiple dependencies need to be delimited by a comma.
Supported Values: ( See pts/distro-xml/generic-packages.xml for a list of currently supported external dependencies. )

Tag: Extends
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_CTPEXTENDS
Required: No - Defaults to assume there is no extension
Description: If the test relies upon another test within the Phoronix Test Suite, list its name in this tag. This is similar to extends in programming languages in the capabilities that it provides. When running a test that extends another one, it will require all test extensions be built as well as the PATH environment variable will contain each of the test directories below this test. This is part of the Cascading Test Profiles functionality.
Supported Values: ( Any name of another test identifer shipping with the Phoronix Test Suite. )

Tag: RequiresRoot
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_ROOTNEEDED
Required: No - Defaults to assume false
Description: If root access is needed by the software encapsulated by this profile, the value of this tag should be set to TRUE.
Supported Values: FALSE, TRUE

Tag: EnvironmentSize
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_ENVIRONMENTSIZE
Required: No
Description: The estimated size (in megabytes) that this test takes up in its environment directory when downloaded files have been extracted, etc.
Supported Values: ( Values must be in MB )

Tag: EnvironmentTestingSize
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_ENVIRONMENT_TESTING_SIZE
Required: No
Description: If the test involves creating a significantly large file(s) during the testing process, this tag should be used to ensure that there is sufficient space available. For example, if during the testing process a 2GB file is temporarily created, this tag should be set to 2048. If there is not sufficient room available on the hard drive prior to the test running, the test will not run. This tag does not impact the EnvironmentSize tag nor is it used at the test's install time.
Supported Values: ( Values must be in MB )

Tag: EstimatedLength
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_ESTIMATEDTIME
Required: No
Description: The estimated length (in minutes) that it takes this benchmark to run in total.
Supported Values: ( Values must be in minutes )

Tag: ProjectURL
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_PROJECTURL
Required: No
Description: The web-site for the software vendor or project of the software being used for testing.
Supported Values: ( The URL must start with http:// and be a valid address )

Tag: RequiresCoreVersion
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_SUPPORTS_COREVERSION
Required: No
Description: If this test profile will only work with specific versions of the Phoronix Test Suite, they should be specified in this tag.
Supported Values: ( Values must be in the PTS version format of X.X.X. If this test profile requires version 1.4.0 of the Phoronix Test Suite or later, the value would simply be 1.4.0. If it requires a version between 1.4.0 and 1.8.0, it must be written as 1.4.0 - 1.8.0. )

Tag: ReferenceSystems
Location: PhoronixTestSuite/TestProfile/
Internal Reference: P_TEST_REFERENCE_SYSTEMS
Required: No
Description: To provide reference test results for comparison purposes, supply Phoronix Global IDs to this tag. For optimal comparisons, the reference IDs should be results generated when running in the Phoronix Test Suite batch mode.
Supported Values: ( Values must be valid Phoronix Global IDs. Multiple IDs can be delimited by a comma. )

PhoronixTestSuite/TestInformation/

Tag: Title
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_TITLE
Required: Yes
Description: The string title of the test, used for display purposes.

Tag: SubTitle
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_SUBTITLE
Required: No
Description: If a value is supplied, this is a static sub-title when rendering the graph results. Otherwise the test options (if applicable) are displayed.

Tag: Version
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_VERSION
Required: Recommended
Description: The version of the software game/application/program itself

Tag: Description
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_DESCRIPTION
Required: Recommended
Description: A few sentence description of the test.

Tag: ResultScale
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_SCALE
Required: Yes
Description: The scale the results are reported in.
Supported Values: ( Examples: Seconds, Average FPS, Mflops )

Tag: ResultQuantifier
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_QUANTIFIER
Required: No - Defaults to AVG
Description: When running a test multiple times, how should the final result be calculated.
Supported Values: AVG (Average), MAX (Maximum), MIN (Minimum)

Tag: ResultFormat
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_RESULTFORMAT
Required: No - Defaults to BAR_GRAPH
Description: The type of graph/visual representation to be used when viewing the test results.
Supported Values: BAR_GRAPH (Bar Graph), LINE_GRAPH (Line Graph), PASS_FAIL (Pass-Fail Graph)

Tag: Proportion
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_PROPORTION
Required: No - Defaults to HIB
Description: The scale for the test, whether lower or higher results yield better performance.
Supported Values: LIB (Lower Is Better), HIB (Higher Is Better)

Tag: Executable
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_EXECUTABLE
Required: No - Defaults to name (identifier) of test
Description: When looking for the test to execute in the path directory or the benchmarking environment directory, this is the string of the file-name.

Tag: TimesToRun
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_RUNCOUNT
Required: No - Defaults to 1
Description: The number of times to run the test.
Supported Values: ( Integer Values )

Tag: IgnoreRuns
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_IGNORERUNS
Required: No - Defaults to not ignoring any runs
Description: If this tag is present, any integer values passed for the run numbers will be ignored. For example, if the value is 1, the results from the first run will be ignored and not recorded. Multiple values can be defined with their values being separated by a comma.
Supported Values: Integers between one and the value of TimesToRun.

Tag: IgnoreFirstRun
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_IGNOREFIRSTRUN
Required: No - Defaults to 0
Description: As of Phoronix Test Suite 1.8, this option has been removed and is replaced with the IgnoreRuns tag.
Supported Values: 0, 1

Tag: PreInstallMessage
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_PREINSTALLMSG
Required: No
Description: If the user needs to be alerted about something dealing with this test, this tag can contain a string that will be shown to the user before the test is installed but after the files are downloaded.

Tag: PostInstallMessage
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_POSTINSTALLMSG
Required: No
Description: If the user needs to be alerted about something dealing with this test, this tag can contain a string that will be shown to the user after the test installation script has been executed.

Tag: InstallationAgreement
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_INSTALLAGREEMENT
Required: No
Description: If there is a user license agreement or installation agreement, it should be specified within this tag. The agreement is then displayed prior to the test being installed. Once the text has been shown, the user is asked whether they agree to the terms with a yes / no option. If the user agrees, the test is installed otherwise the test installation is aborted. Instead of supplying the full-text to the agreement in this tag, an HTTP URL to where the agreement can be found in a plain text format can be given, which will then be fetched by the Phoronix Test Suite.

Tag: PreRunMessage
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_PRERUNMSG
Required: No
Description: If the user needs to be alerted about something dealing with this test, this tag can contain a string that will be shown to the user before the test begins.

Tag: PostRunMessage
Location: PhoronixTestSuite/TestInformation/
Internal Reference: P_TEST_POSTRUNMSG
Required: No
Description: If the user needs to be alerted about something dealing with this test, this tag can contain a string that will be shown to the user after the test ends.

PhoronixTestSuite/TestSettings/Default/

Tag: Arguments
Location: PhoronixTestSuite/TestSettings/Default/
Internal Reference: P_TEST_DEFAULTARGUMENTS
Required: No
Description: Any arguments to supply to the test executable by default each and every time.

Tag: PossiblePaths
Location: PhoronixTestSuite/TestSettings/Default/
Internal Reference: P_TEST_POSSIBLEPATHS
Required: No
Description: As an alternative to using the testing environment directory, this tag can containe different paths to check for the test executable. Multiple paths need to be separated by a comma.

Tag: AllowCacheShare
Location: PhoronixTestSuite/TestSettings/Default/
Internal Reference: P_TEST_ALLOW_CACHE_SHARE
Required: No
Description: If this tag is present and is set to TRUE, the result output from the test run will be cached and then submitted to the test profile again later on if it's being run multiple times in the suite or test run. This is used for preventing a test from running multiple times with different settings, but all of the different options are tested in that single run, and the only change in options is what from the result output is returned.

Tag: MinimumLength
Location: PhoronixTestSuite/TestSettings/Default/
Internal Reference: P_TEST_MIN_LENGTH
Required: No
Description: If a test must run for at least a certain amount of time to produce meaningful results, enter the time (in seconds) into this tag. If the actual test run-time ends up being less than this value, the results will not be considered valid. This is to address the possibility of a test ending prematurely, crashes (segmentation faults, etc), or the process being killed by the user. This tag should only be used if the test will always at least run for the specified amount of time when properly executed.

Tag: MaximumLength
Location: PhoronixTestSuite/TestSettings/Default/
Internal Reference: P_TEST_MAX_LENGTH
Required: No
Description: This option is similar to the MinimumLength tag, but is to be used for invalidating a test result if the test takes too long to complete. Enter the time (in seconds) into this tag. If the actual test run-time is greater than this value, the results will not be considered valid. This tag should only be used if the test will always be less than the specified amount of time when properly executed.

PhoronixTestSuite/TestSettings/Option/
NOTE: Multiple instances of these tags are allowed for multiple options in the test

Tag: DisplayName
Location: PhoronixTestSuite/TestSettings/Option/
Internal Reference: P_TEST_OPTIONS_DISPLAYNAME
Required: No
Description: If the end-user is presented with options for this test, this tag is the full string shown to the user to describe this option.

Tag: ArgumentPrefix
Location: PhoronixTestSuite/TestSettings/Option/
Internal Reference: P_TEST_OPTIONS_ARGPREFIX
Required: No
Description: The argument that will be appended to the Arguments tag and is pre-fixed to the Identifier tag selected by the user.

Tag: ArgumentPostfix
Location: PhoronixTestSuite/TestSettings/Option/
Internal Reference: P_TEST_OPTIONS_ARGPOSTFIX
Required: No
Description: The argument that will be appended to the Arguments tag and is post-fixed to the Identifier tag selected by the user.

Tag: Identifier
Location: PhoronixTestSuite/TestSettings/Option/
Internal Reference: P_TEST_OPTIONS_IDENTIFIER
Required: No
Description: The internal ID to describe this option.

Tag: DefaultEntry
Location: PhoronixTestSuite/TestSettings/Option/
Internal Reference: P_TEST_OPTIONS_DEFAULTENTRY
Required: No
Description: The integer option (starting from 1) of the menu entry to select for that option when running in defaults mode. For tests that have no menu options or just one, this tag isn't needed. For menus with two entries, both of them are used in the defaults mode. When dealing with more than two entries, use this tag to select the option to use otherwise the last entry will automatically be used.

Tag: Name
Location: PhoronixTestSuite/TestSettings/Option/Menu/
Internal Reference: P_TEST_OPTIONS_MENU_GROUP_NAME
Required: No
Description: The string shown to the user to describe one of the available options within the option.

Tag: Value
Location: PhoronixTestSuite/TestSettings/Option/Menu/
Internal Reference: P_TEST_OPTIONS_MENU_GROUP_VALUE
Required: No
Description: The string post-fixed to the ArgumentName tag and appended to the end of the Arguments tag, which is then supplied to the executable binary as arguments.

Phoronix-Test-Suite.com
Copyright © 2008 by Phoronix Media.