python logo

Evaluating Python Implementations

This page described the details of each program suite that compose our performance evaluation work. For that, the individual test names of each program suite described here are listed. The code of each one of these tests can be downloaded here. The test detail is divided in four sections, one for each program suite group:

  1. Microbenchmarks
  2. Benchmarks
  3. Programs
  4. Large Scale Applications

Microbenchmarks

A description of every listed microbenchmark suite can be found here.

Tommti
1 TommtiIntArithmetic
2 TommtiDoubleArithmetic
3 TommtiLongArithmetic
4 TommtiTrig
5 TommtiHashtest
6 TommtiHashes
7 TommtiHeapsort
8 TommtiVector
9 TommtiMatrixMultiply
10 TommtiNestedLoop
11 TommtiStringConcat
12 TommtiIO
13 TommtiExcept
Pybench
Arithmetic
1 SimpleIntegerArithmetic
2 SimpleFloatArithmetic
3 SimpleLongArithmetic
4 SimpleComplexArithmetic
Calls
1 PythonFunctionCalls
2 ComplexPythonFunctionCalls
3 BuiltinFunctionCalls
4 Recursion
5 PythonInnerClassMethodCalls
Constructs
1 IfThenElse
2 NestedForLoops
3 SequentialForLoops
Dict
1 DictCreation
2 DictWithStringKeys
3 DictWithFloatKeys
4 DictWithIntegerKeys
5 SimpleDictManipulation
Exceptions
1 TryExcept
2 TryRaiseExcept
Imports
1 SecondImport
2 SecondPackageImport
3 SecondSubmoduleImport
Instances
1 CreateInstances
2 CreateNewInstances
Lists
1 ListManagement
2 ListSlicing
3 SmallLists
4 SimpleListComprehensions
5 NestedListComprehensions
Lookups
1 NormalInstanceAttribute
2 NormalClassAttribute
3 SpecialClassAttribute
4 SpecialInstanceAttribute
NewInstances
1 CreateNewInstances
Numbers
1 CompareIntegers
2 CompareFloats
3 CompareFloatsIntegers
4 CompareLongs
Strings
1 StringCreation
2 StringConcat
3 StringSubstring
4 StringCompare
5 StringCompareInterned
6 StringMethods
7 StringMappings
Tuples
1 TupleSlicing
2 SmallTuples
Unicode
1 StringCreationUnicodeWithConcat
2 StringConcatUnicode
3 StringSubstringUnicode
4 StringCompareUnicode
5 StringUnicodeMethods
6 StringUnicodeMappings
7 StringUnicodeProperties
With
1 WithFinally
2 TryFinally
3 WithRaiseExcept
JavaGrande Microbenchmarks (section 1)
JGFArithBench
1 JGFArithAddInt
2 JGFArithAddLong
3 JGFArithMultInt
4 JGFArithMultLong
5 JGFArithDivInt
6 JGFArithDivLong
7 JGFArithAddFloat
8 JGFArithAddDouble
9 JGFArithMultFloat
10 JGFArithMultDouble
11 JGFArithDivFloat
12 JGFArithDivDouble
JGFAssignBench
1 AssignSameScalarLocal
2 AssignSameScalarInstance
3 AssignSameScalarClass
4 AssignSameArrayLocal
5 AssignSameArrayInstance
6 AssignSameArrayClass
7 AssignOtherScalarInstance
8 AssignOtherScalarClass
9 AssignOtherArrayInstance
10 AssignOtherArrayClass
JGFCreateBench
1 CreateArrayInt
2 CreateArrayLong
3 CreateArrayFloat
4 CreateArrayObject
5 CreateObjectBase
6 CreateObjectSimple
7 CreateObjectSimpleConstructor
8 CreateObjectSimple1Field
9 CreateObjectSimple2Field
10 CreateObjectSimple4Field
11 CreateObjectSimple4fField
12 CreateObjectSimple4LField
13 CreateObjectSubclass
14 CreateObjectComplex
JGFExceptionBench
1 ExceptionThrow
2 ExceptionNew
3 ExceptionMethod
JGFLoopBench
1 LoopFor
2 LoopWhile
3 LoopReverseFor
JGFMathBench
1 JGFMathAbsInt
2 JGFMathAbsLong
3 JGFMathAbsFloat
4 JGFMathAbsDouble
5 JGFMathMaxInt
6 JGFMathMaxLong
7 JGFMathMaxFloat
8 JGFMathMaxDouble
9 JGFMathMinInt
10 JGFMathMinLong
11 JGFMathMinFloat
12 JGFMathMinDouble
13 JGFMathSinDouble
14 JGFMathCosDouble
15 JGFMathTanDouble
16 JGFMathAsinDouble
17 JGFMathAcosDouble
18 JGFMathAtanDouble
19 JGFMathAtan2Double
20 JGFMathFloorDouble
21 JGFMathCeilDouble
22 JGFMathSqrtDouble
23 JGFMathExpDouble
24 JGFMathLogDouble
25 JGFMathPowDouble
26 JGFMathRintDouble
27 JGFMathRandom
28 JGFMathRoundFloat
29 JGFMathRoundDouble
30 JGFMathIEEERemainderDouble
JGFMethodBench
1 MethodSameInstance
2 MethodSameSynchronizedInstance
3 MethodSameClass
4 MethodSameSynchronizedClass
5 MethodOtherInstance
6 MethodOtherInstanceOfAbstract
7 MethodOtherClass
JGFSerialBench
1 IOLinkList
2 IOBinaryTree
3 IOVector
4 IOArray
Pypy translator Microbenchmarks
test_bltn
1 test_isinstance1
2 test_isinstance2
3 test_isinstance3
4 test_isinstance1_old_style
5 test_isinstance3_old_style
test_call
1 test_call_1
2 test_call_2
3 test_call_3
4 test_call_4
5 test_call_default_1
6 test_call_default_2
7 test_call_keyword_1
8 test_call_keyword_2
9 test_call_keyword_3
test_count1
1 test_loop
2 test_loop_other_count
3 test_loop_unrolled
4 test_count_in_slot
5 test_count_in_dict
6 test_call_function
7 test_call_nested_function
8 test_call_nested_function_other_count
9 test_call_nested_function_many_args
10 test_call_method_of_old_style_class
11 test_call_method_of_new_style_class
12 test_call_function_with_arguments_in_cellvars
13 test_call_function_without_arguments_in_cellvars
14 test_count_in_attr
15 test_count_in_global
16 test_count_increment_in_global
17 test_count_in_global2
18 test_count_with_True
19 test_count_with_global_increment
test_create1
1 test_simple_loop_with_old_style_class_creation
2 test_simple_loop_with_new_style_class_creation
3 test_simple_loop_with_new_style_class_new
test_dict
1 test_dict_setitem1
2 test_dict_setitem2
3 test_dict_creation_mode1
4 test_dict_creation_mode2
5 test_dict_creation_mode3
6 test_dict_getitem
7 test_dict_raw_range
8 test_dict_class_dict_getmethod
9 test_dict_instance_getattr_instance_dict
10 test_dict_instance_setattr_instance_dict
11 test_dict_instance_setnewattr_instance_dict
12 test_dict_class_dict_getmethod_old_style
13 test_dict_instance_getattr_instance_dict_old_style
14 test_dict_instance_setattr_instance_dict_old_style
15 test_dict_instance_setnewattr_instance_dict_old_style
test_dispatch
1 test_dispatch_nop
test_exception
1 test_try_except
2 test_try_except_else
3 test_try_except_finally
4 test_instantiate_builtin_exception
5 test_instantiate_user_exception
6 test_raise_builtin_exception
7 test_raise_user_exception
8 test_except_specific_builtin_exception
9 test_except_multiple_builtin_exception
10 test_except_specific_user_exception
11 test_except_multiple_user_exception
12 test_reraise
test_formatting
1 test_simple_formatting
2 test_dict_formatting
3 test_number_formatting
4 test_repr_formatting
5 test_format_unicode
6 test_format_long
test_list
1 test_list_append
2 test_list_setitem
3 test_list_slice
4 test_list_getitem
5 test_list_extend
test_unicode
1 test_find_worstcase
2 test_count_worstcase
3 test_find_pattern16
4 test_find_pattern8
5 test_find_pattern4
6 test_find_pattern2
7 test_find_pattern1
Custom Microbenchmarks
Process
1 ProcessCall
2 ProcessCheckCall
3 ProcessCheckOutput
4 ProcessPopen
Thread
1 ThreadCreation
2 ThreadRunAndJoin
3 ThreadRunAndJoinSynchronization
Bit operations
1 GGBench
2 LLBench
3 BORBench
4 BAndBench
5 BXorBench
Object comparations
1 CompareObjectsReference
2 CompareObjectsByValue
Complex arithmetic
1 AddComplex
2 MulComplex
3 DivComplex
Rational arithmetic
1 RationalArithAddInt
2 RationalArithMultInt
3 RationalArithDivInt
Complementary variable access
1 VariableAccess
Set management
1 SetManagement
Complementary String management
1 StringAccess
2 StringSearch
Functional Programming Microbenchmarks
1 LambdaCreation
2 HigherOrder
3 LambdaInvokation
4 FunctionCurrying
5 FunctionMap
6 FunctionFilter
7 FunctionReduce
Metaprogramming Microbenchmarks
Introspection
1 IntrospectionClassLookupBench
2 FunctionsInvoke
3 MethodsInvokeObject
4 MethodsInvokeClass
5 DynamicAttributeLookup
6 DynamicVariableLookup
7 DynamicAttributeWrite
8 DynamicVariableWrite
Intercession
1 CreateClass
2 CreateClassWithInstance
3 CreateEmptyClass
4 CreateEmptyClassWithInstance
5 AttributeAddIntClass
6 AttributeAddObjectClass
7 AttributeAccessAddedClassInt
8 AttributeAccessAddedClassObject
9 AttributeAddIntObject
10 AttributeAddObjectObject
11 AttributeAccessAddedClassObject
12 AttributeAccessAddedObjectObject
13 AttributeDeleteIntClass
14 AttributeDeleteIntObject
15 AttributeDeleteObjectClass
16 AttributeDeleteObjectObject
17 MethodsAddClass
18 MethodsAddInvokeClass
19 MethodsAddObject
20 MethodsAddInvokeObject
21 MethodsDeleteAddedClass
22 MethodsDeleteAddedObject
23 TypeChange
24 InheritanceTreeChange
Computational Reflection
1 GetAttrExistingBench
2 GetAttrNonExistingBench
3 SetAttrExistingBench
4 SetAttrNonExistingBench
5 HasAttrExistingBench
6 HasAttrNonExistingBench
7 CallBench
Dynamic code execution
1 EvalBench
2 ExecBench
3 ExecFileBench

Benchmarks

A description of every listed benchmark suite can be found here.
Unlanden swallow Benchmarks
1 CallMethod
2 CallMethodSlots
3 CallMethodUnknown
4 CallSimple
5 Float
6 Json
7 Logging
8 LoggingSimpleOutput
9 LoggingNoOutput
10 NQueens
11 Pickle
12 PickleList
13 PickleUnpickle
14 PickleUnpickleList
15 PickleDict
16 Pidigits
17 RegexCompile
18 RegexEffbot
19 RegexV8
20 Richards
21 ThreadCount
22 Threading
23 UnpackSequence
24 JsonLoad
Java Grande Benchmarks (section 2 & 3)
Section 2
1 JGFCryptBench
2 JGFFFTBench
3 JGFHeapSortBench
4 JGFLUFactBench
5 JGFSeriesBench
6 JGFSparseMatmultBench
Section 3
1 JGFMolDynBench
2 JGFRayTracerBench
3 JGFSearchBench
Crypto Benchmarks
1 AESBench
2 MD5Bench
Jolden Benchmarks
1 TSPBench
Parrot Benchmarks
1 B0Compiler
2 B1UnboundedRecursion
3 B2GeneratePI
4 B3Sorting
5 B4Compiler
6 B5StandardLibrary
7 B6Iterators
Computer Shootout Language Benchmarks
1 NBody
2 BinaryTrees
3 ThreadRing
4 ChameneosRedux
5 RegexDNA
6 ReverseComplement
7 FannkuchRedux
8 SpectralNorm
9 Fasta
10 MeteorContest
Intercessive Pybench Benchmarks
1 instancesDynamic
2 lookupsDynamic
3 callsDynamic
Dynamic inheritance Benchmarks
1 PybenchDynamicInheritance
2 ShootoutDynamicInheritance
3 WordcountDynamicInheritance
4 ProductsManager
The Pystone Benchmark
1 Pystone

Programs

For a description of every measured program we used please click here.

BioPython Programs

A description of the BioPython program suite can be found here.
BioPyhon Program Suite
1 test_Ace
2 test_AlignIO_FastaIO
3 test_AlignIO_convert
4 test_BioSQL
5 test_BioSQL_SeqIO
6 test_CAPS
7 test_Chi2
8 test_CodonTable
9 test_CodonUsage
10 test_Compass
11 test_Crystal
12 test_EmbossPrimer
13 test_Entrez
14 test_Enzyme
15 test_File
16 test_FSSP
17 test_GACrossover
18 test_GAOrganism
19 test_GAQueens
20 test_GARepair
21 test_GASelection
22 test_GenBank
23 test_HMMCasino
24 test_HMMGeneral
25 test_IsoelectricPoint
26 test_KEGG
27 test_KeyWList
28 test_Location
29 test_Medline
30 test_Motif
31 test_NCBIStandalone
32 test_NCBITextParser
33 test_NCBIXML
34 test_Nexus
35 test_NNExclusiveOr
36 test_NNGene
37 test_NNGeneral
38 test_PAML_baseml
39 test_PAML_codeml
40 test_PAML_yn00
41 test_ParserSupport
42 test_Phd
43 test_Phylo
44 test_PhyloXML
45 test_PopGen_FDist_nodepend
46 test_PopGen_GenePop_nodepend
47 test_PopGen_SimCoal_nodepend
48 test_ProtParam
49 test_Restriction
50 test_SCOP_Astral
51 test_SCOP_Cla
52 test_SCOP_Des
53 test_SCOP_Dom
54 test_SCOP_Hie
55 test_SCOP_Raf
56 test_SCOP_Residues
57 test_SCOP_Scop
58 test_SeqIO
59 test_SeqIO_AbiIO
60 test_SeqIO_convert
61 test_SeqIO_FastaIO
62 test_SeqIO_features
63 test_SeqIO_index
64 test_SeqIO_QualityIO
65 test_SeqIO_SeqXML
66 test_SeqIO_write
67 test_SeqRecord
68 test_SeqUtils
69 test_Seq_objs
70 test_SffIO
71 test_SwissProt
72 test_translate
73 test_Tutorial
74 test_Uniprot
75 test_align
76 test_bgzf
77 test_geo
78 test_pairwise2
79 test_prodoc
80 test_prosite1
81 test_prosite2
82 test_seq
83 test_PopGen_SimCoal_nodepend
84 test_UniGene

Large Scale Applications

The large scale applications we used in our measurements are detailed here.