Performance Indicators Computation¶
Dedicated scripts are provided to compute performance indicators from the outcome of the different pipelines (strategic, pretactical and tactical).
The indicators focus mainly on passenger-centric mobility aspects as defined in the Digital Catalogue of Indicators developed as part of the Multimodal and Passengers Experience Performance Framework.
The main script to estimate the performance indicators is the mmx_kpis.py.
This with different parameters in the configuration TOML file can compute Strategic (planned network), Replanned and
Tactical. Ad-hoc scripts are also provided for particular computations for the pre-tactical and tactical evaluation.
Strategic indicators¶
The Strategic Indicators are computed using the mmx_kpis.py script.
Script mmx_kpis.py parameters
The script accepts some parameters:
- -tf: path to configuration TOML file
- -ex: name of the folder with processed data (after running strategic pipeline)
- -c: if we want to compare 2 experiments (need to have computed the indicators first for each experiment individually)
- -ppv: post-processing version (default 0): defines the number that is in file names, e.g. possible_itineraries_1.csv
- -sf: sufix to be added to the figures when generating them.
Examples of usage
python3 mmx_kpis.py -ex processed_cs10.pp00.so00_c1
python3 mmx_kpis.py -c processed_cs10.pp00.so00_c1 processed_cs10.pp10.so00_c1
python3 mmx_kpis.py -c processed_cs10.pp00.so00_c2 processed_c1_replan -ppv 0 1
Which indicators to compute and where to store them (as csv or plots) is controlled by a dedicated
TOML file (mmx_kpis.toml), described here:
MMX PIs TOML description for strategic and pre-tactical evaluator (mmx_kpis.toml)
[input]
# Define the input folders, i.e. where the results of the pipeline are located, and where some other input files
# are located (needed for example to generate plots)
path_to_strategic_output = "../data/CS10/v=0.16/output/"
path_to_tactical_output = "/home/michal/Documents/westminster/multimodx/results/3.1_9_1_0_ground_mobility__delay_mean_0/"
path_to_tactical_input = "/home/michal/Documents/westminster/multimodx/input/scenario=1/"
nuts_data_path = "/home/luis/MultiModX/data/EUROSTAT/NUTS_RG_01M_2021_4326.shp/NUTS_RG_01M_2021_4326.shp"
[output]
# Define where to save the results of running the script
# config['output']['path_to_output'] = Path(config['output']['path_to_output']) / args.experiment / 'indicators'
# config['output']['path_to_output_figs'] = (config['output']['path_to_output'] / 'figures')
# config['input']['preprocessed_version'] = args.preprocessed_version[0]
path_to_output = "../data/CS10/v=0.16/output/"
[indicators]
# This includes the list of indicators to be computed. These are grouped between:
# - strategic indicators (to be computed from the strategic pipeline (or the pre-tactical, as a replanning but
# computed in the replanned network only, i.e., not comparing with the original planned
# network).
# - replanned: specific indicators for replanned networks.
#
# See for each indicator examples of different variants (e.g. total_journey_time comptued as sum or average with
# different aggreations), and if a plot is to be generated or not. Some additional parameters might be possible
# e.g. when plotting indluceing min and max values for axis, etc, filtering for particular NUTS or aiports, etc.
# These configuarations are used by the libraries of kpi_lib_strategic.py and kpi_lib_replanned.py to compute
# the indicators. See the mmx_kpis.py code to see how this works. The code mainly computes all the results indicated
# in this TOML in a dictionary that then is used to gernerate the output (CSV/plots).
[indicators.strategic]
[[indicators.strategic.strategic_total_journey_time]]
variant = "sum"
plot = true
#name = "my_sum"
[[indicators.strategic.strategic_total_journey_time]]
variant = "avg_connecting_itineraries"
[[indicators.strategic.strategic_total_journey_time]]
variant = "avg_by_nuts"
plot = true
plot_matrix = true
vmin_matrix = 0
vmax_matrix = 800
vmin_matrix_nuts2 = 100
vmax_matrix_nuts2 = 600
[[indicators.strategic.strategic_total_journey_time]]
variant = "sum_per_region_archetype"
[[indicators.strategic.strategic_total_journey_time]]
variant = "avg_per_region_archetype"
[[indicators.strategic.diversity_of_destinations]]
variant = "nuts"
plot = true
[[indicators.strategic.diversity_of_destinations]]
variant = "hubs"
plot = true
[[indicators.strategic.modal_share]]
variant = "total"
[[indicators.strategic.modal_share]]
variant = "by_regional_archetype"
[[indicators.strategic.modal_share]]
variant = "by_nuts"
[[indicators.strategic.modal_share]]
variant = "between_nuts"
plot = true
plot_top = 50
[[indicators.strategic.modal_share]]
variant = "between_nuts_level2"
plot = true
plot_top = 20
[[indicators.strategic.pax_time_efficiency]]
variant = "total"
[[indicators.strategic.demand]]
variant = "all"
plot = true
top_od = 20
[[indicators.strategic.demand_served]]
variant = "total_connecting_itineraries"
[[indicators.strategic.demand_served]]
variant = "total"
[[indicators.strategic.demand_served]]
variant = "by_nuts"
vmin_matrix = 0
vmax_matrix = 1
plot = true
[[indicators.strategic.demand_served]]
variant = "by_regional_archetype"
vmin_matrix = 0.7
vmax_matrix = 1
plot = true
[[indicators.strategic.demand_served]]
variant = "by_od"
od = [['ES617', 'ES111'],
['ES111', 'ES617'],
['ES111', 'ES512'],
['ES512', 'ES111']]
figure_ending = '0'
[[indicators.strategic.demand_served]]
variant = "by_od"
od = [['ES511', 'ES300'],
['ES300', 'ES511'],
]
figure_ending = '1'
[[indicators.strategic.load_factor]]
variant = "modes"
[[indicators.strategic.load_factor]]
variant = "total"
[[indicators.strategic.resilience_alternatives]]
variant = "by_nuts"
vmin_matrix = 0
vmax_matrix = 60
vmin_matrix_air = 0
vmax_matrix_air = 36
vmin_matrix_rail = 10
vmax_matrix_rail = 60
vmin_matrix_multimodal = 0
vmax_matrix_multimodal = 30
vmin_matrix_nuts2 = 25
vmax_matrix_nuts2 = 500
vmin_matrix_nuts2_air = 25
vmax_matrix_nuts2_air = 300
vmin_matrix_nuts2_rail = 25
vmax_matrix_nuts2_rail = 300
vmin_matrix_nuts2_multimodal = 10
vmax_matrix_nuts2_multimodal = 200
plot = true
[[indicators.strategic.resilience_alternatives]]
variant = "by_regional_archetype"
plot = true
vmin_matrix = 400
vmax_matrix = 6000
[[indicators.strategic.resilience_alternatives]]
variant = "by_regional_archetype"
[[indicators.strategic.buffer_in_itineraries]]
variant = "avg"
plot = true
[[indicators.strategic.buffer_in_itineraries]]
variant = "sum"
[[indicators.strategic.catchment_area]]
variant = "hubs_rail_time"
plot = true
[[indicators.strategic.catchment_area]]
variant = "access_egress"
plot = true
vmin = 0
vmax = 1000
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEMD'
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEBL'
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEVD'
vmin = 0
vmax = 100
topleft = [43.67, -9]
bottomright = [39.6, -2.3]
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEMG'
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEZL'
[[indicators.strategic.catchment_area]]
variant = "rail_stop_pax"
plot = true
vmin = 0
vmax = 1000
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEMD'
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEBL'
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEVD'
vmin = 0
vmax = 100
topleft = [43.67, -9]
bottomright = [39.6, -2.3]
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEMG'
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEZL'
[[indicators.strategic.catchment_area]]
variant = "access_egress_rail_stop"
plot = true
vmin = 0
vmax = 1000
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEMD'
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEBL'
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEVD'
vmin = 0
vmax = 100
topleft = [43.67, -9]
bottomright = [39.6, -2.3]
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEMG'
[[indicators.strategic.catchment_area.plot_airports]]
airport= 'LEZL'
[[indicators.strategic.cost_per_user]]
variant = "avg"
[[indicators.strategic.co2_emissions]]
variant = "avg"
[[indicators.strategic.seamless_of_travel]]
variant = "avg"
[[indicators.strategic.pax_processes_time]]
variant = "avg"
[[indicators.strategic.capacity_available]]
variant = "all"
plot = true
vmax_nuts2 = 20000
vmax_nuts3 = 3000
od_capacity_modes_nuts3 = [ ['ES511', 'ES300'],
['ES300', 'ES511'],
['ES617', 'ES111'],
['ES111', 'ES617'],
['ES111', 'ES512'],
['ES512', 'ES111']]
od_capacity_modes_nuts2 = [['ES30', 'ES51']]
[[indicators.strategic.connectivity_infrastructure]]
variant = 'all'
plot = true
top_rail = 15
top_air = 20
plot_map = true
top_rail_map = 10
top_air_map = 15
exclude_nuts = ['ES703', 'ES704', 'ES705', 'ES706', 'ES707', 'ES708', 'ES709']
topleft = [44, -9.5]
bottomright = [35.9, 4.5]
[indicators.replanned]
[[indicators.replanned.resilience_replanned]]
variant = "total"
[[indicators.replanned.pax_resilience_replanned]]
variant = "total"
The mmx_kpis.py script computes the indicators that are defined in the TOML considering any variant (e.g. sum, avg) and
possible filtering (e.g. NUTS, airports), stores the results in a dictionary and then save these in CSV and/or plots (again)
as indicated in the TOML.
Pre-tactical (replanned operations) indicators¶
For the Pre-tactical pipeline, the result PIs are computed using the same script and TOML as for the Strategic indicators case. Therefore, see that section for more information.
PI computation is performed after:
- Passenger reassignment is completed
- Final itineraries and service loads are available
PIs depend on the final state of the replanned network and passenger assignments.
MMX Replanning PIs¶
Besides the network indicators, the MultiModX repository provides an ad-hoc script mmx_replanning_pis.py which computes
indicators to describe the replanning of the network. Please, refer to that script to have more information. This script
does not have a dedicated TOML configuration file, but it is controlled by using parameters when calling the
script.
Script mmx_replanning_pis.py parameters
- -pr: Folder with the replanned results
- -v: Version of the case study
- -cs: Which case study to analyse
- -nd: Which network definition parameters to do
- -pp: Which policy package to do
- -so: Which scheduler optimiser to do
- -dp: Which disruption package to do
- -dm: Which disruption management to do
- -pa: Which version of pax assigment to do
- -pamin: If provided instead of computing indicators compute aggregation across PAs
- -pamax: If provided instead of computing indicators compute aggregation across PAs
Purpose¶
The PI module evaluates the system-level and passenger-level impacts of disruptions and replanning decisions.
It enables: - Comparison of alternative replanning strategies - Quantification of passenger inconvenience - Assessment of operational robustness
Usage¶
Typical workflow:
- Run the pre-tactical replanning pipeline (see Pre-Tactical Passenger Replanning Pipeline)
- Collect final passenger and service CSV outputs
- Invoke PI routines from
kpi_lib_replanned.py - Analyse PIs for scenario comparison and reporting
Intended Use¶
The PI module supports: - Research analysis - Scenario benchmarking - Policy and operational evaluation - Input generation for higher-level decision support tools
Inputs¶
KPIs are computed using outputs from the replanning pipeline, including:
- Reassigned passenger itineraries
- Final replanned schedules (air and rail)
- Passenger delays and missed connections
- Service capacity utilisation
Key Indicators¶
Typical PIs computed on replanned network
-
Passenger-Centric
- Total travel time increase
- Delay distributions
- Missed connections
- Denied boarding events
- Number of stranded passengers
-
Service-Centric
- Load factor variation
- Capacity utilisation
- Service saturation levels
-
System-Level
- Mode share changes (air / rail / multimodal)
- Reaccommodated vs unmet demand
- Network resilience metrics
Tactical indicators¶
In addition to the mmx_kpis.py script, a postprocessing_tactical.py
script is also provided. This script post-process the execution of the tactical evaluator to add the modelling of the
passenger itineraries which are not directly supported by the Tactical Evaluator. This script is configured with a
dedicated TOML file.
Tactical post-processing TOML description (postprocessing.toml)
[input]
path_to_strategic_output = "../../data/CS10/v=0.16/output/processed_cs10.pp00.nd00.so00.00/paths_itineraries"
path_to_tactical_output = "/home/michal/Documents/westminster/multimodx/results/"
tactical_output_name = "3.1_1_0"
parameter_name = ""#"ground_mobility__delay_mean_30"
ppv = 0
iterations = 10
path_to_tactical_input = "/home/michal/Documents/westminster/multimodx/input/scenario=1/"
path_to_scenario = "/home/michal/Documents/westminster/multimodx/input/scenario=1/scenario_config.toml"
Finally, as mentioned, the mmx_kpis.py
can compute the performance indicators related to the tactical execution of the mobility network. The input required is
the TOML configuration file defining the indicators for the tactical execution. Note that the definition of the input
also requires a few more parameters to characterise the tactical execution.
MMX PIs TOML description for tactical evaluator (mmx_kpis_tactical.toml)
[input]
# Define the input folders, i.e. where the results of the pipeline are located, and where some other input files
# are located (needed for example to generate plots). Some additional information for the tactical evaluator needed.
path_to_strategic_output = "../../data/CS10/v=0.16/output/"
path_to_tactical_output = "/home/michal/Documents/westminster/multimodx/results/"
tactical_output_name = "3.1_1_0"
parameter_name = ""#"ground_mobility__delay_mean_30"
iterations = 10
path_to_tactical_input = "/home/michal/Documents/westminster/multimodx/input/scenario=1/"
nuts_data_path = "/home/michal/Documents/westminster/multimodx/data/demand/data/NUTS_RG_01M_2021_4326_shp.shp"
[output]
# Define where to save the results of running the script
path_to_output = "../../data/CS10/v=0.16/output/"
[indicators]
# This includes the list of indicators to be computed. Same principle as in mmx_kpis.tom (see documentation there)
# but with the category tactical for tactical indicators.
[indicators.tactical]
[[indicators.tactical.total_arrival_delay]]
variant = "total"
#name = "my_sum"
[[indicators.tactical.total_arrival_delay]]
variant = "missed_connection"
[[indicators.tactical.stranded_pax]]
variant = "total"
[[indicators.tactical.stranded_pax]]
variant = "missed_air2rail"
[[indicators.tactical.stranded_pax]]
variant = "missed_rail2air"
[[indicators.tactical.stranded_pax]]
variant = "abs"
[[indicators.tactical.ratio_stranded_pax]]
variant = "total"
[[indicators.tactical.missed_connections]]
variant = "all"
[[indicators.tactical.total_journey_time]]
variant = "total"
[[indicators.tactical.flight_arrival_delay]]
variant = "total"
[[indicators.tactical.variability]]
variant = "total"