google-research
2240 строк · 173.5 Кб
1{2"cells": [3{4"cell_type": "markdown",5"metadata": {6"id": "YajxPiJCuy5j"7},8"source": [9"##### Copyright 2020 Google LLC.\n",10"Licensed under the Apache License, Version 2.0 (the \"License\")"11]12},13{14"cell_type": "code",15"execution_count": null,16"metadata": {17"id": "7mLBFZ-uu-Bh"18},19"outputs": [],20"source": [21"# Copyright 2022 The Google Research Authors.\n",22"#\n",23"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",24"# you may not use this file except in compliance with the License.\n",25"# You may obtain a copy of the License at\n",26"#\n",27"# https://www.apache.org/licenses/LICENSE-2.0\n",28"#\n",29"# Unless required by applicable law or agreed to in writing, software\n",30"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",31"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",32"# See the License for the specific language governing permissions and\n",33"# limitations under the License."34]35},36{37"cell_type": "markdown",38"metadata": {39"id": "9CimtUIDwna4"40},41"source": [42"# Human evaluation of visual metrics\n",43"\n",44"This colab explores correlations between the mucped22 dataset and various objective visual metrics.\n",45"\n",46"Unlike many other datasets, these evaluations:\n",47" * Are made using only compression distortions (since the distortions were created using AVIF, JPEG, and JXL encoders), which will benefit metrics designed for compression artifacts.\n",48" * Are made by human evaluators experienced with image quality comparisons, which will benefit smaller distortions, relatively speaking, since unexperienced evaluators often don't notice them.\n",49" * Are made using two-alternative-forced-choice with a reference image instead of MOS, which will reduce the noise, since evaluators don't need to calibrate their MOS scores.\n",50"\n",51"For each image, each distortion (method and quality setting) has then been ranked using ELO, to provide an expected human-rated ranking for each distortion.\n",52"\n",53"This ranking will allow a comparison of the various metrics across different levels of distortion, e.g. near just-noticeable-differences vs far from just-noticeable-differences.\n",54"\n",55" "56]57},58{59"cell_type": "code",60"execution_count": null,61"metadata": {62"colab": {63"base_uri": "https://localhost:8080/"64},65"id": "mVjmAAMuLszT",66"outputId": "6c5207c9-0da0-4896-dc66-fcc00af78e37"67},68"outputs": [69{70"name": "stdout",71"output_type": "stream",72"text": [73"Collecting pandas_bokeh\n",74" Downloading pandas_bokeh-0.5.5-py2.py3-none-any.whl (29 kB)\n",75"Requirement already satisfied: bokeh\u003e=2.0 in /usr/local/lib/python3.7/dist-packages (from pandas_bokeh) (2.3.3)\n",76"Requirement already satisfied: pandas\u003e=0.22.0 in /usr/local/lib/python3.7/dist-packages (from pandas_bokeh) (1.3.5)\n",77"Requirement already satisfied: numpy\u003e=1.11.3 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (1.21.5)\n",78"Requirement already satisfied: Jinja2\u003e=2.9 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (2.11.3)\n",79"Requirement already satisfied: python-dateutil\u003e=2.1 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (2.8.2)\n",80"Requirement already satisfied: tornado\u003e=5.1 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (5.1.1)\n",81"Requirement already satisfied: PyYAML\u003e=3.10 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (3.13)\n",82"Requirement already satisfied: pillow\u003e=7.1.0 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (7.1.2)\n",83"Requirement already satisfied: typing-extensions\u003e=3.7.4 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (3.10.0.2)\n",84"Requirement already satisfied: packaging\u003e=16.8 in /usr/local/lib/python3.7/dist-packages (from bokeh\u003e=2.0-\u003epandas_bokeh) (21.3)\n",85"Requirement already satisfied: MarkupSafe\u003e=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2\u003e=2.9-\u003ebokeh\u003e=2.0-\u003epandas_bokeh) (2.0.1)\n",86"Requirement already satisfied: pyparsing!=3.0.5,\u003e=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging\u003e=16.8-\u003ebokeh\u003e=2.0-\u003epandas_bokeh) (3.0.7)\n",87"Requirement already satisfied: pytz\u003e=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas\u003e=0.22.0-\u003epandas_bokeh) (2018.9)\n",88"Requirement already satisfied: six\u003e=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil\u003e=2.1-\u003ebokeh\u003e=2.0-\u003epandas_bokeh) (1.15.0)\n",89"Installing collected packages: pandas-bokeh\n",90"Successfully installed pandas-bokeh-0.5.5\n"91]92}93],94"source": [95"import pandas as pd\n",96"import functools\n",97"import bokeh.io\n",98"!pip install pandas_bokeh\n",99"import pandas_bokeh\n",100"import requests\n",101"import json\n",102"import numpy as np\n",103"bokeh.io.output_notebook()"104]105},106{107"cell_type": "markdown",108"metadata": {109"id": "1jPN8OpOy0Kn"110},111"source": [112"First download the dataset containing all evaluations."113]114},115{116"cell_type": "code",117"execution_count": null,118"metadata": {119"id": "N4R1dG01sVEJ"120},121"outputs": [],122"source": [123"!wget --quiet --no-check-certificate https://storage.googleapis.com/gresearch/mucped22/evaluations.json"124]125},126{127"cell_type": "code",128"execution_count": null,129"metadata": {130"id": "osV-uT-Kh4HK"131},132"outputs": [],133"source": [134"with open('evaluations.json') as f:\n",135" data = pd.DataFrame(json.load(f))"136]137},138{139"cell_type": "code",140"execution_count": null,141"metadata": {142"colab": {143"base_uri": "https://localhost:8080/",144"height": 1000145},146"id": "eGIkAdlWo-di",147"outputId": "b21eefd6-cff2-444a-ae08-ea5556e78e19"148},149"outputs": [150{151"data": {152"text/html": [153"\n",154" \u003cdiv id=\"df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32\"\u003e\n",155" \u003cdiv class=\"colab-df-container\"\u003e\n",156" \u003cdiv\u003e\n",157"\u003cstyle scoped\u003e\n",158" .dataframe tbody tr th:only-of-type {\n",159" vertical-align: middle;\n",160" }\n",161"\n",162" .dataframe tbody tr th {\n",163" vertical-align: top;\n",164" }\n",165"\n",166" .dataframe thead th {\n",167" text-align: right;\n",168" }\n",169"\u003c/style\u003e\n",170"\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",171" \u003cthead\u003e\n",172" \u003ctr style=\"text-align: right;\"\u003e\n",173" \u003cth\u003e\u003c/th\u003e\n",174" \u003cth\u003ecrop\u003c/th\u003e\n",175" \u003cth\u003egreater\u003c/th\u003e\n",176" \u003cth\u003eimage\u003c/th\u003e\n",177" \u003cth\u003elesser\u003c/th\u003e\n",178" \u003cth\u003erandom_choice\u003c/th\u003e\n",179" \u003cth\u003erater_time_ms\u003c/th\u003e\n",180" \u003cth\u003eimage_dims\u003c/th\u003e\n",181" \u003cth\u003egreater_butteraugli_max\u003c/th\u003e\n",182" \u003cth\u003egreater_butteraugli_6\u003c/th\u003e\n",183" \u003cth\u003elesser_butteraugli_max\u003c/th\u003e\n",184" \u003cth\u003e...\u003c/th\u003e\n",185" \u003cth\u003elesser_fsim-rgb\u003c/th\u003e\n",186" \u003cth\u003egreater_msssim-y\u003c/th\u003e\n",187" \u003cth\u003elesser_msssim-y\u003c/th\u003e\n",188" \u003cth\u003egreater_nlpd-y\u003c/th\u003e\n",189" \u003cth\u003elesser_nlpd-y\u003c/th\u003e\n",190" \u003cth\u003egreater_ssimulacra\u003c/th\u003e\n",191" \u003cth\u003elesser_ssimulacra\u003c/th\u003e\n",192" \u003cth\u003egreater_elo\u003c/th\u003e\n",193" \u003cth\u003elesser_elo\u003c/th\u003e\n",194" \u003cth\u003erater_flips\u003c/th\u003e\n",195" \u003c/tr\u003e\n",196" \u003c/thead\u003e\n",197" \u003ctbody\u003e\n",198" \u003ctr\u003e\n",199" \u003cth\u003e0\u003c/th\u003e\n",200" \u003ctd\u003e[176, 279, 768, 768]\u003c/td\u003e\n",201" \u003ctd\u003edist_6\u003c/td\u003e\n",202" \u003ctd\u003eedgar-castrejon-J_NbCcGPBOw-unsplash.png\u003c/td\u003e\n",203" \u003ctd\u003edist_17\u003c/td\u003e\n",204" \u003ctd\u003eFalse\u003c/td\u003e\n",205" \u003ctd\u003e75125\u003c/td\u003e\n",206" \u003ctd\u003e[1280, 1920]\u003c/td\u003e\n",207" \u003ctd\u003e6.792748\u003c/td\u003e\n",208" \u003ctd\u003e3.291877\u003c/td\u003e\n",209" \u003ctd\u003e2.287096\u003c/td\u003e\n",210" \u003ctd\u003e...\u003c/td\u003e\n",211" \u003ctd\u003e0.998617\u003c/td\u003e\n",212" \u003ctd\u003e0.975003\u003c/td\u003e\n",213" \u003ctd\u003e0.993973\u003c/td\u003e\n",214" \u003ctd\u003e0.161716\u003c/td\u003e\n",215" \u003ctd\u003e0.078657\u003c/td\u003e\n",216" \u003ctd\u003e0.096484\u003c/td\u003e\n",217" \u003ctd\u003e0.045761\u003c/td\u003e\n",218" \u003ctd\u003e1537.137226\u003c/td\u003e\n",219" \u003ctd\u003e2172.213746\u003c/td\u003e\n",220" \u003ctd\u003e9\u003c/td\u003e\n",221" \u003c/tr\u003e\n",222" \u003ctr\u003e\n",223" \u003cth\u003e1\u003c/th\u003e\n",224" \u003ctd\u003e[149, 32, 768, 768]\u003c/td\u003e\n",225" \u003ctd\u003edist_6\u003c/td\u003e\n",226" \u003ctd\u003e4KK2_20150823_130650_314.png\u003c/td\u003e\n",227" \u003ctd\u003edist_11\u003c/td\u003e\n",228" \u003ctd\u003eFalse\u003c/td\u003e\n",229" \u003ctd\u003e41883\u003c/td\u003e\n",230" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",231" \u003ctd\u003e6.830727\u003c/td\u003e\n",232" \u003ctd\u003e3.382573\u003c/td\u003e\n",233" \u003ctd\u003e1.990106\u003c/td\u003e\n",234" \u003ctd\u003e...\u003c/td\u003e\n",235" \u003ctd\u003e0.998584\u003c/td\u003e\n",236" \u003ctd\u003e0.985169\u003c/td\u003e\n",237" \u003ctd\u003e0.997969\u003c/td\u003e\n",238" \u003ctd\u003e0.145104\u003c/td\u003e\n",239" \u003ctd\u003e0.050339\u003c/td\u003e\n",240" \u003ctd\u003e0.072111\u003c/td\u003e\n",241" \u003ctd\u003e0.019148\u003c/td\u003e\n",242" \u003ctd\u003e1972.598657\u003c/td\u003e\n",243" \u003ctd\u003e2291.390798\u003c/td\u003e\n",244" \u003ctd\u003e13\u003c/td\u003e\n",245" \u003c/tr\u003e\n",246" \u003ctr\u003e\n",247" \u003cth\u003e2\u003c/th\u003e\n",248" \u003ctd\u003e[1089, 489, 768, 768]\u003c/td\u003e\n",249" \u003ctd\u003edist_11\u003c/td\u003e\n",250" \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",251" \u003ctd\u003edist_17\u003c/td\u003e\n",252" \u003ctd\u003eFalse\u003c/td\u003e\n",253" \u003ctd\u003e48385\u003c/td\u003e\n",254" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",255" \u003ctd\u003e1.760315\u003c/td\u003e\n",256" \u003ctd\u003e0.828016\u003c/td\u003e\n",257" \u003ctd\u003e1.656251\u003c/td\u003e\n",258" \u003ctd\u003e...\u003c/td\u003e\n",259" \u003ctd\u003e0.999163\u003c/td\u003e\n",260" \u003ctd\u003e0.999303\u003c/td\u003e\n",261" \u003ctd\u003e0.999264\u003c/td\u003e\n",262" \u003ctd\u003e0.030033\u003c/td\u003e\n",263" \u003ctd\u003e0.030637\u003c/td\u003e\n",264" \u003ctd\u003e0.010733\u003c/td\u003e\n",265" \u003ctd\u003e0.011180\u003c/td\u003e\n",266" \u003ctd\u003e2268.583209\u003c/td\u003e\n",267" \u003ctd\u003e2309.359159\u003c/td\u003e\n",268" \u003ctd\u003e4\u003c/td\u003e\n",269" \u003c/tr\u003e\n",270" \u003ctr\u003e\n",271" \u003cth\u003e3\u003c/th\u003e\n",272" \u003ctd\u003e[433, 286, 768, 768]\u003c/td\u003e\n",273" \u003ctd\u003edist_14\u003c/td\u003e\n",274" \u003ctd\u003emichael-niessl-KraoHdRYrRE-unsplash.png\u003c/td\u003e\n",275" \u003ctd\u003edist_11\u003c/td\u003e\n",276" \u003ctd\u003eTrue\u003c/td\u003e\n",277" \u003ctd\u003e36774\u003c/td\u003e\n",278" \u003ctd\u003e[1920, 1080]\u003c/td\u003e\n",279" \u003ctd\u003e4.110695\u003c/td\u003e\n",280" \u003ctd\u003e1.995982\u003c/td\u003e\n",281" \u003ctd\u003e2.432354\u003c/td\u003e\n",282" \u003ctd\u003e...\u003c/td\u003e\n",283" \u003ctd\u003e0.998708\u003c/td\u003e\n",284" \u003ctd\u003e0.993285\u003c/td\u003e\n",285" \u003ctd\u003e0.998220\u003c/td\u003e\n",286" \u003ctd\u003e0.071748\u003c/td\u003e\n",287" \u003ctd\u003e0.037420\u003c/td\u003e\n",288" \u003ctd\u003e0.043709\u003c/td\u003e\n",289" \u003ctd\u003e0.019095\u003c/td\u003e\n",290" \u003ctd\u003e1991.945557\u003c/td\u003e\n",291" \u003ctd\u003e2241.368691\u003c/td\u003e\n",292" \u003ctd\u003e10\u003c/td\u003e\n",293" \u003c/tr\u003e\n",294" \u003ctr\u003e\n",295" \u003cth\u003e4\u003c/th\u003e\n",296" \u003ctd\u003e[4, 99, 768, 768]\u003c/td\u003e\n",297" \u003ctd\u003edist_6\u003c/td\u003e\n",298" \u003ctd\u003e0127_20161022_144117_906.png\u003c/td\u003e\n",299" \u003ctd\u003edist_14\u003c/td\u003e\n",300" \u003ctd\u003eTrue\u003c/td\u003e\n",301" \u003ctd\u003e22468\u003c/td\u003e\n",302" \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",303" \u003ctd\u003e5.866006\u003c/td\u003e\n",304" \u003ctd\u003e2.645947\u003c/td\u003e\n",305" \u003ctd\u003e3.550395\u003c/td\u003e\n",306" \u003ctd\u003e...\u003c/td\u003e\n",307" \u003ctd\u003e0.994832\u003c/td\u003e\n",308" \u003ctd\u003e0.988954\u003c/td\u003e\n",309" \u003ctd\u003e0.994369\u003c/td\u003e\n",310" \u003ctd\u003e0.095547\u003c/td\u003e\n",311" \u003ctd\u003e0.069588\u003c/td\u003e\n",312" \u003ctd\u003e0.049872\u003c/td\u003e\n",313" \u003ctd\u003e0.035815\u003c/td\u003e\n",314" \u003ctd\u003e1851.898409\u003c/td\u003e\n",315" \u003ctd\u003e1945.269660\u003c/td\u003e\n",316" \u003ctd\u003e9\u003c/td\u003e\n",317" \u003c/tr\u003e\n",318" \u003ctr\u003e\n",319" \u003cth\u003e...\u003c/th\u003e\n",320" \u003ctd\u003e...\u003c/td\u003e\n",321" \u003ctd\u003e...\u003c/td\u003e\n",322" \u003ctd\u003e...\u003c/td\u003e\n",323" \u003ctd\u003e...\u003c/td\u003e\n",324" \u003ctd\u003e...\u003c/td\u003e\n",325" \u003ctd\u003e...\u003c/td\u003e\n",326" \u003ctd\u003e...\u003c/td\u003e\n",327" \u003ctd\u003e...\u003c/td\u003e\n",328" \u003ctd\u003e...\u003c/td\u003e\n",329" \u003ctd\u003e...\u003c/td\u003e\n",330" \u003ctd\u003e...\u003c/td\u003e\n",331" \u003ctd\u003e...\u003c/td\u003e\n",332" \u003ctd\u003e...\u003c/td\u003e\n",333" \u003ctd\u003e...\u003c/td\u003e\n",334" \u003ctd\u003e...\u003c/td\u003e\n",335" \u003ctd\u003e...\u003c/td\u003e\n",336" \u003ctd\u003e...\u003c/td\u003e\n",337" \u003ctd\u003e...\u003c/td\u003e\n",338" \u003ctd\u003e...\u003c/td\u003e\n",339" \u003ctd\u003e...\u003c/td\u003e\n",340" \u003ctd\u003e...\u003c/td\u003e\n",341" \u003c/tr\u003e\n",342" \u003ctr\u003e\n",343" \u003cth\u003e12978\u003c/th\u003e\n",344" \u003ctd\u003e[887, 59, 768, 768]\u003c/td\u003e\n",345" \u003ctd\u003edist_23\u003c/td\u003e\n",346" \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",347" \u003ctd\u003edist_40\u003c/td\u003e\n",348" \u003ctd\u003eFalse\u003c/td\u003e\n",349" \u003ctd\u003e7983\u003c/td\u003e\n",350" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",351" \u003ctd\u003e1.877585\u003c/td\u003e\n",352" \u003ctd\u003e0.923585\u003c/td\u003e\n",353" \u003ctd\u003e2.143382\u003c/td\u003e\n",354" \u003ctd\u003e...\u003c/td\u003e\n",355" \u003ctd\u003e0.999173\u003c/td\u003e\n",356" \u003ctd\u003e0.998298\u003c/td\u003e\n",357" \u003ctd\u003e0.998512\u003c/td\u003e\n",358" \u003ctd\u003e0.051131\u003c/td\u003e\n",359" \u003ctd\u003e0.048955\u003c/td\u003e\n",360" \u003ctd\u003e0.020006\u003c/td\u003e\n",361" \u003ctd\u003e0.017322\u003c/td\u003e\n",362" \u003ctd\u003e2356.875932\u003c/td\u003e\n",363" \u003ctd\u003e2388.690716\u003c/td\u003e\n",364" \u003ctd\u003e6\u003c/td\u003e\n",365" \u003c/tr\u003e\n",366" \u003ctr\u003e\n",367" \u003cth\u003e12979\u003c/th\u003e\n",368" \u003ctd\u003e[1121, 175, 768, 768]\u003c/td\u003e\n",369" \u003ctd\u003edist_32\u003c/td\u003e\n",370" \u003ctd\u003e47L8_20150518_144020_604.png\u003c/td\u003e\n",371" \u003ctd\u003edist_30\u003c/td\u003e\n",372" \u003ctd\u003eTrue\u003c/td\u003e\n",373" \u003ctd\u003e1702517\u003c/td\u003e\n",374" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",375" \u003ctd\u003e4.943964\u003c/td\u003e\n",376" \u003ctd\u003e2.460808\u003c/td\u003e\n",377" \u003ctd\u003e8.950997\u003c/td\u003e\n",378" \u003ctd\u003e...\u003c/td\u003e\n",379" \u003ctd\u003e0.969582\u003c/td\u003e\n",380" \u003ctd\u003e0.983741\u003c/td\u003e\n",381" \u003ctd\u003e0.955344\u003c/td\u003e\n",382" \u003ctd\u003e0.127071\u003c/td\u003e\n",383" \u003ctd\u003e0.201221\u003c/td\u003e\n",384" \u003ctd\u003e0.071460\u003c/td\u003e\n",385" \u003ctd\u003e0.155907\u003c/td\u003e\n",386" \u003ctd\u003e1937.790267\u003c/td\u003e\n",387" \u003ctd\u003e1402.672706\u003c/td\u003e\n",388" \u003ctd\u003e5\u003c/td\u003e\n",389" \u003c/tr\u003e\n",390" \u003ctr\u003e\n",391" \u003cth\u003e12980\u003c/th\u003e\n",392" \u003ctd\u003e[1121, 175, 768, 768]\u003c/td\u003e\n",393" \u003ctd\u003edist_21\u003c/td\u003e\n",394" \u003ctd\u003e0006_20160721_220618_556.png\u003c/td\u003e\n",395" \u003ctd\u003edist_32\u003c/td\u003e\n",396" \u003ctd\u003eTrue\u003c/td\u003e\n",397" \u003ctd\u003e1430\u003c/td\u003e\n",398" \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",399" \u003ctd\u003e2.339564\u003c/td\u003e\n",400" \u003ctd\u003e1.123564\u003c/td\u003e\n",401" \u003ctd\u003e4.024082\u003c/td\u003e\n",402" \u003ctd\u003e...\u003c/td\u003e\n",403" \u003ctd\u003e0.997534\u003c/td\u003e\n",404" \u003ctd\u003e0.999049\u003c/td\u003e\n",405" \u003ctd\u003e0.998433\u003c/td\u003e\n",406" \u003ctd\u003e0.029906\u003c/td\u003e\n",407" \u003ctd\u003e0.036302\u003c/td\u003e\n",408" \u003ctd\u003e0.011507\u003c/td\u003e\n",409" \u003ctd\u003e0.012152\u003c/td\u003e\n",410" \u003ctd\u003e2202.608359\u003c/td\u003e\n",411" \u003ctd\u003e1870.660112\u003c/td\u003e\n",412" \u003ctd\u003e0\u003c/td\u003e\n",413" \u003c/tr\u003e\n",414" \u003ctr\u003e\n",415" \u003cth\u003e12981\u003c/th\u003e\n",416" \u003ctd\u003e[84, 358, 768, 768]\u003c/td\u003e\n",417" \u003ctd\u003edist_10\u003c/td\u003e\n",418" \u003ctd\u003ericardo-gomez-angel-2GglDxer-Ew-unsplash.png\u003c/td\u003e\n",419" \u003ctd\u003edist_32\u003c/td\u003e\n",420" \u003ctd\u003eTrue\u003c/td\u003e\n",421" \u003ctd\u003e5731\u003c/td\u003e\n",422" \u003ctd\u003e[1920, 1319]\u003c/td\u003e\n",423" \u003ctd\u003e3.816417\u003c/td\u003e\n",424" \u003ctd\u003e1.697080\u003c/td\u003e\n",425" \u003ctd\u003e5.742501\u003c/td\u003e\n",426" \u003ctd\u003e...\u003c/td\u003e\n",427" \u003ctd\u003e0.985294\u003c/td\u003e\n",428" \u003ctd\u003e0.991662\u003c/td\u003e\n",429" \u003ctd\u003e0.980832\u003c/td\u003e\n",430" \u003ctd\u003e0.105602\u003c/td\u003e\n",431" \u003ctd\u003e0.189351\u003c/td\u003e\n",432" \u003ctd\u003e0.048339\u003c/td\u003e\n",433" \u003ctd\u003e0.077644\u003c/td\u003e\n",434" \u003ctd\u003e2174.169512\u003c/td\u003e\n",435" \u003ctd\u003e2083.620449\u003c/td\u003e\n",436" \u003ctd\u003e1\u003c/td\u003e\n",437" \u003c/tr\u003e\n",438" \u003ctr\u003e\n",439" \u003cth\u003e12982\u003c/th\u003e\n",440" \u003ctd\u003e[84, 358, 768, 768]\u003c/td\u003e\n",441" \u003ctd\u003edist_10\u003c/td\u003e\n",442" \u003ctd\u003e6G7M_20150403_174605_886.png\u003c/td\u003e\n",443" \u003ctd\u003edist_21\u003c/td\u003e\n",444" \u003ctd\u003eTrue\u003c/td\u003e\n",445" \u003ctd\u003e339\u003c/td\u003e\n",446" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",447" \u003ctd\u003e3.647443\u003c/td\u003e\n",448" \u003ctd\u003e1.543176\u003c/td\u003e\n",449" \u003ctd\u003e3.500668\u003c/td\u003e\n",450" \u003ctd\u003e...\u003c/td\u003e\n",451" \u003ctd\u003e0.996085\u003c/td\u003e\n",452" \u003ctd\u003e0.995072\u003c/td\u003e\n",453" \u003ctd\u003e0.991573\u003c/td\u003e\n",454" \u003ctd\u003e0.067868\u003c/td\u003e\n",455" \u003ctd\u003e0.094789\u003c/td\u003e\n",456" \u003ctd\u003e0.022608\u003c/td\u003e\n",457" \u003ctd\u003e0.036871\u003c/td\u003e\n",458" \u003ctd\u003e2235.728150\u003c/td\u003e\n",459" \u003ctd\u003e2219.418325\u003c/td\u003e\n",460" \u003ctd\u003e0\u003c/td\u003e\n",461" \u003c/tr\u003e\n",462" \u003c/tbody\u003e\n",463"\u003c/table\u003e\n",464"\u003cp\u003e12983 rows × 24 columns\u003c/p\u003e\n",465"\u003c/div\u003e\n",466" \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32')\"\n",467" title=\"Convert this dataframe to an interactive table.\"\n",468" style=\"display:none;\"\u003e\n",469" \n",470" \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",471" width=\"24px\"\u003e\n",472" \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",473" \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",474" \u003c/svg\u003e\n",475" \u003c/button\u003e\n",476" \n",477" \u003cstyle\u003e\n",478" .colab-df-container {\n",479" display:flex;\n",480" flex-wrap:wrap;\n",481" gap: 12px;\n",482" }\n",483"\n",484" .colab-df-convert {\n",485" background-color: #E8F0FE;\n",486" border: none;\n",487" border-radius: 50%;\n",488" cursor: pointer;\n",489" display: none;\n",490" fill: #1967D2;\n",491" height: 32px;\n",492" padding: 0 0 0 0;\n",493" width: 32px;\n",494" }\n",495"\n",496" .colab-df-convert:hover {\n",497" background-color: #E2EBFA;\n",498" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",499" fill: #174EA6;\n",500" }\n",501"\n",502" [theme=dark] .colab-df-convert {\n",503" background-color: #3B4455;\n",504" fill: #D2E3FC;\n",505" }\n",506"\n",507" [theme=dark] .colab-df-convert:hover {\n",508" background-color: #434B5C;\n",509" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",510" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",511" fill: #FFFFFF;\n",512" }\n",513" \u003c/style\u003e\n",514"\n",515" \u003cscript\u003e\n",516" const buttonEl =\n",517" document.querySelector('#df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32 button.colab-df-convert');\n",518" buttonEl.style.display =\n",519" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",520"\n",521" async function convertToInteractive(key) {\n",522" const element = document.querySelector('#df-9c5949dd-e0f8-42d9-8efe-46c4aa30bb32');\n",523" const dataTable =\n",524" await google.colab.kernel.invokeFunction('convertToInteractive',\n",525" [key], {});\n",526" if (!dataTable) return;\n",527"\n",528" const docLinkHtml = 'Like what you see? Visit the ' +\n",529" '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",530" + ' to learn more about interactive tables.';\n",531" element.innerHTML = '';\n",532" dataTable['output_type'] = 'display_data';\n",533" await google.colab.output.renderOutput(dataTable, element);\n",534" const docLink = document.createElement('div');\n",535" docLink.innerHTML = docLinkHtml;\n",536" element.appendChild(docLink);\n",537" }\n",538" \u003c/script\u003e\n",539" \u003c/div\u003e\n",540" \u003c/div\u003e\n",541" "542],543"text/plain": [544" crop greater \\\n",545"0 [176, 279, 768, 768] dist_6 \n",546"1 [149, 32, 768, 768] dist_6 \n",547"2 [1089, 489, 768, 768] dist_11 \n",548"3 [433, 286, 768, 768] dist_14 \n",549"4 [4, 99, 768, 768] dist_6 \n",550"... ... ... \n",551"12978 [887, 59, 768, 768] dist_23 \n",552"12979 [1121, 175, 768, 768] dist_32 \n",553"12980 [1121, 175, 768, 768] dist_21 \n",554"12981 [84, 358, 768, 768] dist_10 \n",555"12982 [84, 358, 768, 768] dist_10 \n",556"\n",557" image lesser random_choice \\\n",558"0 edgar-castrejon-J_NbCcGPBOw-unsplash.png dist_17 False \n",559"1 4KK2_20150823_130650_314.png dist_11 False \n",560"2 6G7M_20150404_121844_208.png dist_17 False \n",561"3 michael-niessl-KraoHdRYrRE-unsplash.png dist_11 True \n",562"4 0127_20161022_144117_906.png dist_14 True \n",563"... ... ... ... \n",564"12978 6G7M_20150404_121844_208.png dist_40 False \n",565"12979 47L8_20150518_144020_604.png dist_30 True \n",566"12980 0006_20160721_220618_556.png dist_32 True \n",567"12981 ricardo-gomez-angel-2GglDxer-Ew-unsplash.png dist_32 True \n",568"12982 6G7M_20150403_174605_886.png dist_21 True \n",569"\n",570" rater_time_ms image_dims greater_butteraugli_max \\\n",571"0 75125 [1280, 1920] 6.792748 \n",572"1 41883 [1920, 1424] 6.830727 \n",573"2 48385 [1920, 1424] 1.760315 \n",574"3 36774 [1920, 1080] 4.110695 \n",575"4 22468 [1920, 1440] 5.866006 \n",576"... ... ... ... \n",577"12978 7983 [1920, 1424] 1.877585 \n",578"12979 1702517 [1920, 1424] 4.943964 \n",579"12980 1430 [1920, 1440] 2.339564 \n",580"12981 5731 [1920, 1319] 3.816417 \n",581"12982 339 [1920, 1424] 3.647443 \n",582"\n",583" greater_butteraugli_6 lesser_butteraugli_max ... lesser_fsim-rgb \\\n",584"0 3.291877 2.287096 ... 0.998617 \n",585"1 3.382573 1.990106 ... 0.998584 \n",586"2 0.828016 1.656251 ... 0.999163 \n",587"3 1.995982 2.432354 ... 0.998708 \n",588"4 2.645947 3.550395 ... 0.994832 \n",589"... ... ... ... ... \n",590"12978 0.923585 2.143382 ... 0.999173 \n",591"12979 2.460808 8.950997 ... 0.969582 \n",592"12980 1.123564 4.024082 ... 0.997534 \n",593"12981 1.697080 5.742501 ... 0.985294 \n",594"12982 1.543176 3.500668 ... 0.996085 \n",595"\n",596" greater_msssim-y lesser_msssim-y greater_nlpd-y lesser_nlpd-y \\\n",597"0 0.975003 0.993973 0.161716 0.078657 \n",598"1 0.985169 0.997969 0.145104 0.050339 \n",599"2 0.999303 0.999264 0.030033 0.030637 \n",600"3 0.993285 0.998220 0.071748 0.037420 \n",601"4 0.988954 0.994369 0.095547 0.069588 \n",602"... ... ... ... ... \n",603"12978 0.998298 0.998512 0.051131 0.048955 \n",604"12979 0.983741 0.955344 0.127071 0.201221 \n",605"12980 0.999049 0.998433 0.029906 0.036302 \n",606"12981 0.991662 0.980832 0.105602 0.189351 \n",607"12982 0.995072 0.991573 0.067868 0.094789 \n",608"\n",609" greater_ssimulacra lesser_ssimulacra greater_elo lesser_elo \\\n",610"0 0.096484 0.045761 1537.137226 2172.213746 \n",611"1 0.072111 0.019148 1972.598657 2291.390798 \n",612"2 0.010733 0.011180 2268.583209 2309.359159 \n",613"3 0.043709 0.019095 1991.945557 2241.368691 \n",614"4 0.049872 0.035815 1851.898409 1945.269660 \n",615"... ... ... ... ... \n",616"12978 0.020006 0.017322 2356.875932 2388.690716 \n",617"12979 0.071460 0.155907 1937.790267 1402.672706 \n",618"12980 0.011507 0.012152 2202.608359 1870.660112 \n",619"12981 0.048339 0.077644 2174.169512 2083.620449 \n",620"12982 0.022608 0.036871 2235.728150 2219.418325 \n",621"\n",622" rater_flips \n",623"0 9 \n",624"1 13 \n",625"2 4 \n",626"3 10 \n",627"4 9 \n",628"... ... \n",629"12978 6 \n",630"12979 5 \n",631"12980 0 \n",632"12981 1 \n",633"12982 0 \n",634"\n",635"[12983 rows x 24 columns]"636]637},638"execution_count": 4,639"metadata": {},640"output_type": "execute_result"641}642],643"source": [644"data"645]646},647{648"cell_type": "markdown",649"metadata": {650"id": "iNn8ft-xy6zE"651},652"source": [653"Then decorate it with whether the crop settings were actually compatible with the image size (a few, ~15, evaluations have this bug), and the worst ELO of both distortions.\n",654"Finally filter out all evaluations where the evaluator didn't seem to do a good job (didn't flip between distortions more than 2 times, didn't spend more than 3 seconds on the evaluation)."655]656},657{658"cell_type": "code",659"execution_count": null,660"metadata": {661"colab": {662"base_uri": "https://localhost:8080/",663"height": 1000664},665"id": "0sFK6hCgMl_9",666"outputId": "a6603c4b-d55a-48a6-89cc-cb10a0324207"667},668"outputs": [669{670"data": {671"text/html": [672"\n",673" \u003cdiv id=\"df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d\"\u003e\n",674" \u003cdiv class=\"colab-df-container\"\u003e\n",675" \u003cdiv\u003e\n",676"\u003cstyle scoped\u003e\n",677" .dataframe tbody tr th:only-of-type {\n",678" vertical-align: middle;\n",679" }\n",680"\n",681" .dataframe tbody tr th {\n",682" vertical-align: top;\n",683" }\n",684"\n",685" .dataframe thead th {\n",686" text-align: right;\n",687" }\n",688"\u003c/style\u003e\n",689"\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",690" \u003cthead\u003e\n",691" \u003ctr style=\"text-align: right;\"\u003e\n",692" \u003cth\u003e\u003c/th\u003e\n",693" \u003cth\u003ecrop\u003c/th\u003e\n",694" \u003cth\u003egreater\u003c/th\u003e\n",695" \u003cth\u003eimage\u003c/th\u003e\n",696" \u003cth\u003elesser\u003c/th\u003e\n",697" \u003cth\u003erandom_choice\u003c/th\u003e\n",698" \u003cth\u003erater_time_ms\u003c/th\u003e\n",699" \u003cth\u003eimage_dims\u003c/th\u003e\n",700" \u003cth\u003egreater_butteraugli_max\u003c/th\u003e\n",701" \u003cth\u003egreater_butteraugli_6\u003c/th\u003e\n",702" \u003cth\u003elesser_butteraugli_max\u003c/th\u003e\n",703" \u003cth\u003e...\u003c/th\u003e\n",704" \u003cth\u003elesser_msssim-y\u003c/th\u003e\n",705" \u003cth\u003egreater_nlpd-y\u003c/th\u003e\n",706" \u003cth\u003elesser_nlpd-y\u003c/th\u003e\n",707" \u003cth\u003egreater_ssimulacra\u003c/th\u003e\n",708" \u003cth\u003elesser_ssimulacra\u003c/th\u003e\n",709" \u003cth\u003egreater_elo\u003c/th\u003e\n",710" \u003cth\u003elesser_elo\u003c/th\u003e\n",711" \u003cth\u003erater_flips\u003c/th\u003e\n",712" \u003cth\u003ecomplete_crop\u003c/th\u003e\n",713" \u003cth\u003eworst_elo\u003c/th\u003e\n",714" \u003c/tr\u003e\n",715" \u003c/thead\u003e\n",716" \u003ctbody\u003e\n",717" \u003ctr\u003e\n",718" \u003cth\u003e0\u003c/th\u003e\n",719" \u003ctd\u003e[176, 279, 768, 768]\u003c/td\u003e\n",720" \u003ctd\u003edist_6\u003c/td\u003e\n",721" \u003ctd\u003eedgar-castrejon-J_NbCcGPBOw-unsplash.png\u003c/td\u003e\n",722" \u003ctd\u003edist_17\u003c/td\u003e\n",723" \u003ctd\u003eFalse\u003c/td\u003e\n",724" \u003ctd\u003e75125\u003c/td\u003e\n",725" \u003ctd\u003e[1280, 1920]\u003c/td\u003e\n",726" \u003ctd\u003e6.792748\u003c/td\u003e\n",727" \u003ctd\u003e3.291877\u003c/td\u003e\n",728" \u003ctd\u003e2.287096\u003c/td\u003e\n",729" \u003ctd\u003e...\u003c/td\u003e\n",730" \u003ctd\u003e0.993973\u003c/td\u003e\n",731" \u003ctd\u003e0.161716\u003c/td\u003e\n",732" \u003ctd\u003e0.078657\u003c/td\u003e\n",733" \u003ctd\u003e0.096484\u003c/td\u003e\n",734" \u003ctd\u003e0.045761\u003c/td\u003e\n",735" \u003ctd\u003e1537.137226\u003c/td\u003e\n",736" \u003ctd\u003e2172.213746\u003c/td\u003e\n",737" \u003ctd\u003e9\u003c/td\u003e\n",738" \u003ctd\u003eTrue\u003c/td\u003e\n",739" \u003ctd\u003e2172.213746\u003c/td\u003e\n",740" \u003c/tr\u003e\n",741" \u003ctr\u003e\n",742" \u003cth\u003e1\u003c/th\u003e\n",743" \u003ctd\u003e[149, 32, 768, 768]\u003c/td\u003e\n",744" \u003ctd\u003edist_6\u003c/td\u003e\n",745" \u003ctd\u003e4KK2_20150823_130650_314.png\u003c/td\u003e\n",746" \u003ctd\u003edist_11\u003c/td\u003e\n",747" \u003ctd\u003eFalse\u003c/td\u003e\n",748" \u003ctd\u003e41883\u003c/td\u003e\n",749" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",750" \u003ctd\u003e6.830727\u003c/td\u003e\n",751" \u003ctd\u003e3.382573\u003c/td\u003e\n",752" \u003ctd\u003e1.990106\u003c/td\u003e\n",753" \u003ctd\u003e...\u003c/td\u003e\n",754" \u003ctd\u003e0.997969\u003c/td\u003e\n",755" \u003ctd\u003e0.145104\u003c/td\u003e\n",756" \u003ctd\u003e0.050339\u003c/td\u003e\n",757" \u003ctd\u003e0.072111\u003c/td\u003e\n",758" \u003ctd\u003e0.019148\u003c/td\u003e\n",759" \u003ctd\u003e1972.598657\u003c/td\u003e\n",760" \u003ctd\u003e2291.390798\u003c/td\u003e\n",761" \u003ctd\u003e13\u003c/td\u003e\n",762" \u003ctd\u003eTrue\u003c/td\u003e\n",763" \u003ctd\u003e2291.390798\u003c/td\u003e\n",764" \u003c/tr\u003e\n",765" \u003ctr\u003e\n",766" \u003cth\u003e2\u003c/th\u003e\n",767" \u003ctd\u003e[1089, 489, 768, 768]\u003c/td\u003e\n",768" \u003ctd\u003edist_11\u003c/td\u003e\n",769" \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",770" \u003ctd\u003edist_17\u003c/td\u003e\n",771" \u003ctd\u003eFalse\u003c/td\u003e\n",772" \u003ctd\u003e48385\u003c/td\u003e\n",773" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",774" \u003ctd\u003e1.760315\u003c/td\u003e\n",775" \u003ctd\u003e0.828016\u003c/td\u003e\n",776" \u003ctd\u003e1.656251\u003c/td\u003e\n",777" \u003ctd\u003e...\u003c/td\u003e\n",778" \u003ctd\u003e0.999264\u003c/td\u003e\n",779" \u003ctd\u003e0.030033\u003c/td\u003e\n",780" \u003ctd\u003e0.030637\u003c/td\u003e\n",781" \u003ctd\u003e0.010733\u003c/td\u003e\n",782" \u003ctd\u003e0.011180\u003c/td\u003e\n",783" \u003ctd\u003e2268.583209\u003c/td\u003e\n",784" \u003ctd\u003e2309.359159\u003c/td\u003e\n",785" \u003ctd\u003e4\u003c/td\u003e\n",786" \u003ctd\u003eTrue\u003c/td\u003e\n",787" \u003ctd\u003e2309.359159\u003c/td\u003e\n",788" \u003c/tr\u003e\n",789" \u003ctr\u003e\n",790" \u003cth\u003e3\u003c/th\u003e\n",791" \u003ctd\u003e[433, 286, 768, 768]\u003c/td\u003e\n",792" \u003ctd\u003edist_14\u003c/td\u003e\n",793" \u003ctd\u003emichael-niessl-KraoHdRYrRE-unsplash.png\u003c/td\u003e\n",794" \u003ctd\u003edist_11\u003c/td\u003e\n",795" \u003ctd\u003eTrue\u003c/td\u003e\n",796" \u003ctd\u003e36774\u003c/td\u003e\n",797" \u003ctd\u003e[1920, 1080]\u003c/td\u003e\n",798" \u003ctd\u003e4.110695\u003c/td\u003e\n",799" \u003ctd\u003e1.995982\u003c/td\u003e\n",800" \u003ctd\u003e2.432354\u003c/td\u003e\n",801" \u003ctd\u003e...\u003c/td\u003e\n",802" \u003ctd\u003e0.998220\u003c/td\u003e\n",803" \u003ctd\u003e0.071748\u003c/td\u003e\n",804" \u003ctd\u003e0.037420\u003c/td\u003e\n",805" \u003ctd\u003e0.043709\u003c/td\u003e\n",806" \u003ctd\u003e0.019095\u003c/td\u003e\n",807" \u003ctd\u003e1991.945557\u003c/td\u003e\n",808" \u003ctd\u003e2241.368691\u003c/td\u003e\n",809" \u003ctd\u003e10\u003c/td\u003e\n",810" \u003ctd\u003eTrue\u003c/td\u003e\n",811" \u003ctd\u003e2241.368691\u003c/td\u003e\n",812" \u003c/tr\u003e\n",813" \u003ctr\u003e\n",814" \u003cth\u003e4\u003c/th\u003e\n",815" \u003ctd\u003e[4, 99, 768, 768]\u003c/td\u003e\n",816" \u003ctd\u003edist_6\u003c/td\u003e\n",817" \u003ctd\u003e0127_20161022_144117_906.png\u003c/td\u003e\n",818" \u003ctd\u003edist_14\u003c/td\u003e\n",819" \u003ctd\u003eTrue\u003c/td\u003e\n",820" \u003ctd\u003e22468\u003c/td\u003e\n",821" \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",822" \u003ctd\u003e5.866006\u003c/td\u003e\n",823" \u003ctd\u003e2.645947\u003c/td\u003e\n",824" \u003ctd\u003e3.550395\u003c/td\u003e\n",825" \u003ctd\u003e...\u003c/td\u003e\n",826" \u003ctd\u003e0.994369\u003c/td\u003e\n",827" \u003ctd\u003e0.095547\u003c/td\u003e\n",828" \u003ctd\u003e0.069588\u003c/td\u003e\n",829" \u003ctd\u003e0.049872\u003c/td\u003e\n",830" \u003ctd\u003e0.035815\u003c/td\u003e\n",831" \u003ctd\u003e1851.898409\u003c/td\u003e\n",832" \u003ctd\u003e1945.269660\u003c/td\u003e\n",833" \u003ctd\u003e9\u003c/td\u003e\n",834" \u003ctd\u003eTrue\u003c/td\u003e\n",835" \u003ctd\u003e1945.269660\u003c/td\u003e\n",836" \u003c/tr\u003e\n",837" \u003ctr\u003e\n",838" \u003cth\u003e...\u003c/th\u003e\n",839" \u003ctd\u003e...\u003c/td\u003e\n",840" \u003ctd\u003e...\u003c/td\u003e\n",841" \u003ctd\u003e...\u003c/td\u003e\n",842" \u003ctd\u003e...\u003c/td\u003e\n",843" \u003ctd\u003e...\u003c/td\u003e\n",844" \u003ctd\u003e...\u003c/td\u003e\n",845" \u003ctd\u003e...\u003c/td\u003e\n",846" \u003ctd\u003e...\u003c/td\u003e\n",847" \u003ctd\u003e...\u003c/td\u003e\n",848" \u003ctd\u003e...\u003c/td\u003e\n",849" \u003ctd\u003e...\u003c/td\u003e\n",850" \u003ctd\u003e...\u003c/td\u003e\n",851" \u003ctd\u003e...\u003c/td\u003e\n",852" \u003ctd\u003e...\u003c/td\u003e\n",853" \u003ctd\u003e...\u003c/td\u003e\n",854" \u003ctd\u003e...\u003c/td\u003e\n",855" \u003ctd\u003e...\u003c/td\u003e\n",856" \u003ctd\u003e...\u003c/td\u003e\n",857" \u003ctd\u003e...\u003c/td\u003e\n",858" \u003ctd\u003e...\u003c/td\u003e\n",859" \u003ctd\u003e...\u003c/td\u003e\n",860" \u003c/tr\u003e\n",861" \u003ctr\u003e\n",862" \u003cth\u003e12975\u003c/th\u003e\n",863" \u003ctd\u003e[361, 268, 768, 768]\u003c/td\u003e\n",864" \u003ctd\u003edist_23\u003c/td\u003e\n",865" \u003ctd\u003e4KK2_20150912_125525_384.png\u003c/td\u003e\n",866" \u003ctd\u003edist_21\u003c/td\u003e\n",867" \u003ctd\u003eFalse\u003c/td\u003e\n",868" \u003ctd\u003e15533\u003c/td\u003e\n",869" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",870" \u003ctd\u003e2.678031\u003c/td\u003e\n",871" \u003ctd\u003e1.201943\u003c/td\u003e\n",872" \u003ctd\u003e3.711323\u003c/td\u003e\n",873" \u003ctd\u003e...\u003c/td\u003e\n",874" \u003ctd\u003e0.988685\u003c/td\u003e\n",875" \u003ctd\u003e0.076957\u003c/td\u003e\n",876" \u003ctd\u003e0.128533\u003c/td\u003e\n",877" \u003ctd\u003e0.025589\u003c/td\u003e\n",878" \u003ctd\u003e0.051236\u003c/td\u003e\n",879" \u003ctd\u003e2334.257408\u003c/td\u003e\n",880" \u003ctd\u003e2105.427273\u003c/td\u003e\n",881" \u003ctd\u003e7\u003c/td\u003e\n",882" \u003ctd\u003eTrue\u003c/td\u003e\n",883" \u003ctd\u003e2334.257408\u003c/td\u003e\n",884" \u003c/tr\u003e\n",885" \u003ctr\u003e\n",886" \u003cth\u003e12976\u003c/th\u003e\n",887" \u003ctd\u003e[374, 425, 768, 768]\u003c/td\u003e\n",888" \u003ctd\u003edist_23\u003c/td\u003e\n",889" \u003ctd\u003ejustin-lim-OiFRMM3o7l0-unsplash.png\u003c/td\u003e\n",890" \u003ctd\u003edist_22\u003c/td\u003e\n",891" \u003ctd\u003eFalse\u003c/td\u003e\n",892" \u003ctd\u003e7306\u003c/td\u003e\n",893" \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",894" \u003ctd\u003e2.273484\u003c/td\u003e\n",895" \u003ctd\u003e1.044854\u003c/td\u003e\n",896" \u003ctd\u003e2.454072\u003c/td\u003e\n",897" \u003ctd\u003e...\u003c/td\u003e\n",898" \u003ctd\u003e0.995344\u003c/td\u003e\n",899" \u003ctd\u003e0.060886\u003c/td\u003e\n",900" \u003ctd\u003e0.078061\u003c/td\u003e\n",901" \u003ctd\u003e0.024588\u003c/td\u003e\n",902" \u003ctd\u003e0.033907\u003c/td\u003e\n",903" \u003ctd\u003e2347.775076\u003c/td\u003e\n",904" \u003ctd\u003e2209.434355\u003c/td\u003e\n",905" \u003ctd\u003e4\u003c/td\u003e\n",906" \u003ctd\u003eTrue\u003c/td\u003e\n",907" \u003ctd\u003e2347.775076\u003c/td\u003e\n",908" \u003c/tr\u003e\n",909" \u003ctr\u003e\n",910" \u003cth\u003e12977\u003c/th\u003e\n",911" \u003ctd\u003e[1093, 224, 768, 768]\u003c/td\u003e\n",912" \u003ctd\u003edist_9\u003c/td\u003e\n",913" \u003ctd\u003e0127_20161022_144117_906.png\u003c/td\u003e\n",914" \u003ctd\u003edist_23\u003c/td\u003e\n",915" \u003ctd\u003eTrue\u003c/td\u003e\n",916" \u003ctd\u003e12069\u003c/td\u003e\n",917" \u003ctd\u003e[1920, 1440]\u003c/td\u003e\n",918" \u003ctd\u003e2.618022\u003c/td\u003e\n",919" \u003ctd\u003e1.303337\u003c/td\u003e\n",920" \u003ctd\u003e1.548064\u003c/td\u003e\n",921" \u003ctd\u003e...\u003c/td\u003e\n",922" \u003ctd\u003e0.998452\u003c/td\u003e\n",923" \u003ctd\u003e0.050813\u003c/td\u003e\n",924" \u003ctd\u003e0.037381\u003c/td\u003e\n",925" \u003ctd\u003e0.023579\u003c/td\u003e\n",926" \u003ctd\u003e0.016908\u003c/td\u003e\n",927" \u003ctd\u003e2187.406607\u003c/td\u003e\n",928" \u003ctd\u003e2373.387899\u003c/td\u003e\n",929" \u003ctd\u003e12\u003c/td\u003e\n",930" \u003ctd\u003eTrue\u003c/td\u003e\n",931" \u003ctd\u003e2373.387899\u003c/td\u003e\n",932" \u003c/tr\u003e\n",933" \u003ctr\u003e\n",934" \u003cth\u003e12978\u003c/th\u003e\n",935" \u003ctd\u003e[887, 59, 768, 768]\u003c/td\u003e\n",936" \u003ctd\u003edist_23\u003c/td\u003e\n",937" \u003ctd\u003e6G7M_20150404_121844_208.png\u003c/td\u003e\n",938" \u003ctd\u003edist_40\u003c/td\u003e\n",939" \u003ctd\u003eFalse\u003c/td\u003e\n",940" \u003ctd\u003e7983\u003c/td\u003e\n",941" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",942" \u003ctd\u003e1.877585\u003c/td\u003e\n",943" \u003ctd\u003e0.923585\u003c/td\u003e\n",944" \u003ctd\u003e2.143382\u003c/td\u003e\n",945" \u003ctd\u003e...\u003c/td\u003e\n",946" \u003ctd\u003e0.998512\u003c/td\u003e\n",947" \u003ctd\u003e0.051131\u003c/td\u003e\n",948" \u003ctd\u003e0.048955\u003c/td\u003e\n",949" \u003ctd\u003e0.020006\u003c/td\u003e\n",950" \u003ctd\u003e0.017322\u003c/td\u003e\n",951" \u003ctd\u003e2356.875932\u003c/td\u003e\n",952" \u003ctd\u003e2388.690716\u003c/td\u003e\n",953" \u003ctd\u003e6\u003c/td\u003e\n",954" \u003ctd\u003eTrue\u003c/td\u003e\n",955" \u003ctd\u003e2388.690716\u003c/td\u003e\n",956" \u003c/tr\u003e\n",957" \u003ctr\u003e\n",958" \u003cth\u003e12979\u003c/th\u003e\n",959" \u003ctd\u003e[1121, 175, 768, 768]\u003c/td\u003e\n",960" \u003ctd\u003edist_32\u003c/td\u003e\n",961" \u003ctd\u003e47L8_20150518_144020_604.png\u003c/td\u003e\n",962" \u003ctd\u003edist_30\u003c/td\u003e\n",963" \u003ctd\u003eTrue\u003c/td\u003e\n",964" \u003ctd\u003e1702517\u003c/td\u003e\n",965" \u003ctd\u003e[1920, 1424]\u003c/td\u003e\n",966" \u003ctd\u003e4.943964\u003c/td\u003e\n",967" \u003ctd\u003e2.460808\u003c/td\u003e\n",968" \u003ctd\u003e8.950997\u003c/td\u003e\n",969" \u003ctd\u003e...\u003c/td\u003e\n",970" \u003ctd\u003e0.955344\u003c/td\u003e\n",971" \u003ctd\u003e0.127071\u003c/td\u003e\n",972" \u003ctd\u003e0.201221\u003c/td\u003e\n",973" \u003ctd\u003e0.071460\u003c/td\u003e\n",974" \u003ctd\u003e0.155907\u003c/td\u003e\n",975" \u003ctd\u003e1937.790267\u003c/td\u003e\n",976" \u003ctd\u003e1402.672706\u003c/td\u003e\n",977" \u003ctd\u003e5\u003c/td\u003e\n",978" \u003ctd\u003eTrue\u003c/td\u003e\n",979" \u003ctd\u003e1937.790267\u003c/td\u003e\n",980" \u003c/tr\u003e\n",981" \u003c/tbody\u003e\n",982"\u003c/table\u003e\n",983"\u003cp\u003e12402 rows × 26 columns\u003c/p\u003e\n",984"\u003c/div\u003e\n",985" \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d')\"\n",986" title=\"Convert this dataframe to an interactive table.\"\n",987" style=\"display:none;\"\u003e\n",988" \n",989" \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",990" width=\"24px\"\u003e\n",991" \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",992" \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",993" \u003c/svg\u003e\n",994" \u003c/button\u003e\n",995" \n",996" \u003cstyle\u003e\n",997" .colab-df-container {\n",998" display:flex;\n",999" flex-wrap:wrap;\n",1000" gap: 12px;\n",1001" }\n",1002"\n",1003" .colab-df-convert {\n",1004" background-color: #E8F0FE;\n",1005" border: none;\n",1006" border-radius: 50%;\n",1007" cursor: pointer;\n",1008" display: none;\n",1009" fill: #1967D2;\n",1010" height: 32px;\n",1011" padding: 0 0 0 0;\n",1012" width: 32px;\n",1013" }\n",1014"\n",1015" .colab-df-convert:hover {\n",1016" background-color: #E2EBFA;\n",1017" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",1018" fill: #174EA6;\n",1019" }\n",1020"\n",1021" [theme=dark] .colab-df-convert {\n",1022" background-color: #3B4455;\n",1023" fill: #D2E3FC;\n",1024" }\n",1025"\n",1026" [theme=dark] .colab-df-convert:hover {\n",1027" background-color: #434B5C;\n",1028" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",1029" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",1030" fill: #FFFFFF;\n",1031" }\n",1032" \u003c/style\u003e\n",1033"\n",1034" \u003cscript\u003e\n",1035" const buttonEl =\n",1036" document.querySelector('#df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d button.colab-df-convert');\n",1037" buttonEl.style.display =\n",1038" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",1039"\n",1040" async function convertToInteractive(key) {\n",1041" const element = document.querySelector('#df-73737e54-fd5b-44a9-a4dc-5e3c92c1719d');\n",1042" const dataTable =\n",1043" await google.colab.kernel.invokeFunction('convertToInteractive',\n",1044" [key], {});\n",1045" if (!dataTable) return;\n",1046"\n",1047" const docLinkHtml = 'Like what you see? Visit the ' +\n",1048" '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",1049" + ' to learn more about interactive tables.';\n",1050" element.innerHTML = '';\n",1051" dataTable['output_type'] = 'display_data';\n",1052" await google.colab.output.renderOutput(dataTable, element);\n",1053" const docLink = document.createElement('div');\n",1054" docLink.innerHTML = docLinkHtml;\n",1055" element.appendChild(docLink);\n",1056" }\n",1057" \u003c/script\u003e\n",1058" \u003c/div\u003e\n",1059" \u003c/div\u003e\n",1060" "1061],1062"text/plain": [1063" crop greater \\\n",1064"0 [176, 279, 768, 768] dist_6 \n",1065"1 [149, 32, 768, 768] dist_6 \n",1066"2 [1089, 489, 768, 768] dist_11 \n",1067"3 [433, 286, 768, 768] dist_14 \n",1068"4 [4, 99, 768, 768] dist_6 \n",1069"... ... ... \n",1070"12975 [361, 268, 768, 768] dist_23 \n",1071"12976 [374, 425, 768, 768] dist_23 \n",1072"12977 [1093, 224, 768, 768] dist_9 \n",1073"12978 [887, 59, 768, 768] dist_23 \n",1074"12979 [1121, 175, 768, 768] dist_32 \n",1075"\n",1076" image lesser random_choice \\\n",1077"0 edgar-castrejon-J_NbCcGPBOw-unsplash.png dist_17 False \n",1078"1 4KK2_20150823_130650_314.png dist_11 False \n",1079"2 6G7M_20150404_121844_208.png dist_17 False \n",1080"3 michael-niessl-KraoHdRYrRE-unsplash.png dist_11 True \n",1081"4 0127_20161022_144117_906.png dist_14 True \n",1082"... ... ... ... \n",1083"12975 4KK2_20150912_125525_384.png dist_21 False \n",1084"12976 justin-lim-OiFRMM3o7l0-unsplash.png dist_22 False \n",1085"12977 0127_20161022_144117_906.png dist_23 True \n",1086"12978 6G7M_20150404_121844_208.png dist_40 False \n",1087"12979 47L8_20150518_144020_604.png dist_30 True \n",1088"\n",1089" rater_time_ms image_dims greater_butteraugli_max \\\n",1090"0 75125 [1280, 1920] 6.792748 \n",1091"1 41883 [1920, 1424] 6.830727 \n",1092"2 48385 [1920, 1424] 1.760315 \n",1093"3 36774 [1920, 1080] 4.110695 \n",1094"4 22468 [1920, 1440] 5.866006 \n",1095"... ... ... ... \n",1096"12975 15533 [1920, 1424] 2.678031 \n",1097"12976 7306 [1920, 1440] 2.273484 \n",1098"12977 12069 [1920, 1440] 2.618022 \n",1099"12978 7983 [1920, 1424] 1.877585 \n",1100"12979 1702517 [1920, 1424] 4.943964 \n",1101"\n",1102" greater_butteraugli_6 lesser_butteraugli_max ... lesser_msssim-y \\\n",1103"0 3.291877 2.287096 ... 0.993973 \n",1104"1 3.382573 1.990106 ... 0.997969 \n",1105"2 0.828016 1.656251 ... 0.999264 \n",1106"3 1.995982 2.432354 ... 0.998220 \n",1107"4 2.645947 3.550395 ... 0.994369 \n",1108"... ... ... ... ... \n",1109"12975 1.201943 3.711323 ... 0.988685 \n",1110"12976 1.044854 2.454072 ... 0.995344 \n",1111"12977 1.303337 1.548064 ... 0.998452 \n",1112"12978 0.923585 2.143382 ... 0.998512 \n",1113"12979 2.460808 8.950997 ... 0.955344 \n",1114"\n",1115" greater_nlpd-y lesser_nlpd-y greater_ssimulacra lesser_ssimulacra \\\n",1116"0 0.161716 0.078657 0.096484 0.045761 \n",1117"1 0.145104 0.050339 0.072111 0.019148 \n",1118"2 0.030033 0.030637 0.010733 0.011180 \n",1119"3 0.071748 0.037420 0.043709 0.019095 \n",1120"4 0.095547 0.069588 0.049872 0.035815 \n",1121"... ... ... ... ... \n",1122"12975 0.076957 0.128533 0.025589 0.051236 \n",1123"12976 0.060886 0.078061 0.024588 0.033907 \n",1124"12977 0.050813 0.037381 0.023579 0.016908 \n",1125"12978 0.051131 0.048955 0.020006 0.017322 \n",1126"12979 0.127071 0.201221 0.071460 0.155907 \n",1127"\n",1128" greater_elo lesser_elo rater_flips complete_crop worst_elo \n",1129"0 1537.137226 2172.213746 9 True 2172.213746 \n",1130"1 1972.598657 2291.390798 13 True 2291.390798 \n",1131"2 2268.583209 2309.359159 4 True 2309.359159 \n",1132"3 1991.945557 2241.368691 10 True 2241.368691 \n",1133"4 1851.898409 1945.269660 9 True 1945.269660 \n",1134"... ... ... ... ... ... \n",1135"12975 2334.257408 2105.427273 7 True 2334.257408 \n",1136"12976 2347.775076 2209.434355 4 True 2347.775076 \n",1137"12977 2187.406607 2373.387899 12 True 2373.387899 \n",1138"12978 2356.875932 2388.690716 6 True 2388.690716 \n",1139"12979 1937.790267 1402.672706 5 True 1937.790267 \n",1140"\n",1141"[12402 rows x 26 columns]"1142]1143},1144"execution_count": 5,1145"metadata": {},1146"output_type": "execute_result"1147}1148],1149"source": [1150"data['complete_crop'] = data.apply(lambda row: row.crop[0] + row.crop[2] \u003c= row.image_dims[0] and row.crop[1] + row.crop[3] \u003c= row.image_dims[1], axis=1)\n",1151"data['worst_elo'] = data.apply(lambda row: row.greater_elo if row.greater_elo \u003e row.lesser_elo else row.lesser_elo, axis=1)\n",1152"data = data[(data.rater_flips \u003e 2) \u0026 (data.rater_time_ms \u003e 3000) \u0026 (data.complete_crop == True)]\n",1153"data"1154]1155},1156{1157"cell_type": "code",1158"execution_count": null,1159"metadata": {1160"id": "RNT0sDkLQJnd"1161},1162"outputs": [],1163"source": [1164"def strip(ary, n):\n",1165" def stripfun(sum, el):\n",1166" sum[el] = el[n:]\n",1167" return sum\n",1168" return functools.reduce(stripfun, ary, {})"1169]1170},1171{1172"cell_type": "code",1173"execution_count": null,1174"metadata": {1175"id": "63zgtekaR4fj"1176},1177"outputs": [],1178"source": [1179"greater_metric_cols = list(filter(lambda el: el.startswith('greater_') and not el.endswith('_file'), list(data.columns)))"1180]1181},1182{1183"cell_type": "code",1184"execution_count": null,1185"metadata": {1186"id": "Kt71n9RsS1E-"1187},1188"outputs": [],1189"source": [1190"lesser_metric_cols = list(filter(lambda el: el.startswith('lesser_') and not el.endswith('_file'), list(data.columns)))"1191]1192},1193{1194"cell_type": "code",1195"execution_count": null,1196"metadata": {1197"id": "dRTUD1m6PTlq"1198},1199"outputs": [],1200"source": [1201"greater_metrics = data[greater_metric_cols]\n",1202"greater_metrics = greater_metrics.rename(columns=strip(greater_metric_cols, 8))"1203]1204},1205{1206"cell_type": "code",1207"execution_count": null,1208"metadata": {1209"id": "iDOc7lrzS-2n"1210},1211"outputs": [],1212"source": [1213"lesser_metrics = data[lesser_metric_cols]\n",1214"lesser_metrics = lesser_metrics.rename(columns=strip(lesser_metric_cols, 7))"1215]1216},1217{1218"cell_type": "markdown",1219"metadata": {1220"id": "4ZyN9BMbzZQb"1221},1222"source": [1223"To allow a rank correlation, like Spearman, combine the metrics of the worse distortion (lesser), and the better distortion (greater), into one dataframe. To also allow comparing correlation in different regions of quality, sort by ELO score."1224]1225},1226{1227"cell_type": "code",1228"execution_count": null,1229"metadata": {1230"colab": {1231"base_uri": "https://localhost:8080/",1232"height": 4231233},1234"id": "8flMfeykR1wL",1235"outputId": "39d9cb1b-2bd1-445b-bf83-ad61b15179ab"1236},1237"outputs": [1238{1239"data": {1240"text/html": [1241"\n",1242" \u003cdiv id=\"df-ede9f7e9-1552-4956-af74-848c6cd1bfaa\"\u003e\n",1243" \u003cdiv class=\"colab-df-container\"\u003e\n",1244" \u003cdiv\u003e\n",1245"\u003cstyle scoped\u003e\n",1246" .dataframe tbody tr th:only-of-type {\n",1247" vertical-align: middle;\n",1248" }\n",1249"\n",1250" .dataframe tbody tr th {\n",1251" vertical-align: top;\n",1252" }\n",1253"\n",1254" .dataframe thead th {\n",1255" text-align: right;\n",1256" }\n",1257"\u003c/style\u003e\n",1258"\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",1259" \u003cthead\u003e\n",1260" \u003ctr style=\"text-align: right;\"\u003e\n",1261" \u003cth\u003e\u003c/th\u003e\n",1262" \u003cth\u003ebutteraugli_max\u003c/th\u003e\n",1263" \u003cth\u003ebutteraugli_6\u003c/th\u003e\n",1264" \u003cth\u003efsim-y\u003c/th\u003e\n",1265" \u003cth\u003efsim-rgb\u003c/th\u003e\n",1266" \u003cth\u003emsssim-y\u003c/th\u003e\n",1267" \u003cth\u003enlpd-y\u003c/th\u003e\n",1268" \u003cth\u003essimulacra\u003c/th\u003e\n",1269" \u003cth\u003eelo\u003c/th\u003e\n",1270" \u003c/tr\u003e\n",1271" \u003c/thead\u003e\n",1272" \u003ctbody\u003e\n",1273" \u003ctr\u003e\n",1274" \u003cth\u003e0\u003c/th\u003e\n",1275" \u003ctd\u003e8.867950\u003c/td\u003e\n",1276" \u003ctd\u003e4.764297\u003c/td\u003e\n",1277" \u003ctd\u003e0.978342\u003c/td\u003e\n",1278" \u003ctd\u003e0.974386\u003c/td\u003e\n",1279" \u003ctd\u003e0.965125\u003c/td\u003e\n",1280" \u003ctd\u003e0.242242\u003c/td\u003e\n",1281" \u003ctd\u003e0.144204\u003c/td\u003e\n",1282" \u003ctd\u003e824.179269\u003c/td\u003e\n",1283" \u003c/tr\u003e\n",1284" \u003ctr\u003e\n",1285" \u003cth\u003e1\u003c/th\u003e\n",1286" \u003ctd\u003e8.879775\u003c/td\u003e\n",1287" \u003ctd\u003e4.580521\u003c/td\u003e\n",1288" \u003ctd\u003e0.979431\u003c/td\u003e\n",1289" \u003ctd\u003e0.976825\u003c/td\u003e\n",1290" \u003ctd\u003e0.960813\u003c/td\u003e\n",1291" \u003ctd\u003e0.228356\u003c/td\u003e\n",1292" \u003ctd\u003e0.125745\u003c/td\u003e\n",1293" \u003ctd\u003e824.179269\u003c/td\u003e\n",1294" \u003c/tr\u003e\n",1295" \u003ctr\u003e\n",1296" \u003cth\u003e2\u003c/th\u003e\n",1297" \u003ctd\u003e8.815258\u003c/td\u003e\n",1298" \u003ctd\u003e4.479465\u003c/td\u003e\n",1299" \u003ctd\u003e0.980187\u003c/td\u003e\n",1300" \u003ctd\u003e0.977970\u003c/td\u003e\n",1301" \u003ctd\u003e0.961923\u003c/td\u003e\n",1302" \u003ctd\u003e0.215990\u003c/td\u003e\n",1303" \u003ctd\u003e0.119023\u003c/td\u003e\n",1304" \u003ctd\u003e824.179269\u003c/td\u003e\n",1305" \u003c/tr\u003e\n",1306" \u003ctr\u003e\n",1307" \u003cth\u003e3\u003c/th\u003e\n",1308" \u003ctd\u003e8.913489\u003c/td\u003e\n",1309" \u003ctd\u003e4.727989\u003c/td\u003e\n",1310" \u003ctd\u003e0.976972\u003c/td\u003e\n",1311" \u003ctd\u003e0.973410\u003c/td\u003e\n",1312" \u003ctd\u003e0.959204\u003c/td\u003e\n",1313" \u003ctd\u003e0.240950\u003c/td\u003e\n",1314" \u003ctd\u003e0.142649\u003c/td\u003e\n",1315" \u003ctd\u003e824.179269\u003c/td\u003e\n",1316" \u003c/tr\u003e\n",1317" \u003ctr\u003e\n",1318" \u003cth\u003e4\u003c/th\u003e\n",1319" \u003ctd\u003e9.732282\u003c/td\u003e\n",1320" \u003ctd\u003e4.798093\u003c/td\u003e\n",1321" \u003ctd\u003e0.977469\u003c/td\u003e\n",1322" \u003ctd\u003e0.973566\u003c/td\u003e\n",1323" \u003ctd\u003e0.958279\u003c/td\u003e\n",1324" \u003ctd\u003e0.250352\u003c/td\u003e\n",1325" \u003ctd\u003e0.148506\u003c/td\u003e\n",1326" \u003ctd\u003e824.179269\u003c/td\u003e\n",1327" \u003c/tr\u003e\n",1328" \u003ctr\u003e\n",1329" \u003cth\u003e...\u003c/th\u003e\n",1330" \u003ctd\u003e...\u003c/td\u003e\n",1331" \u003ctd\u003e...\u003c/td\u003e\n",1332" \u003ctd\u003e...\u003c/td\u003e\n",1333" \u003ctd\u003e...\u003c/td\u003e\n",1334" \u003ctd\u003e...\u003c/td\u003e\n",1335" \u003ctd\u003e...\u003c/td\u003e\n",1336" \u003ctd\u003e...\u003c/td\u003e\n",1337" \u003ctd\u003e...\u003c/td\u003e\n",1338" \u003c/tr\u003e\n",1339" \u003ctr\u003e\n",1340" \u003cth\u003e24799\u003c/th\u003e\n",1341" \u003ctd\u003e1.762231\u003c/td\u003e\n",1342" \u003ctd\u003e0.863336\u003c/td\u003e\n",1343" \u003ctd\u003e0.998921\u003c/td\u003e\n",1344" \u003ctd\u003e0.998753\u003c/td\u003e\n",1345" \u003ctd\u003e0.998144\u003c/td\u003e\n",1346" \u003ctd\u003e0.053619\u003c/td\u003e\n",1347" \u003ctd\u003e0.014470\u003c/td\u003e\n",1348" \u003ctd\u003e2699.407630\u003c/td\u003e\n",1349" \u003c/tr\u003e\n",1350" \u003ctr\u003e\n",1351" \u003cth\u003e24800\u003c/th\u003e\n",1352" \u003ctd\u003e1.857362\u003c/td\u003e\n",1353" \u003ctd\u003e0.921340\u003c/td\u003e\n",1354" \u003ctd\u003e0.998864\u003c/td\u003e\n",1355" \u003ctd\u003e0.998748\u003c/td\u003e\n",1356" \u003ctd\u003e0.998029\u003c/td\u003e\n",1357" \u003ctd\u003e0.051740\u003c/td\u003e\n",1358" \u003ctd\u003e0.016668\u003c/td\u003e\n",1359" \u003ctd\u003e2699.407630\u003c/td\u003e\n",1360" \u003c/tr\u003e\n",1361" \u003ctr\u003e\n",1362" \u003cth\u003e24801\u003c/th\u003e\n",1363" \u003ctd\u003e2.087677\u003c/td\u003e\n",1364" \u003ctd\u003e0.921140\u003c/td\u003e\n",1365" \u003ctd\u003e0.999216\u003c/td\u003e\n",1366" \u003ctd\u003e0.999160\u003c/td\u003e\n",1367" \u003ctd\u003e0.998273\u003c/td\u003e\n",1368" \u003ctd\u003e0.042458\u003c/td\u003e\n",1369" \u003ctd\u003e0.013809\u003c/td\u003e\n",1370" \u003ctd\u003e2699.407630\u003c/td\u003e\n",1371" \u003c/tr\u003e\n",1372" \u003ctr\u003e\n",1373" \u003cth\u003e24802\u003c/th\u003e\n",1374" \u003ctd\u003e1.668185\u003c/td\u003e\n",1375" \u003ctd\u003e0.847164\u003c/td\u003e\n",1376" \u003ctd\u003e0.998240\u003c/td\u003e\n",1377" \u003ctd\u003e0.998118\u003c/td\u003e\n",1378" \u003ctd\u003e0.996968\u003c/td\u003e\n",1379" \u003ctd\u003e0.056177\u003c/td\u003e\n",1380" \u003ctd\u003e0.016632\u003c/td\u003e\n",1381" \u003ctd\u003e2699.407630\u003c/td\u003e\n",1382" \u003c/tr\u003e\n",1383" \u003ctr\u003e\n",1384" \u003cth\u003e24803\u003c/th\u003e\n",1385" \u003ctd\u003e1.859033\u003c/td\u003e\n",1386" \u003ctd\u003e0.889231\u003c/td\u003e\n",1387" \u003ctd\u003e0.999222\u003c/td\u003e\n",1388" \u003ctd\u003e0.999119\u003c/td\u003e\n",1389" \u003ctd\u003e0.998511\u003c/td\u003e\n",1390" \u003ctd\u003e0.045151\u003c/td\u003e\n",1391" \u003ctd\u003e0.015015\u003c/td\u003e\n",1392" \u003ctd\u003e2699.407630\u003c/td\u003e\n",1393" \u003c/tr\u003e\n",1394" \u003c/tbody\u003e\n",1395"\u003c/table\u003e\n",1396"\u003cp\u003e24804 rows × 8 columns\u003c/p\u003e\n",1397"\u003c/div\u003e\n",1398" \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ede9f7e9-1552-4956-af74-848c6cd1bfaa')\"\n",1399" title=\"Convert this dataframe to an interactive table.\"\n",1400" style=\"display:none;\"\u003e\n",1401" \n",1402" \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",1403" width=\"24px\"\u003e\n",1404" \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",1405" \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",1406" \u003c/svg\u003e\n",1407" \u003c/button\u003e\n",1408" \n",1409" \u003cstyle\u003e\n",1410" .colab-df-container {\n",1411" display:flex;\n",1412" flex-wrap:wrap;\n",1413" gap: 12px;\n",1414" }\n",1415"\n",1416" .colab-df-convert {\n",1417" background-color: #E8F0FE;\n",1418" border: none;\n",1419" border-radius: 50%;\n",1420" cursor: pointer;\n",1421" display: none;\n",1422" fill: #1967D2;\n",1423" height: 32px;\n",1424" padding: 0 0 0 0;\n",1425" width: 32px;\n",1426" }\n",1427"\n",1428" .colab-df-convert:hover {\n",1429" background-color: #E2EBFA;\n",1430" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",1431" fill: #174EA6;\n",1432" }\n",1433"\n",1434" [theme=dark] .colab-df-convert {\n",1435" background-color: #3B4455;\n",1436" fill: #D2E3FC;\n",1437" }\n",1438"\n",1439" [theme=dark] .colab-df-convert:hover {\n",1440" background-color: #434B5C;\n",1441" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",1442" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",1443" fill: #FFFFFF;\n",1444" }\n",1445" \u003c/style\u003e\n",1446"\n",1447" \u003cscript\u003e\n",1448" const buttonEl =\n",1449" document.querySelector('#df-ede9f7e9-1552-4956-af74-848c6cd1bfaa button.colab-df-convert');\n",1450" buttonEl.style.display =\n",1451" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",1452"\n",1453" async function convertToInteractive(key) {\n",1454" const element = document.querySelector('#df-ede9f7e9-1552-4956-af74-848c6cd1bfaa');\n",1455" const dataTable =\n",1456" await google.colab.kernel.invokeFunction('convertToInteractive',\n",1457" [key], {});\n",1458" if (!dataTable) return;\n",1459"\n",1460" const docLinkHtml = 'Like what you see? Visit the ' +\n",1461" '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",1462" + ' to learn more about interactive tables.';\n",1463" element.innerHTML = '';\n",1464" dataTable['output_type'] = 'display_data';\n",1465" await google.colab.output.renderOutput(dataTable, element);\n",1466" const docLink = document.createElement('div');\n",1467" docLink.innerHTML = docLinkHtml;\n",1468" element.appendChild(docLink);\n",1469" }\n",1470" \u003c/script\u003e\n",1471" \u003c/div\u003e\n",1472" \u003c/div\u003e\n",1473" "1474],1475"text/plain": [1476" butteraugli_max butteraugli_6 fsim-y fsim-rgb msssim-y nlpd-y \\\n",1477"0 8.867950 4.764297 0.978342 0.974386 0.965125 0.242242 \n",1478"1 8.879775 4.580521 0.979431 0.976825 0.960813 0.228356 \n",1479"2 8.815258 4.479465 0.980187 0.977970 0.961923 0.215990 \n",1480"3 8.913489 4.727989 0.976972 0.973410 0.959204 0.240950 \n",1481"4 9.732282 4.798093 0.977469 0.973566 0.958279 0.250352 \n",1482"... ... ... ... ... ... ... \n",1483"24799 1.762231 0.863336 0.998921 0.998753 0.998144 0.053619 \n",1484"24800 1.857362 0.921340 0.998864 0.998748 0.998029 0.051740 \n",1485"24801 2.087677 0.921140 0.999216 0.999160 0.998273 0.042458 \n",1486"24802 1.668185 0.847164 0.998240 0.998118 0.996968 0.056177 \n",1487"24803 1.859033 0.889231 0.999222 0.999119 0.998511 0.045151 \n",1488"\n",1489" ssimulacra elo \n",1490"0 0.144204 824.179269 \n",1491"1 0.125745 824.179269 \n",1492"2 0.119023 824.179269 \n",1493"3 0.142649 824.179269 \n",1494"4 0.148506 824.179269 \n",1495"... ... ... \n",1496"24799 0.014470 2699.407630 \n",1497"24800 0.016668 2699.407630 \n",1498"24801 0.013809 2699.407630 \n",1499"24802 0.016632 2699.407630 \n",1500"24803 0.015015 2699.407630 \n",1501"\n",1502"[24804 rows x 8 columns]"1503]1504},1505"execution_count": 11,1506"metadata": {},1507"output_type": "execute_result"1508}1509],1510"source": [1511"metrics = pd.concat([greater_metrics, lesser_metrics])\n",1512"metrics = metrics.sort_values('elo').reset_index(drop=True)\n",1513"metrics"1514]1515},1516{1517"cell_type": "markdown",1518"metadata": {1519"id": "gKClTtjRzjj_"1520},1521"source": [1522"Then compute the correlation matrix for these, using Spearman's rank correlation coeffient."1523]1524},1525{1526"cell_type": "code",1527"execution_count": null,1528"metadata": {1529"colab": {1530"base_uri": "https://localhost:8080/",1531"height": 3001532},1533"id": "8bTKAKn6ToJD",1534"outputId": "b355b0a9-1f06-435a-da1f-3ec3c1cbe552"1535},1536"outputs": [1537{1538"data": {1539"text/html": [1540"\n",1541" \u003cdiv id=\"df-df0a5791-d639-462a-94ac-26e812c3d0d1\"\u003e\n",1542" \u003cdiv class=\"colab-df-container\"\u003e\n",1543" \u003cdiv\u003e\n",1544"\u003cstyle scoped\u003e\n",1545" .dataframe tbody tr th:only-of-type {\n",1546" vertical-align: middle;\n",1547" }\n",1548"\n",1549" .dataframe tbody tr th {\n",1550" vertical-align: top;\n",1551" }\n",1552"\n",1553" .dataframe thead th {\n",1554" text-align: right;\n",1555" }\n",1556"\u003c/style\u003e\n",1557"\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",1558" \u003cthead\u003e\n",1559" \u003ctr style=\"text-align: right;\"\u003e\n",1560" \u003cth\u003e\u003c/th\u003e\n",1561" \u003cth\u003ebutteraugli_max\u003c/th\u003e\n",1562" \u003cth\u003ebutteraugli_6\u003c/th\u003e\n",1563" \u003cth\u003efsim-y\u003c/th\u003e\n",1564" \u003cth\u003efsim-rgb\u003c/th\u003e\n",1565" \u003cth\u003emsssim-y\u003c/th\u003e\n",1566" \u003cth\u003enlpd-y\u003c/th\u003e\n",1567" \u003cth\u003essimulacra\u003c/th\u003e\n",1568" \u003cth\u003eelo\u003c/th\u003e\n",1569" \u003c/tr\u003e\n",1570" \u003c/thead\u003e\n",1571" \u003ctbody\u003e\n",1572" \u003ctr\u003e\n",1573" \u003cth\u003ebutteraugli_max\u003c/th\u003e\n",1574" \u003ctd\u003e1.000000\u003c/td\u003e\n",1575" \u003ctd\u003e0.986256\u003c/td\u003e\n",1576" \u003ctd\u003e-0.886318\u003c/td\u003e\n",1577" \u003ctd\u003e-0.892921\u003c/td\u003e\n",1578" \u003ctd\u003e-0.823469\u003c/td\u003e\n",1579" \u003ctd\u003e0.792767\u003c/td\u003e\n",1580" \u003ctd\u003e0.842150\u003c/td\u003e\n",1581" \u003ctd\u003e-0.874967\u003c/td\u003e\n",1582" \u003c/tr\u003e\n",1583" \u003ctr\u003e\n",1584" \u003cth\u003ebutteraugli_6\u003c/th\u003e\n",1585" \u003ctd\u003e0.986256\u003c/td\u003e\n",1586" \u003ctd\u003e1.000000\u003c/td\u003e\n",1587" \u003ctd\u003e-0.877144\u003c/td\u003e\n",1588" \u003ctd\u003e-0.883633\u003c/td\u003e\n",1589" \u003ctd\u003e-0.833595\u003c/td\u003e\n",1590" \u003ctd\u003e0.807155\u003c/td\u003e\n",1591" \u003ctd\u003e0.859254\u003c/td\u003e\n",1592" \u003ctd\u003e-0.863846\u003c/td\u003e\n",1593" \u003c/tr\u003e\n",1594" \u003ctr\u003e\n",1595" \u003cth\u003efsim-y\u003c/th\u003e\n",1596" \u003ctd\u003e-0.886318\u003c/td\u003e\n",1597" \u003ctd\u003e-0.877144\u003c/td\u003e\n",1598" \u003ctd\u003e1.000000\u003c/td\u003e\n",1599" \u003ctd\u003e0.998044\u003c/td\u003e\n",1600" \u003ctd\u003e0.900061\u003c/td\u003e\n",1601" \u003ctd\u003e-0.845079\u003c/td\u003e\n",1602" \u003ctd\u003e-0.840729\u003c/td\u003e\n",1603" \u003ctd\u003e0.859066\u003c/td\u003e\n",1604" \u003c/tr\u003e\n",1605" \u003ctr\u003e\n",1606" \u003cth\u003efsim-rgb\u003c/th\u003e\n",1607" \u003ctd\u003e-0.892921\u003c/td\u003e\n",1608" \u003ctd\u003e-0.883633\u003c/td\u003e\n",1609" \u003ctd\u003e0.998044\u003c/td\u003e\n",1610" \u003ctd\u003e1.000000\u003c/td\u003e\n",1611" \u003ctd\u003e0.898808\u003c/td\u003e\n",1612" \u003ctd\u003e-0.850361\u003c/td\u003e\n",1613" \u003ctd\u003e-0.848226\u003c/td\u003e\n",1614" \u003ctd\u003e0.862459\u003c/td\u003e\n",1615" \u003c/tr\u003e\n",1616" \u003ctr\u003e\n",1617" \u003cth\u003emsssim-y\u003c/th\u003e\n",1618" \u003ctd\u003e-0.823469\u003c/td\u003e\n",1619" \u003ctd\u003e-0.833595\u003c/td\u003e\n",1620" \u003ctd\u003e0.900061\u003c/td\u003e\n",1621" \u003ctd\u003e0.898808\u003c/td\u003e\n",1622" \u003ctd\u003e1.000000\u003c/td\u003e\n",1623" \u003ctd\u003e-0.958732\u003c/td\u003e\n",1624" \u003ctd\u003e-0.942831\u003c/td\u003e\n",1625" \u003ctd\u003e0.766435\u003c/td\u003e\n",1626" \u003c/tr\u003e\n",1627" \u003ctr\u003e\n",1628" \u003cth\u003enlpd-y\u003c/th\u003e\n",1629" \u003ctd\u003e0.792767\u003c/td\u003e\n",1630" \u003ctd\u003e0.807155\u003c/td\u003e\n",1631" \u003ctd\u003e-0.845079\u003c/td\u003e\n",1632" \u003ctd\u003e-0.850361\u003c/td\u003e\n",1633" \u003ctd\u003e-0.958732\u003c/td\u003e\n",1634" \u003ctd\u003e1.000000\u003c/td\u003e\n",1635" \u003ctd\u003e0.933383\u003c/td\u003e\n",1636" \u003ctd\u003e-0.699681\u003c/td\u003e\n",1637" \u003c/tr\u003e\n",1638" \u003ctr\u003e\n",1639" \u003cth\u003essimulacra\u003c/th\u003e\n",1640" \u003ctd\u003e0.842150\u003c/td\u003e\n",1641" \u003ctd\u003e0.859254\u003c/td\u003e\n",1642" \u003ctd\u003e-0.840729\u003c/td\u003e\n",1643" \u003ctd\u003e-0.848226\u003c/td\u003e\n",1644" \u003ctd\u003e-0.942831\u003c/td\u003e\n",1645" \u003ctd\u003e0.933383\u003c/td\u003e\n",1646" \u003ctd\u003e1.000000\u003c/td\u003e\n",1647" \u003ctd\u003e-0.766894\u003c/td\u003e\n",1648" \u003c/tr\u003e\n",1649" \u003ctr\u003e\n",1650" \u003cth\u003eelo\u003c/th\u003e\n",1651" \u003ctd\u003e-0.874967\u003c/td\u003e\n",1652" \u003ctd\u003e-0.863846\u003c/td\u003e\n",1653" \u003ctd\u003e0.859066\u003c/td\u003e\n",1654" \u003ctd\u003e0.862459\u003c/td\u003e\n",1655" \u003ctd\u003e0.766435\u003c/td\u003e\n",1656" \u003ctd\u003e-0.699681\u003c/td\u003e\n",1657" \u003ctd\u003e-0.766894\u003c/td\u003e\n",1658" \u003ctd\u003e1.000000\u003c/td\u003e\n",1659" \u003c/tr\u003e\n",1660" \u003c/tbody\u003e\n",1661"\u003c/table\u003e\n",1662"\u003c/div\u003e\n",1663" \u003cbutton class=\"colab-df-convert\" onclick=\"convertToInteractive('df-df0a5791-d639-462a-94ac-26e812c3d0d1')\"\n",1664" title=\"Convert this dataframe to an interactive table.\"\n",1665" style=\"display:none;\"\u003e\n",1666" \n",1667" \u003csvg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",1668" width=\"24px\"\u003e\n",1669" \u003cpath d=\"M0 0h24v24H0V0z\" fill=\"none\"/\u003e\n",1670" \u003cpath d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/\u003e\u003cpath d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/\u003e\n",1671" \u003c/svg\u003e\n",1672" \u003c/button\u003e\n",1673" \n",1674" \u003cstyle\u003e\n",1675" .colab-df-container {\n",1676" display:flex;\n",1677" flex-wrap:wrap;\n",1678" gap: 12px;\n",1679" }\n",1680"\n",1681" .colab-df-convert {\n",1682" background-color: #E8F0FE;\n",1683" border: none;\n",1684" border-radius: 50%;\n",1685" cursor: pointer;\n",1686" display: none;\n",1687" fill: #1967D2;\n",1688" height: 32px;\n",1689" padding: 0 0 0 0;\n",1690" width: 32px;\n",1691" }\n",1692"\n",1693" .colab-df-convert:hover {\n",1694" background-color: #E2EBFA;\n",1695" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",1696" fill: #174EA6;\n",1697" }\n",1698"\n",1699" [theme=dark] .colab-df-convert {\n",1700" background-color: #3B4455;\n",1701" fill: #D2E3FC;\n",1702" }\n",1703"\n",1704" [theme=dark] .colab-df-convert:hover {\n",1705" background-color: #434B5C;\n",1706" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",1707" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",1708" fill: #FFFFFF;\n",1709" }\n",1710" \u003c/style\u003e\n",1711"\n",1712" \u003cscript\u003e\n",1713" const buttonEl =\n",1714" document.querySelector('#df-df0a5791-d639-462a-94ac-26e812c3d0d1 button.colab-df-convert');\n",1715" buttonEl.style.display =\n",1716" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",1717"\n",1718" async function convertToInteractive(key) {\n",1719" const element = document.querySelector('#df-df0a5791-d639-462a-94ac-26e812c3d0d1');\n",1720" const dataTable =\n",1721" await google.colab.kernel.invokeFunction('convertToInteractive',\n",1722" [key], {});\n",1723" if (!dataTable) return;\n",1724"\n",1725" const docLinkHtml = 'Like what you see? Visit the ' +\n",1726" '\u003ca target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb\u003edata table notebook\u003c/a\u003e'\n",1727" + ' to learn more about interactive tables.';\n",1728" element.innerHTML = '';\n",1729" dataTable['output_type'] = 'display_data';\n",1730" await google.colab.output.renderOutput(dataTable, element);\n",1731" const docLink = document.createElement('div');\n",1732" docLink.innerHTML = docLinkHtml;\n",1733" element.appendChild(docLink);\n",1734" }\n",1735" \u003c/script\u003e\n",1736" \u003c/div\u003e\n",1737" \u003c/div\u003e\n",1738" "1739],1740"text/plain": [1741" butteraugli_max butteraugli_6 fsim-y fsim-rgb msssim-y \\\n",1742"butteraugli_max 1.000000 0.986256 -0.886318 -0.892921 -0.823469 \n",1743"butteraugli_6 0.986256 1.000000 -0.877144 -0.883633 -0.833595 \n",1744"fsim-y -0.886318 -0.877144 1.000000 0.998044 0.900061 \n",1745"fsim-rgb -0.892921 -0.883633 0.998044 1.000000 0.898808 \n",1746"msssim-y -0.823469 -0.833595 0.900061 0.898808 1.000000 \n",1747"nlpd-y 0.792767 0.807155 -0.845079 -0.850361 -0.958732 \n",1748"ssimulacra 0.842150 0.859254 -0.840729 -0.848226 -0.942831 \n",1749"elo -0.874967 -0.863846 0.859066 0.862459 0.766435 \n",1750"\n",1751" nlpd-y ssimulacra elo \n",1752"butteraugli_max 0.792767 0.842150 -0.874967 \n",1753"butteraugli_6 0.807155 0.859254 -0.863846 \n",1754"fsim-y -0.845079 -0.840729 0.859066 \n",1755"fsim-rgb -0.850361 -0.848226 0.862459 \n",1756"msssim-y -0.958732 -0.942831 0.766435 \n",1757"nlpd-y 1.000000 0.933383 -0.699681 \n",1758"ssimulacra 0.933383 1.000000 -0.766894 \n",1759"elo -0.699681 -0.766894 1.000000 "1760]1761},1762"execution_count": 12,1763"metadata": {},1764"output_type": "execute_result"1765}1766],1767"source": [1768"corrs = metrics.corr(method='spearman')\n",1769"corrs"1770]1771},1772{1773"cell_type": "code",1774"execution_count": null,1775"metadata": {1776"id": "tk-8SPyiD7Bd"1777},1778"outputs": [],1779"source": [1780"metric_cols = list(map(lambda name: name[7:], lesser_metric_cols))\n",1781"metric_cols.remove('elo')"1782]1783},1784{1785"cell_type": "code",1786"execution_count": null,1787"metadata": {1788"id": "VAdFdjD0Dtmb"1789},1790"outputs": [],1791"source": [1792"def rollingcorr(df, method, window_size, step_size):\n",1793" res = []\n",1794" for start in range(0, df.shape[0] - window_size, step_size):\n",1795" window = df[start:start+window_size]\n",1796" row = [window.iloc[-1]['elo']]\n",1797" for metric_name in metric_cols:\n",1798" row.append(np.abs(window[metric_name].corr(window['elo'], method=method)))\n",1799" res.append(row)\n",1800" return pd.DataFrame(res, dtype=np.float, columns=['elo'] + list(map(lambda name: f\"{name}\", metric_cols)))"1801]1802},1803{1804"cell_type": "markdown",1805"metadata": {1806"id": "p5axWasVGP1E"1807},1808"source": [1809"Plot the correlation in a rolling window of 5000 evaluations with a step of 1000 evaluations for each metric, to see how they behave across a range of ELO scores."1810]1811},1812{1813"cell_type": "code",1814"execution_count": null,1815"metadata": {1816"colab": {1817"base_uri": "https://localhost:8080/",1818"height": 5061819},1820"id": "kBEIDlJDEIIo",1821"outputId": "c7ed5ac9-30f3-4da3-896e-912380e5955b"1822},1823"outputs": [1824{1825"name": "stderr",1826"output_type": "stream",1827"text": [1828"/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:9: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",1829"Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",1830" if __name__ == '__main__':\n"1831]1832},1833{1834"data": {1835"application/javascript": [1836"\n",1837"(function(root) {\n",1838" function now() {\n",1839" return new Date();\n",1840" }\n",1841"\n",1842" var force = true;\n",1843"\n",1844" if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n",1845" root._bokeh_onload_callbacks = [];\n",1846" root._bokeh_is_loading = undefined;\n",1847" }\n",1848"\n",1849" var JS_MIME_TYPE = 'application/javascript';\n",1850" var HTML_MIME_TYPE = 'text/html';\n",1851" var EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n",1852" var CLASS_NAME = 'output_bokeh rendered_html';\n",1853"\n",1854" /**\n",1855" * Render data to the DOM node\n",1856" */\n",1857" function render(props, node) {\n",1858" var script = document.createElement(\"script\");\n",1859" node.appendChild(script);\n",1860" }\n",1861"\n",1862" /**\n",1863" * Handle when an output is cleared or removed\n",1864" */\n",1865" function handleClearOutput(event, handle) {\n",1866" var cell = handle.cell;\n",1867"\n",1868" var id = cell.output_area._bokeh_element_id;\n",1869" var server_id = cell.output_area._bokeh_server_id;\n",1870" // Clean up Bokeh references\n",1871" if (id != null \u0026\u0026 id in Bokeh.index) {\n",1872" Bokeh.index[id].model.document.clear();\n",1873" delete Bokeh.index[id];\n",1874" }\n",1875"\n",1876" if (server_id !== undefined) {\n",1877" // Clean up Bokeh references\n",1878" var cmd = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n",1879" cell.notebook.kernel.execute(cmd, {\n",1880" iopub: {\n",1881" output: function(msg) {\n",1882" var id = msg.content.text.trim();\n",1883" if (id in Bokeh.index) {\n",1884" Bokeh.index[id].model.document.clear();\n",1885" delete Bokeh.index[id];\n",1886" }\n",1887" }\n",1888" }\n",1889" });\n",1890" // Destroy server and session\n",1891" var cmd = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n",1892" cell.notebook.kernel.execute(cmd);\n",1893" }\n",1894" }\n",1895"\n",1896" /**\n",1897" * Handle when a new output is added\n",1898" */\n",1899" function handleAddOutput(event, handle) {\n",1900" var output_area = handle.output_area;\n",1901" var output = handle.output;\n",1902"\n",1903" // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n",1904" if ((output.output_type != \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n",1905" return\n",1906" }\n",1907"\n",1908" var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n",1909"\n",1910" if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n",1911" toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n",1912" // store reference to embed id on output_area\n",1913" output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n",1914" }\n",1915" if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n",1916" var bk_div = document.createElement(\"div\");\n",1917" bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n",1918" var script_attrs = bk_div.children[0].attributes;\n",1919" for (var i = 0; i \u003c script_attrs.length; i++) {\n",1920" toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n",1921" toinsert[toinsert.length - 1].firstChild.textContent = bk_div.children[0].textContent\n",1922" }\n",1923" // store reference to server id on output_area\n",1924" output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n",1925" }\n",1926" }\n",1927"\n",1928" function register_renderer(events, OutputArea) {\n",1929"\n",1930" function append_mime(data, metadata, element) {\n",1931" // create a DOM node to render to\n",1932" var toinsert = this.create_output_subarea(\n",1933" metadata,\n",1934" CLASS_NAME,\n",1935" EXEC_MIME_TYPE\n",1936" );\n",1937" this.keyboard_manager.register_events(toinsert);\n",1938" // Render to node\n",1939" var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n",1940" render(props, toinsert[toinsert.length - 1]);\n",1941" element.append(toinsert);\n",1942" return toinsert\n",1943" }\n",1944"\n",1945" /* Handle when an output is cleared or removed */\n",1946" events.on('clear_output.CodeCell', handleClearOutput);\n",1947" events.on('delete.Cell', handleClearOutput);\n",1948"\n",1949" /* Handle when a new output is added */\n",1950" events.on('output_added.OutputArea', handleAddOutput);\n",1951"\n",1952" /**\n",1953" * Register the mime type and append_mime function with output_area\n",1954" */\n",1955" OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n",1956" /* Is output safe? */\n",1957" safe: true,\n",1958" /* Index of renderer in `output_area.display_order` */\n",1959" index: 0\n",1960" });\n",1961" }\n",1962"\n",1963" // register the mime type if in Jupyter Notebook environment and previously unregistered\n",1964" if (root.Jupyter !== undefined) {\n",1965" var events = require('base/js/events');\n",1966" var OutputArea = require('notebook/js/outputarea').OutputArea;\n",1967"\n",1968" if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n",1969" register_renderer(events, OutputArea);\n",1970" }\n",1971" }\n",1972"\n",1973" \n",1974" if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",1975" root._bokeh_timeout = Date.now() + 5000;\n",1976" root._bokeh_failed_load = false;\n",1977" }\n",1978"\n",1979" var NB_LOAD_WARNING = {'data': {'text/html':\n",1980" \"\u003cdiv style='background-color: #fdd'\u003e\\n\"+\n",1981" \"\u003cp\u003e\\n\"+\n",1982" \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",1983" \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",1984" \"\u003c/p\u003e\\n\"+\n",1985" \"\u003cul\u003e\\n\"+\n",1986" \"\u003cli\u003ere-rerun `output_notebook()` to attempt to load from CDN again, or\u003c/li\u003e\\n\"+\n",1987" \"\u003cli\u003euse INLINE resources instead, as so:\u003c/li\u003e\\n\"+\n",1988" \"\u003c/ul\u003e\\n\"+\n",1989" \"\u003ccode\u003e\\n\"+\n",1990" \"from bokeh.resources import INLINE\\n\"+\n",1991" \"output_notebook(resources=INLINE)\\n\"+\n",1992" \"\u003c/code\u003e\\n\"+\n",1993" \"\u003c/div\u003e\"}};\n",1994"\n",1995" function display_loaded() {\n",1996" var el = document.getElementById(null);\n",1997" if (el != null) {\n",1998" el.textContent = \"BokehJS is loading...\";\n",1999" }\n",2000" if (root.Bokeh !== undefined) {\n",2001" if (el != null) {\n",2002" el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n",2003" }\n",2004" } else if (Date.now() \u003c root._bokeh_timeout) {\n",2005" setTimeout(display_loaded, 100)\n",2006" }\n",2007" }\n",2008"\n",2009"\n",2010" function run_callbacks() {\n",2011" try {\n",2012" root._bokeh_onload_callbacks.forEach(function(callback) {\n",2013" if (callback != null)\n",2014" callback();\n",2015" });\n",2016" } finally {\n",2017" delete root._bokeh_onload_callbacks\n",2018" }\n",2019" console.debug(\"Bokeh: all callbacks have finished\");\n",2020" }\n",2021"\n",2022" function load_libs(css_urls, js_urls, callback) {\n",2023" if (css_urls == null) css_urls = [];\n",2024" if (js_urls == null) js_urls = [];\n",2025"\n",2026" root._bokeh_onload_callbacks.push(callback);\n",2027" if (root._bokeh_is_loading \u003e 0) {\n",2028" console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",2029" return null;\n",2030" }\n",2031" if (js_urls == null || js_urls.length === 0) {\n",2032" run_callbacks();\n",2033" return null;\n",2034" }\n",2035" console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",2036" root._bokeh_is_loading = css_urls.length + js_urls.length;\n",2037"\n",2038" function on_load() {\n",2039" root._bokeh_is_loading--;\n",2040" if (root._bokeh_is_loading === 0) {\n",2041" console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n",2042" run_callbacks()\n",2043" }\n",2044" }\n",2045"\n",2046" function on_error(url) {\n",2047" console.error(\"failed to load \" + url);\n",2048" }\n",2049"\n",2050" for (let i = 0; i \u003c css_urls.length; i++) {\n",2051" const url = css_urls[i];\n",2052" const element = document.createElement(\"link\");\n",2053" element.onload = on_load;\n",2054" element.onerror = on_error.bind(null, url);\n",2055" element.rel = \"stylesheet\";\n",2056" element.type = \"text/css\";\n",2057" element.href = url;\n",2058" console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n",2059" document.body.appendChild(element);\n",2060" }\n",2061"\n",2062" const hashes = {\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\": \"dM3QQsP+wXdHg42wTqW85BjZQdLNNIXqlPw/BgKoExPmTG7ZLML4EGqLMfqHT6ON\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\": \"8x57I4YuIfu8XyZfFo0XVr2WAT8EK4rh/uDe3wF7YuW2FNUSNEpJbsPaB1nJ2fz2\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\": \"3QTqdz9LyAm2i0sG5XTePsHec3UHWwVsrOL68SYRoAXsafvfAyqtQ+h440+qIBhS\"};\n",2063"\n",2064" for (let i = 0; i \u003c js_urls.length; i++) {\n",2065" const url = js_urls[i];\n",2066" const element = document.createElement('script');\n",2067" element.onload = on_load;\n",2068" element.onerror = on_error.bind(null, url);\n",2069" element.async = false;\n",2070" element.src = url;\n",2071" if (url in hashes) {\n",2072" element.crossOrigin = \"anonymous\";\n",2073" element.integrity = \"sha384-\" + hashes[url];\n",2074" }\n",2075" console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",2076" document.head.appendChild(element);\n",2077" }\n",2078" };\n",2079"\n",2080" function inject_raw_css(css) {\n",2081" const element = document.createElement(\"style\");\n",2082" element.appendChild(document.createTextNode(css));\n",2083" document.body.appendChild(element);\n",2084" }\n",2085"\n",2086" \n",2087" var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\"];\n",2088" var css_urls = [];\n",2089" \n",2090"\n",2091" var inline_js = [\n",2092" function(Bokeh) {\n",2093" Bokeh.set_log_level(\"info\");\n",2094" },\n",2095" function(Bokeh) {\n",2096" \n",2097" \n",2098" }\n",2099" ];\n",2100"\n",2101" function run_inline_js() {\n",2102" \n",2103" if (root.Bokeh !== undefined || force === true) {\n",2104" \n",2105" for (var i = 0; i \u003c inline_js.length; i++) {\n",2106" inline_js[i].call(root, root.Bokeh);\n",2107" }\n",2108" } else if (Date.now() \u003c root._bokeh_timeout) {\n",2109" setTimeout(run_inline_js, 100);\n",2110" } else if (!root._bokeh_failed_load) {\n",2111" console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",2112" root._bokeh_failed_load = true;\n",2113" } else if (force !== true) {\n",2114" var cell = $(document.getElementById(null)).parents('.cell').data().cell;\n",2115" cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",2116" }\n",2117"\n",2118" }\n",2119"\n",2120" if (root._bokeh_is_loading === 0) {\n",2121" console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",2122" run_inline_js();\n",2123" } else {\n",2124" load_libs(css_urls, js_urls, function() {\n",2125" console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n",2126" run_inline_js();\n",2127" });\n",2128" }\n",2129"}(window));"2130],2131"application/vnd.bokehjs_load.v0+json": "\n(function(root) {\n function now() {\n return new Date();\n }\n\n var force = true;\n\n if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n root._bokeh_onload_callbacks = [];\n root._bokeh_is_loading = undefined;\n }\n\n \n\n \n if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n root._bokeh_timeout = Date.now() + 5000;\n root._bokeh_failed_load = false;\n }\n\n var NB_LOAD_WARNING = {'data': {'text/html':\n \"\u003cdiv style='background-color: #fdd'\u003e\\n\"+\n \"\u003cp\u003e\\n\"+\n \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n \"\u003c/p\u003e\\n\"+\n \"\u003cul\u003e\\n\"+\n \"\u003cli\u003ere-rerun `output_notebook()` to attempt to load from CDN again, or\u003c/li\u003e\\n\"+\n \"\u003cli\u003euse INLINE resources instead, as so:\u003c/li\u003e\\n\"+\n \"\u003c/ul\u003e\\n\"+\n \"\u003ccode\u003e\\n\"+\n \"from bokeh.resources import INLINE\\n\"+\n \"output_notebook(resources=INLINE)\\n\"+\n \"\u003c/code\u003e\\n\"+\n \"\u003c/div\u003e\"}};\n\n function display_loaded() {\n var el = document.getElementById(null);\n if (el != null) {\n el.textContent = \"BokehJS is loading...\";\n }\n if (root.Bokeh !== undefined) {\n if (el != null) {\n el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n }\n } else if (Date.now() \u003c root._bokeh_timeout) {\n setTimeout(display_loaded, 100)\n }\n }\n\n\n function run_callbacks() {\n try {\n root._bokeh_onload_callbacks.forEach(function(callback) {\n if (callback != null)\n callback();\n });\n } finally {\n delete root._bokeh_onload_callbacks\n }\n console.debug(\"Bokeh: all callbacks have finished\");\n }\n\n function load_libs(css_urls, js_urls, callback) {\n if (css_urls == null) css_urls = [];\n if (js_urls == null) js_urls = [];\n\n root._bokeh_onload_callbacks.push(callback);\n if (root._bokeh_is_loading \u003e 0) {\n console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n return null;\n }\n if (js_urls == null || js_urls.length === 0) {\n run_callbacks();\n return null;\n }\n console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n function on_load() {\n root._bokeh_is_loading--;\n if (root._bokeh_is_loading === 0) {\n console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n run_callbacks()\n }\n }\n\n function on_error(url) {\n console.error(\"failed to load \" + url);\n }\n\n for (let i = 0; i \u003c css_urls.length; i++) {\n const url = css_urls[i];\n const element = document.createElement(\"link\");\n element.onload = on_load;\n element.onerror = on_error.bind(null, url);\n element.rel = \"stylesheet\";\n element.type = \"text/css\";\n element.href = url;\n console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n document.body.appendChild(element);\n }\n\n const hashes = {\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\": \"dM3QQsP+wXdHg42wTqW85BjZQdLNNIXqlPw/BgKoExPmTG7ZLML4EGqLMfqHT6ON\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\": \"8x57I4YuIfu8XyZfFo0XVr2WAT8EK4rh/uDe3wF7YuW2FNUSNEpJbsPaB1nJ2fz2\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\": \"3QTqdz9LyAm2i0sG5XTePsHec3UHWwVsrOL68SYRoAXsafvfAyqtQ+h440+qIBhS\"};\n\n for (let i = 0; i \u003c js_urls.length; i++) {\n const url = js_urls[i];\n const element = document.createElement('script');\n element.onload = on_load;\n element.onerror = on_error.bind(null, url);\n element.async = false;\n element.src = url;\n if (url in hashes) {\n element.crossOrigin = \"anonymous\";\n element.integrity = \"sha384-\" + hashes[url];\n }\n console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n document.head.appendChild(element);\n }\n };\n\n function inject_raw_css(css) {\n const element = document.createElement(\"style\");\n element.appendChild(document.createTextNode(css));\n document.body.appendChild(element);\n }\n\n \n var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.3.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.3.3.min.js\"];\n var css_urls = [];\n \n\n var inline_js = [\n function(Bokeh) {\n Bokeh.set_log_level(\"info\");\n },\n function(Bokeh) {\n \n \n }\n ];\n\n function run_inline_js() {\n \n if (root.Bokeh !== undefined || force === true) {\n \n for (var i = 0; i \u003c inline_js.length; i++) {\n inline_js[i].call(root, root.Bokeh);\n }\n } else if (Date.now() \u003c root._bokeh_timeout) {\n setTimeout(run_inline_js, 100);\n } else if (!root._bokeh_failed_load) {\n console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n root._bokeh_failed_load = true;\n } else if (force !== true) {\n var cell = $(document.getElementById(null)).parents('.cell').data().cell;\n cell.output_area.append_execute_result(NB_LOAD_WARNING)\n }\n\n }\n\n if (root._bokeh_is_loading === 0) {\n console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n run_inline_js();\n } else {\n load_libs(css_urls, js_urls, function() {\n console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n run_inline_js();\n });\n }\n}(window));"2132},2133"metadata": {},2134"output_type": "display_data"2135},2136{2137"data": {2138"text/html": [2139"\n",2140"\n",2141"\n",2142"\n",2143"\n",2144"\n",2145" \u003cdiv class=\"bk-root\" id=\"2f2d37ba-1716-4efd-ab9a-af3959c20f92\" data-root-id=\"1002\"\u003e\u003c/div\u003e\n"2146]2147},2148"metadata": {},2149"output_type": "display_data"2150},2151{2152"data": {2153"application/javascript": [2154"(function(root) {\n",2155" function embed_document(root) {\n",2156" \n",2157" var docs_json = {\"f8e0cf83-1ec8-4efd-aa6f-6ea4ccc965c7\":{\"defs\":[],\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1013\"}],\"center\":[{\"id\":\"1016\"},{\"id\":\"1020\"},{\"id\":\"1051\"}],\"height\":400,\"left\":[{\"id\":\"1017\"}],\"output_backend\":\"webgl\",\"renderers\":[{\"id\":\"1039\"},{\"id\":\"1059\"},{\"id\":\"1080\"},{\"id\":\"1103\"},{\"id\":\"1128\"},{\"id\":\"1155\"},{\"id\":\"1184\"}],\"sizing_mode\":\"fixed\",\"title\":{\"id\":\"1003\"},\"toolbar\":{\"id\":\"1028\"},\"width\":1400,\"x_range\":{\"id\":\"1005\"},\"x_scale\":{\"id\":\"1009\"},\"y_range\":{\"id\":\"1007\"},\"y_scale\":{\"id\":\"1011\"}},\"id\":\"1002\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"line_color\":\"#2ca02c\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-y\"}},\"id\":\"1078\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1025\",\"type\":\"ResetTool\"},{\"attributes\":{},\"id\":\"1026\",\"type\":\"HelpTool\"},{\"attributes\":{\"label\":{\"value\":\" butteraugli_6\"},\"renderers\":[{\"id\":\"1059\"}]},\"id\":\"1073\",\"type\":\"LegendItem\"},{\"attributes\":{\"source\":{\"id\":\"1055\"}},\"id\":\"1060\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1044\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1071\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1072\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1059\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"butteraugli_6\",\"@{butteraugli_6}\"]]},\"id\":\"1074\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1049\",\"type\":\"Selection\"},{\"attributes\":{\"source\":{\"id\":\"1035\"}},\"id\":\"1040\",\"type\":\"CDSView\"},{\"attributes\":{\"data_source\":{\"id\":\"1035\"},\"glyph\":{\"id\":\"1037\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1038\"},\"view\":{\"id\":\"1040\"}},\"id\":\"1039\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1042\",\"type\":\"AllLabels\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1094\"},\"selection_policy\":{\"id\":\"1095\"}},\"id\":\"1076\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#1f77b4\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_max\"}},\"id\":\"1038\",\"type\":\"Line\"},{\"attributes\":{\"label\":{\"value\":\" fsim-y\"},\"renderers\":[{\"id\":\"1080\"}]},\"id\":\"1096\",\"type\":\"LegendItem\"},{\"attributes\":{\"data_source\":{\"id\":\"1124\"},\"glyph\":{\"id\":\"1126\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1127\"},\"view\":{\"id\":\"1129\"}},\"id\":\"1128\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"click_policy\":\"hide\",\"items\":[{\"id\":\"1052\"},{\"id\":\"1073\"},{\"id\":\"1096\"},{\"id\":\"1121\"},{\"id\":\"1148\"},{\"id\":\"1177\"},{\"id\":\"1208\"}]},\"id\":\"1051\",\"type\":\"Legend\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":0.5,\"fill_color\":\"lightgrey\",\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":1.0,\"line_color\":\"black\",\"line_dash\":[4,4],\"line_width\":2,\"right_units\":\"screen\",\"syncable\":false,\"top_units\":\"screen\"},\"id\":\"1027\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"source\":{\"id\":\"1076\"}},\"id\":\"1081\",\"type\":\"CDSView\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#2ca02c\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-y\"}},\"id\":\"1079\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1076\"},\"glyph\":{\"id\":\"1078\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1079\"},\"view\":{\"id\":\"1081\"}},\"id\":\"1080\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1094\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1005\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1095\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1080\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"fsim-y\",\"@{fsim-y}\"]]},\"id\":\"1097\",\"type\":\"HoverTool\"},{\"attributes\":{\"line_color\":\"#d62728\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-rgb\"}},\"id\":\"1101\",\"type\":\"Line\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1119\"},\"selection_policy\":{\"id\":\"1120\"}},\"id\":\"1099\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"label\":{\"value\":\" fsim-rgb\"},\"renderers\":[{\"id\":\"1103\"}]},\"id\":\"1121\",\"type\":\"LegendItem\"},{\"attributes\":{},\"id\":\"1003\",\"type\":\"Title\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1049\"},\"selection_policy\":{\"id\":\"1050\"}},\"id\":\"1035\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#9467bd\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"msssim-y\"}},\"id\":\"1127\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1047\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1045\",\"type\":\"AllLabels\"},{\"attributes\":{\"source\":{\"id\":\"1099\"}},\"id\":\"1104\",\"type\":\"CDSView\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#d62728\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"fsim-rgb\"}},\"id\":\"1102\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1099\"},\"glyph\":{\"id\":\"1101\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1102\"},\"view\":{\"id\":\"1104\"}},\"id\":\"1103\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1050\",\"type\":\"UnionRenderers\"},{\"attributes\":{},\"id\":\"1119\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1120\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"label\":{\"value\":\" butteraugli_max\"},\"renderers\":[{\"id\":\"1039\"}]},\"id\":\"1052\",\"type\":\"LegendItem\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1103\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"fsim-rgb\",\"@{fsim-rgb}\"]]},\"id\":\"1122\",\"type\":\"HoverTool\"},{\"attributes\":{\"line_color\":\"#ff7f0e\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_6\"}},\"id\":\"1057\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"1007\",\"type\":\"DataRange1d\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1071\"},\"selection_policy\":{\"id\":\"1072\"}},\"id\":\"1055\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"line_color\":\"#9467bd\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"msssim-y\"}},\"id\":\"1126\",\"type\":\"Line\"},{\"attributes\":{\"label\":{\"value\":\" msssim-y\"},\"renderers\":[{\"id\":\"1128\"}]},\"id\":\"1148\",\"type\":\"LegendItem\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1146\"},\"selection_policy\":{\"id\":\"1147\"}},\"id\":\"1124\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"data_source\":{\"id\":\"1055\"},\"glyph\":{\"id\":\"1057\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1058\"},\"view\":{\"id\":\"1060\"}},\"id\":\"1059\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#ff7f0e\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_6\"}},\"id\":\"1058\",\"type\":\"Line\"},{\"attributes\":{\"source\":{\"id\":\"1124\"}},\"id\":\"1129\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1147\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"axis\":{\"id\":\"1013\"},\"ticker\":null},\"id\":\"1016\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1146\",\"type\":\"Selection\"},{\"attributes\":{\"formatter\":{\"id\":\"1044\"},\"major_label_policy\":{\"id\":\"1042\"},\"ticker\":{\"id\":\"1018\"}},\"id\":\"1017\",\"type\":\"LinearAxis\"},{\"attributes\":{\"line_color\":\"#8c564b\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"nlpd-y\"}},\"id\":\"1153\",\"type\":\"Line\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1128\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"msssim-y\",\"@{msssim-y}\"]]},\"id\":\"1149\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1018\",\"type\":\"BasicTicker\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1175\"},\"selection_policy\":{\"id\":\"1176\"}},\"id\":\"1151\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"axis\":{\"id\":\"1017\"},\"dimension\":1,\"ticker\":null},\"id\":\"1020\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1176\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"label\":{\"value\":\" nlpd-y\"},\"renderers\":[{\"id\":\"1155\"}]},\"id\":\"1177\",\"type\":\"LegendItem\"},{\"attributes\":{\"line_color\":\"#1f77b4\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"butteraugli_max\"}},\"id\":\"1037\",\"type\":\"Line\"},{\"attributes\":{\"source\":{\"id\":\"1151\"}},\"id\":\"1156\",\"type\":\"CDSView\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#8c564b\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"nlpd-y\"}},\"id\":\"1154\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1151\"},\"glyph\":{\"id\":\"1153\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1154\"},\"view\":{\"id\":\"1156\"}},\"id\":\"1155\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1175\",\"type\":\"Selection\"},{\"attributes\":{\"line_color\":\"#e377c2\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"ssimulacra\"}},\"id\":\"1182\",\"type\":\"Line\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1039\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"butteraugli_max\",\"@{butteraugli_max}\"]]},\"id\":\"1053\",\"type\":\"HoverTool\"},{\"attributes\":{\"active_multi\":null,\"active_scroll\":{\"id\":\"1022\"},\"tools\":[{\"id\":\"1021\"},{\"id\":\"1022\"},{\"id\":\"1023\"},{\"id\":\"1024\"},{\"id\":\"1025\"},{\"id\":\"1026\"},{\"id\":\"1053\"},{\"id\":\"1074\"},{\"id\":\"1097\"},{\"id\":\"1122\"},{\"id\":\"1149\"},{\"id\":\"1178\"},{\"id\":\"1209\"}]},\"id\":\"1028\",\"type\":\"Toolbar\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1155\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"nlpd-y\",\"@{nlpd-y}\"]]},\"id\":\"1178\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1014\",\"type\":\"BasicTicker\"},{\"attributes\":{\"data\":{\"__x__values\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"__x__values_original\":{\"__ndarray__\":\"xLEugsd7mkDKMsQdF4ybQKK0N2gMW5xAUrgeRaWknUDnHadWxXKeQAfwFgJNH59ArK3Y7+f4n0D5oGdKvU6gQDqSy5dzpaBAyxDHotHeoEDWVuxXsiOhQPp+at50VKFAi2znlcGGoUA/NV7I7q+hQEHxY+R/4qFAKqkTrOsSokCvJeS9SkmiQJCg+EU3f6JA+Q/plza4okBAE2EZ/xOjQA==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_6\":{\"__ndarray__\":\"3UbgWG2r5T958AIRhxPhP36eEdskz9g/bfewLeGI4D9CGtifOPrhP15pOq62FeI/N/VgkFHA3D/v2RhlDgzXP0DX2xtzD9c/Gy3j0F/72D9JGhd+z3fYP63MwDfrUts/4MSH+ShY2D87d77Qo5/PPzZjxgU1f8c/PQjodvo1wD/SKI7c66HKP+Fs1S12E7M/lbQY42kXpz+jbD44Tii8Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"butteraugli_max\":{\"__ndarray__\":\"bEylh/P55D/NU0l5bcHgP46bB80d3Ng/K3SuzVhC3z9YVr43DtLhP3tweg3KvOI/fHN3P+FX3j87yKCq6YTXP0JBFlflstY/duOPWPd42D8wN4ylbunXP+B/Q1jibNs/K/7e/GIM2T8Awb5t/bjRP+VP5jCj/Mk/0P0Fsz/cwj+++zkB9tDPP2j4so00dLg/ggGo0nfFrT8vBk05no25Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-rgb\":{\"__ndarray__\":\"aPjtJ2W84T/8LtQW3XXeP7YvPrLVbNQ/5Og/rCJw2z/r0xVQLmbaP7J1GbqTqNs/DJUkNoSS3T/1uVPNsEDNPxkTOsf2q9M/EfygMDmm0T+klgX0iWbTP/05ZiHddd0/u/LBHwNg1j93G/ecUC7UPwIKimHnEck/Qi/IOnv/zj/5fDgkRu7VP+wBlEeWSMM/953QR9RCrj8wYVcd9gm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"fsim-y\":{\"__ndarray__\":\"+C0HyzSw4T+r5NOZxXXePyG2yU7mNdQ/ZTUZ8vtc2z9/d2VCYGnaPySLTb6RDNs/02Z+02Sa3D9MbnOKMxTLP0NVvhJUTdM/p/j6LGnc0D/iZnDpiWbSP0jsbMt2cNw/WZ2iS4Ax1j8TyIBNo6bUP5pYa6Uk5sg/ikIYjyChzT/xutc4LbHUPw1Se7Mbw8E/VIewKJbErT8V6+HxXRa6Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"msssim-y\":{\"__ndarray__\":\"9Nk8Kkx82z9CjJDtRCjVP217MuF9H8g/LY4oE1Vm0z+eT2n7UOfQP6vSK1VZW9A/0rFIBM0O0j+/Ai03ZCe7P/XDjygcINE/iX5bEOwY0D9wxgwG7MzNP/orMX+9kdI//A85PFJdyz8+hJ8pGBXMPxSeGcOW0Lo/CGlcZ5h6wT9gDhROVCfLP7jizoAGGaM/tUqOcZWMqD9CXYW7LUm4Pw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"nlpd-y\":{\"__ndarray__\":\"SingXqt+1j9iy5bdcsbQP5gtKeFZncM/+t2r1gouzz8LVKMJZibKP636pkQHUMo/YsKziv5D0D8hcGHT2Bu7P2xupNljxcs/kCy7Janozj+fSf25C4nKP644MS0tlNI/eY11v1mpxT/RMwd/HenCP8m5Yn3WorU/X3M6w1DjvD8aQ+8eLtPGP+j2l9ZwsYg/N/FKmxK9lj8CzEWRrEyqPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]},\"ssimulacra\":{\"__ndarray__\":\"SUb/cOmA2z80Yr/R4krVP0J4oF2ViMw/iSreuUgS1T/6j6z7LyLTP23Kl1DoadA/0mp55oFzzz/iVI3cHhvGP284Dd+hqtE/wJtkh6hE0j86YDFuVHHSP10k0WjrXtE/Qc8pUWDhyj/pQjQXJD3IP+/8WYbLJb8/girrFokkxT/KT74rOPfHP/I+XHIqCIA/fM29XIH3dD/MXuK0u9OvPw==\",\"dtype\":\"float64\",\"order\":\"little\",\"shape\":[20]}},\"selected\":{\"id\":\"1206\"},\"selection_policy\":{\"id\":\"1207\"}},\"id\":\"1180\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"label\":{\"value\":\" ssimulacra\"},\"renderers\":[{\"id\":\"1184\"}]},\"id\":\"1208\",\"type\":\"LegendItem\"},{\"attributes\":{},\"id\":\"1021\",\"type\":\"PanTool\"},{\"attributes\":{\"source\":{\"id\":\"1180\"}},\"id\":\"1185\",\"type\":\"CDSView\"},{\"attributes\":{\"axis_label\":\"elo\",\"formatter\":{\"id\":\"1047\"},\"major_label_policy\":{\"id\":\"1045\"},\"ticker\":{\"id\":\"1014\"}},\"id\":\"1013\",\"type\":\"LinearAxis\"},{\"attributes\":{\"line_alpha\":0.1,\"line_color\":\"#e377c2\",\"line_width\":2,\"x\":{\"field\":\"__x__values\"},\"y\":{\"field\":\"ssimulacra\"}},\"id\":\"1183\",\"type\":\"Line\"},{\"attributes\":{\"data_source\":{\"id\":\"1180\"},\"glyph\":{\"id\":\"1182\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1183\"},\"view\":{\"id\":\"1185\"}},\"id\":\"1184\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1009\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1206\",\"type\":\"Selection\"},{\"attributes\":{},\"id\":\"1011\",\"type\":\"LinearScale\"},{\"attributes\":{\"overlay\":{\"id\":\"1027\"}},\"id\":\"1023\",\"type\":\"BoxZoomTool\"},{\"attributes\":{},\"id\":\"1207\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"renderers\":[{\"id\":\"1184\"}],\"tooltips\":[[\"elo\",\"@__x__values_original\"],[\"ssimulacra\",\"@{ssimulacra}\"]]},\"id\":\"1209\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1024\",\"type\":\"SaveTool\"},{\"attributes\":{},\"id\":\"1022\",\"type\":\"WheelZoomTool\"}],\"root_ids\":[\"1002\"]},\"title\":\"Bokeh Application\",\"version\":\"2.3.3\"}};\n",2158" var render_items = [{\"docid\":\"f8e0cf83-1ec8-4efd-aa6f-6ea4ccc965c7\",\"root_ids\":[\"1002\"],\"roots\":{\"1002\":\"2f2d37ba-1716-4efd-ab9a-af3959c20f92\"}}];\n",2159" root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",2160"\n",2161" }\n",2162" if (root.Bokeh !== undefined) {\n",2163" embed_document(root);\n",2164" } else {\n",2165" var attempts = 0;\n",2166" var timer = setInterval(function(root) {\n",2167" if (root.Bokeh !== undefined) {\n",2168" clearInterval(timer);\n",2169" embed_document(root);\n",2170" } else {\n",2171" attempts++;\n",2172" if (attempts \u003e 100) {\n",2173" clearInterval(timer);\n",2174" console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n",2175" }\n",2176" }\n",2177" }, 10, root)\n",2178" }\n",2179"})(window);"2180],2181"application/vnd.bokehjs_exec.v0+json": ""2182},2183"metadata": {2184"application/vnd.bokehjs_exec.v0+json": {2185"id": "1002"2186}2187},2188"output_type": "display_data"2189},2190{2191"data": {2192"text/html": [2193"\u003cdiv style=\"display: table;\"\u003e\u003cdiv style=\"display: table-row;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003cb title=\"bokeh.plotting.figure.Figure\"\u003eFigure\u003c/b\u003e(\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eid\u0026nbsp;=\u0026nbsp;'1002', \u003cspan id=\"1344\" style=\"cursor: pointer;\"\u003e\u0026hellip;)\u003c/span\u003e\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eabove\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ealign\u0026nbsp;=\u0026nbsp;'start',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003easpect_ratio\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003easpect_scale\u0026nbsp;=\u0026nbsp;1,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebackground\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebackground_fill_alpha\u0026nbsp;=\u0026nbsp;1.0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebackground_fill_color\u0026nbsp;=\u0026nbsp;'#ffffff',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ebelow\u0026nbsp;=\u0026nbsp;[LinearAxis(id='1013', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eborder_fill_alpha\u0026nbsp;=\u0026nbsp;1.0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eborder_fill_color\u0026nbsp;=\u0026nbsp;'#ffffff',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ecenter\u0026nbsp;=\u0026nbsp;[Grid(id='1016', ...), Grid(id='1020', ...), Legend(id='1051', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ecss_classes\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003edisabled\u0026nbsp;=\u0026nbsp;False,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eextra_x_ranges\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eextra_y_ranges\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eframe_height\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eframe_width\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eheight\u0026nbsp;=\u0026nbsp;400,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eheight_policy\u0026nbsp;=\u0026nbsp;'auto',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ehidpi\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003einner_height\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003einner_width\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ejs_event_callbacks\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ejs_property_callbacks\u0026nbsp;=\u0026nbsp;{},\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eleft\u0026nbsp;=\u0026nbsp;[LinearAxis(id='1017', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_factor\u0026nbsp;=\u0026nbsp;10,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_interval\u0026nbsp;=\u0026nbsp;300,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_threshold\u0026nbsp;=\u0026nbsp;2000,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003elod_timeout\u0026nbsp;=\u0026nbsp;500,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emargin\u0026nbsp;=\u0026nbsp;(0, 0, 0, 0),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ematch_aspect\u0026nbsp;=\u0026nbsp;False,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emax_height\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emax_width\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border\u0026nbsp;=\u0026nbsp;5,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_bottom\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_left\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_right\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_border_top\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_height\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003emin_width\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ename\u0026nbsp;=\u0026nbsp;None,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eouter_height\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eouter_width\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_alpha\u0026nbsp;=\u0026nbsp;1.0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_cap\u0026nbsp;=\u0026nbsp;'butt',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_color\u0026nbsp;=\u0026nbsp;'#e5e5e5',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_dash\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_dash_offset\u0026nbsp;=\u0026nbsp;0,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_join\u0026nbsp;=\u0026nbsp;'bevel',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutline_line_width\u0026nbsp;=\u0026nbsp;1,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eoutput_backend\u0026nbsp;=\u0026nbsp;'webgl',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003erenderers\u0026nbsp;=\u0026nbsp;[GlyphRenderer(id='1039', ...), GlyphRenderer(id='1059', ...), GlyphRenderer(id='1080', ...), GlyphRenderer(id='1103', ...), GlyphRenderer(id='1128', ...), GlyphRenderer(id='1155', ...), GlyphRenderer(id='1184', ...)],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ereset_policy\u0026nbsp;=\u0026nbsp;'standard',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003eright\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003esizing_mode\u0026nbsp;=\u0026nbsp;'fixed',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003esubscribed_events\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003esyncable\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etags\u0026nbsp;=\u0026nbsp;[],\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etitle\u0026nbsp;=\u0026nbsp;Title(id='1003', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etitle_location\u0026nbsp;=\u0026nbsp;'above',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etoolbar\u0026nbsp;=\u0026nbsp;Toolbar(id='1028', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etoolbar_location\u0026nbsp;=\u0026nbsp;'right',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003etoolbar_sticky\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003evisible\u0026nbsp;=\u0026nbsp;True,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ewidth\u0026nbsp;=\u0026nbsp;1400,\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ewidth_policy\u0026nbsp;=\u0026nbsp;'auto',\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ex_range\u0026nbsp;=\u0026nbsp;DataRange1d(id='1005', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ex_scale\u0026nbsp;=\u0026nbsp;LinearScale(id='1009', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ey_range\u0026nbsp;=\u0026nbsp;DataRange1d(id='1007', ...),\u003c/div\u003e\u003c/div\u003e\u003cdiv class=\"1343\" style=\"display: none;\"\u003e\u003cdiv style=\"display: table-cell;\"\u003e\u003c/div\u003e\u003cdiv style=\"display: table-cell;\"\u003ey_scale\u0026nbsp;=\u0026nbsp;LinearScale(id='1011', ...))\u003c/div\u003e\u003c/div\u003e\u003c/div\u003e\n",2194"\u003cscript\u003e\n",2195"(function() {\n",2196" var expanded = false;\n",2197" var ellipsis = document.getElementById(\"1344\");\n",2198" ellipsis.addEventListener(\"click\", function() {\n",2199" var rows = document.getElementsByClassName(\"1343\");\n",2200" for (var i = 0; i \u003c rows.length; i++) {\n",2201" var el = rows[i];\n",2202" el.style.display = expanded ? \"none\" : \"table-row\";\n",2203" }\n",2204" ellipsis.innerHTML = expanded ? \"\u0026hellip;)\" : \"\u0026lsaquo;\u0026lsaquo;\u0026lsaquo;\";\n",2205" expanded = !expanded;\n",2206" });\n",2207"})();\n",2208"\u003c/script\u003e\n"2209],2210"text/plain": [2211"Figure(id='1002', ...)"2212]2213},2214"execution_count": 15,2215"metadata": {},2216"output_type": "execute_result"2217}2218],2219"source": [2220"rollingcorr(metrics, 'spearman', 5000, 1000).plot_bokeh(x='elo', figsize=(1400, 400))"2221]2222}2223],2224"metadata": {2225"colab": {2226"collapsed_sections": [],2227"name": "correlations.ipynb",2228"provenance": []2229},2230"kernelspec": {2231"display_name": "Python 3",2232"name": "python3"2233},2234"language_info": {2235"name": "python"2236}2237},2238"nbformat": 4,2239"nbformat_minor": 02240}2241