{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Quickstart: Pandas API on Spark\n", "\n", "This is a short introduction to pandas API on Spark, geared mainly for new users. This notebook shows you some key differences between pandas and pandas API on Spark. You can run this examples by yourself in 'Live Notebook: pandas API on Spark' at [the quickstart page](https://spark.apache.org/docs/latest/api/python/getting_started/index.html).\n", "\n", "Customarily, we import pandas API on Spark as follows:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import pyspark.pandas as ps\n", "from pyspark.sql import SparkSession" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Object Creation\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a pandas-on-Spark Series by passing a list of values, letting pandas API on Spark create a default integer index:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "s = ps.Series([1, 3, 5, np.nan, 6, 8])" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 1.0\n", "1 3.0\n", "2 5.0\n", "3 NaN\n", "4 6.0\n", "5 8.0\n", "dtype: float64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a pandas-on-Spark DataFrame by passing a dict of objects that can be converted to series-like." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "psdf = ps.DataFrame(\n", " {'a': [1, 2, 3, 4, 5, 6],\n", " 'b': [100, 200, 300, 400, 500, 600],\n", " 'c': [\"one\", \"two\", \"three\", \"four\", \"five\", \"six\"]},\n", " index=[10, 20, 30, 40, 50, 60])" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	a	b	c
10	1	100	one
20	2	200	two
30	3	300	three
40	4	400	four
50	5	500	five
60	6	600	six

\n", "

" ], "text/plain": [ " a b c\n", "10 1 100 one\n", "20 2 200 two\n", "30 3 300 three\n", "40 4 400 four\n", "50 5 500 five\n", "60 6 600 six" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a pandas DataFrame by passing a numpy array, with a datetime index and labeled columns:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "dates = pd.date_range('20130101', periods=6)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',\n", " '2013-01-05', '2013-01-06'],\n", " dtype='datetime64[ns]', freq='D')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dates" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "pdf = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	A	B	C	D
2013-01-01	0.912558	-0.795645	-0.289115	0.187606
2013-01-02	-0.059703	-1.233897	0.316625	-1.226828
2013-01-03	0.332871	-1.262010	-0.434844	-0.579920
2013-01-04	0.924016	-1.022019	-0.405249	-1.036021
2013-01-05	-0.772209	-1.228099	0.068901	0.896679
2013-01-06	1.485582	-0.709306	-0.202637	-0.248766

\n", "

	A	B	C	D
2013-01-01	0.912558	-0.795645	-0.289115	0.187606
2013-01-02	-0.059703	-1.233897	0.316625	-1.226828
2013-01-03	0.332871	-1.262010	-0.434844	-0.579920
2013-01-04	0.924016	-1.022019	-0.405249	-1.036021
2013-01-05	-0.772209	-1.228099	0.068901	0.896679
2013-01-06	1.485582	-0.709306	-0.202637	-0.248766

\n", "

" ], "text/plain": [ " A B C D\n", "2013-01-01 0.912558 -0.795645 -0.289115 0.187606\n", "2013-01-02 -0.059703 -1.233897 0.316625 -1.226828\n", "2013-01-03 0.332871 -1.262010 -0.434844 -0.579920\n", "2013-01-04 0.924016 -1.022019 -0.405249 -1.036021\n", "2013-01-05 -0.772209 -1.228099 0.068901 0.896679\n", "2013-01-06 1.485582 -0.709306 -0.202637 -0.248766" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Also, it is possible to create a pandas-on-Spark DataFrame from Spark DataFrame easily.\n", "\n", "Creating a Spark DataFrame from pandas DataFrame" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "spark = SparkSession.builder.getOrCreate()" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "sdf = spark.createDataFrame(pdf)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "+--------------------+-------------------+--------------------+--------------------+\n", "| A| B| C| D|\n", "+--------------------+-------------------+--------------------+--------------------+\n", "| 0.91255803205208|-0.7956452608556638|-0.28911463069772175| 0.18760566615081622|\n", "|-0.05970271470242...| -1.233896949308984| 0.3166246451758431| -1.2268284000402265|\n", "| 0.33287106947536615|-1.2620100816441786| -0.4348444277082644| -0.5799199651437185|\n", "| 0.9240158461589916|-1.0220190956326003| -0.4052488880650239| -1.0360212104348547|\n", "| -0.7722090016558953|-1.2280986385313222| 0.0689011451939635| 0.8966790729426755|\n", "| 1.4855822995785612|-0.7093056426018517| -0.2026366848847041|-0.24876619876451092|\n", "+--------------------+-------------------+--------------------+--------------------+\n", "\n" ] } ], "source": [ "sdf.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating pandas-on-Spark DataFrame from Spark DataFrame." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "psdf = sdf.pandas_api()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D
0	0.912558	-0.795645	-0.289115	0.187606
1	-0.059703	-1.233897	0.316625	-1.226828
2	0.332871	-1.262010	-0.434844	-0.579920
3	0.924016	-1.022019	-0.405249	-1.036021
4	-0.772209	-1.228099	0.068901	0.896679
5	1.485582	-0.709306	-0.202637	-0.248766

\n", "

" ], "text/plain": [ " A B C D\n", "0 0.912558 -0.795645 -0.289115 0.187606\n", "1 -0.059703 -1.233897 0.316625 -1.226828\n", "2 0.332871 -1.262010 -0.434844 -0.579920\n", "3 0.924016 -1.022019 -0.405249 -1.036021\n", "4 -0.772209 -1.228099 0.068901 0.896679\n", "5 1.485582 -0.709306 -0.202637 -0.248766" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Having specific [dtypes](http://pandas.pydata.org/pandas-docs/stable/basics.html#basics-dtypes) . Types that are common to both Spark and pandas are currently supported." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "A float64\n", "B float64\n", "C float64\n", "D float64\n", "dtype: object" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is how to show top rows from the frame below.\n", "\n", "Note that the data in a Spark dataframe does not preserve the natural order by default. The natural order can be preserved by setting `compute.ordered_head` option but it causes a performance overhead with sorting internally." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D
0	0.912558	-0.795645	-0.289115	0.187606
1	-0.059703	-1.233897	0.316625	-1.226828
2	0.332871	-1.262010	-0.434844	-0.579920
3	0.924016	-1.022019	-0.405249	-1.036021
4	-0.772209	-1.228099	0.068901	0.896679

\n", "

" ], "text/plain": [ " A B C D\n", "0 0.912558 -0.795645 -0.289115 0.187606\n", "1 -0.059703 -1.233897 0.316625 -1.226828\n", "2 0.332871 -1.262010 -0.434844 -0.579920\n", "3 0.924016 -1.022019 -0.405249 -1.036021\n", "4 -0.772209 -1.228099 0.068901 0.896679" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Displaying the index, columns, and the underlying numpy data." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index([0, 1, 2, 3, 4, 5], dtype='int64')" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.index" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['A', 'B', 'C', 'D'], dtype='object')" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.columns" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ 0.91255803, -0.79564526, -0.28911463, 0.18760567],\n", " [-0.05970271, -1.23389695, 0.31662465, -1.2268284 ],\n", " [ 0.33287107, -1.26201008, -0.43484443, -0.57991997],\n", " [ 0.92401585, -1.0220191 , -0.40524889, -1.03602121],\n", " [-0.772209 , -1.22809864, 0.06890115, 0.89667907],\n", " [ 1.4855823 , -0.70930564, -0.20263668, -0.2487662 ]])" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.to_numpy()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Showing a quick statistic summary of your data" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D
count	6.000000	6.000000	6.000000	6.000000
mean	0.470519	-1.041829	-0.157720	-0.334542
std	0.809428	0.241511	0.294520	0.793014
min	-0.772209	-1.262010	-0.434844	-1.226828
25%	-0.059703	-1.233897	-0.405249	-1.036021
50%	0.332871	-1.228099	-0.289115	-0.579920
75%	0.924016	-0.795645	0.068901	0.187606
max	1.485582	-0.709306	0.316625	0.896679

\n", "

" ], "text/plain": [ " A B C D\n", "count 6.000000 6.000000 6.000000 6.000000\n", "mean 0.470519 -1.041829 -0.157720 -0.334542\n", "std 0.809428 0.241511 0.294520 0.793014\n", "min -0.772209 -1.262010 -0.434844 -1.226828\n", "25% -0.059703 -1.233897 -0.405249 -1.036021\n", "50% 0.332871 -1.228099 -0.289115 -0.579920\n", "75% 0.924016 -0.795645 0.068901 0.187606\n", "max 1.485582 -0.709306 0.316625 0.896679" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transposing your data" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	0	1	2	3	4	5
A	0.912558	-0.059703	0.332871	0.924016	-0.772209	1.485582
B	-0.795645	-1.233897	-1.262010	-1.022019	-1.228099	-0.709306
C	-0.289115	0.316625	-0.434844	-0.405249	0.068901	-0.202637
D	0.187606	-1.226828	-0.579920	-1.036021	0.896679	-0.248766

\n", "

" ], "text/plain": [ " 0 1 2 3 4 5\n", "A 0.912558 -0.059703 0.332871 0.924016 -0.772209 1.485582\n", "B -0.795645 -1.233897 -1.262010 -1.022019 -1.228099 -0.709306\n", "C -0.289115 0.316625 -0.434844 -0.405249 0.068901 -0.202637\n", "D 0.187606 -1.226828 -0.579920 -1.036021 0.896679 -0.248766" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sorting by its index" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D
5	1.485582	-0.709306	-0.202637	-0.248766
4	-0.772209	-1.228099	0.068901	0.896679
3	0.924016	-1.022019	-0.405249	-1.036021
2	0.332871	-1.262010	-0.434844	-0.579920
1	-0.059703	-1.233897	0.316625	-1.226828
0	0.912558	-0.795645	-0.289115	0.187606

\n", "

" ], "text/plain": [ " A B C D\n", "5 1.485582 -0.709306 -0.202637 -0.248766\n", "4 -0.772209 -1.228099 0.068901 0.896679\n", "3 0.924016 -1.022019 -0.405249 -1.036021\n", "2 0.332871 -1.262010 -0.434844 -0.579920\n", "1 -0.059703 -1.233897 0.316625 -1.226828\n", "0 0.912558 -0.795645 -0.289115 0.187606" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.sort_index(ascending=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sorting by value" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D
2	0.332871	-1.262010	-0.434844	-0.579920
1	-0.059703	-1.233897	0.316625	-1.226828
4	-0.772209	-1.228099	0.068901	0.896679
3	0.924016	-1.022019	-0.405249	-1.036021
0	0.912558	-0.795645	-0.289115	0.187606
5	1.485582	-0.709306	-0.202637	-0.248766

\n", "

" ], "text/plain": [ " A B C D\n", "2 0.332871 -1.262010 -0.434844 -0.579920\n", "1 -0.059703 -1.233897 0.316625 -1.226828\n", "4 -0.772209 -1.228099 0.068901 0.896679\n", "3 0.924016 -1.022019 -0.405249 -1.036021\n", "0 0.912558 -0.795645 -0.289115 0.187606\n", "5 1.485582 -0.709306 -0.202637 -0.248766" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.sort_values(by='B')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Missing Data\n", "Pandas API on Spark primarily uses the value `np.nan` to represent missing data. It is by default not included in computations. \n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "pdf1 = pdf.reindex(index=dates[0:4], columns=list(pdf.columns) + ['E'])" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "pdf1.loc[dates[0]:dates[1], 'E'] = 1" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "psdf1 = ps.from_pandas(pdf1)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D	E
2013-01-01	0.912558	-0.795645	-0.289115	0.187606	1.0
2013-01-02	-0.059703	-1.233897	0.316625	-1.226828	1.0
2013-01-03	0.332871	-1.262010	-0.434844	-0.579920	NaN
2013-01-04	0.924016	-1.022019	-0.405249	-1.036021	NaN

\n", "

	A	B	C	D	E
2013-01-01	0.912558	-0.795645	-0.289115	0.187606	1.0
2013-01-02	-0.059703	-1.233897	0.316625	-1.226828	1.0

\n", "

" ], "text/plain": [ " A B C D E\n", "2013-01-01 0.912558 -0.795645 -0.289115 0.187606 1.0\n", "2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 1.0" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf1.dropna(how='any')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Filling missing data." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D	E
2013-01-01	0.912558	-0.795645	-0.289115	0.187606	1.0
2013-01-02	-0.059703	-1.233897	0.316625	-1.226828	1.0
2013-01-03	0.332871	-1.262010	-0.434844	-0.579920	5.0
2013-01-04	0.924016	-1.022019	-0.405249	-1.036021	5.0

\n", "

" ], "text/plain": [ " A B C D E\n", "2013-01-01 0.912558 -0.795645 -0.289115 0.187606 1.0\n", "2013-01-02 -0.059703 -1.233897 0.316625 -1.226828 1.0\n", "2013-01-03 0.332871 -1.262010 -0.434844 -0.579920 5.0\n", "2013-01-04 0.924016 -1.022019 -0.405249 -1.036021 5.0" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf1.fillna(value=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Operations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Stats\n", "Performing a descriptive statistic:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "A 0.470519\n", "B -1.041829\n", "C -0.157720\n", "D -0.334542\n", "dtype: float64" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Spark Configurations\n", "\n", "Various configurations in PySpark could be applied internally in pandas API on Spark.\n", "For example, you can enable Arrow optimization to hugely speed up internal pandas conversion. See also PySpark Usage Guide for Pandas with Apache Arrow in PySpark documentation." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "prev = spark.conf.get(\"spark.sql.execution.arrow.pyspark.enabled\") # Keep its default value.\n", "ps.set_option(\"compute.default_index_type\", \"distributed\") # Use default index prevent overhead.\n", "import warnings\n", "warnings.filterwarnings(\"ignore\") # Ignore warnings coming from Arrow optimizations." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "900 ms ± 186 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", True)\n", "%timeit ps.range(300000).to_pandas()" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3.08 s ± 227 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", False)\n", "%timeit ps.range(300000).to_pandas()" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "ps.reset_option(\"compute.default_index_type\")\n", "spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", prev) # Set its default value back." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Grouping\n", "By “group by” we are referring to a process involving one or more of the following steps:\n", "\n", "- Splitting the data into groups based on some criteria\n", "- Applying a function to each group independently\n", "- Combining the results into a data structure" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "psdf = ps.DataFrame({'A': ['foo', 'bar', 'foo', 'bar',\n", " 'foo', 'bar', 'foo', 'foo'],\n", " 'B': ['one', 'one', 'two', 'three',\n", " 'two', 'two', 'one', 'three'],\n", " 'C': np.random.randn(8),\n", " 'D': np.random.randn(8)})" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	A	B	C	D
0	foo	one	1.039632	-0.571950
1	bar	one	0.972089	1.085353
2	foo	two	-1.931621	-2.579164
3	bar	three	-0.654371	-0.340704
4	foo	two	-0.157080	0.893736
5	bar	two	0.882795	0.024978
6	foo	one	-0.149384	0.201667
7	foo	three	-1.355136	0.693883

\n", "

" ], "text/plain": [ " A B C D\n", "0 foo one 1.039632 -0.571950\n", "1 bar one 0.972089 1.085353\n", "2 foo two -1.931621 -2.579164\n", "3 bar three -0.654371 -0.340704\n", "4 foo two -0.157080 0.893736\n", "5 bar two 0.882795 0.024978\n", "6 foo one -0.149384 0.201667\n", "7 foo three -1.355136 0.693883" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Grouping and then applying the [sum()](https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.groupby.GroupBy.sum.html) function to the resulting groups." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	B	C	D
A
bar	onethreetwo	1.200513	0.769627
foo	onetwotwoonethree	-2.553589	-1.361828

\n", "

" ], "text/plain": [ " C D\n", "A \n", "bar 1.200513 0.769627\n", "foo -2.553589 -1.361828" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.groupby('A').sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Grouping by multiple columns forms a hierarchical index, and again we can apply the sum function." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

		C	D
A	B
foo	one	0.890248	-0.370283
two	-2.088701	-1.685428
bar	three	-0.654371	-0.340704
foo	three	-1.355136	0.693883
bar	two	0.882795	0.024978
one	0.972089	1.085353

\n", "

" ], "text/plain": [ " C D\n", "A B \n", "foo one 0.890248 -0.370283\n", " two -2.088701 -1.685428\n", "bar three -0.654371 -0.340704\n", "foo three -1.355136 0.693883\n", "bar two 0.882795 0.024978\n", " one 0.972089 1.085353" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "psdf.groupby(['A', 'B']).sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "pser = pd.Series(np.random.randn(1000),\n", " index=pd.date_range('1/1/2000', periods=1000))" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "psser = ps.Series(pser)" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "psser = psser.cummax()" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/html": [ "