promptflow

1
{
2
 "cells": [
3
  {
4
   "cell_type": "markdown",
5
   "metadata": {},
6
   "source": [
7
    "# Evaluate with langchain's evaluator"
8
   ]
9
  },
10
  {
11
   "cell_type": "markdown",
12
   "metadata": {},
13
   "source": [
14
    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
15
    "\n",
16
    "- Convert LangChain criteria evaluator applications to `flex flow`.\n",
17
    "- Use `CustomConnection` to store secrets.\n",
18
    "\n",
19
    "## 0. Install dependent packages"
20
   ]
21
  },
22
  {
23
   "cell_type": "code",
24
   "execution_count": null,
25
   "metadata": {},
26
   "outputs": [],
27
   "source": [
28
    "%%capture --no-stderr\n",
29
    "%pip install -r ./requirements.txt"
30
   ]
31
  },
32
  {
33
   "cell_type": "markdown",
34
   "metadata": {},
35
   "source": [
36
    "## 1. Trace your langchain evaluator with prompt flow\n",
37
    "### Initialize a pf client"
38
   ]
39
  },
40
  {
41
   "cell_type": "code",
42
   "execution_count": null,
43
   "metadata": {},
44
   "outputs": [],
45
   "source": [
46
    "from promptflow.client import PFClient\n",
47
    "\n",
48
    "pf = PFClient()"
49
   ]
50
  },
51
  {
52
   "cell_type": "markdown",
53
   "metadata": {},
54
   "source": [
55
    "### Create a custom connection to protect your API key\n",
56
    "\n",
57
    "You can protect your API key in custom connection's secrets."
58
   ]
59
  },
60
  {
61
   "cell_type": "code",
62
   "execution_count": null,
63
   "metadata": {},
64
   "outputs": [],
65
   "source": [
66
    "import os\n",
67
    "from dotenv import load_dotenv\n",
68
    "\n",
69
    "from promptflow.entities import CustomConnection\n",
70
    "\n",
71
    "conn_name = \"my_llm_connection\"\n",
72
    "\n",
73
    "try:\n",
74
    "    conn = pf.connections.get(name=conn_name)\n",
75
    "    print(\"using existing connection\")\n",
76
    "except:\n",
77
    "    if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
78
    "        # load environment variables from .env file\n",
79
    "        load_dotenv()\n",
80
    "\n",
81
    "    # put API key in secrets\n",
82
    "    connection = CustomConnection(\n",
83
    "        name=conn_name,\n",
84
    "        configs={\n",
85
    "            \"azure_endpoint\": os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
86
    "        },\n",
87
    "        secrets={\n",
88
    "            # store API key\n",
89
    "            # \"anthropic_api_key\": \"<your-api-key>\",\n",
90
    "            \"openai_api_key\": os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
91
    "        },\n",
92
    "    )\n",
93
    "    # Create the connection, note that all secret values will be scrubbed in the returned result\n",
94
    "    conn = pf.connections.create_or_update(connection)\n",
95
    "    print(\"successfully created connection\")\n",
96
    "print(conn)"
97
   ]
98
  },
99
  {
100
   "cell_type": "markdown",
101
   "metadata": {},
102
   "source": [
103
    "### Test the evaluator with trace"
104
   ]
105
  },
106
  {
107
   "cell_type": "code",
108
   "execution_count": null,
109
   "metadata": {},
110
   "outputs": [],
111
   "source": [
112
    "from eval_conciseness import LangChainEvaluator\n",
113
    "\n",
114
    "\n",
115
    "evaluator = LangChainEvaluator(custom_connection=conn)\n",
116
    "result = evaluator(\n",
117
    "    prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
118
    "    input=\"What's 2+2?\",\n",
119
    ")\n",
120
    "print(result)"
121
   ]
122
  },
123
  {
124
   "cell_type": "markdown",
125
   "metadata": {},
126
   "source": [
127
    "## 2. Batch run the evaluator with flow yaml\n",
128
    "Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/eval-criteria-with-langchain/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
129
   ]
130
  },
131
  {
132
   "cell_type": "code",
133
   "execution_count": null,
134
   "metadata": {},
135
   "outputs": [],
136
   "source": [
137
    "data = \"./data.jsonl\"  # path to the data file\n",
138
    "# create run with the flow function and data\n",
139
    "base_run = pf.run(\n",
140
    "    flow=\"./flow.flex.yaml\",\n",
141
    "    # reference custom connection by name\n",
142
    "    init={\n",
143
    "        \"custom_connection\": \"my_llm_connection\",\n",
144
    "    },\n",
145
    "    data=data,\n",
146
    "    column_mapping={\n",
147
    "        \"prediction\": \"${data.prediction}\",\n",
148
    "        \"input\": \"${data.input}\",\n",
149
    "    },\n",
150
    "    stream=True,\n",
151
    ")"
152
   ]
153
  },
154
  {
155
   "cell_type": "code",
156
   "execution_count": null,
157
   "metadata": {},
158
   "outputs": [],
159
   "source": [
160
    "details = pf.get_details(base_run)\n",
161
    "details.head(10)"
162
   ]
163
  },
164
  {
165
   "cell_type": "code",
166
   "execution_count": null,
167
   "metadata": {},
168
   "outputs": [],
169
   "source": [
170
    "pf.visualize([base_run])"
171
   ]
172
  }
173
 ],
174
 "metadata": {
175
  "build_doc": {
176
   "author": [
177
    "D-W-@github.com",
178
    "wangchao1230@github.com"
179
   ],
180
   "category": "local",
181
   "section": "Flow",
182
   "weight": 60
183
  },
184
  "description": "A tutorial to converting LangChain criteria evaluator application to flex flow.",
185
  "kernelspec": {
186
   "display_name": "prompt_flow",
187
   "language": "python",
188
   "name": "python3"
189
  },
190
  "language_info": {
191
   "codemirror_mode": {
192
    "name": "ipython",
193
    "version": 3
194
   },
195
   "file_extension": ".py",
196
   "mimetype": "text/x-python",
197
   "name": "python",
198
   "nbconvert_exporter": "python",
199
   "pygments_lexer": "ipython3",
200
   "version": "3.9.18"
201
  },
202
  "resources": "examples/flex-flows/eval-criteria-with-langchain"
203
 },
204
 "nbformat": 4,
205
 "nbformat_minor": 2
206
}
207