promptflow

1
{
2
 "cells": [
3
  {
4
   "cell_type": "markdown",
5
   "metadata": {},
6
   "source": [
7
    "# Getting started with flex flow"
8
   ]
9
  },
10
  {
11
   "cell_type": "markdown",
12
   "metadata": {},
13
   "source": [
14
    "\n",
15
    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
16
    "\n",
17
    "- Write LLM application using notebook and visualize the trace of your application.\n",
18
    "- Convert the application into a flow and batch run against multi lines of data.\n"
19
   ]
20
  },
21
  {
22
   "cell_type": "markdown",
23
   "metadata": {},
24
   "source": [
25
    "## 0. Install dependent packages"
26
   ]
27
  },
28
  {
29
   "cell_type": "code",
30
   "execution_count": null,
31
   "metadata": {},
32
   "outputs": [],
33
   "source": [
34
    "%%capture --no-stderr\n",
35
    "%pip install -r ./requirements.txt"
36
   ]
37
  },
38
  {
39
   "cell_type": "markdown",
40
   "metadata": {},
41
   "source": [
42
    "## 1. Trace your application with promptflow\n",
43
    "\n",
44
    "Assume we already have a python function that calls OpenAI API. "
45
   ]
46
  },
47
  {
48
   "cell_type": "code",
49
   "execution_count": null,
50
   "metadata": {},
51
   "outputs": [],
52
   "source": [
53
    "with open(\"llm.py\") as fin:\n",
54
    "    print(fin.read())"
55
   ]
56
  },
57
  {
58
   "cell_type": "markdown",
59
   "metadata": {},
60
   "source": [
61
    "Note: before running below cell, please configure required environment variable `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` by create an `.env` file. Please refer to `../.env.example` as an template."
62
   ]
63
  },
64
  {
65
   "cell_type": "code",
66
   "execution_count": null,
67
   "metadata": {},
68
   "outputs": [],
69
   "source": [
70
    "# control the AOAI deployment (model) used in this example\n",
71
    "deployment_name = \"gpt-35-turbo\""
72
   ]
73
  },
74
  {
75
   "cell_type": "code",
76
   "execution_count": null,
77
   "metadata": {},
78
   "outputs": [],
79
   "source": [
80
    "from llm import my_llm_tool\n",
81
    "\n",
82
    "# pls configure `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` environment variables first\n",
83
    "result = my_llm_tool(\n",
84
    "    prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
85
    "    deployment_name=deployment_name,\n",
86
    ")\n",
87
    "result"
88
   ]
89
  },
90
  {
91
   "cell_type": "markdown",
92
   "metadata": {},
93
   "source": [
94
    "### Visualize trace by using start_trace\n",
95
    "\n",
96
    "Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
97
   ]
98
  },
99
  {
100
   "cell_type": "code",
101
   "execution_count": null,
102
   "metadata": {},
103
   "outputs": [],
104
   "source": [
105
    "from promptflow.tracing import start_trace\n",
106
    "\n",
107
    "# start a trace session, and print a url for user to check trace\n",
108
    "start_trace()\n",
109
    "# rerun the function, which will be recorded in the trace\n",
110
    "result = my_llm_tool(\n",
111
    "    prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
112
    "    deployment_name=deployment_name,\n",
113
    ")\n",
114
    "result"
115
   ]
116
  },
117
  {
118
   "cell_type": "markdown",
119
   "metadata": {},
120
   "source": [
121
    "Now, let's add another layer of function call. In `programmer.py` there is a function called `write_simple_program`, which calls a new function called `load_prompt` and previous `my_llm_tool` function."
122
   ]
123
  },
124
  {
125
   "cell_type": "code",
126
   "execution_count": null,
127
   "metadata": {},
128
   "outputs": [],
129
   "source": [
130
    "# show the programmer.py content\n",
131
    "with open(\"programmer.py\") as fin:\n",
132
    "    print(fin.read())"
133
   ]
134
  },
135
  {
136
   "cell_type": "code",
137
   "execution_count": null,
138
   "metadata": {},
139
   "outputs": [],
140
   "source": [
141
    "# call the flow entry function\n",
142
    "from programmer import write_simple_program\n",
143
    "\n",
144
    "result = write_simple_program(\"Java Hello, world!\")\n",
145
    "result"
146
   ]
147
  },
148
  {
149
   "cell_type": "markdown",
150
   "metadata": {},
151
   "source": [
152
    "### Setup model configuration with environment variables\n",
153
    "\n",
154
    "When used in local, create a model configuration object with environment variables."
155
   ]
156
  },
157
  {
158
   "cell_type": "code",
159
   "execution_count": null,
160
   "metadata": {},
161
   "outputs": [],
162
   "source": [
163
    "import os\n",
164
    "from dotenv import load_dotenv\n",
165
    "\n",
166
    "from promptflow.core import AzureOpenAIModelConfiguration\n",
167
    "\n",
168
    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
169
    "    # load environment variables from .env file\n",
170
    "    load_dotenv()\n",
171
    "\n",
172
    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
173
    "    raise Exception(\"Please specify environment variables: AZURE_OPENAI_API_KEY\")\n",
174
    "model_config = AzureOpenAIModelConfiguration(\n",
175
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
176
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
177
    "    azure_deployment=deployment_name,\n",
178
    "    api_version=\"2023-07-01-preview\",\n",
179
    ")"
180
   ]
181
  },
182
  {
183
   "cell_type": "markdown",
184
   "metadata": {},
185
   "source": [
186
    "### Eval the result "
187
   ]
188
  },
189
  {
190
   "cell_type": "code",
191
   "execution_count": null,
192
   "metadata": {},
193
   "outputs": [],
194
   "source": [
195
    "%load_ext autoreload\n",
196
    "%autoreload 2\n",
197
    "\n",
198
    "import paths  # add the code_quality module to the path\n",
199
    "from code_quality import CodeEvaluator\n",
200
    "\n",
201
    "evaluator = CodeEvaluator(model_config=model_config)\n",
202
    "eval_result = evaluator(result)\n",
203
    "eval_result"
204
   ]
205
  },
206
  {
207
   "cell_type": "markdown",
208
   "metadata": {},
209
   "source": [
210
    "## 2. Batch run the function as flow with multi-line data\n",
211
    "\n",
212
    "Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
213
   ]
214
  },
215
  {
216
   "cell_type": "code",
217
   "execution_count": null,
218
   "metadata": {},
219
   "outputs": [],
220
   "source": [
221
    "# show the flow.flex.yaml content\n",
222
    "with open(\"flow.flex.yaml\") as fin:\n",
223
    "    print(fin.read())"
224
   ]
225
  },
226
  {
227
   "cell_type": "markdown",
228
   "metadata": {},
229
   "source": [
230
    "### Batch run with a data file (with multiple lines of test data)\n"
231
   ]
232
  },
233
  {
234
   "cell_type": "code",
235
   "execution_count": null,
236
   "metadata": {},
237
   "outputs": [],
238
   "source": [
239
    "from promptflow.client import PFClient\n",
240
    "\n",
241
    "pf = PFClient()"
242
   ]
243
  },
244
  {
245
   "cell_type": "code",
246
   "execution_count": null,
247
   "metadata": {},
248
   "outputs": [],
249
   "source": [
250
    "data = \"./data.jsonl\"  # path to the data file\n",
251
    "# create run with the flow function and data\n",
252
    "base_run = pf.run(\n",
253
    "    flow=write_simple_program,\n",
254
    "    data=data,\n",
255
    "    column_mapping={\n",
256
    "        \"text\": \"${data.text}\",\n",
257
    "    },\n",
258
    "    stream=True,\n",
259
    ")"
260
   ]
261
  },
262
  {
263
   "cell_type": "code",
264
   "execution_count": null,
265
   "metadata": {},
266
   "outputs": [],
267
   "source": [
268
    "details = pf.get_details(base_run)\n",
269
    "details.head(10)"
270
   ]
271
  },
272
  {
273
   "cell_type": "markdown",
274
   "metadata": {},
275
   "source": [
276
    "## 3. Evaluate your flow\n",
277
    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
278
   ]
279
  },
280
  {
281
   "cell_type": "markdown",
282
   "metadata": {},
283
   "source": [
284
    "### Run evaluation on the previous batch run\n",
285
    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
286
   ]
287
  },
288
  {
289
   "cell_type": "code",
290
   "execution_count": null,
291
   "metadata": {},
292
   "outputs": [],
293
   "source": [
294
    "# we can also run flow pointing to yaml file\n",
295
    "eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
296
    "\n",
297
    "eval_run = pf.run(\n",
298
    "    flow=eval_flow,\n",
299
    "    init={\"model_config\": model_config},\n",
300
    "    data=\"./data.jsonl\",  # path to the data file\n",
301
    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
302
    "    column_mapping={\n",
303
    "        \"code\": \"${run.outputs.output}\",\n",
304
    "    },\n",
305
    "    stream=True,\n",
306
    ")"
307
   ]
308
  },
309
  {
310
   "cell_type": "code",
311
   "execution_count": null,
312
   "metadata": {},
313
   "outputs": [],
314
   "source": [
315
    "details = pf.get_details(eval_run)\n",
316
    "details.head(10)"
317
   ]
318
  },
319
  {
320
   "cell_type": "code",
321
   "execution_count": null,
322
   "metadata": {},
323
   "outputs": [],
324
   "source": [
325
    "import json\n",
326
    "\n",
327
    "metrics = pf.get_metrics(eval_run)\n",
328
    "print(json.dumps(metrics, indent=4))"
329
   ]
330
  },
331
  {
332
   "cell_type": "code",
333
   "execution_count": null,
334
   "metadata": {},
335
   "outputs": [],
336
   "source": [
337
    "pf.visualize([base_run, eval_run])"
338
   ]
339
  },
340
  {
341
   "cell_type": "markdown",
342
   "metadata": {},
343
   "source": [
344
    "## Next steps\n",
345
    "\n",
346
    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
347
    "\n",
348
    "You can check out more examples:\n",
349
    "- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
350
   ]
351
  }
352
 ],
353
 "metadata": {
354
  "build_doc": {
355
   "author": [
356
    "D-W-@github.com",
357
    "wangchao1230@github.com"
358
   ],
359
   "category": "local",
360
   "section": "Flow",
361
   "weight": 10
362
  },
363
  "description": "A quickstart tutorial to run a flex flow and evaluate it.",
364
  "kernelspec": {
365
   "display_name": "prompt_flow",
366
   "language": "python",
367
   "name": "python3"
368
  },
369
  "language_info": {
370
   "codemirror_mode": {
371
    "name": "ipython",
372
    "version": 3
373
   },
374
   "file_extension": ".py",
375
   "mimetype": "text/x-python",
376
   "name": "python",
377
   "nbconvert_exporter": "python",
378
   "pygments_lexer": "ipython3",
379
   "version": "3.9.18"
380
  },
381
  "resources": "examples/requirements.txt, examples/flex-flows/basic"
382
 },
383
 "nbformat": 4,
384
 "nbformat_minor": 2
385
}
386