llama

1
ied 4 ½ months
2
__ggml_vocab_test__
3
Führer
4
__ggml_vocab_test__
5

6
__ggml_vocab_test__
7
 
8
__ggml_vocab_test__
9
  
10
__ggml_vocab_test__
11
   
12
__ggml_vocab_test__
13
	
14
__ggml_vocab_test__
15

16

17
__ggml_vocab_test__
18

19

20

21
__ggml_vocab_test__
22

23

24

25

26
__ggml_vocab_test__
27
	
28

29
__ggml_vocab_test__
30
Hello world
31
__ggml_vocab_test__
32
 Hello world
33
__ggml_vocab_test__
34
Hello World
35
__ggml_vocab_test__
36
 Hello World
37
__ggml_vocab_test__
38
 Hello World!
39
__ggml_vocab_test__
40
Hello, world!
41
__ggml_vocab_test__
42
 Hello, world!
43
__ggml_vocab_test__
44
 this is 🦙.cpp
45
__ggml_vocab_test__
46
w048 7tuijk dsdfhu
47
__ggml_vocab_test__
48
нещо на Български
49
__ggml_vocab_test__
50
កាន់តែពិសេសអាចខលចេញ
51
__ggml_vocab_test__
52
🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)
53
__ggml_vocab_test__
54
Hello
55
__ggml_vocab_test__
56
 Hello
57
__ggml_vocab_test__
58
  Hello
59
__ggml_vocab_test__
60
   Hello
61
__ggml_vocab_test__
62
    Hello
63
__ggml_vocab_test__
64
    Hello
65
    Hello
66
__ggml_vocab_test__
67
 (
68
__ggml_vocab_test__
69

70
 =
71
__ggml_vocab_test__
72
' era
73
__ggml_vocab_test__
74
Hello, y'all! How are you 😁 ?我想在apple工作1314151天～
75
__ggml_vocab_test__
76
!!!!!!
77
__ggml_vocab_test__
78
3
79
__ggml_vocab_test__
80
33
81
__ggml_vocab_test__
82
333
83
__ggml_vocab_test__
84
3333
85
__ggml_vocab_test__
86
33333
87
__ggml_vocab_test__
88
333333
89
__ggml_vocab_test__
90
3333333
91
__ggml_vocab_test__
92
33333333
93
__ggml_vocab_test__
94
333333333
95
__ggml_vocab_test__
96
Cửa Việt
97
__ggml_vocab_test__
98
 discards
99
__ggml_vocab_test__
100

101
 
102

103
 
104

105

106
 	 		 	
107
  
108
   
109
    
110
     
111
🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL
112
__ggml_vocab_test__
113