1
// Copyright (c) 2012-2020 Ugorji Nwoke. All rights reserved.
2
// Use of this source code is governed by a MIT license found in the LICENSE file.
5
Package codec provides a
6
High Performance, Feature-Rich Idiomatic Go 1.4+ codec/encoding library
7
for binc, msgpack, cbor, json.
9
Supported Serialization formats are:
11
- msgpack: https://github.com/msgpack/msgpack
12
- binc: http://github.com/ugorji/binc
13
- cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
14
- json: http://json.org http://tools.ietf.org/html/rfc7159
17
This package will carefully use 'package unsafe' for performance reasons in specific places.
18
You can build without unsafe use by passing the safe or appengine tag
19
i.e. 'go install -tags=codec.safe ...'.
21
This library works with both the standard `gc` and the `gccgo` compilers.
23
For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer .
25
The idiomatic Go support is as seen in other encoding packages in
26
the standard library (ie json, xml, gob, etc).
28
Rich Feature Set includes:
30
- Simple but extremely powerful and feature-rich API
31
- Support for go 1.4 and above, while selectively using newer APIs for later releases
32
- Excellent code coverage ( > 90% )
33
- Very High Performance.
34
Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
35
- Careful selected use of 'unsafe' for targeted performance gains.
36
- 100% safe mode supported, where 'unsafe' is not used at all.
37
- Lock-free (sans mutex) concurrency for scaling to 100's of cores
38
- In-place updates during decode, with option to zero value in maps and slices prior to decode
39
- Coerce types where appropriate
40
e.g. decode an int in the stream into a float, decode numbers from formatted strings, etc
42
Overflows, nil maps/slices, nil values in streams are handled correctly
43
- Standard field renaming via tags
44
- Support for omitting empty fields during an encoding
45
- Encoding from any value and decoding into pointer to any value
46
(struct, slice, map, primitives, pointers, interface{}, etc)47
- Extensions to support efficient encoding/decoding of any named types
48
- Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
49
- Support using existence of `IsZero() bool` to determine if a value is a zero value.
50
Analogous to time.Time.IsZero() bool.
51
- Decoding without a schema (into a interface{}).52
Includes Options to configure what specific map or slice type to use
53
when decoding an encoded list or map into a nil interface{}54
- Mapping a non-interface type to an interface, so we can decode appropriately
55
into any interface type with a correctly configured non-interface value.
56
- Encode a struct as an array, and decode struct from an array in the data stream
57
- Option to encode struct keys as numbers (instead of strings)
58
(to support structured streams with fields encoded as numeric codes)
59
- Comprehensive support for anonymous fields
60
- Fast (no-reflection) encoding/decoding of common maps and slices
61
- Code-generation for faster performance, supported in go 1.6+
62
- Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
63
- Support indefinite-length formats to enable true streaming
64
(for formats which support it e.g. json, cbor)
65
- Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
66
This mostly applies to maps, where iteration order is non-deterministic.
67
- NIL in data stream decoded as zero value
68
- Never silently skip data when decoding.
69
User decides whether to return an error or silently skip data when keys or indexes
70
in the data stream do not map to fields in the struct.
71
- Detect and error when encoding a cyclic reference (instead of stack overflow shutdown)
72
- Encode/Decode from/to chan types (for iterative streaming support)
73
- Drop-in replacement for encoding/json. `json:` key in struct tag supported.
74
- Provides a RPC Server and Client Codec for net/rpc communication protocol.
75
- Handle unique idiosyncrasies of codecs e.g.
76
For messagepack, configure how ambiguities in handling raw bytes are resolved and
77
provide rpc server/client codec to support
78
msgpack-rpc protocol defined at:
79
https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
83
Users can register a function to handle the encoding or decoding of
86
There are no restrictions on what the custom type can be. Some examples:
91
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }92
type GifImage struct { ... }94
As an illustration, MyStructWithUnexportedFields would normally be
95
encoded as an empty map because it has no exported fields, while UUID
96
would be encoded as a string. However, with extension support, you can
97
encode any of these however you like.
99
There is also seamless support provided for registering an extension (with a tag)
100
but letting the encoding mechanism default to the standard way.
102
# Custom Encoding and Decoding
104
This package maintains symmetry in the encoding and decoding halfs.
105
We determine how to encode or decode by walking this decision tree
107
- is there an extension registered for the type?
108
- is type a codec.Selfer?
109
- is format binary, and is type a encoding.BinaryMarshaler and BinaryUnmarshaler?
110
- is format specifically json, and is type a encoding/json.Marshaler and Unmarshaler?
111
- is format text-based, and type an encoding.TextMarshaler and TextUnmarshaler?
112
- else we use a pair of functions based on the "kind" of the type e.g. map, slice, int64, etc
114
This symmetry is important to reduce chances of issues happening because the
115
encoding and decoding sides are out of sync e.g. decoded via very specific
116
encoding.TextUnmarshaler but encoded via kind-specific generalized mode.
118
Consequently, if a type only defines one-half of the symmetry
119
(e.g. it implements UnmarshalJSON() but not MarshalJSON() ),
120
then that type doesn't satisfy the check and we will continue walking down the
125
RPC Client and Server Codecs are implemented, so the codecs can be used
126
with the standard net/rpc package.
130
The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification.
132
The Encoder and Decoder are NOT safe for concurrent use.
134
Consequently, the usage model is basically:
136
- Create and initialize the Handle before any use.
137
Once created, DO NOT modify it.
138
- Multiple Encoders or Decoders can now use the Handle concurrently.
139
They only read information off the Handle (never write).
140
- However, each Encoder or Decoder MUST not be used concurrently
141
- To re-use an Encoder/Decoder, call Reset(...) on it first.
142
This allows you use state maintained on the Encoder/Decoder.
146
// create and configure Handle
149
mh codec.MsgpackHandle
153
mh.MapType = reflect.TypeOf(map[string]interface{}(nil))155
// configure extensions
156
// e.g. for msgpack, define functions and enable Time support for tag 1
157
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)159
// create and use decoder/encoder
164
h = &bh // or mh to use msgpack
167
dec = codec.NewDecoder(r, h)
168
dec = codec.NewDecoderBytes(b, h)
171
enc = codec.NewEncoder(w, h)
172
enc = codec.NewEncoderBytes(&b, h)
178
conn, err := listener.Accept()
179
rpcCodec := codec.GoRpc.ServerCodec(conn, h)
180
//OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
181
rpc.ServeCodec(rpcCodec)
185
//RPC Communication (client side)
186
conn, err = net.Dial("tcp", "localhost:5555")187
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
188
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
189
client := rpc.NewClientWithCodec(rpcCodec)
193
To run tests, use the following:
197
To run the full suite of tests, use the following:
199
go test -tags alltests -run Suite
201
You can run the tag 'codec.safe' to run tests or build in safe mode. e.g.
203
go test -tags codec.safe -run Json
204
go test -tags "alltests codec.safe" -run Suite
209
go test -bench . -benchmem -benchtime 1s
211
Please see http://github.com/ugorji/go-codec-bench .
215
Struct fields matching the following are ignored during encoding and decoding
216
- struct tag value set to -
217
- func, complex numbers, unsafe pointers
218
- unexported and not embedded
219
- unexported and embedded and not struct kind
220
- unexported and embedded pointers (from go1.10)
222
Every other field in a struct will be encoded/decoded.
224
Embedded fields are encoded as if they exist in the top-level struct,
225
with some caveats. See Encode documentation.