Clever Geek Handbook

在开发过程中在Go中寻找失控的内存

问题

堆内存转义具有以下潜在可解决的问题：

  1. 由于内存分配成本而导致性能降低

  2. 由于垃圾收集成本导致性能下降

  3. Out of Memory如果垃圾出现率超过清除率，则会出现错误

这些问题可以通过几种方式解决：

  1. 计算资源（内存，处理器）数量的增加

  2. 微调垃圾收集器

  3. 通过减少堆中的芽数

在本文中，我将仅考虑第三种方式。

从头开始

, , , . , : make , new , map ,go . , "", - .

make new . sync.Pool(), habr . , []byte , , - 32 ( go1.14.6 windows/amd64). , ; , , butebufferpool @valyala.

map . map , . , map, , - Discord Rust. .. , , , - . @valyala fastcache, , , , , , slices maps.

go - . , .

, - "" , , fastcache "". , , []byte->[]byte, , . , , - , "rps", "95th percentile latency" .. , , "" , . " " . .. "".

" " : 

git clone https://github.com/valyala/fasthttp

, , , , , :

go test -bench=PServerGet10Req -benchmem -memprofile netmem.out

go test -bench=kServerGet10Req -benchmem  -memprofile fastmem.out

http.Server, - fasthttp.Server. , fasthttp zero-allocation . , netmem.out fastmem.out. -, :

echo top | go tool pprof netmem.out

10 "" :

Showing top 10 nodes out of 53
      flat  flat%   sum%        cum   cum%
  698.15MB 21.85% 21.85%   710.15MB 22.22%  net/textproto.(*Reader).ReadMIMEHeader
  466.13MB 14.59% 36.43%   466.13MB 14.59%  net/http.Header.Clone
  423.07MB 13.24% 49.67%  1738.32MB 54.39%  net/http.(*conn).readRequest
  384.12MB 12.02% 61.69%   384.12MB 12.02%  net/textproto.MIMEHeader.Set
  299.07MB  9.36% 71.05%  1186.24MB 37.12%  net/http.readRequest
  137.02MB  4.29% 75.33%   137.02MB  4.29%  bufio.NewReaderSize
  134.02MB  4.19% 79.53%   134.02MB  4.19%  net/url.parse
  122.45MB  3.83% 83.36%   122.45MB  3.83%  bufio.NewWriterSize (inline)
   99.51MB  3.11% 86.47%   133.01MB  4.16%  context.WithCancel
   87.11MB  2.73% 89.20%    87.11MB  2.73%  github.com/andybalholm/brotli.(*h5).Initialize

: 

go tool pprof -svg netmem.out > netmem.svg

netmem.svg ():

:

go tool pprof -http=:8088 netmem.out

, , , - : Could not execute dot; may need to install graphviz. , Unix- , , Windows chocolatey , , cinst graphviz. - . VIEW/Source:

, make, []byte string. , " ", []byte. "", , , .. return &localVar . , - .

fasthttp , . - fasthttp HTTP/2.0, websockets , fasthttp ( , ), , , . c5.4xlarge 250.000 RPS fasthttp.Server 190.000 RPS http.Server . , , 190.000 RPS? , , , , .

, , . " " , json/yaml , , . - , , , .

, flatbuffers, dynobuffers, - , . , , , .

" ":

Avro         23394 ns/op    11257 B/op
Dyno_Untyped  6437 ns/op      808 B/op
Dyno_Typed    3776 ns/op        0 B/op
Flat          1132 ns/op        0 B/op
Json         87331 ns/op    14145 B/op

" ":

Avro         19311 ns/op    11257 B/op
Dyno_Typed    62.2 ns/op        0 B/op
Flat          19.8 ns/op        0 B/op
Json         83824 ns/op    11073 B/op

, , , linkedin/goavro - .

- . , () , avro "" , flatbuffer.

可以解决导致Go性能下降的问题，但是请记住，这种斗争是有代价的。在开始之前，最好将所选数据处理方法的功能与系统要求进行实验比较-也许一切都将以最小的调整“即开即用”地工作。

链接



More articles:


All Articles