golang/go

all: reduce init-time CPU & memory usage

Open

#26,775 opened on Aug 2, 2018

View on GitHub
 (21 comments) (6 reactions) (0 assignees)Go (19,008 forks)batch import
Performancehelp wanted

Repository metrics

Stars
 (133,883 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Tracking bug to reduce init-time CPU & memory usage.

Previously:

https://go-review.googlesource.com/c/go/+/127075 - html: lazily populate Unescape tables https://go-review.googlesource.com/c/net/+/127275 - http2/hpack: lazily build huffman table on first use

Open:

#26752 - x/text/unicode/norm: reduce init-time memory usage

The program https://play.golang.org/p/9ervXCWzV_z is useful to find offenders:

$ go build all.go
$ GODEBUG=memprofilerate=1  ./all
346768
$ go tool pprof /tmp/all.mem.prof 
(pprof) top 50
Showing nodes accounting for 286.24kB, 88.87% of 322.09kB total
Dropped 228 nodes (cum <= 1.61kB)
Showing top 50 nodes out of 133
      flat  flat%   sum%        cum   cum%
   37.03kB 11.50% 11.50%    37.03kB 11.50%  runtime.procresize
      32kB  9.94% 21.43%       32kB  9.94%  hash/crc64.makeSlicingBy8Table
   21.39kB  6.64% 28.07%    21.39kB  6.64%  vendor/golang_org/x/text/unicode/norm.init
   19.98kB  6.20% 34.28%    19.98kB  6.20%  strings.NewReplacer
   18.95kB  5.88% 40.16%    18.97kB  5.89%  encoding/xml.init
   12.17kB  3.78% 43.94%    12.17kB  3.78%  unicode.init
   10.92kB  3.39% 47.33%    10.92kB  3.39%  regexp/syntax.appendRange
   10.61kB  3.29% 50.63%    10.61kB  3.29%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
   10.38kB  3.22% 53.85%    12.62kB  3.92%  sync.(*Map).LoadOrStore
    9.38kB  2.91% 56.76%     9.38kB  2.91%  runtime.malg
    9.17kB  2.85% 59.61%     9.22kB  2.86%  html/template.init
    8.70kB  2.70% 62.31%    69.47kB 21.57%  net/http.init
    6.84kB  2.12% 64.43%     6.84kB  2.12%  regexp/syntax.(*compiler).inst (inline)
    6.12kB  1.90% 66.34%    13.12kB  4.07%  runtime.allocm
    5.25kB  1.63% 67.97%     5.25kB  1.63%  math/rand.NewSource
    5.06kB  1.57% 69.54%     5.47kB  1.70%  net/http.init.0
    4.03kB  1.25% 70.79%     4.22kB  1.31%  net/http.init.2
    3.94kB  1.22% 72.01%     3.95kB  1.23%  debug/dwarf.init
    3.72kB  1.15% 73.17%     3.72kB  1.15%  errors.New
    3.62kB  1.13% 74.29%    12.13kB  3.77%  encoding/gob.validUserType
    3.62kB  1.13% 75.42%     3.62kB  1.13%  regexp/syntax.init
    2.91kB   0.9% 76.32%     2.91kB   0.9%  go/types.(*Scope).Insert
    2.70kB  0.84% 77.16%    26.97kB  8.37%  encoding/gob.init
    2.69kB  0.83% 77.99%     2.69kB  0.83%  flag.(*FlagSet).Var
    2.59kB  0.81% 78.80%     2.59kB  0.81%  net/mail.init.0
    2.25kB   0.7% 79.50%     2.25kB   0.7%  compress/flate.(*huffmanEncoder).generate
    2.25kB   0.7% 80.20%     2.25kB   0.7%  net/textproto.init.0
    2.09kB  0.65% 80.85%     2.09kB  0.65%  image/jpeg.(*huffmanLUT).init
    1.89kB  0.59% 81.43%     1.89kB  0.59%  compress/flate.newHuffmanEncoder
    1.80kB  0.56% 81.99%     1.80kB  0.56%  go/types.NewTypeName
    1.77kB  0.55% 82.54%     1.77kB  0.55%  text/template/parse.(*Tree).newText (inline)
    1.75kB  0.54% 83.08%     4.38kB  1.36%  runtime.mcommoninit
    1.66kB  0.51% 83.60%     8.84kB  2.74%  go/types.init
    1.64kB  0.51% 84.11%    28.67kB  8.90%  go/doc.init
    1.59kB  0.49% 84.60%     3.57kB  1.11%  encoding/gob.bootstrapType
    1.55kB  0.48% 85.08%     2.98kB  0.93%  net.init
    1.39kB  0.43% 85.51%    32.34kB 10.04%  go/build.init
    1.17kB  0.36% 85.88%     3.09kB  0.96%  text/template.init
    1.16kB  0.36% 86.24%     3.55kB  1.10%  encoding/gob.newTypeObject
    1.06kB  0.33% 86.57%    14.55kB  4.52%  encoding/gob.RegisterName
    1.02kB  0.32% 86.88%     4.57kB  1.42%  encoding/gob.buildTypeInfo
    0.94kB  0.29% 87.17%    21.06kB  6.54%  regexp.compile
    0.91kB  0.28% 87.45%    16.88kB  5.24%  encoding/gob.init.1
    0.75kB  0.23% 87.69%     1.84kB  0.57%  html/template.New
    0.69kB  0.21% 87.90%     2.13kB  0.66%  reflect.(*rtype).ptrTo
    0.67kB  0.21% 88.11%     7.20kB  2.24%  crypto/tls.init
    0.67kB  0.21% 88.32%    22.42kB  6.96%  vendor/golang_org/x/net/http/httpguts.init
    0.61kB  0.19% 88.51%    14.49kB  4.50%  archive/tar.init
    0.61kB  0.19% 88.70%     5.40kB  1.68%  net/http/pprof.init
    0.56kB  0.17% 88.87%     7.81kB  2.43%  runtime.systemstack

(pprof) top --cum 50
Showing nodes accounting for 222.77kB, 69.16% of 322.09kB total
Dropped 228 nodes (cum <= 1.61kB)
Showing top 50 nodes out of 133
      flat  flat%   sum%        cum   cum%
    0.09kB 0.029% 0.029%   268.74kB 83.44%  runtime.main
         0     0% 0.029%   266.68kB 82.80%  main.init
         0     0% 0.029%    70.38kB 21.85%  expvar.init
    8.70kB  2.70%  2.73%    69.47kB 21.57%  net/http.init
         0     0%  2.73%    37.41kB 11.61%  runtime.rt0_go
   37.03kB 11.50% 14.23%    37.03kB 11.50%  runtime.procresize
         0     0% 14.23%    37.03kB 11.50%  runtime.schedinit
    1.39kB  0.43% 14.66%    32.34kB 10.04%  go/build.init
         0     0% 14.66%       32kB  9.94%  hash/crc64.init
      32kB  9.94% 24.60%       32kB  9.94%  hash/crc64.makeSlicingBy8Table
    1.64kB  0.51% 25.10%    28.67kB  8.90%  go/doc.init
    2.70kB  0.84% 25.94%    26.97kB  8.37%  encoding/gob.init
    0.67kB  0.21% 26.15%    22.42kB  6.96%  vendor/golang_org/x/net/http/httpguts.init
         0     0% 26.15%    21.75kB  6.75%  vendor/golang_org/x/net/idna.init
   21.39kB  6.64% 32.79%    21.39kB  6.64%  vendor/golang_org/x/text/unicode/norm.init
         0     0% 32.79%    21.06kB  6.54%  regexp.Compile
         0     0% 32.79%    21.06kB  6.54%  regexp.MustCompile
    0.94kB  0.29% 33.09%    21.06kB  6.54%  regexp.compile
   19.98kB  6.20% 39.29%    19.98kB  6.20%  strings.NewReplacer
   18.95kB  5.88% 45.17%    18.97kB  5.89%  encoding/xml.init
    0.91kB  0.28% 45.46%    16.88kB  5.24%  encoding/gob.init.1
         0     0% 45.46%    14.55kB  4.52%  encoding/gob.Register
    1.06kB  0.33% 45.79%    14.55kB  4.52%  encoding/gob.RegisterName
         0     0% 45.79%    14.55kB  4.52%  encoding/gob.registerBasics
    0.61kB  0.19% 45.97%    14.49kB  4.50%  archive/tar.init
         0     0% 45.97%    13.70kB  4.25%  fmt.init
    6.12kB  1.90% 47.88%    13.12kB  4.07%  runtime.allocm
         0     0% 47.88%    12.72kB  3.95%  regexp/syntax.Parse
   10.38kB  3.22% 51.10%    12.62kB  3.92%  sync.(*Map).LoadOrStore
         0     0% 51.10%    12.17kB  3.78%  reflect.init
   12.17kB  3.78% 54.88%    12.17kB  3.78%  unicode.init
         0     0% 54.88%    12.13kB  3.77%  encoding/gob.userType
    3.62kB  1.13% 56.00%    12.13kB  3.77%  encoding/gob.validUserType
         0     0% 56.00%    11.94kB  3.71%  runtime.mstart
         0     0% 56.00%    11.50kB  3.57%  regexp/syntax.(*parser).parseClass
         0     0% 56.00%    11.25kB  3.49%  runtime.newm
   10.92kB  3.39% 59.39%    10.92kB  3.39%  regexp/syntax.appendRange
         0     0% 59.39%    10.81kB  3.36%  vendor/golang_org/x/net/http2/hpack.init
    0.05kB 0.015% 59.41%    10.75kB  3.34%  vendor/golang_org/x/net/http2/hpack.newStaticTable
   10.61kB  3.29% 62.70%    10.61kB  3.29%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
         0     0% 62.70%    10.50kB  3.26%  regexp/syntax.(*parser).parseUnicodeClass
         0     0% 62.70%    10.50kB  3.26%  regexp/syntax.appendTable
    9.38kB  2.91% 65.61%     9.38kB  2.91%  runtime.malg
    9.17kB  2.85% 68.46%     9.22kB  2.86%  html/template.init
         0     0% 68.46%     9.05kB  2.81%  go/importer.init
         0     0% 68.46%     9.05kB  2.81%  go/internal/gccgoimporter.init
    1.66kB  0.51% 68.97%     8.84kB  2.74%  go/types.init
    0.05kB 0.015% 68.99%     8.80kB  2.73%  mime/multipart.init
    0.56kB  0.17% 69.16%     7.81kB  2.43%  runtime.systemstack
         0     0% 69.16%     7.69kB  2.39%  html/template.(*Template).Parse

Contributor guide