x/text: support API for Unicode word breaking and word extraction (Annex #29)

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ioe/sources/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/user/1000/go-build353744209=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Trying to split text at word boundaries like mentioned at http://unicode.org/reports/tr29/#Word_Boundaries and also trying to extract words from strings as mentioned in the same document.

What did you expect to see?

Given the sentence "The quick (“brown”) fox can’t jump 32.3 feet, right?"

• detecting word boundaries at all places marked with "|"

The| |quick| |(|“|brown|”|)| |fox| |can’t| |jump| |32.3| |feet|,| |right|?

• support word extraction to a string slice with the following content

words := []string{"The", "quick", "brown", "fox", "can’t", "jump", "32.3", "feet", "right"}

What did you see instead?

That depends on the API used. example with strings.Fields at https://play.golang.org/p/dhJtlR-b3w displays:

[]string{"The", "quick", "(“brown”)", "fox", "can’t", "jump", "32.3", "feet,", "right?"}

Note: Proper test vectors are here: Test vectors are here: http://www.unicode.org/Public/UCD/latest/ucd/auxiliary/WordBreakTest.txt

An implementation using Ruby magic and a state machine generated by Ragel can be found here: github.com/blevesearch/segment