Michael Munday [Tue, 14 Apr 2020 14:46:26 +0000 (15:46 +0100)]
[release-branch.go1.13] cmd/compile: fix deallocation of live value copies in regalloc
When deallocating the input register to a phi so that the phi
itself could be allocated to that register the code was also
deallocating all copies of that phi input value. Those copies
of the value could still be live and if they were the register
allocator could reuse them incorrectly to hold speculative
copies of other phi inputs. This causes strange bugs.
No test because this is a very obscure scenario that is hard
to replicate but CL 228060 adds an assertion to the compiler
that does trigger when running the std tests on linux/s390x
without this CL applied. Hopefully that assertion will prevent
future regressions.
Fixes #38442.
Change-Id: Id975dadedd731c7bb21933b9ea6b17daaa5c9e1d
Reviewed-on: https://go-review.googlesource.com/c/go/+/228061
Run-TryBot: Michael Munday <mike.munday@ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
(cherry picked from commit 382fe3e2498f2066400e7e7007aa9903440e339d)
Reviewed-on: https://go-review.googlesource.com/c/go/+/230358
Austin Clements [Wed, 26 Feb 2020 20:12:33 +0000 (15:12 -0500)]
[release-branch.go1.13] runtime: fix rounding in materializeGCProg
materializeGCProg allocates a temporary buffer for unrolling a GC
program. Unfortunately, when computing the size of the buffer, it
rounds *down* the number of bytes needed to store bitmap before
rounding up the number of pages needed to store those bytes. The fact
that it rounds up to pages usually mitigates the rounding down, but
the type from #37470 exists right on the boundary where this doesn't
work:
On 64-bit, this GC bitmap is exactly 8 KiB of zeros, followed by three
one bits. Hence, this needs 8193 bytes of storage, but the current
math in materializeGCProg rounds *down* the three one bits to 8192
bytes. Since this is exactly pageSize, the next step of rounding up to
the page size doesn't mitigate this error, and materializeGCProg
allocates a buffer that is one byte too small. runGCProg then writes
one byte past the end of this buffer, causing either a segfault (if
you're lucky!) or memory corruption.
Jay Conrod [Fri, 28 Feb 2020 21:31:19 +0000 (16:31 -0500)]
[release-branch.go1.13] cmd/go: make module zip extraction more robust
Currently, we extract module zip files to temporary directories, then
atomically rename them into place. On Windows, this can fail with
ERROR_ACCESS_DENIED if another process (antivirus) has files open
before the rename. In CL 220978, we repeated the rename operation in a
loop over 500 ms, but this didn't solve the problem for everyone.
A better solution will extract module zip files to their permanent
locations in the cache and will keep a ".partial" marker file,
indicating when a module hasn't been fully extracted (CL 221157).
This approach is not safe if current versions of Go access the module
cache concurrently, since the module directory is detected with a
single os.Stat.
In the interim, this CL makes two changes:
1. Flaky file system operations are repeated over 2000 ms to reduce
the chance of this error occurring.
2. cmd/go will now check for .partial files created by future
versions. If a .partial file is found, it will lock the lock file,
then remove the .partial file and directory if needed.
After some time has passed and Go versions lacking this CL are no
longer supported, we can start extracting module zip files in place.
Updates #37802
Change-Id: I467ee11aa59a90b63cf0e3e761c4fec89d57d3b6
Reviewed-on: https://go-review.googlesource.com/c/go/+/221820
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
(cherry picked from commit 093049b3709eda7537ece92a2991918cf53782d6)
Reviewed-on: https://go-review.googlesource.com/c/go/+/223146
Carlos Eduardo Seo [Fri, 17 Jan 2020 20:59:59 +0000 (17:59 -0300)]
[release-branch.go1.13] runtime: fix wrong offset when calling ppc64x nanotime syscall
There is a wrong offset when getting the results of a clock_gettime
syscall. Although the syscall will never be called in native ppc64x,
QEMU doesn't implement VDSO, so it will return wrong values.
For #36592
Fixes #38236
Change-Id: Icf838075228dcdd62cf2c1279aa983e5993d66ee
Reviewed-on: https://go-review.googlesource.com/c/go/+/215397 Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
(cherry picked from commit 71239b4f491698397149868c88d2c851de2cd49b)
Reviewed-on: https://go-review.googlesource.com/c/go/+/227179 Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Dmitri Shuralyov [Tue, 17 Mar 2020 21:50:25 +0000 (17:50 -0400)]
[release-branch.go1.13] cmd/go: fix and skip known Windows test failures
These non-short Windows test failures were resolved fully in CL 206144.
Both TestScript/build_trimpath and TestScript/version tests can be fixed
by backporting the changes to test scripts only, so that is done here.
Fixing TestScript/mod_list_dir requires backporting non-test changes in
addition to the test script changes, which is unlikely to be appropriate
this late in Go 1.13 release cycle. A failing test can cover up other
regressions, so skip this known failing test to fix the builder.
For #36181.
Change-Id: I4f140bd373554eb4664f04638666dee77986ec3e
Reviewed-on: https://go-review.googlesource.com/c/go/+/223782
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Dmitri Shuralyov [Tue, 17 Mar 2020 14:22:41 +0000 (10:22 -0400)]
[release-branch.go1.13] cmd/doc: skip failing TestDotSlashLookup on Windows
This test was fixed by changing cmd/doc behavior in CL 204442.
Backporting that non-test code change is unlikely to be appropriate
this late in Go 1.13 release cycle. A failing test can cover up other
regressions, so skip this known failing test to fix the builder.
For #35236.
For #36181.
Change-Id: I07e795e75d7e37bc96ab68607d5d5cc9254342f8
Reviewed-on: https://go-review.googlesource.com/c/go/+/223780
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
[release-branch.go1.13] os/exec: use environment variables for user token when present
Builds upon the changes from #32000 which supported sourcing environment
variables for a new process from the environment of a Windows user token
when supplied.
But due to the logic of os/exec, the Env field of a process was
always non-nil when it reached that change.
This change moves the logic up to os/exec, specifically when
os.ProcAttr is being built for the os.StartProcess call, this
ensures that if a user token has been supplied and no Env slice has
been provided on the command it will be sourced from the user's
environment.
If no token is provided, or the program is compiled for any other
platform than Windows, the default environment will be sourced from
syscall.Environ().
For #35314
Fixes #37433
Change-Id: I4c1722e90b91945eb6980d5c5928183269b50487
GitHub-Last-Rev: 32216b7291418f9285147a93ed6d0ba028f94ef2
GitHub-Pull-Request: golang/go#37402
Reviewed-on: https://go-review.googlesource.com/c/go/+/220587
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/226280
Bryan C. Mills [Thu, 26 Mar 2020 02:24:44 +0000 (22:24 -0400)]
[release-branch.go1.13] cmd/go: do not append to the global cfg.OrigEnv slice
Appending to a global slice is only safe if its length is already
equal to its capacity. That property is not guaranteed for slices in
general, and empirically does not hold for this one.
This is a minimal fix to make it easier to backport.
A more robust cleanup of the base.EnvForDir function will be sent in a
subsequent CL.
Fixes #38082
Updates #38077
Change-Id: I731d5bbd0e516642c2cf43e713eeea15402604e5
Reviewed-on: https://go-review.googlesource.com/c/go/+/225577
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
(cherry picked from commit bfb1342a40216cba0ff5ae3a1b102823b7603068)
Reviewed-on: https://go-review.googlesource.com/c/go/+/225660
Alex Brainman [Sun, 16 Feb 2020 01:01:02 +0000 (12:01 +1100)]
[release-branch.go1.13] runtime: ignore error returned by PowerRegisterSuspendResumeNotification
It appears that PowerRegisterSuspendResumeNotification is not supported
when running inside Docker - see issues #35447, #36557 and #37149.
Our current code relies on error number to determine Docker environment.
But we already saw PowerRegisterSuspendResumeNotification return
ERROR_FILE_NOT_FOUND, ERROR_INVALID_PARAMETERS and ERROR_ACCESS_DENIED
(see issues above). So this approach is not sustainable.
Just ignore PowerRegisterSuspendResumeNotification returned error.
For #37149
Fixes #37230
Change-Id: I2beba9d45cdb8c1efac5e974e747827a6261915a
Reviewed-on: https://go-review.googlesource.com/c/go/+/219657
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
(cherry picked from commit d467f3bbc9c76805ae16ab1924c28ec3be487875)
Reviewed-on: https://go-review.googlesource.com/c/go/+/224585
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Jason A. Donenfeld [Sat, 26 Oct 2019 21:05:22 +0000 (23:05 +0200)]
[release-branch.go1.13] internal/syscall/windows/registry: remove TestWalkFullRegistry due to false assumptions
This test's existence was predicated upon assumptions about the full
range of known data types and known data into those types. However,
we've learned from Microsoft that there are several undocumented secret
registry types that are in use by various parts of Windows, and we've
learned from inspection that many Microsoft uses of registry types don't
strictly adhere to the recommended value size. It's therefore foolhardy
to make any assumptions about what goes in and out of the registry, and
so this test, as well as its "blacklist", are meaningless.
For #35084.
Fixes #37826.
Change-Id: I6c3fe5fb0e740e88858321b3b042c0ff1a23284e
Reviewed-on: https://go-review.googlesource.com/c/go/+/203604
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
(cherry picked from commit 0d3092ffa7e7f613429ddcfd596d26ccbc84766f)
Reviewed-on: https://go-review.googlesource.com/c/go/+/223237
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Hana (Hyang-Ah) Kim [Wed, 19 Feb 2020 03:41:20 +0000 (22:41 -0500)]
[release-branch.go1.13] cmd/trace: update to use WebComponents V0 polyfill
Old trace viewer stopped working with Chrome M80+ because the
old trace viewer heavily depended on WebComponents V0 which are deprecated.
Trace viewer recently migrated to use WebComponents V0 polyfill
(crbug.com/1036492). This CL brings in the newly updated trace_viewer_full.html
(sync'd @ 9508452e)
and updates the javascript snippet included in the /trace endpoint
to use the polyfill.
This brings in webcomponents.min.js copied from
https://chromium.googlesource.com/catapult/+/9508452e18f130c98499cb4c4f1e1efaedee8962/third_party/polymer/components/webcomponentsjs/webcomponents.min.js
That is necessary because the /trace endpoint needs to import
the vulcanized trace_viewer_full.html.
It's possible that some features are not working correctly with
this polyfill. In that case, report the issue to crbug.com/1036492.
There will be a warning message in the UI (yellow banner above the timeline)
which can be hidden by clicking the 'hide' button.
This allows to render the trace in browsers other than chrome in theory,
but I observed some buttons and functions still don't work outside
chrome.
Updates #34374.
Fixes #37342.
Change-Id: Ib575f756f5e6b22ad904ede6e4d224a995ebe259
Reviewed-on: https://go-review.googlesource.com/c/go/+/219997
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
(cherry picked from commit 75ea964b3f6073076e1a86a0de2be9a2f159da24)
Reviewed-on: https://go-review.googlesource.com/c/go/+/220321
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Jay Conrod [Fri, 24 Jan 2020 16:25:52 +0000 (08:25 -0800)]
[release-branch.go1.13] cmd/go: fix cgo test when min macOS version is set
Regression tests for #24161 use a macro to conditionally compile some
stub definitions. The macro tests that the minimum macOS version is
less than 10.12.
We get duplicate definitions when building this test with
CGO_CFLAGS=-mmacosx-version-min=10.x where 10.x < 10.12. With this
change, we use a different macro, __MAC_OS_X_VERSION_MAX_ALLOWED__,
which tests the SDK version instead of the minimum macOS version. This
checks whether these definitions are present in headers.
After this change, 'go tool dist test cgo_test' should pass with
CGO_FLAGS=-mmacosx-version-min=10.10.
Updates #36846
Updates #35459
Change-Id: I88d63601c94b0369c73c38d216a2d41ba7d4e579
Reviewed-on: https://go-review.googlesource.com/c/go/+/216243
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
(cherry picked from commit 1f9f88b95eaec50c72c8595ca9f52b7b876e28f9)
Reviewed-on: https://go-review.googlesource.com/c/go/+/217059
Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Jay Conrod [Tue, 21 Jan 2020 22:50:36 +0000 (14:50 -0800)]
[release-branch.go1.13] cmd/go: add -d flag to mod_get_test
'go get all' was run in this test without -d. This caused some std
packages to be reinstalled if the test is run in a slightly different
configuration than make.bash was run. run.bash would fail in some
situations because of this. Nothing in the cmd/go tests should modify
installed std or cmd packages.
Updates #36846
Updates #35459
Change-Id: Idd259a27d55502923b7fc54f361a77f0ac11eea2
Reviewed-on: https://go-review.googlesource.com/c/go/+/215721
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org>
(cherry picked from commit 71bbffbc48d03b447c73da1f54ac57350fc9b36a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/218597
Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Carlos Amedee [Thu, 23 Jan 2020 21:05:29 +0000 (16:05 -0500)]
[release-branch.go1.13] cmd/link: ensure cgo cflags do not leak into dwarf tests
Running the dwarf tests with CGO_CFLAGS set
with certain values would cause the test to fail. all.bash
would fail when CGO_CFLAGS was set to '-mmacosx-version-min=10.10'
because the --macosx-version-min flag is incompatible with some dwarf
tests. The change guards against using an unintended flag in the unit test.
Updates #36846
Updates #35459
Change-Id: Idc9b354aba44fdab424cb0081a4b3ea7a6d0f8e3
Reviewed-on: https://go-review.googlesource.com/c/go/+/216177
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit e948d2b73ede67f12bff9e4d050f0e1425163010)
Reviewed-on: https://go-review.googlesource.com/c/go/+/217057
Carlos Amedee [Wed, 22 Jan 2020 20:30:52 +0000 (15:30 -0500)]
[release-branch.go1.13] cmd/link: ensure cgo cflags do not leak into tvOS test
Running the 'TestBuildForTvOS' test with CGO_CFLAGS set
with certain values would cause the test to fail. all.bash
would fail when CGO_CFLAGS was set to '-mmacosx-version-min=10.10'
because the --macosx-version-min flag is incompatible with tvOS.
The change guards against using an unintended flag in the unit test.
Updates #36846
Updated #35459
Change-Id: Ifc43f3ebfb23d37aabeaac2ea9efae5b877991bf
Reviewed-on: https://go-review.googlesource.com/c/go/+/215957
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit ace25f82df0a27eb26a518e1883eb56c1bec6c5e)
Reviewed-on: https://go-review.googlesource.com/c/go/+/218598
Filippo Valsorda [Fri, 24 Jan 2020 23:04:20 +0000 (18:04 -0500)]
[release-branch.go1.13-security] src/go.mod: import x/crypto/cryptobyte security fix for 32-bit archs
cryptobyte: fix panic due to malformed ASN.1 inputs on 32-bit archs
When int is 32 bits wide (on 32-bit architectures like 386 and arm), an
overflow could occur, causing a panic, due to malformed ASN.1 being
passed to any of the ASN1 methods of String.
Tested on linux/386 and darwin/amd64.
This fixes CVE-2020-7919 and was found thanks to the Project Wycheproof
test vectors.
Change-Id: I8c9696a8bfad1b40ec877cd740dba3467d66ab54
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/645211 Reviewed-by: Katie Hockman <katiehockman@google.com> Reviewed-by: Adam Langley <agl@google.com>
x/crypto/cryptobyte is used in crypto/x509 for parsing certificates.
Malformed certificates might cause a panic during parsing on 32-bit
architectures (like arm and 386).
Filippo Valsorda [Tue, 21 Jan 2020 19:45:15 +0000 (14:45 -0500)]
[release-branch.go1.13-security] crypto/x509: mitigate CVE-2020-0601 verification bypass on Windows
An attacker can trick the Windows system verifier to use a poisoned set
of elliptic curve parameters for a trusted root, allowing it to generate
spoofed signatures. When this happens, the returned chain will present
the unmodified original root, so the actual signatures won't verify (as
they are invalid for the correct parameters). Simply double check them
as a safety measure and mitigation.
Windows users should still install the system security patch ASAP.
Cherry Zhang [Fri, 27 Dec 2019 17:02:00 +0000 (12:02 -0500)]
[release-branch.go1.13] runtime: ensure memmove write pointer atomically on ARM64
If a pointer write is not atomic, if the GC is running
concurrently, it may observe a partially updated pointer, which
may point to unallocated or already dead memory. Most pointer
writes, like the store instructions generated by the compiler,
are already atomic. But we still need to be careful in places
like memmove. In memmove, we don't know which bits are pointers
(or too expensive to query), so we ensure that all aligned
pointer-sized units are written atomically.
Bryan C. Mills [Tue, 7 Jan 2020 17:03:28 +0000 (12:03 -0500)]
[release-branch.go1.13] net/http: avoid writing to Transport.ProxyConnectHeader
Previously, we accidentally wrote the Proxy-Authorization header for
the initial CONNECT request to the shared ProxyConnectHeader map when
it was non-nil.
Jason A. Donenfeld [Thu, 21 Nov 2019 15:16:56 +0000 (16:16 +0100)]
[release-branch.go1.13] runtime: do not use PowerRegisterSuspendResumeNotification on systems with "program time" timer
Systems where PowerRegisterSuspendResumeNotification returns ERROR_
FILE_NOT_FOUND are also systems where nanotime() is on "program time"
rather than "real time". The chain for this is:
Opening \\.\PdcPort fails with STATUS_OBJECT_NAME_NOT_FOUND when pdc.sys
hasn't been initialized. Pdc.sys also provides the various hooks for
sleep resumption events, which means if it's not loaded, then our "real
time" timer is actually on "program time". Finally STATUS_OBJECT_NAME_
NOT_FOUND is passed through RtlNtStatusToDosError, which returns ERROR_
FILE_NOT_FOUND. Therefore, in the case where the function returns ERROR_
FILE_NOT_FOUND, we don't mind, since the timer we're using will
correspond fine with the lack of sleep resumption notifications. This
applies, for example, to Docker users.
Updates #35447
Updates #35482
Fixes #35746
Change-Id: I9e1ce5bbc54b9da55ff7a3918b5da28112647eee
Reviewed-on: https://go-review.googlesource.com/c/go/+/211280
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Michael Anthony Knyszek [Mon, 14 Oct 2019 17:33:36 +0000 (17:33 +0000)]
[release-branch.go1.13] runtime: call goready in wakeScavenger instead of ready
This changes fixes an oversight in wakeScavenger which would cause ready
to be called off of the system stack. This change makes it so that
wakeScavenger calls goready, which switches to the system stack before
calling ready.
Fixes #36127.
Change-Id: Icb13f180b4d8fdd47c921eac1b896e3dd49e43b3
Reviewed-on: https://go-review.googlesource.com/c/go/+/200999
Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Keith Randall <khr@golang.org>
(cherry picked from commit 2c87be436bddd9b49f11959adee1ae817cb48ee1)
Reviewed-on: https://go-review.googlesource.com/c/go/+/212103
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
Bryan C. Mills [Tue, 5 Nov 2019 16:01:10 +0000 (11:01 -0500)]
[release-branch.go1.13] cmd/go/internal/modfetch: remove non-hermetic test
The test for gopkg.in/yaml.v2@v2 assumes that there are
no future upstream releases. That assumption empirically
does not hold. Backporting fixes to this test is annoying,
and other gopkg.in cases are already reasonably covered,
so remove the problematic test.
Updates #28856
Change-Id: I6455baa1816ac69e02d1ad5d03b82a93e1481a17
Reviewed-on: https://go-review.googlesource.com/c/go/+/205437
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
(cherry picked from commit f0390ffc9d461cb84207b5a94c4b645c87673406)
Reviewed-on: https://go-review.googlesource.com/c/go/+/205438 Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Dmitri Shuralyov [Thu, 5 Dec 2019 14:13:21 +0000 (17:13 +0300)]
[release-branch.go1.13] doc: add CherryPickApproved filter to Release History links
Not all closed issues in a given minor milestone are included in that
release, only the ones that have been labeled as CherryPickApproved are.
Update the links to the GitHub issue tracker to include a filter on the
CherryPickApproved label, so that the default view shows only the
backports that were included in a given release. This should more useful
to most people than seeing all backports (considered and approved).
Do this only for Go 1.9.1 and newer releases, as that is when we started
using the CherryPickCandidate and CherryPickApproved labels.
Updates #35988
Fixes #36003
Change-Id: I51e07c1bc3ab9c4a5744e8f668c5470adf78bffe
Reviewed-on: https://go-review.googlesource.com/c/go/+/210117
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Than McIntosh [Thu, 10 Oct 2019 16:11:06 +0000 (12:11 -0400)]
[release-branch.go1.13] cmd/compile: fix spurious R_TLE_LE reloc on android/386
When compiling for GOARCH=386 GOOS=android, the compiler was attaching
R_TLS_LE relocations inappropriately -- as of Go 1.13 the TLS access
recipe for Android refers to a runtime symbol and no longer needs this
type of relocation (which was causing a crash when the linker tried to
process it).
Lynn Boger [Mon, 28 Oct 2019 13:29:40 +0000 (09:29 -0400)]
[release-branch.go1.13] runtime: fix textOff for multiple text sections
If a compilation has multiple text sections, code in
textOff must compare the offset argument against the range
for each text section to determine which one it is in.
The comparison looks like this:
if uintptr(off) >= sectaddr && uintptr(off) <= sectaddr+sectlen
If the off value being compared is equal to sectaddr+sectlen then it
is not within the range of the text section but after it. The
comparison should be just '<'.
Fixes #35211
Change-Id: I114633fd734563d38f4e842dd884c6c239f73c95
Reviewed-on: https://go-review.googlesource.com/c/go/+/203817
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
(cherry picked from commit 0ae9389609f23dc905c58fc2ad7bcc16b770f337)
Reviewed-on: https://go-review.googlesource.com/c/go/+/203819
Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
Andrew [Wed, 20 Nov 2019 17:06:51 +0000 (12:06 -0500)]
[release-branch.go1.13] all: base64-encode binaries that will cause Apple notarization to fail
Starting with macOS 10.15 (Catalina), Apple now requires all software
distributed outside of the App Store to be notarized. Any binaries we
distribute must abide by a strict set of requirements like code-signing
and having a minimum target SDK of 10.9 (amongst others).
Apple’s notarization service will recursively inspect archives looking to
find notarization candidate binaries. If it finds a binary that does not
meet the requirements or is unable to decompress an archive, it will
reject the entire distribution. From cursory testing, it seems that the
service uses content sniffing to determine file types, so changing
the file extension will not work.
There are some binaries and archives included in our distribution that
are being detected by Apple’s service as potential candidates for
notarization or decompression. As these are files used by tests and some
are intentionally invalid, we don’t intend to ever make them compliant.
As a workaround for this, we base64-encode any binaries or archives that
Apple’s notarization service issues a warning for, as these warnings will
become errors in January 2020.
Updates #34986
Fixes #35748
Change-Id: I106fbb6227b61eb221755568f047ee11103c1680
Reviewed-on: https://go-review.googlesource.com/c/go/+/208118
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
(cherry picked from commit 8bbfc51d9ac9ce9472e126cc3654c9a45eceb236)
Reviewed-on: https://go-review.googlesource.com/c/go/+/208219 Reviewed-by: Alexander Rakoczy <alex@golang.org>
Jason A. Donenfeld [Wed, 23 Oct 2019 11:08:46 +0000 (13:08 +0200)]
[release-branch.go1.13] syscall: reenable sysctl on iOS
This was disabled due to a report that the App Store rejects the symbol
__sysctl. However, we use the sysctl symbol, which is fine. The __sysctl
symbol is used by x/sys/unix, which needs fixing instead. So, this
commit reenables sysctl on iOS, so that things like net.InterfaceByName
can work again.
This reverts CL 193843, CL 193844, CL 193845, and CL 193846.
Agniva De Sarker [Sun, 13 Oct 2019 15:03:47 +0000 (20:33 +0530)]
[release-branch.go1.13] cmd/go/internal/work: fix error while passing custom vet tool
For GOROOT packages, we were adding -unsafeptr=false to prevent unsafe.Pointer
checks. But the flag also got passed to invocations of go vet with a custom
vet tool. To prevent this from happening, we add this flag only when no
tools are passed.
Updates #34053
Fixes #34922
Change-Id: I8bcd637fd8ec423d597fcdab2a0ceedd20786019
Reviewed-on: https://go-review.googlesource.com/c/go/+/200957
Run-TryBot: Agniva De Sarker <agniva.quicksilver@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
(cherry picked from commit 902d5aa84f8340752c20b93bfd450a6cefcf3952)
Reviewed-on: https://go-review.googlesource.com/c/go/+/201237
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Agniva De Sarker <agniva.quicksilver@gmail.com>
zdjones [Fri, 11 Oct 2019 15:04:47 +0000 (16:04 +0100)]
[release-branch.go1.13-security] cmd/compile: make poset use sufficient conditions for OrderedOrEqual
When assessing whether A <= B, the poset's OrderedOrEqual has a passing
condition which permits A <= B, but is not sufficient to infer that A <= B.
This CL removes that incorrect passing condition.
Having identified that A and B are in the poset, the method will report that
A <= B if any of these three conditions are true:
(1) A and B are the same node in the poset.
- This means we know that A == B.
(2) There is a directed path, strict or not, from A -> B
- This means we know that, at least, A <= B, but A < B is possible.
(3) There is a directed path from B -> A, AND that path has no strict edges.
- This means we know that B <= A, but do not know that B < A.
In condition (3), we do not have enough information to say that A <= B, rather
we only know that B == A (which satisfies A <= B) is possible. The way I
understand it, a strict edge shows a known, strictly-ordered relation (<) but
the lack of a strict edge does not show the lack of a strictly-ordered relation.
The difference is highlighted by the example in #34802, where a bounds check is
incorrectly removed by prove, such that negative indexes into a slice
succeed:
n := make([]int, 1)
for i := -1; i <= 0; i++ {
fmt.Printf("i is %d\n", i)
n[i] = 1 // No Bounds check, program runs, assignment to n[-1] succeeds!!
}
When prove is checking the negative/failed branch from the bounds check at n[i],
in the signed domain we learn (0 > i || i >= len(n)). Because prove can't learn
the OR condition, we check whether we know that i is non-negative so we can
learn something, namely that i >= len(n). Prove uses the poset to check whether
we know that i is non-negative. At this point the poset holds the following
relations as a directed graph:
-1 <= i <= 0
-1 < 0
In poset.OrderedOrEqual, we are testing for 0 <= i. In this case, condition (3)
above is true because there is a non-strict path from i -> 0, and that path
does NOT have any strict edges. Because this condition is true, the poset
reports to prove that i is known to be >= 0. Knowing, incorrectly, that i >= 0,
prove learns from the failed bounds check that i >= len(n) in the signed domain.
When the slice, n, was created, prove learned that len(n) == 1. Because i is
also the induction variable for the loop, upon entering the loop, prove previously
learned that i is in [-1,0]. So when prove attempts to learn from the failed
bounds check, it finds the new fact, i > len(n), unsatisfiable given that it
previously learned that i <= 0 and len(n) = 1.
Fixes #34807
Change-Id: I235f4224bef97700c3aa5c01edcc595eb9f13afc
Reviewed-on: https://go-review.googlesource.com/c/go/+/200759
Run-TryBot: Zach Jones <zachj1@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Giovanni Bajo <rasky@develer.com> Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/201060
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/575398 Reviewed-by: Filippo Valsorda <valsorda@google.com>
zdjones [Fri, 30 Aug 2019 13:41:09 +0000 (14:41 +0100)]
[release-branch.go1.13-security] cmd/compile: rename poset method dominates to reaches
The partially ordered set uses a method named 'dominates' to determine whether
two nodes are partially ordered. Dominates does a depth-first search of the
DAG, beginning at the source node, and returns true as soon as it finds a path
to the target node. In the context of the forest-of-DAGs that makes up the
poset, dominates is not necessarily checking dominance, but is checking
reachability. See the issue tracker for a more detailed discussion of the
difference.
Fortunately, reachability is logically correct everywhere dominates is currently
used in poset.go. Reachability within a DAG is sufficient to establish the
partial ordering (source < target).
This CL changes the name of the method (dominates -> reaches) and updates
all the comments in the file accordingly.
Updates #34807
Change-Id: Ia3a34f7b14b363801d75b05099cfc686035f7d96
Reviewed-on: https://go-review.googlesource.com/c/go/+/192617 Reviewed-by: Giovanni Bajo <rasky@develer.com>
Run-TryBot: Giovanni Bajo <rasky@develer.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/201059
Run-TryBot: Alexander Rakoczy <alex@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/575397 Reviewed-by: Filippo Valsorda <valsorda@google.com>
Katie Hockman [Mon, 14 Oct 2019 20:42:21 +0000 (16:42 -0400)]
[release-branch.go1.13-security] crypto/dsa: prevent bad public keys from causing panic
dsa.Verify might currently use a nil s inverse in a
multiplication if the public key contains a non-prime Q,
causing a panic. Change this to check that the mod
inverse exists before using it.
This is a revert of CL 174437 and equivalent to CL 201360.
The size of the params block passed into the KDSA instruction is
incorrect and this appears to result in out-of-bounds writes
that cause a panic in the crypto/x509 tests when run on a machine
that supports KDSA.
Remove this assembly for now. We can revisit the use of the KDSA
instruction in a future release.
zdjones [Fri, 11 Oct 2019 15:04:47 +0000 (16:04 +0100)]
[release-branch.go1.13] cmd/compile: make poset use sufficient conditions for OrderedOrEqual
When assessing whether A <= B, the poset's OrderedOrEqual has a passing
condition which permits A <= B, but is not sufficient to infer that A <= B.
This CL removes that incorrect passing condition.
Having identified that A and B are in the poset, the method will report that
A <= B if any of these three conditions are true:
(1) A and B are the same node in the poset.
- This means we know that A == B.
(2) There is a directed path, strict or not, from A -> B
- This means we know that, at least, A <= B, but A < B is possible.
(3) There is a directed path from B -> A, AND that path has no strict edges.
- This means we know that B <= A, but do not know that B < A.
In condition (3), we do not have enough information to say that A <= B, rather
we only know that B == A (which satisfies A <= B) is possible. The way I
understand it, a strict edge shows a known, strictly-ordered relation (<) but
the lack of a strict edge does not show the lack of a strictly-ordered relation.
The difference is highlighted by the example in #34802, where a bounds check is
incorrectly removed by prove, such that negative indexes into a slice
succeed:
n := make([]int, 1)
for i := -1; i <= 0; i++ {
fmt.Printf("i is %d\n", i)
n[i] = 1 // No Bounds check, program runs, assignment to n[-1] succeeds!!
}
When prove is checking the negative/failed branch from the bounds check at n[i],
in the signed domain we learn (0 > i || i >= len(n)). Because prove can't learn
the OR condition, we check whether we know that i is non-negative so we can
learn something, namely that i >= len(n). Prove uses the poset to check whether
we know that i is non-negative. At this point the poset holds the following
relations as a directed graph:
-1 <= i <= 0
-1 < 0
In poset.OrderedOrEqual, we are testing for 0 <= i. In this case, condition (3)
above is true because there is a non-strict path from i -> 0, and that path
does NOT have any strict edges. Because this condition is true, the poset
reports to prove that i is known to be >= 0. Knowing, incorrectly, that i >= 0,
prove learns from the failed bounds check that i >= len(n) in the signed domain.
When the slice, n, was created, prove learned that len(n) == 1. Because i is
also the induction variable for the loop, upon entering the loop, prove previously
learned that i is in [-1,0]. So when prove attempts to learn from the failed
bounds check, it finds the new fact, i > len(n), unsatisfiable given that it
previously learned that i <= 0 and len(n) = 1.
Fixes #34807
Change-Id: I235f4224bef97700c3aa5c01edcc595eb9f13afc
Reviewed-on: https://go-review.googlesource.com/c/go/+/200759
Run-TryBot: Zach Jones <zachj1@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Giovanni Bajo <rasky@develer.com> Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/201060
Run-TryBot: Alexander Rakoczy <alex@golang.org>
zdjones [Fri, 30 Aug 2019 13:41:09 +0000 (14:41 +0100)]
[release-branch.go1.13] cmd/compile: rename poset method dominates to reaches
The partially ordered set uses a method named 'dominates' to determine whether
two nodes are partially ordered. Dominates does a depth-first search of the
DAG, beginning at the source node, and returns true as soon as it finds a path
to the target node. In the context of the forest-of-DAGs that makes up the
poset, dominates is not necessarily checking dominance, but is checking
reachability. See the issue tracker for a more detailed discussion of the
difference.
Fortunately, reachability is logically correct everywhere dominates is currently
used in poset.go. Reachability within a DAG is sufficient to establish the
partial ordering (source < target).
This CL changes the name of the method (dominates -> reaches) and updates
all the comments in the file accordingly.
Updates #34807
Change-Id: Ia3a34f7b14b363801d75b05099cfc686035f7d96
Reviewed-on: https://go-review.googlesource.com/c/go/+/192617 Reviewed-by: Giovanni Bajo <rasky@develer.com>
Run-TryBot: Giovanni Bajo <rasky@develer.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/201059
Run-TryBot: Alexander Rakoczy <alex@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
Emmanuel T Odeke [Sun, 13 Oct 2019 22:07:06 +0000 (15:07 -0700)]
[release-branch.go1.13] net/http: fix Transport panic with nil Request.Header
For Go 1.13 we introduced Header.Clone and it returns
nil if a nil Header is cloned. Unfortunately, though,
this exported Header.Clone nil behavior differed from
the old Go 1.12 and earlier internal header clone
behavior which always returned non-nil Headers.
This CL fixes the places where that distinction mattered.
Shenghou Ma [Mon, 30 Sep 2019 13:44:37 +0000 (09:44 -0400)]
[release-branch.go1.13] runtime: fix darwin syscall performance regression
While understanding why syscall.Read is 2x slower on darwin/amd64, I found
out that, contrary to popular belief, the slowdown is not due to the migration
to use libSystem.dylib instead of direct SYSCALLs, i.e., CL 141639 (and #17490),
but due to a subtle change introduced in CL 141639.
Previously, syscall.Read used syscall.Syscall(SYS_READ), whose preamble called
runtime.entersyscall, but after CL 141639, syscall.Read changes to call
runtime.syscall_syscall instead, which in turn calls runtime.entersyscallblock
instead of runtime.entersyscall. And the entire 2x slow down can be attributed
to this change.
I think this is unnecessary as even though syscalls like Read might block, it
does not always block, so there is no need to handoff P proactively for each
Read. Additionally, we have been fine with not handing off P for each Read
prior to Go 1.12, so we probably don't need to change it. This changes restores
the pre-Go 1.12 behavior, where syscall preamble uses runtime.entersyscall,
and we rely on sysmon to take P back from g blocked in syscalls.
Updates #34712
Change-Id: If76e97b5a7040cf1c10380a567c4f5baec3121ba
Reviewed-on: https://go-review.googlesource.com/c/go/+/197938
Run-TryBot: Minux Ma <minux@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit c1635ad8f0bb9fbe5bfbf0a633c78a03930758c4)
Reviewed-on: https://go-review.googlesource.com/c/go/+/200103
Run-TryBot: Alexander Rakoczy <alex@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
Bryan C. Mills [Tue, 8 Oct 2019 18:23:43 +0000 (14:23 -0400)]
[release-branch.go1.13] cmd/go: suppress more errors in package-to-module loading
In CL 197059, I suppressed errors if the target package was already found.
However, that does not cover the case of passing a '/v2' module path to
'go get' when the module does not contain a package at its root.
This CL is a minimal fix for that case, intended to be backportable to 1.13.
(Longer term, I intend to rework the version-validation check to treat
all mismatched paths as ErrNotExist.)
Updates #34746
Fixes #34747
Change-Id: Ia963c2ea00fae424812b8f46a4d6c2c668252147
Reviewed-on: https://go-review.googlesource.com/c/go/+/199839
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
(cherry picked from commit 421d35cf69f4a18edf96004ba99c01e629a0f79f)
Reviewed-on: https://go-review.googlesource.com/c/go/+/199997
Duco van Amstel [Fri, 4 Oct 2019 13:05:13 +0000 (13:05 +0000)]
[release-branch.go1.13] cmd/go: fix listing of ambiguous paths
Passing ambiguous patterns, ending in `.go`, to `go list` results in them
being interpreted as Go files despite potentially being package references.
This can then result in errors on other package references.
The parsing logic is modified to check for a locally present file
corresponding to any pattern ending in `.go`. If no such file is present
the pattern is considered to be a package reference.
We're also adding a variety of non-regression tests that fail with the
original parsing code but passes after applying the fix.
Updates #34653
Fixes #34694
Change-Id: I073871da0dfc5641a359643f95ac14608fdca09b
GitHub-Last-Rev: 5abc200103ffc122df05422d79cf30c3ba0ee646
GitHub-Pull-Request: golang/go#34663
Reviewed-on: https://go-review.googlesource.com/c/go/+/198459
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
(cherry picked from commit 33683f1d64df0cef2c598a84b741abb5af8abe5e)
Reviewed-on: https://go-review.googlesource.com/c/go/+/198957 Reviewed-by: Jay Conrod <jayconrod@google.com>
Jason A. Donenfeld [Fri, 4 Oct 2019 14:48:17 +0000 (16:48 +0200)]
[release-branch.go1.13] syscall: replace mksyscall_windows.go with wrapper to new x/sys home
We replace the existing file with a thin wrapper around its target so
that we don't break anybody's workflow.
This combines CL 198977 and CL 199277.
Fixes #34388
Change-Id: I0d00371c483cb78f4be18fe987df33c79cd40f05
Reviewed-on: https://go-review.googlesource.com/c/go/+/199538
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Michael Anthony Knyszek [Tue, 25 Jun 2019 19:06:57 +0000 (19:06 +0000)]
[release-branch.go1.13] runtime: scavenge on growth instead of inline with allocation
Inline scavenging causes significant performance regressions in tail
latency for k8s and has relatively little benefit for RSS footprint.
We disabled inline scavenging in Go 1.12.5 (CL 174102) as well, but
we thought other changes in Go 1.13 had mitigated the issues with
inline scavenging. Apparently we were wrong.
This CL switches back to only doing foreground scavenging on heap
growth, rather than doing it when allocation tries to allocate from
scavenged space.
Fixes #34556
Change-Id: I1f5df44046091f0b4f89fec73c2cde98bf9448cb
Reviewed-on: https://go-review.googlesource.com/c/go/+/183857
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
(cherry picked from commit eb96f8a57444d174bba500b3a5d2a8b21b7e6d1e)
Reviewed-on: https://go-review.googlesource.com/c/go/+/198486 Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Andrew Bonventre <andybons@golang.org>
Michael Anthony Knyszek [Tue, 3 Sep 2019 19:54:32 +0000 (19:54 +0000)]
[release-branch.go1.13] runtime: redefine scavenge goal in terms of heap_inuse
This change makes it so that the scavenge goal is defined primarily in
terms of heap_inuse at the end of the last GC rather than next_gc. The
reason behind this change is that next_gc doesn't take into account
fragmentation, and we can fall into situation where the scavenger thinks
it should have work to do but there's no free and unscavenged memory
available.
In order to ensure the scavenge goal still tracks next_gc, we multiply
heap_inuse by the ratio between the current heap goal and the last heap
goal, which describes whether the heap is growing or shrinking, and by
how much.
Finally, this change updates the documentation for scavenging and
elaborates on why the scavenge goal is defined the way it is.
Fixes #34149
Change-Id: I8deaf87620b5dc12a40ab8a90bf27932868610da
Reviewed-on: https://go-review.googlesource.com/c/go/+/193040
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Keith Randall <khr@golang.org>
(cherry picked from commit 9b30811280427a6d50d2558f316d62210e948656)
Reviewed-on: https://go-review.googlesource.com/c/go/+/198487
Run-TryBot: Andrew Bonventre <andybons@golang.org>
Austin Clements [Mon, 12 Aug 2019 18:54:28 +0000 (14:54 -0400)]
[release-branch.go1.13] runtime: grow the heap incrementally
Currently, we map and grow the heap a whole arena (64MB) at a time.
Unfortunately, in order to fix #32828, we need to switch from
scavenging inline with allocation back to scavenging on heap growth,
but heap-growth scavenging happens in large jumps because we grow the
heap in large jumps.
In order to prepare for better heap-growth scavenging, this CL
separates mapping more space for the heap from actually "growing" it
(tracking the new space with spans). Instead, growing the heap keeps
track of the "current arena" it's growing into. It track that with new
spans as needed, and only maps more arena space when the current arena
is inadequate. The effect to the user is the same, but this will let
us scavenge on much smaller increments of heap growth.
There are two slightly subtleties to this change:
1. If an allocation requires mapping a new arena and that new arena
isn't contiguous with the current arena, we don't want to lose the
unused space in the current arena, so we have to immediately track
that with a span.
2. The mapped space must be accounted as released and idle, even
though it isn't actually tracked in a span.
For #32828, since this makes heap-growth scavenging far more
effective, especially at small heap sizes. For example, this change is
necessary for TestPhysicalMemoryUtilization to pass once we remove
inline scavenging.
Updates #34556
Change-Id: I300e74a0534062467e4ce91cdc3508e5ef9aa73a
Reviewed-on: https://go-review.googlesource.com/c/go/+/189957
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
(cherry picked from commit f18109d7e30c8d1a6e1c87ba3458499ac7ab2a79)
Reviewed-on: https://go-review.googlesource.com/c/go/+/198485
Run-TryBot: Andrew Bonventre <andybons@golang.org> Reviewed-by: Austin Clements <austin@google.com>
Jason A. Donenfeld [Tue, 27 Aug 2019 12:46:16 +0000 (06:46 -0600)]
[release-branch.go1.13] runtime: monitor for suspend/resume to kick timeouts
Starting in Windows 8, the wait functions don't take into account
suspend time, even though the monotonic counters do. This results in
timer buckets stalling on resume. Therefore, this commit makes it so
that on resume, we return from the wait functions and recalculate the
amount of time left to wait.
This is a cherry pick of CL 191957 and its cleanup, CL 198417.
Updates #31528
Fixes #34130
Change-Id: I0db02cc72188cb620954e87a0180e0a3c83f4a56
Reviewed-on: https://go-review.googlesource.com/c/go/+/193607
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
Michael Hendricks [Wed, 2 Oct 2019 15:30:51 +0000 (09:30 -0600)]
[release-branch.go1.13] net: avoid an infinite loop in LookupAddr
If a request for a PTR record returned a response with a non-PTR
answer, goLookupPTR would loop forever. Skipping non-PTR answers
guarantees progress through the DNS response.
Michael Anthony Knyszek [Thu, 5 Sep 2019 16:34:00 +0000 (16:34 +0000)]
[release-branch.go1.13] runtime: fix lock acquire cycles related to scavenge.lock
There are currently two edges in the lock cycle graph caused by
scavenge.lock: with sched.lock and mheap_.lock. These edges appear
because of the call to ready() and stack growths respectively.
Furthermore, there's already an invariant in the code wherein
mheap_.lock must be acquired before scavenge.lock, hence the cycle.
The fix to this is to bring scavenge.lock higher in the lock cycle
graph, such that sched.lock and mheap_.lock are only acquired once
scavenge.lock is already held.
To faciliate this change, we move scavenger waking outside of
gcSetTriggerRatio such that it doesn't have to happen with the heap
locked. Furthermore, we check scavenge generation numbers with the heap
locked by using gopark instead of goparkunlock, and specify a function
which aborts the park should there be any skew in generation count.
Fixes #34150.
Change-Id: I3519119214bac66375e2b1262b36ce376c820d12
Reviewed-on: https://go-review.googlesource.com/c/go/+/191977
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
(cherry picked from commit 62e415655238a3c0103c1b70e6805edf8193c543)
Reviewed-on: https://go-review.googlesource.com/c/go/+/197501 Reviewed-by: Austin Clements <austin@google.com>
Jay Conrod [Fri, 13 Sep 2019 17:58:58 +0000 (13:58 -0400)]
[release-branch.go1.13] cmd/go: don't include package dir in cache key when -trimpath is set
The '-trimpath' flag tells 'go build' to trim any paths from the
output files that are tied to the current workspace or toolchain. When
this flag is set, we do not need to include the package directory in
the text hashed to construct the action ID for each package.
Updates #33772
Fixes #34326
Change-Id: I20b902d2f58019709b15864ca79aa0d9255ae707
Reviewed-on: https://go-review.googlesource.com/c/go/+/195318
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
(cherry picked from commit aa680c0c49b55722a72ad3772e590cd2f9af541d)
Reviewed-on: https://go-review.googlesource.com/c/go/+/198259
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>
Also added a test to ensure that any interactions
between TimeoutHandler and Flusher result in the
correct status code and body, but also that we don't
get superfluous logs from stray writes as was seen
in the bug report.
[release-branch.go1.13-security] net/textproto: don't normalize headers with spaces before the colon
RFC 7230 is clear about headers with a space before the colon, like
X-Answer : 42
being invalid, but we've been accepting and normalizing them for compatibility
purposes since CL 5690059 in 2012.
On the client side, this is harmless and indeed most browsers behave the same
to this day. On the server side, this becomes a security issue when the
behavior doesn't match that of a reverse proxy sitting in front of the server.
For example, if a WAF accepts them without normalizing them, it might be
possible to bypass its filters, because the Go server would interpret the
header differently. Worse, if the reverse proxy coalesces requests onto a
single HTTP/1.1 connection to a Go server, the understanding of the request
boundaries can get out of sync between them, allowing an attacker to tack an
arbitrary method and path onto a request by other clients, including
authentication headers unknown to the attacker.
This was recently presented at multiple security conferences:
https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn
net/http servers already reject header keys with invalid characters.
Simply stop normalizing extra spaces in net/textproto, let it return them
unchanged like it does for other invalid headers, and let net/http enforce
RFC 7230, which is HTTP specific. This loses us normalization on the client
side, but there's no right answer on the client side anyway, and hiding the
issue sounds worse than letting the application decide.
Fixes CVE-2019-16276
Change-Id: I6d272de827e0870da85d93df770d6a0e161bbcf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/549719 Reviewed-by: Brad Fitzpatrick <bradfitz@google.com>
(cherry picked from commit 1280b868e82bf173ea3e988be3092d160ee66082)
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/558935 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Tom Thorogood [Sat, 14 Sep 2019 01:16:20 +0000 (01:16 +0000)]
[release-branch.go1.13] net/http: fix HTTP/2 idle pool tracing
CL 140357 caused HTTP/2 connections to be put in the idle pool, but
failed to properly guard the trace.GotConn call in getConn. dialConn
returns a minimal persistConn with conn == nil for HTTP/2 connections.
This persistConn was then returned from queueForIdleConn and caused the
httptrace.GotConnInfo passed into GotConn to have a nil Conn field.
HTTP/2 connections call GotConn themselves so leave it for HTTP/2 to call
GotConn as is done directly below.
Fixes #34285
Change-Id: If54bfaf6edb14f5391463f908efbef5bb8a5d78e
GitHub-Last-Rev: 2b7d66a1ce66b4424c4d0fca2b8e8b547d874136
GitHub-Pull-Request: golang/go#34283
Reviewed-on: https://go-review.googlesource.com/c/go/+/195237 Reviewed-by: Michael Fraenkel <michael.fraenkel@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
(cherry picked from commit 582d5194faec20c475ab93b45cf0520253dec4a9)
Reviewed-on: https://go-review.googlesource.com/c/go/+/196579 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
[release-branch.go1.13] cmd/go/internal/modfetch/codehost: work around an apparent bug in 'git fetch --unshallow'
When 'git fetch' is passed the '--unshallow' flag, it assumes that the
local and remote refs are equal.¹ However, we were fetching an
expanded set of refs explicitly in the same command, violating that
assumption.
Now we first expand the set of refs, then unshallow the repo in a
separate fetch. Empirically, this seems to work, whereas the opposite
order does not.
Jay Conrod [Fri, 13 Sep 2019 15:10:36 +0000 (11:10 -0400)]
[release-branch.go1.13] cmd/go: fix link error for -coverpkg in GOPATH mode
If a generated test main package transitively depends on a main
package, the main package will now always be rebuilt as a library and
will not be compiled with '-p main'.
This expands the fix for #30907, which only applied to packages with
the BuildInfo set (main packages built in module mode). Linking
multiple packages with BuildInfo caused link errors, but it appears
these errors apply to some symbols in GOPATH mode.
Fixes #34223
Change-Id: Ic1e53437942269a950dd7e45d163707922c92edd
Reviewed-on: https://go-review.googlesource.com/c/go/+/195279
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
(cherry picked from commit 24781a1faf62ac5c9a553af6f46b787e972f5539)
Reviewed-on: https://go-review.googlesource.com/c/go/+/195281
[release-branch.go1.13] cmd/go: suppress errors in package-to-module queries if the package is already found
In CL 173017, I changed the package-to-module query logic to query all
possible module paths in parallel in order to reduce latency. (For
long package paths, most such paths will not exist and will fail with
little overhead.)
The module resolution algorithm treats various kinds of non-existence
as “soft errors”, to be reported only if package resolution fails, but
treats any remaining errors as hard errors that should fail the query.
Unfortunately, that interacted badly with the +incompatible version
validation added in CL 181881, causing a regression in the 'direct'
fetch path for modules using the “major branch” layout¹ with a post-v1
version on the repository's default branch. Because we did not
interpret a mismatched module path as “no such module”, a go.mod file
specifying the path 'example.com/foo/v2' would cause the search for
module 'example.com/foo' to error out. (That regression was not caught
ahead of time due to a lack of test coverage for 'go get' on a package
within a /vN module.)
The promotion of hard errors during parallel search also made the 'go'
command less tolerant of servers that advertise 'go-import' tags for
nonexistent repositories. CL 194561 mitigated that problem for HTTP
servers that return code 404 or 410 for a nonexistent repository, but
unfortunately a few servers in common use (notably GitLab and
pre-1.9.3 releases of Gitea) do not.
This change mitigates both of those failure modes by ignoring
“miscellaneous” errors from shorter module paths if the requested
package pattern was successfully matched against a module with a
longer path.
Rhys Hiltner [Thu, 29 Aug 2019 16:49:36 +0000 (09:49 -0700)]
[release-branch.go1.13] cmd/cover: skip go list when profile is empty
Only call "go list" when explicitly listing packages. An empty coverage
profile references no packages, and would otherwise lead to "go list"
implicitly looking at the package in "." (which might not exist).
Fixes #33984
Change-Id: I02d4e374405d86f03d105fe14648aa03b4d2284c
Reviewed-on: https://go-review.googlesource.com/c/go/+/192340 Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
(cherry picked from commit 9d480edadc6144d9f9f5a896d729d1642e46083b)
Reviewed-on: https://go-review.googlesource.com/c/go/+/192722