Upgrade to openssl-1.0.2a (Part2: final)
This is a final PR for upgrading to openssl-1.0.2a. I'm sorry for this is a really big patch for I could not succeed separating. This PR consist of 4 type of commits. I think we can focus to review 2nd and 3rd part.
1. source updates and private patches
e5624ef41a657b4add3c1780ac635c64d54901f6 deps: upgrade openssl to 1.0.2a 2a011557f0a4ec46f28abc174931f7ab90a81bf6 deps: fix openssl assembly error on ia32 win32 752636f64098e2e762d5f6b66f4f62c06cd34ff3 deps: fixed asm build error of openssl in x86_win32 0e6fad7311003c57710d96c00802e7ad214cf41a deps: backport openssl patch of alt cert chains 1 6b3a7c9fe4eafd3f0462ca797a0f04c126462e8f openssl: fix keypress requirement in apps on win32
2. openssconf.h and openssl.gyp/gypi updates
c3e3c4dc9fd37fdb23e53372f91f371b2f1fa992 deps: replace all headers in openssl b581ae23dbe0d44727d3f090bc46110124634906 deps: add x32 and arm64 support for opensslconf.h c75888ef4760a3708c6a8fb86b728387a861cec9 deps: update opensslconf.h for 1.0.2a 0d19c08be2d50993880a8758d0639fa8ef55da00 deps: update openssl.gyp/gypi for openssl-1.0.2a
3. asm updates
f2327d0e94ffc0be56748cd4d89dcf386e6c2672 deps: update asm Makefile for openssl-1.0.2a 208e06a12a574d3db22f8200209c650e31ab1c47 deps, build: add support older asm compiler e31145972eefb155d3631640cc560429a8360b73 deps: update asm files for openssl-1.0.2a
4. others
2a925d5ad1d7656c62b7d2f4486a0eaf7027df60 build: enable ssl support on arm64 5f532c2850555582595eff825ea5aff362d8538e test: fix error message check for openssl-1.0.2a d4a0c2743aa7b8d397c1ed56795f75ab21ffcac9 deps: add docs to upgrade openssl
Ci is https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/486/ and it seems fine . The test failures in armv7 and wins are not related this PR.
Here attached is the file of deps/openssl/doc/UPGRADING.md
below
I hope this helps to review this PR.
R= @bnoordhuis @indutny
How to upgrade openssl library in io.js
This document is intended to describe the procedure to upgrade openssl from 1.0.1m to 1.0.2a in io.js.
Build System and Upgrading Overview
The openssl build system is based on the perl script of Configure.
For example, running Configure linux_x86-64
in the openssl
repository generates Makefile
and opensslconf.h
for the target
architectures of linux_x86_64.
The Makefile
contains the list of asm files which are generated by
perl scripts during build so that we can get the most of use of the
hardware performance according to the type of cpus.
Configure TABLE
shows various build parameters that depend on each
os and arch.
In io.js, build target is defined as --dest-os
and --dest-cpu
in
configure options which are different from the one that is defined in
openssl and it's build system is gyp that is based on python,
therefore we cannot use the openssl build system directly.
In order to build openssl with gyp in iojs, files of opensslconf.h and asm are generated in advance for several supported platforms.
Here is a map table to show conf(opensslconf.h) and asm between the openssl target and configuration parameters of os and cpu in iojs. The tested platform in CI are also listed.
--dest-os | --dest-cpu | conf | asm | openssl target | CI |
---|---|---|---|---|---|
linux | ia32 | o | o | linux-elf | o |
linux | x32 | o | x(*2) | linux-x32 | x |
linux | x64 | o | o | linux-x86_64 | o |
linux | arm | o | o | linux-arm | o |
linux | arm64 | o | o | linux-aarch64 | o |
mac | ia32 | o | o | darwin-i386-cc | o |
mac | x64 | o | o | darwin64-x86_64-cc | o |
win | ia32 | o | o(*3) | VC-WIN32 | x |
win | x64 | o | o | VC-WIN64A | o |
solaris | ia32 | o | o | solaris-x86-gcc | o |
solaris | x64 | o | o | solaris64-x86_64-gcc | o |
freebsd | ia32 | o | o | BSD-x86 | o |
freebsd | x64 | o | o | BSD-x86_64 | o |
openbsd | ia32 | o | o | BSD-x86 | x |
openbsd | x64 | o | o | BSD-x86_64 | x |
others | ia32 | x(*1) | o | - | x |
others | x64 | x(*1) | o | - | x |
others | arm | x(*1) | o | - | x |
others | arm64 | x(*1) | o | - | x |
others | others | x(*1) | x(*2) | - | x |
- (*1) use linux-elf as a fallback configuration
- (*2) no-asm used
- (*3) currently masm (Microsoft Macro Assembler) is used but it's no longer supported in opnessl. We need to move to use nasm or yasm.
All parameters such as sources, defines, cflags and others generated
in openssl Makefile are wrote down into deps/openssl/openssl.gypi
.
The header file of deps/openssl/openssl/crypto/opensslconf.h
are
generated by Configure
and varies on each os and arch so that we
made a new deps/openssl/config/opensslconf.h
, where it includes each
conf file from deps/openssl/config/archs/*/opensslconf.h
by using
pre-defined compiler macros. This procedure can be processed
automatically with deps/openssl/config/Makefile
Assembler support is one of the key features in openssl, but asm files
are dynamically generated with
deps/openssl/openssl/crypto/*/asm/*.pl
by perl during
build. Furthermore, these perl scripts check the version of asm
compiler and generate asm files according to the supported
instructions in each compiler.
Since perl is not a build requirement in iojs, they all should be
generated in advance and statically stored in the repository. We
provide two sets of asm files, one is asm_latest(avx2 and addx
supported) in deps/openssl/asm
and the other asm_obsolete(without
avx1/2 and addx) in deps/openssl/asm_obsolute
, which depends on
supported features in asm compilers. Each directory has a Makefile
to generate asm files with perl scripts in openssl sources.
configure
and gyp check the version of asm compilers such as gnu
as(gas), llvm and Visual Studio. deps/openssl/openssl.gypi
determines what asm files should be used, in which the asm_latest
needs the version of gas >= 2.23, llvm >= 3.3 or MSVS_VERSION>='2012'
(ml64 >= 12) as defined in
https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/crypto/sha/asm/sha512-x86_64.pl#L112-L129,
otherwise asm_obsolete are used.
The following is the detail instruction steps how to upgrade openssl version from 1.0.1m to 1.0.2a in iojs.
deps/openssl/openssl
1. Replace openssl source in Remove old openssl sources in deps/openssl/openssl
.
Get original openssl sources from
https://www.openssl.org/source/openssl-1.0.2a.tar.gz and extract all
files into deps/openssl/openssl
.
2. Apply private patches
There are three kinds of private patches to be applied in openssl-1.0.2a.
- The two fixes of assembly error on ia32 win32. masm is no longer supported in openssl. We should move to use nasm or yasm in future version of iojs.
- The fix of openssl-cli built on win. Key press requirement of openssl-cli in win causes timeout failures of several tests.
- Backport patches for alt cert feature from openssl-1.1.x. Root certs of 1024bit RSA key length were deprecated in io.js. When a tls server has a cross root cert, io.js client leads CERT_UNTRUSTED error because openssl does not find alternate cert chains. This fix supports its feature but was made the current master which is openssl-1.1.x. We backported them privately into openssl-1.0.2 on iojs.
deps/openssl/openssl/include/openssl
3. Replace openssl header files in all header files in deps/openssl/openssl/include/openssl/*.h
are
symbolic links in the distributed release tar.gz. They cause issues in
Windows. They are replaced into the files to include a real header
file such as
#include "../../crypto/aes/aes.h"
opensslconf.h
so as to fit each platform.
4. Change The opensslconf.h in each target was created in advance by typing
deps/openssl/openssl/Configure {target}
and copied
into deps/openssl/conf/archs/{target}/opensslconf.h
.
deps/openssl/conf/openssconf.h
includes each file according to its
target by checking pre-defined compiler macros. These can be generated
by using deps/openssl/conf/Makefile
We should remove OPENSSL_CPUID_OBJ define in opensslconf.h because it
causes build error when --openss-no-asm option is specified. Instead,
the OPENSSL_CPUID_OBJ is defined in deps/openssl/openssl.gypi
according to the configure options.
One fix of opensslconf.h is needed in 64-bit MacOS.
The current openssl release does not use RC4 asm since it explicitly
specified as $asm=~s/rc4\-[^:]+//;
in
https://github.com/openssl/openssl/blob/OpenSSL_1_0_1-stable/Configure#L584
But iojs has used RC4 asm on MacOS for long time. Fix type of RC4_INT
into unsigned int
in opensslconf.h of darwin64-x86_64-cc to work on
the RC4 asm.
5. Update openssl.gyp and openssl.gypi
Sources, cflags and define parameters that depends on each target can
be obtained via Configure TABLE
. Its list is put in the table of
define and cflags changes in openssl-1.0.2a
There is no way to verify all necessary sources automatically. We can only carefully look at the source list and compiled objects in Makefile of openssl and compare the compiled objects that stored stored under `out/Release/obj.target/openssl/deps/openssl/' in iojs.
6. ASM files for openssl
We provide two sets of asm files. One is for the latest asm compiler and the other is the older one.
6.1. asm files for the latest compiler
This was made in deps/openssl/asm/Makefile
- Updated asm files for each platforms which are required in openssl-1.0.2a.
- Some perl files need CC and ASM envs. Added a check if these envs
exist. Followed asm files are to be generated with CC=gcc and
ASM=nasm on Linux. See
deps/openssl/openssl/crypto/sha/asm/sha512-x86_64.pl
- Added new 32bit targets/rules with a sse2 flag (OPENSSL_IA32_SSE2) to generate asm for use SSE2.
- Generating sha512 asm files in x86_64 need output filename which has 512. Added new rules so as not to use stdout for outputs.
- PERLASM_SCHEME of linux-armv4 is
void
as defined in openssl Configure. Changed its target/rule and all directories are moved from arm-elf-gas to arm-void-gas. - add a new rule for armv8 asm generation
With export environments of CC=gcc and ASM=nasm, then type make command and check if new asm files are generated.
6.2.asm files for the older compiler
For older asm compiler, the version check of CC and ASM should be
skipped in generating asm file with perl scripts.
Copy files from deps/openssl/asm
into
deps/openssl/asm/asm_obsolete
and change rules to generate asm files
into this directories and remove the check of CC and ASM envs.
Without environments of CC and ASM, then type make command and check if new asm files for older compilers are generated.
This PR fixes #589 (closed)