{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "title: Exploring why native executables produced with Mandrel 23.1 are bigger than those produced with Mandrel 23.0\n", "summary: >\n", " Starting with Quarkus 3.5, the default Mandrel version was updated from 23.0 to 23.1.\n", " This update introduced various improvements but also resulted in larger native executables.\n", " To understand the reasons behind this size increase, a comprehensive analysis was conducted, attributing it to specific changes in Mandrel's code base.\n", "date: \"2023-11-03T13:56:45+03:00\"\n", "lastmod: \"2023-11-03T17:44:42+03:00\"\n", "draft: false # Is this a draft? true/false\n", "toc: true # Show table of contents? true/false\n", "type: post # Do not modify.\n", "tags: \n", " - native \n", " - graalvm\n", " - mandrel\n", " - metrics\n", " - quarkus\n", "\n", "# Generate markdown from this file using\n", "# jupyter nbconvert index.ipynb --to markdown --NbConvertApp.output_files_dir=.\n", "---\n", "\n", "This article is a follow-up to [Exploring why native executables produced with Mandrel 23.0 are bigger than with Mandrel 22.3](../quarkus-size-22-3-23-0/).\n", "\n", "Starting with Quarkus 3.5 the default Mandrel version was updated from 23.0 to 23.1.\n", "\n", "This update brought a number of bugfixes as well as new features like:\n", "\n", "1. Preview of [Foreign Function & Memory API downcalls](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/ForeignInterface.md) (part of \"Project Panama\", [JEP 442](https://openjdk.org/jeps/442)) on AMD64. Must be enabled with `--enable-preview`.\n", "2. New option `-H:±IndirectBranchTargetMarker` to mark indirect branch targets on AMD64 with an endbranch instruction. This is a prerequisite for future Intel CET support.\n", "3. Throw `MissingReflectionRegistrationError` when attempting to create a proxy class without having it registered at build-time, instead of a `VMError`.\n", "4. Support for `-XX:+HeapDumpOnOutOfMemoryError`.\n", "5. New `--parallelism` option to control how many threads are used by the build process.\n", "6. Simulation of class initializer: Class initializer of classes that are not marked for initialization at image build time are simulated at image build time to avoid executing them at image run time.\n", "7. and [more](https://github.com/oracle/graal/blob/master/substratevm/CHANGELOG.md#graalvm-for-jdk-21-internal-version-2310)\n", "\n", "However, it also brought an unwanted side effect.\n", "The native executables produced with Mandrel 23.1 are bigger than the ones produced with Mandrel 23.10.\n", "To better understand why that happens we perform a thorough analysis to attribute the size increase to specific changes in Mandrel's code base.\n", "\n", "## TL;DR\n", "\n", "According to our analysis the binary size increase is attributed to two distinct changes, both of which are necessary for getting more accurate profiles when using the [async-sampler](https://github.com/oracle/graal/discussions/7707#discussioncomment-7443058).\n", "\n", "- [**Add support for profiling of topmost frame**](https://github.com/oracle/graal/pull/7003)\n", "- [**ProfilingSampler does not need local variable values**](https://github.com/oracle/graal/pull/6763) (specifically [the commit \"Always store bci in frame info\"](https://github.com/oracle/graal/commit/d747c30c7691012c39989a8597fd850c68b740ad))\n", "\n", "## Better understanding what is different between the generated native executables\n", "\n", "To perform the analysis we use the [Quarkus startstop test](https://github.com/quarkus-qe/quarkus-startstop) (specifically commit `a8bae846881607e376c7c8a96116b6b50ee50b70`) which generates, starts, tests, and stops small Quarkus applications and measures various time-related metrics (e.g. time-to-first-OK-request) and memory usage.\n", "We get the test with:\n", "\n", "```shell\n", "git clone https://github.com/quarkus-qe/quarkus-startstop\n", "cd quarkus-startstop\n", "git checkout a8bae846881607e376c7c8a96116b6b50ee50b70\n", "```\n", "\n", "and build it with:\n", "\n", "```shell\n", "mvn clean package -Pnative -Dquarkus.version=3.5.0\\\n", " -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17\n", "```\n", "\n", "changing the builder image tag to `jdk-20` and `jdk-21` for building with Mandrel 23.0 (based on JDK 20) and Mandrel 23.1 (based on JDK 21) respectively.\n", "\n", "The reason we also use `jdk-20` although deprecated is to see the effects of the base JDK when using the same code base (`jdk-17` and `jdk-20` are based on the same Mandrel source code but are built using a different base JDK version).\n", "\n", "Looking at the build output (generated by Quarkus in `target/my-app-native-image-sources/my-app-build-output-stats.json`) the main differences between the three builds are in the following metrics:\n", "\n", "| Mandrel version | 23.0.2.1 (jdk-17) | 23.0.1.2 (jdk-20) | 23.1.1.0 (jdk-21) | Increase jdk-17 to jdk-20 % | Increase jdk-20 to jdk-21 % |\n", "| --- | --- | --- | --- | --- | --- |\n", "| Image Heap Size | 29790208 | 30982144 | 33546240 | 4 | 8.3 |\n", "| Objects count | 351565 | 353273 | 356059 | 0.5 | 0.8 |\n", "| Resources Size | 169205 | 174761 | 175392 | 3.3 | 0.36 |\n", "| Resources Count | 28 | 28 | 79 | 0 | 182 |\n", "| Total Image Size | 60006728 | 61734352 | 64224536 | 2.88 | 4 |\n", "\n", "Which indicates that the base JDK plays significant role in the image size increase, leaving the question open on whether the further increase in the generated binary size between `jdk-20` and `jdk-21` is due to the JDK difference or due to changes in Mandrel itself.\n", "\n", "It is also interesting that despite the resource count increase between Mandrel 23.0 (jdk-20) and Mandrel 23.1 (jdk-21) the resource size is not affected that much.\n", "As a result, we focus our analysis on the Image Heap Size which increases disproportionally to the objects count between the different Mandrel versions indicating that either some objects became bigger, or the few new objects being added to the heap are quite big.\n", "\n", "### Dashboards\n", "\n", "GraalVM and Mandrel provide the `-H:+DashboardAll` and `-H:+DashboardJson` flags that can be used to generate dashboards that contain more information about the generated native executable.\n", "The resulting dashboard contains a number of metrics and looks like this:\n", "\n", "```json\n", "{\n", " \"points-to\": {\n", " \"type-flows\": [\n", " ...\n", " ]\n", " },\n", " \"code-breakdown\": {\n", " \"code-size\": [\n", " {\n", " \"name\": \"io.smallrye.mutiny.CompositeException.getFirstOrFail(Throwable[]) Throwable\",\n", " \"size\": 575\n", " },\n", " ...\n", " ]\n", " },\n", " \"heap-breakdown\": {\n", " \"heap-size\": [\n", " {\n", " \"name\": \"Lio/vertx/core/impl/VerticleManager$$Lambda$bf09d38f5d19578a0d041ffd0a524c1cbe1843df;\",\n", " \"size\": 24,\n", " \"count\": 1\n", " },\n", " ...\n", " ]\n", " }\n", "}\n", "```\n", "\n", "Using the aforementioned flags we generate dashboards using both Mandrel 23.1 and 23.0 and compare the results.\n", "\n", "To generate the dashboards using Mandrel 23.0 we use the following command:\n", "\n", "```shell\n", "mvn package -Pnative \\\n", " -Dquarkus.version=3.5.0 \\\n", " -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-20 \\\n", " -Dquarkus.native.additional-build-args=-H:+DashboardAll,-H:+DashboardJson,-H:DashboardDump=path/to/23.0.dashboard.json\n", "```\n", "\n", "Similarly to generate the dashboards using Mandrel 23.1 we use the following command:\n", "\n", "```shell\n", "mvn package -Pnative \\\n", " -Dquarkus.version=3.5.0 \\\n", " -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 \\\n", " -Dquarkus.native.additional-build-args=-H:+DashboardAll,-H:+DashboardJson,-H:DashboardDump=path/to/23.1.dashboard.json\n", "```\n", "\n", "Note: Make sure to change `path/to/` to the path where you would like the dashboard json files to be stored, each file is about 370MB big.\n", "\n", "### Analyzing and visualizing the data\n", "\n", "To process the data from the dashboards we used a Jupyter notebook, like we do in this article.\n", "To grab the notebook follow [this link](index.ipynb).\n", "\n", "#### Loading the data from the json files" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import json\n", "import pandas as pd\n", "\n", "# load data from JSON file\n", "with open('23.0.dashboard.json', 'r') as f:\n", " data23_0 = json.load(f)\n", "with open('23.1.dashboard.json', 'r') as f:\n", " data23_1 = json.load(f)\n", "\n", "# create dataframes from json data\n", "df23_0 = pd.DataFrame(data23_0)\n", "df23_1 = pd.DataFrame(data23_1)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Is the heap image bigger because the objects in the heap are bigger than before or because we store more objects in it? " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Get heap-size lists from dataframes\n", "heap_size_23_0 = df23_0['heap-breakdown']['heap-size']\n", "heap_size_23_1 = df23_1['heap-breakdown']['heap-size']\n", "\n", "# create dataframes from heap_size lists\n", "heap_df23_0 = pd.DataFrame(heap_size_23_0).rename(columns={'size': 'size-23.0', 'count': 'count-23.0'})\n", "heap_df23_1 = pd.DataFrame(heap_size_23_1).rename(columns={'size': 'size-23.1', 'count': 'count-23.1'})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### What's the average object size?" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average object size for Mandrel 23.0: 8137.46\n", "Average object size for Mandrel 23.1: 8880.06\n" ] } ], "source": [ "print(\"Average object size for Mandrel 23.0: {:.2f}\".format(heap_df23_0['size-23.0'].mean()))\n", "print(\"Average object size for Mandrel 23.1: {:.2f}\".format(heap_df23_1['size-23.1'].mean()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### What's the minimum and maximum object size in each case?" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Minimum object size for Mandrel 23.0: 16\n", "Minimum object size for Mandrel 23.1: 16\n", "Maximum object size for Mandrel 23.0: 14149496\n", "Maximum object size for Mandrel 23.1: 16933168\n" ] } ], "source": [ "print(\"Minimum object size for Mandrel 23.0:\", heap_df23_0['size-23.0'].min())\n", "print(\"Minimum object size for Mandrel 23.1:\", heap_df23_1['size-23.1'].min())\n", "max_size_23_0 = heap_df23_0['size-23.0'].max()\n", "max_size_23_1 = heap_df23_1['size-23.1'].max()\n", "print(\"Maximum object size for Mandrel 23.0:\", max_size_23_0)\n", "print(\"Maximum object size for Mandrel 23.1:\", max_size_23_1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We observe that the maximum object size when compiling with Mandrel 23.1 is about 2.6MB bigger than the maximum object size when compiling with Mandrel 23.0." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Max size difference in MB: 2.65\n" ] } ], "source": [ "max_size_diff = (max_size_23_1 - max_size_23_0) / (1024 * 1024)\n", "print(\"Max size difference in MB: {:.2f}\".format(max_size_diff))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Which objects are the bigger ones?\n", "\n", "As a result, next we search to see which objects are the bigger ones in both cases and what is their corresponding size in the other Mandrel version." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Objects with size equal to max_size_23_0 in heap_size_23_0:\n", " name size-23.0 count-23.0\n", "1340 [B 14149496 110914\n" ] } ], "source": [ "max_size_23_0_rows = heap_df23_0.loc[heap_df23_0['size-23.0'] == max_size_23_0]\n", "print(\"Objects with size equal to max_size_23_0 in heap_size_23_0:\")\n", "print(max_size_23_0_rows)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Objects with size equal to max_size_23_1 in heap_size_23_1:\n", " name size-23.1 count-23.1\n", "1351 [B 16933168 111454\n" ] } ], "source": [ "max_size_23_1_rows = heap_df23_1.loc[heap_df23_1['size-23.1'] == max_size_23_1]\n", "print(\"Objects with size equal to max_size_23_1 in heap_size_23_1:\")\n", "print(max_size_23_1_rows)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not surprisingly, we detect that the object type with the maximum size in both cases is `[B`, i.e. byte arrays.\n", "\n", "### More byte arrays of similar size or a few larger ones? \n", "\n", "Next we look at the average size of the byte arrays in both versions to see if the increase can be attributed to more similarly sized arrays being added to the image or just a few larger ones." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average size of byte arrays in Mandrel 23.0: 127.57\n", "Average size of byte arrays in Mandrel 23.1: 151.93\n" ] } ], "source": [ "max_size_23_0_row = max_size_23_0_rows.iloc[0]\n", "max_size_23_1_row = max_size_23_1_rows.iloc[0]\n", "\n", "print(\"Average size of byte arrays in Mandrel 23.0: {:.2f}\".format(max_size_23_0_row['size-23.0']/max_size_23_0_row['count-23.0']))\n", "print(\"Average size of byte arrays in Mandrel 23.1: {:.2f}\".format(max_size_23_1_row['size-23.1']/max_size_23_1_row['count-23.1']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We observe that the average byte array size when building with Mandrel 23.1 is bigger, which is an indication that some larger byte arrays are being added to the image heap.\n", "\n", "## Generating heap dumps and analyzing them in Java Mission Control (JMC)\n", "\n", "Since the dashboards don't provide more info we rebuild our test with `-Dquarkus.native.additional-build-args=-R:+DumpHeapAndExit` using both Mandrel versions.\n", "This options instructs the generated native images to create a heap dump and exit.\n", "\n", "```shell\n", "mvn package -Pnative \\\n", " -Dquarkus.version=3.5.0 \\\n", " -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-20 \\\n", " -Dquarkus.native.additional-build-args=-R:+DumpHeapAndExit\n", "...\n", "./target/quarkus-runner\n", "Heap dump created at 'quarkus-runner.hprof'.\n", "mv quarkus-runner.hprof quarkus-runner-23-0.hprof\n", "```\n", "\n", "We do the same with Mandrel 23.1 using the `jdk-21` tag and open the dumps in Java Mission Control (JMC).\n", "To install JMC one may use `sdk install jmc`.\n", "\n", "After starting JMC we navigate to `File->Open` and select the heap dumps we just generated.\n", "\n", "![JMC File -> Open](jmc-file-open.png)\n", "\n", "Once the heap dumps are loaded we click on the `byte[]` class to filter the results and focus on the objects of this type.\n", "\n", "![JMC focus on byte[] for 23.1](jmc-focus-byte-23-1.png)\n", "\n", "At this point on the right side of the window we can see the referrers sorted by the total size of the byte arrays they reference.\n", "\n", "![JMC focus on byte[] for 23.1](jmc-focus-byte-23-1-2.png)\n", "\n", "We observe that the majority of the byte arrays when using Mandrel 23.1 is referenced by `com.oracle.svm.core.code.ImageCodeInfo.codeInfoEncodings` (12%) and `com.oracle.svm.core.code.ImageCodeInfo.frameInfoEncodings` (11%), while when using Mandrerl 23.0 the corresponding percentages are 12% and 6%.\n", "\n", "![JMC focus on byte[] for 23.0](jmc-focus-byte-23-0-2.png)\n", "\n", "We also observe that when using Mandrel 23.1 the size of `com.oracle.svm.core.code.ImageCodeInfo.frameInfoEncodings` is ~2.5MB larger than the corresponding size when using Mandrel 23.0.\n", "\n", "## Attributing Binary Size Increase to Specific Code Changes\n", "\n", "As a result, we focus our search on changes in Mandrel's source code that could affect the frame info encodings using:\n", "\n", "```shell\n", "git log -- substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/code/FrameInfo*\n", "```\n", "\n", "this way we detected the following two pull requests:\n", "\n", "1. [**Add support for profiling of topmost frame**](https://github.com/oracle/graal/pull/7003) which adds ~1MB of data to the image.\n", "2. [**ProfilingSampler does not need local variable values**](https://github.com/oracle/graal/pull/6763) (specifically [the commit \"Always store bci in frame info\"](https://github.com/oracle/graal/commit/d747c30c7691012c39989a8597fd850c68b740ad)) which adds ~1.7MB of data to the image.\n", "\n", "Both of these changes are necessary to improve the accuracy of the [async-sampler](https://github.com/oracle/graal/discussions/7707#discussioncomment-7443058).\n", "\n", "## Conclusion\n", "\n", "Similarly to when Quarkus upgraded from 22.3 to 23.0, we observe an increase in the size of the generated native executables.\n", "Once more the changes resulting to that increase in the binary size appear to be well justified.\n", "As `native-image` becomes more mature and feature rich it seems inevitable to avoid increasing the size of the generated binaries.\n", "\n", "If you think that this kind of info should only be included when the user opts-in, please provide your feedback in https://github.com/oracle/graal/discussions/7707" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 2 }