Long Core and Assets Build Times
I'm working with OpenCPI and have a project that will require several iterations of changes to OpenCPI functionality and to the core and assets projects. Since the core and assets projects need to get recopied and rebuilt for each iteration, this will turn into a time consuming process. For instance, just building the projects once took almost 5 hours on an 8th generation i7 with 32 GB of RAM! The problem is multiplied due to the fact that I have many colleagues that will also have to perform the same process.
Is there any way to speed up this build process? Can the build system utilize more threads? Are there pieces that do not need to be rebuilt for future generations of OpenCPI? Can I copy the pre-compiled projects to other computers and register them? I'm welcome to any feedback or help on this issue.
Thanks,
Andrew Lippolis
there are a lot of things and configurations of things that are built in each of the projects that are supplied.
there are a lot of bitfiles being built in assets that you may or may not be using. try using the --no-assembly option and only build the assemblies as you need them.
if a worker isn't changed there no reason that it needs to be rebuilt (the underlying make system deals with these dependencies for single workers/assemblies/). If you change a worker and don't rebuild the worker and build an assembly that it contains the worker will not get rebuilt as required to reflect the change that was made.
a change to the framework (i.e opencpi version change updates to anything but not limited to in opencpi/runtime or opencpi/tools/ocpigen) will always require all projects to be rebuilt. but, I would expect your development process would just be updating what is in your project and not making a lot of changes to the framework itself not requiring a rebuild of core and assets.
you can copy built projects as long as they are not registered. so you can unregister the project copy it to other computers like you mention.
Is there any way to speed up this build process?
It is possible to greatly reduce the build-time burden for the core, assets and your custom projects by building them in parallel, but it first requires an understanding of their (internal/external project) dependencies. As a point of reference, we've been able to reduce build-times of the core and assets projects targeting the Zedboad down to ~1 hour. (FYI - Targeting the Ultrascale++ parts seem to take longer than Zynq).
Before we go any further, be warned that building too many "assets" (core/assets/custom) in parallel can render your host practically useless for some unknown amount of time, or possibly crash. I recommend incrementally adding parallel builds to gauge your systems capabilities (our systems have 64 GB RAM).
Review the OpenCPI_HDL_Development, Section 4.2, Figure 7. This is the roadmap that you'll use for executing parallel builds and understanding cross project dependencies.
If 'ocpidev build --hdl-platform zed' is executed from the top-level of the project, the HDL build order (+RCC) follows Figure 7:
RCC workers(app & proxies)
HDL primitive libraries
application workers
device workers
adapters workers
cards (device workers)
platform workers (unique device workers)
assemblies
applications
Building a single (Core) project in parallel:
-
For the HDL build discussion, the core project HDL primitives are a dependency within and across all projects and MUST be built first.
$ ocpidev build hdl primitives --hdl-target zynq -
Since Workers (App, Device, Adapter, & Cards) are never dependent on each other, then they can be build in parallel.
$ ocpidev build -d components --hdl-target zynq &
$ ocpidev build -d hdl/devices --hdl-target zynq &
$ ocpidev build -d hdl/adapters --hdl-target zynq &
$ ocpidev build -d hdl/cards --hdl-target zynq & -
Build the desired platform(s) in parallel
$ ocpidev build -d hdl/platforms/zed --hdl-platform zed &
$ ocpidev build -d hdl/platforms/matchstiq_z1 --hdl-platform matchstiq_z1 & -
Build assemblies (NO ASSEMBLIES IN CORE PROJECT, here for completeness)
$ ocpidev build --hdl-platform matchstiq_z1 ('All' in the order in which they are listed in hdl/assemblies/Makefile)
$ ocpidev build --assembly <assm_name> --hdl-platform matchstiq_z1 (A specific assembly) -
Build applications (NO APPLICATION IN CORE PROJECT, here for completeness)
$ ocpidev build application ('All' in the order in which they are listed in applications/Makefile)
Building multiple (Core and Assets) projects in parallel:
-
Core project HDL primitives are a dependency within and across all projects and MUST be built first.
$ cd ./core
$ ocpidev build hdl primitives --hdl-target zynq -
Assets project HDL primitives are a dependency within and (potentially) across all projects and MUST be built first.
$ cd ./assets
$ ocpidev build hdl primitives --hdl-target zynq -
Core and Assets: Since Workers (App, Device, Adapter, & Cards) are never dependent on each other, then they can be build in parallel.
(THIS WILL SLOW YOUR SYSTEM DOWN, RECOMMEND INCREMENTALLY ADDING ocpidev build... to gauge system performance)
$ cd ./core
$ ocpidev build -d components --hdl-target zynq &
$ ocpidev build -d hdl/devices --hdl-target zynq &
$ ocpidev build -d hdl/adapters --hdl-target zynq &
$ ocpidev build -d hdl/cards --hdl-target zynq &
$ cd ./assets
$ ocpidev build -d components/base_comps --hdl-target zynq &
$ ocpidev build -d components/util_comps --hdl-target zynq &
$ ocpidev build -d components/dsp_comps --hdl-target zynq &
$ ocpidev build -d components/misc_comps --hdl-target zynq &
$ ocpidev build -d hdl/devices --hdl-target zynq &
$ ocpidev build -d hdl/adapters --hdl-target zynq &
$ ocpidev build -d hdl/cards --hdl-target zynq & -
Build the desired platform(s) in parallel
$ cd ./assets
$ ocpidev build -d hdl/platforms/zed --hdl-platform zed &
$ ocpidev build -d hdl/platforms/matchstiq_z1 --hdl-platform matchstiq_z1 &
$ cd ./bsp_custom
$ ocpidev build -d hdl/platforms/custom --hdl-platform custom & -
Build assemblies (NO ASSEMBLIES IN CORE PROJECT, here for completeness)
(Will slow system down, depending on number of parallel assemblies being built)
$ cd ./core
$ ocpidev build --hdl-platform matchstiq_z1 ('All' in the order in which they are listed in hdl/assemblies/Makefile)
$ ocpidev build --assembly <assm_name> --hdl-platform matchstiq_z1 (A specific assembly)
$ cd ./assets
$ ocpidev build --hdl-platform matchstiq_z1 ('All' in the order in which they are listed in hdl/assemblies/Makefile)
$ ocpidev build --assembly <assm_name> --hdl-platform matchstiq_z1 (A specific assembly) -
Build applications (NO APPLICATION IN CORE PROJECT, here for completeness)
$ cd ./assets
$ ocpidev build application ('All' in the order in which they are listed in applications/Makefile)
There is another level of parallel building which involves the use of "make -j HdlTargets=<zynq>" but this for the advanced users. We recommend using ocpidev as previously described.
Can the build system utilize more threads?
Vivado does this automatically under the hood. We don't have any tooling to tweak this from what i can tell from looking at the makefiles.
From: discuss discuss-bounces@lists.opencpi.org on behalf of Andrew Lippolis ALippolis@girdsystems.com
Sent: Thursday, March 14, 2019 11:52 AM
To: discuss@lists.opencpi.org
Subject: [Discuss OpenCPI] Long Core and Assets Build Times
I'm working with OpenCPI and have a project that will require several iterations of changes to OpenCPI functionality and to the core and assets projects. Since the core and assets projects need to get recopied and rebuilt for each iteration, this will turn into a time consuming process. For instance, just building the projects once took almost 5 hours on an 8th generation i7 with 32 GB of RAM! The problem is multiplied due to the fact that I have many colleagues that will also have to perform the same process.
Is there any way to speed up this build process? Can the build system utilize more threads? Are there pieces that do not need to be rebuilt for future generations of OpenCPI? Can I copy the pre-compiled projects to other computers and register them? I'm welcome to any feedback or help on this issue.
Thanks,
Andrew Lippolis
Andrew,
According to Xilinx Vivado's UG901 and UG904, the Tcl parameter
"general.maxThreads" can be used to set the number of threads to be used
for synthesis and implementation. That being said, this defaults to its
maximum of 8 on Linux systems. You can check your installation's default
value for this parameter by running the following:
vivado -mode batch -source /dev/stdin <<<'puts [get_param
general.maxThreads]'
So, your installation is likely already using the global maximum of 8
maxThreads. It *could *be the case a lower number of threads could result
in faster builds on your system, but I wouldn't bet on it. If you wish to
change this parameter, you could create an "init.tcl" file for your user
that will be sourced each time you open Vivado (see UG835 for more
information). Note that I do not believe this has ever been tried in
OpenCPI. In that init.tcl script, you could add something like the
following:
set_param general.maxThreads 2
Regards,
David Banks
On Thu, Mar 14, 2019 at 1:48 PM Chris Hinkey <
chinkey@geontech.onmicrosoft.com> wrote:
there are a lot of things and configurations of things that are built in
each of the projects that are supplied.
there are a lot of bitfiles being built in assets that you may or may not
be using. try using the --no-assembly option and only build the assemblies
as you need them.
if a worker isn't changed there no reason that it needs to be rebuilt (the
underlying make system deals with these dependencies for single
workers/assemblies/). If you change a worker and don't rebuild the worker
and build an assembly that it contains the worker will not get rebuilt as
required to reflect the change that was made.
a change to the framework (i.e opencpi version change updates to anything
but not limited to in opencpi/runtime or opencpi/tools/ocpigen) will always
require all projects to be rebuilt. but, I would expect your development
process would just be updating what is in your project and not making a lot
of changes to the framework itself not requiring a rebuild of core and
assets.
you can copy built projects as long as they are not registered. so you
can unregister the project copy it to other computers like you mention.
Is there any way to speed up this build process?
It is possible to greatly reduce the build-time burden for the core,
assets and your custom projects by building them in parallel, but it first
requires an understanding of their (internal/external project)
dependencies. As a point of reference, we've been able to reduce
build-times of the core and assets projects targeting the Zedboad down to
~1 hour. (FYI - Targeting the Ultrascale++ parts seem to take longer than
Zynq).
Before we go any further, be warned that building too many "assets"
(core/assets/custom) in parallel can render your host practically useless
for some unknown amount of time, or possibly crash. I recommend
incrementally adding parallel builds to gauge your systems capabilities
(our systems have 64 GB RAM).
Review the OpenCPI_HDL_Development, Section 4.2, Figure 7. This is the
roadmap that you'll use for executing parallel builds and understanding
cross project dependencies.
If 'ocpidev build --hdl-platform zed' is executed from the top-level of
the project, the HDL build order (+RCC) follows Figure 7:
RCC workers(app & proxies)
HDL primitive libraries
application workers
device workers
adapters workers
cards (device workers)
platform workers (unique device workers)
assemblies
applications
Building a single (Core) project in parallel:
-
For the HDL build discussion, the core project HDL primitives are a
dependency within and across all projects and MUST be built first.
$ ocpidev build hdl primitives --hdl-target zynq -
Since Workers (App, Device, Adapter, & Cards) are never dependent on
each other, then they can be build in parallel.
$ ocpidev build -d components --hdl-target zynq &
$ ocpidev build -d hdl/devices --hdl-target zynq &
$ ocpidev build -d hdl/adapters --hdl-target zynq &
$ ocpidev build -d hdl/cards --hdl-target zynq & -
Build the desired platform(s) in parallel
$ ocpidev build -d hdl/platforms/zed --hdl-platform zed &
$ ocpidev build -d hdl/platforms/matchstiq_z1 --hdl-platform matchstiq_z1 & -
Build assemblies (NO ASSEMBLIES IN CORE PROJECT, here for completeness)
$ ocpidev build --hdl-platform matchstiq_z1 ('All' in the order in which
they are listed in hdl/assemblies/Makefile)
$ ocpidev build --assembly <assm_name> --hdl-platform matchstiq_z1 (A
specific assembly) -
Build applications (NO APPLICATION IN CORE PROJECT, here for
completeness)
$ ocpidev build application ('All' in the order in which they are listed
in applications/Makefile)
Building multiple (Core and Assets) projects in parallel:
-
Core project HDL primitives are a dependency within and across all
projects and MUST be built first.
$ cd ./core
$ ocpidev build hdl primitives --hdl-target zynq -
Assets project HDL primitives are a dependency within and (potentially)
across all projects and MUST be built first.
$ cd ./assets
$ ocpidev build hdl primitives --hdl-target zynq -
Core and Assets: Since Workers (App, Device, Adapter, & Cards) are
never dependent on each other, then they can be build in parallel.
(THIS WILL SLOW YOUR SYSTEM DOWN, RECOMMEND INCREMENTALLY ADDING ocpidev
build... to gauge system performance)
$ cd ./core
$ ocpidev build -d components --hdl-target zynq &
$ ocpidev build -d hdl/devices --hdl-target zynq &
$ ocpidev build -d hdl/adapters --hdl-target zynq &
$ ocpidev build -d hdl/cards --hdl-target zynq &
$ cd ./assets
$ ocpidev build -d components/base_comps --hdl-target zynq &
$ ocpidev build -d components/util_comps --hdl-target zynq &
$ ocpidev build -d components/dsp_comps --hdl-target zynq &
$ ocpidev build -d components/misc_comps --hdl-target zynq &
$ ocpidev build -d hdl/devices --hdl-target zynq &
$ ocpidev build -d hdl/adapters --hdl-target zynq &
$ ocpidev build -d hdl/cards --hdl-target zynq & -
Build the desired platform(s) in parallel
$ cd ./assets
$ ocpidev build -d hdl/platforms/zed --hdl-platform zed &
$ ocpidev build -d hdl/platforms/matchstiq_z1 --hdl-platform matchstiq_z1 &
$ cd ./bsp_custom
$ ocpidev build -d hdl/platforms/custom --hdl-platform custom & -
Build assemblies (NO ASSEMBLIES IN CORE PROJECT, here for completeness)
(Will slow system down, depending on number of parallel assemblies being
built)
$ cd ./core
$ ocpidev build --hdl-platform matchstiq_z1 ('All' in the order in which
they are listed in hdl/assemblies/Makefile)
$ ocpidev build --assembly <assm_name> --hdl-platform matchstiq_z1 (A
specific assembly)
$ cd ./assets
$ ocpidev build --hdl-platform matchstiq_z1 ('All' in the order in which
they are listed in hdl/assemblies/Makefile)
$ ocpidev build --assembly <assm_name> --hdl-platform matchstiq_z1 (A
specific assembly) -
Build applications (NO APPLICATION IN CORE PROJECT, here for
completeness)
$ cd ./assets
$ ocpidev build application ('All' in the order in which they are listed
in applications/Makefile)
There is another level of parallel building which involves the use of
"make -j HdlTargets=<zynq>" but this for the advanced users. We recommend
using ocpidev as previously described.
Can the build system utilize more threads?
Vivado does this automatically under the hood. We don't have any tooling
to tweak this from what i can tell from looking at the makefiles.
From: discuss discuss-bounces@lists.opencpi.org on behalf of Andrew
Lippolis ALippolis@girdsystems.com
Sent: Thursday, March 14, 2019 11:52 AM
To: discuss@lists.opencpi.org
Subject: [Discuss OpenCPI] Long Core and Assets Build Times
I'm working with OpenCPI and have a project that will require several
iterations of changes to OpenCPI functionality and to the core and assets
projects. Since the core and assets projects need to get recopied and
rebuilt for each iteration, this will turn into a time consuming process.
For instance, just building the projects once took almost 5 hours on an 8th
generation i7 with 32 GB of RAM! The problem is multiplied due to the fact
that I have many colleagues that will also have to perform the same process.
Is there any way to speed up this build process? Can the build system
utilize more threads? Are there pieces that do not need to be rebuilt for
future generations of OpenCPI? Can I copy the pre-compiled projects to
other computers and register them? I'm welcome to any feedback or help on
this issue.
Thanks,
Andrew Lippolis