Andi Vajda (JIRA)
2014-09-09 05:58:28 UTC
[ https://issues.apache.org/jira/browse/PYLUCENE-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126652#comment-14126652 ]
Andi Vajda commented on PYLUCENE-31:
------------------------------------
I did indeed eventually forget about this issue. My apologies.
My reservations about maintaining more monkey patches still stand but I need to review the patch to see how 'bad' it actually is.
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Andi Vajda commented on PYLUCENE-31:
------------------------------------
I did indeed eventually forget about this issue. My apologies.
My reservations about maintaining more monkey patches still stand but I need to review the patch to see how 'bad' it actually is.
JCC Parallel/Multiprocess Compilation + Caching
-----------------------------------------------
Key: PYLUCENE-31
URL: https://issues.apache.org/jira/browse/PYLUCENE-31
Project: PyLucene
Issue Type: Improvement
Environment: Linux 3.11.0-19-generic #33-Ubuntu SMP x86_64 GNU/Linux
Reporter: Lee Skillen
Priority: Minor
Labels: build, cache, ccache, distutils, jcc, parallel
Attachments: feature-parallel-build.patch
JCC utilises distutils.Extension() in order to build JCC itself and the packages that it generates for Java wrapping - Unfortunately distutils performs its build sequentially and doesn't take advantage of any additional free cores for parallel building. As discussed on the list this is likely a design decision due to potential issues that may arise when building projects with awkward, cyclic or recursive dependencies.
These issues shouldn't appear within JCC-based projects because of the generative nature of the build; i.e. all dependencies are resolved and generated prior to building, and the build process itself is about compilation and construction of the wrapper alone, of which the wrapper files are contained to a sequence of flattened compilation units.
http://hg.python.org/cpython/file/tip/Lib/distutils/ccompiler.py).
In addition to the distutils changes this patch also includes changes to the wrapper class generation to make it more cache friendly, with the target being that no changes in the wrapped code equals no changes in the wrapper code. So any changes that minimally change the wrapped code mean that with a tool such as ccache the rebuild time would be significantly reduced (almost to a nth, where n is the number of files and only one has changed).
Obviously the maintainers would have to assess this risk and decide whether they would like to accept the patch or not. Code has only been tested on Linux with Python 2.7.5 but should gracefully fail and prevent parallelisation if one of the requirements hasn't been met (not on linux, no multiprocessing support, or monkey patching somehow fails). The change to caching should still benefit everyone regardless.
Please note that an additional dependency on orderedset has been added to achieve the more deterministic ordering - This may not be desirable (i.e. another package might be desired, such as ordered-set, or the code might be inlined into the package instead), as per maintainer comments.
--- [following repeated from mailing list] ---
Performance Statistics :-
real 5m1.502s
user 5m22.887s
sys 0m7.749s
real 1m37.382s
user 7m16.658s
sys 0m8.697s
real 0m43.051s
user 1m10.392s
sys 0m4.547s
This was a run in which nothing changed between runs, so a realistic run in which changes occur it'll be a figure between 0m43.051s and 1m37.382s, depending on how drastic the change was. If many changes are expected and you want to keep it more cache friendly then using a higher --files would probably work (to an extent), or ideally use --files separate, although it doesn't currently work for me (need to investigate).
We're mostly utilising the PyLucene build as a test bed since it is repeatable for others, rather than just showing numbers for own application compilations; we also use it to run the unit test suite after changes to JCC itself to ensure it still works as intended for PyLucene. For illustrative purposes though our application takes 1m53s to compile with JCC from scratch serially, 0m31s in parallel (8 jobs), 0m14s in parallel with ccache enabled and minimal changes, and 0m8s with ccache and no changes. A very agreeable result!
-------------------------------------------------
Key: PYLUCENE-31
URL: https://issues.apache.org/jira/browse/PYLUCENE-31
Project: PyLucene
Issue Type: Improvement
Environment: Linux 3.11.0-19-generic #33-Ubuntu SMP x86_64 GNU/Linux
Reporter: Lee Skillen
Priority: Minor
Labels: build, cache, ccache, distutils, jcc, parallel
Attachments: feature-parallel-build.patch
JCC utilises distutils.Extension() in order to build JCC itself and the packages that it generates for Java wrapping - Unfortunately distutils performs its build sequentially and doesn't take advantage of any additional free cores for parallel building. As discussed on the list this is likely a design decision due to potential issues that may arise when building projects with awkward, cyclic or recursive dependencies.
These issues shouldn't appear within JCC-based projects because of the generative nature of the build; i.e. all dependencies are resolved and generated prior to building, and the build process itself is about compilation and construction of the wrapper alone, of which the wrapper files are contained to a sequence of flattened compilation units.
http://hg.python.org/cpython/file/tip/Lib/distutils/ccompiler.py).
In addition to the distutils changes this patch also includes changes to the wrapper class generation to make it more cache friendly, with the target being that no changes in the wrapped code equals no changes in the wrapper code. So any changes that minimally change the wrapped code mean that with a tool such as ccache the rebuild time would be significantly reduced (almost to a nth, where n is the number of files and only one has changed).
Obviously the maintainers would have to assess this risk and decide whether they would like to accept the patch or not. Code has only been tested on Linux with Python 2.7.5 but should gracefully fail and prevent parallelisation if one of the requirements hasn't been met (not on linux, no multiprocessing support, or monkey patching somehow fails). The change to caching should still benefit everyone regardless.
Please note that an additional dependency on orderedset has been added to achieve the more deterministic ordering - This may not be desirable (i.e. another package might be desired, such as ordered-set, or the code might be inlined into the package instead), as per maintainer comments.
--- [following repeated from mailing list] ---
Performance Statistics :-
real 5m1.502s
user 5m22.887s
sys 0m7.749s
real 1m37.382s
user 7m16.658s
sys 0m8.697s
real 0m43.051s
user 1m10.392s
sys 0m4.547s
This was a run in which nothing changed between runs, so a realistic run in which changes occur it'll be a figure between 0m43.051s and 1m37.382s, depending on how drastic the change was. If many changes are expected and you want to keep it more cache friendly then using a higher --files would probably work (to an extent), or ideally use --files separate, although it doesn't currently work for me (need to investigate).
We're mostly utilising the PyLucene build as a test bed since it is repeatable for others, rather than just showing numbers for own application compilations; we also use it to run the unit test suite after changes to JCC itself to ensure it still works as intended for PyLucene. For illustrative purposes though our application takes 1m53s to compile with JCC from scratch serially, 0m31s in parallel (8 jobs), 0m14s in parallel with ccache enabled and minimal changes, and 0m8s with ccache and no changes. A very agreeable result!
This message was sent by Atlassian JIRA
(v6.3.4#6332)