Installing PyLucene

Post by marco turchi
Dear All,
I'm newer with PyLucene and I'm trying to instal it on my local home on a
cluster. In this environment, I do not have root privilege,

You do not need root privileges if you don't modify the system python. One way to achieve that is to setup a python virtualenv first and install jcc and pylucene into it instead of the system python.

Post by marco turchi
I'm using a
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Post by marco turchi
python setup.py build
python setup.py install --users
and JCC is installed in $home/.local
Then I have started to install pyLucene, I have changed the Makefile (using
the Linux (Debian Jessie 64-bit, Python 2.7.9, Oracle Java 1.8)
[ivy:configure] :: loading settings :: file =
/hltsrv0/turchi/Projects/QT21/WorkingFolder/NMT/software/pylucene-6.2.0/lucene-java-6.2.0/lucene/top-level-ivy-settings.xml

Ah, here, to build Java Lucene, ivy is required and without a web connection, it's going to be more difficult. You need to somehow make sure that all things ivy is going to download during the Lucene build (a one time setup only) are already there when you build Lucene.
You could do this on an equivalent machine that has a web connection and then copy the local ivy tree to the machine that doesn't.

Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

marco turchi

2016-12-30 23:07:42 UTC

Dear Andi,
thanks a lot for you answers!

Post by Andi Vajda
You do not need root privileges if you don't modify the system python. One
way to achieve that is to setup a python virtualenv first and install jcc
and pylucene into it instead of the system python.

Do you mean to install a new version of python in one of my folders and us
it for installing JCC and pyLucene?

Post by marco turchi
I'm using a
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to the
Web to download the required libraries

Post by Andi Vajda
Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make sure
that all things ivy is going to download during the Lucene build (a one
time setup only) are already there when you build Lucene.
You could do this on an equivalent machine that has a web connection and
then copy the local ivy tree to the machine that doesn't.

This is a great suggestion, thanks a lot! I'm going to try this in the next
days!!

Best,
Marco

Post by Andi Vajda
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

Andi Vajda

2016-12-31 03:09:09 UTC

Post by marco turchi
Dear Andi,
thanks a lot for you answers!

Do you mean to install a new version of python in one of my folders and us
it for installing JCC and pyLucene?

No, I mean to setup a python virtualenv.

Andi..

Post by marco turchi
I'm using a
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to the
Web to download the required libraries

This is a great suggestion, thanks a lot! I'm going to try this in the next
days!!
Best,
Marco

Post by Andi Vajda
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

marco turchi

2017-01-03 15:01:54 UTC

Dear Andi,
following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.

Running the make test I have the following errors:
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_PythonException.py", line 34, in testThroughLayerException
qp.parse("foo bar")
JavaError: <super: <class 'JavaError'>, <JavaError object>>
Java stacktrace:
java.lang.RuntimeException: TestException
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery_quoted(Native
Method)
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery(Unknown
Source
at
org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(QueryParser.java:585)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:198)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:187)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:111)

and:
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...

I have checked on the mailing list but I have not found solutions that work
for me. I'm using:
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60

do you have an idea of what it is not working?

Thanks a lot in advance for your help!
Marco

Post by marco turchi
Dear Andi,
thanks a lot for you answers!

Post by Andi Vajda
You do not need root privileges if you don't modify the system python.

One

Post by Andi Vajda
way to achieve that is to setup a python virtualenv first and install

jcc

Post by Andi Vajda
and pylucene into it instead of the system python.

Do you mean to install a new version of python in one of my folders and

Post by marco turchi
it for installing JCC and pyLucene?

No, I mean to setup a python virtualenv.
Andi..

Post by marco turchi
I'm using a
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to

the

Post by marco turchi
Web to download the required libraries

Post by Andi Vajda
Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make

sure

Post by Andi Vajda
that all things ivy is going to download during the Lucene build (a one
time setup only) are already there when you build Lucene.
You could do this on an equivalent machine that has a web connection and
then copy the local ivy tree to the machine that doesn't.

This is a great suggestion, thanks a lot! I'm going to try this in the

Post by marco turchi
days!!
Best,
Marco

Post by Andi Vajda
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

marco turchi

2017-01-03 15:05:35 UTC

Hi
I have also run the following commands to check if I'm using the shared

from jcc import config
config.SHARED

True

Best,
Marco

Post by marco turchi
Dear Andi,
following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
----------------------------------------------------------------------
File "test/test_PythonException.py", line 34, in
testThroughLayerException
qp.parse("foo bar")
JavaError: <super: <class 'JavaError'>, <JavaError object>>
java.lang.RuntimeException: TestException
at org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery_quoted(Native
Method)
at org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery(Unknown
Source
at org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(
QueryParser.java:585)
at org.apache.lucene.queryparser.classic.QueryParser.Query(
QueryParser.java:198)
at org.apache.lucene.queryparser.classic.QueryParser.
TopLevelQuery(QueryParser.java:187)
at org.apache.lucene.queryparser.classic.QueryParserBase.parse(
QueryParserBase.java:111)
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...
I have checked on the mailing list but I have not found solutions that
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60
do you have an idea of what it is not working?
Thanks a lot in advance for your help!
Marco

Dear Andi,
thanks a lot for you answers!

Post by Andi Vajda
You do not need root privileges if you don't modify the system python.

One

Post by Andi Vajda
way to achieve that is to setup a python virtualenv first and install

jcc

Post by Andi Vajda
and pylucene into it instead of the system python.

Do you mean to install a new version of python in one of my folders and

it for installing JCC and pyLucene?

No, I mean to setup a python virtualenv.
Andi..

Post by marco turchi
I'm using a
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to

the

Web to download the required libraries

Post by Andi Vajda
Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make

sure

and

Post by Andi Vajda
then copy the local ivy tree to the machine that doesn't.

This is a great suggestion, thanks a lot! I'm going to try this in the

days!!
Best,
Marco

Post by Andi Vajda
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

Andi Vajda

2017-01-03 21:05:40 UTC

Post by marco turchi
Dear Andi,
following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
----------------------------------------------------------------------
File "test/test_PythonException.py", line 34, in testThroughLayerException
qp.parse("foo bar")
JavaError: <super: <class 'JavaError'>, <JavaError object>>
java.lang.RuntimeException: TestException
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery_quoted(Native
Method)
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery(Unknown
Source
at
org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(QueryParser.java:585)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:198)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:187)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:111)
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...

Are both machines (the one on the web and the one offline) running the samev
version of Windows and Python ?

Andi..

Post by marco turchi
I have checked on the mailing list but I have not found solutions that work
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60
do you have an idea of what it is not working?
Thanks a lot in advance for your help!
Marco

Post by marco turchi
Dear Andi,
thanks a lot for you answers!

Post by Andi Vajda
You do not need root privileges if you don't modify the system python.

One

Post by Andi Vajda
way to achieve that is to setup a python virtualenv first and install

jcc

Post by Andi Vajda
and pylucene into it instead of the system python.

Do you mean to install a new version of python in one of my folders and

Post by marco turchi
it for installing JCC and pyLucene?

No, I mean to setup a python virtualenv.
Andi..

Post by marco turchi
I'm using a
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to

the

Post by marco turchi
Web to download the required libraries

Post by Andi Vajda
Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make

sure

This is a great suggestion, thanks a lot! I'm going to try this in the

Post by marco turchi
days!!
Best,
Marco

Post by Andi Vajda
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

marco turchi

2017-01-03 21:26:01 UTC

Nope different linux kernels and python versions (2.7.11 and 2.7.12).

Btw I have also similar problem on the machine with Web access.

Best,
Marco

Post by marco turchi
Dear Andi,

Post by marco turchi
following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
----------------------------------------------------------------------
File "test/test_PythonException.py", line 34, in
testThroughLayerException
qp.parse("foo bar")
JavaError: <super: <class 'JavaError'>, <JavaError object>>
java.lang.RuntimeException: TestException
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.ge
tFieldQuery_quoted(Native
Method)
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.ge
tFieldQuery(Unknown
Source
at
org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(
QueryParser.java:585)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(Quer
yParser.java:198)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQu
ery(QueryParser.java:187)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(
QueryParserBase.java:111)
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...

Are both machines (the one on the web and the one offline) running the
samev version of Windows and Python ?
Andi..
I have checked on the mailing list but I have not found solutions that work

Post by marco turchi
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60
do you have an idea of what it is not working?
Thanks a lot in advance for your help!
Marco

Post by marco turchi
Dear Andi,
thanks a lot for you answers!
You do not need root privileges if you don't modify the system python.
One
way to achieve that is to setup a python virtualenv first and install
jcc
and pylucene into it instead of the system python.

Post by marco turchi
Do you mean to install a new version of python in one of my folders and

Post by marco turchi
it for installing JCC and pyLucene?

No, I mean to setup a python virtualenv.
Andi..

Post by marco turchi
I'm using a

Post by marco turchi
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to

the

Post by marco turchi
Web to download the required libraries

Post by marco turchi
Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make

sure
that all things ivy is going to download during the Lucene build (a one

Post by marco turchi
time setup only) are already there when you build Lucene.
You could do this on an equivalent machine that has a web connection and
then copy the local ivy tree to the machine that doesn't.

This is a great suggestion, thanks a lot! I'm going to try this in the

Post by marco turchi
days!!
Best,
Marco

Post by marco turchi
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

Andi Vajda

2017-01-03 22:48:24 UTC

Post by marco turchi
Nope different linux kernels and python versions (2.7.11 and 2.7.12).
Btw I have also similar problem on the machine with Web access.

I suspect you don't have shared mode enabled when running tests.
The WindowsError exception is a bug that no one hits until the tests fails
to remove its STORE_DIR tree. I need to remove this...
You can remove that try/except bracket since it doesn't apply to you anyway.

Usually, errors with test_PythonException.py have to do with missing shared
mode. If PyLucene is otherwise working for you, you can ignore this or make
sure shared mode is actually deployed with PyLucene.

Andi..

Post by marco turchi
Best,
Marco

Post by marco turchi
Dear Andi,

Are both machines (the one on the web and the one offline) running the
samev version of Windows and Python ?
Andi..
I have checked on the mailing list but I have not found solutions that work

Post by marco turchi
Do you mean to install a new version of python in one of my folders and

Post by marco turchi
it for installing JCC and pyLucene?

No, I mean to setup a python virtualenv.
Andi..

Post by marco turchi
I'm using a

Post by marco turchi
version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.

You can use setuptools without a web connection, why not ?

Sorry, you are right I thought that setuptools needs to be connected to

the

Post by marco turchi
Web to download the required libraries

Post by marco turchi
Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make

sure
that all things ivy is going to download during the Lucene build (a one

This is a great suggestion, thanks a lot! I'm going to try this in the

Post by marco turchi
days!!
Best,
Marco

Post by marco turchi
Andi..

Post by marco turchi
Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?
Thanks a lot in advance for your help!
Best Regards,
Marco

Thomas Koch

2017-01-04 08:13:03 UTC

Post by marco turchi
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)

I've downloaded current version
pylucene-6.2.0-src.tar.gz 2016-09-18 09:04 30M
and was able to build JCC and PyLucene on MacOS X10.11 (64-bit, Python 2.7.10 , Java 1.8.0)

'make test' runs without any issues.

From the stacktrace and the error:
NameError: global name 'WindowsError' is not defined
I assume that one test fails to cleanup the store directory and then runs in the undefined 'WindowsError' in test_Python.py

def tearDown(self):

if os.path.exists(self.STORE_DIR):
try:
shutil.rmtree(self.STORE_DIR)
except WindowsError:
# maybe leaking file handles in closing stores
# does not affect other tests
pass

Please look for a directory named 'testrepoâ , then check it's file permissions and remove it (maybe you did ran the tests as root initially?) - finally run the test again.

Of course the error should be catched better on Linux... You could file a bug report here:
https://issues.apache.org/jira/browse/PYLUCENE <https://issues.apache.org/jira/browse/PYLUCENE>

Note that 'WindowsError' is only available on Windows platform.

Post by marco turchi
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile gcc)

Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered the patches needed to JCC and PyLucene for use/review on the list - but didn't get any feedback so far.
cf. https://www.mail-archive.com/pylucene-***@lucene.apache.org/msg02167.html <https://www.mail-archive.com/pylucene-***@lucene.apache.org/msg02167.html>

Regards,
Thomas
--

Post by marco turchi
Dear Andi,
following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
----------------------------------------------------------------------
File "test/test_PythonException.py", line 34, in testThroughLayerException
qp.parse("foo bar")
JavaError: <super: <class 'JavaError'>, <JavaError object>>
java.lang.RuntimeException: TestException
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery_quoted(Native
Method)
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery(Unknown
Source
at
org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(QueryParser.java:585)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:198)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:187)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:111)
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...
I have checked on the mailing list but I have not found solutions that work
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60
do you have an idea of what it is not working?
Thanks a lot in advance for your help!
Marco

Petrus Hyvönen

2017-01-04 09:59:08 UTC

Dear Thomas,

I would be very interested in a python 3 port of JCC. I am not a very
skilled developer, looked at starting a development based on the old
python-3 version but it's beyond my current skills.

I would be happy to help and test and review the JCC patches, I think your
patches would be a valuable contribution to JCC.

With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered the
patches needed to JCC and PyLucene for use/review on the list - but didn't
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus HyvÃ¶nen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

marco turchi

2017-01-04 12:24:23 UTC

Dear Andi and Thomas,
following your advice I have removed the Windows error.

I still have this

ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)

To answer Andi, I have printed the config.SHARED just before the error and
the output is true, in my opinion, showing that the shared mode is enabled
when running tests. Is this that you were mentioning in your email?

Thanks a lot for your help!
Marco

Post by Petrus HyvÃ¶nen
Dear Thomas,
I would be very interested in a python 3 port of JCC. I am not a very
skilled developer, looked at starting a development based on the old
python-3 version but it's beyond my current skills.
I would be happy to help and test and review the JCC patches, I think your
patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus HyvÃ¶nen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

Andi Vajda

2017-01-04 18:42:51 UTC

Post by marco turchi
Dear Andi and Thomas,
following your advice I have removed the Windows error.
I still have this
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
To answer Andi, I have printed the config.SHARED just before the error and
the output is true, in my opinion, showing that the shared mode is enabled
when running tests. Is this that you were mentioning in your email?

When you built PyLucene did you include --shared on the jcc invocation command line ?

Andi..

Post by marco turchi
Thanks a lot for your help!
Marco

Post by marco turchi
NameError: global name 'WindowsError' is not defined

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus Hyvönen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

marco turchi

2017-01-04 21:51:27 UTC

No I didn't.

I have run the codes in sample and they work. For my project the
functionalities in the samples are enough. If necessary I can recompile jcc
with --shared. What do you suggest?

Best
Marco

When you built PyLucene did you include --shared on the jcc invocation
command line ?

Andi..

Post by marco turchi
Thanks a lot for your help!
Marco

Post by marco turchi
NameError: global name 'WindowsError' is not defined

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus HyvÃ¶nen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

Andi Vajda

2017-01-05 01:27:34 UTC

Post by marco turchi
No I didn't.
I have run the codes in sample and they work. For my project the
functionalities in the samples are enough. If necessary I can recompile jcc
with --shared. What do you suggest?

If you don't use --shared then the jcc that is linked into PyLucene is not running shared mode and the test failure you're seeing is due to that.

It's easy enough to rebuild PyLucene with --shared.
Up to you !

Andi..

Post by marco turchi
Best
Marco

When you built PyLucene did you include --shared on the jcc invocation command line ?
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco

your

Post by Petrus HyvÃ¶nen
patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus Hyvönen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

marco turchi

2017-01-05 15:27:03 UTC

Perfect!!!

For now, I keep the version as it is. I'll try later to install jcc with
--shared flag, because I'm not sure if the patch for the setuptools
requires root access.

Thanks a lot for your help!
Marco

Post by marco turchi
No I didn't.
I have run the codes in sample and they work. For my project the
functionalities in the samples are enough. If necessary I can recompile

jcc

Post by marco turchi
with --shared. What do you suggest?

If you don't use --shared then the jcc that is linked into PyLucene is not
running shared mode and the test failure you're seeing is due to that.
It's easy enough to rebuild PyLucene with --shared.
Up to you !
Andi..

Post by marco turchi
Best
Marco

and

Post by marco turchi
the output is true, in my opinion, showing that the shared mode is

enabled

Post by marco turchi
when running tests. Is this that you were mentioning in your email?

When you built PyLucene did you include --shared on the jcc invocation command line ?
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco
On Wed, Jan 4, 2017 at 10:59 AM, Petrus HyvÃ¶nen <

your

Post by Petrus HyvÃ¶nen
patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered

the

Post by Thomas Koch
patches needed to JCC and PyLucene for use/review on the list - but

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus HyvÃ¶nen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

Andi Vajda

2017-01-05 16:21:06 UTC

Post by marco turchi
Perfect!!!
For now, I keep the version as it is. I'll try later to install jcc with
--shared flag, because I'm not sure if the patch for the setuptools
requires root access.

Your JCC install is fine. It's PyLucene that needs to be rebuilt by adding a --shared arg to its jcc invocation command line in its Makefile. No setuptools patching necessary.

Andi..

Post by marco turchi
Thanks a lot for your help!
Marco

Post by marco turchi
No I didn't.
I have run the codes in sample and they work. For my project the
functionalities in the samples are enough. If necessary I can recompile

jcc

Post by marco turchi
with --shared. What do you suggest?

If you don't use --shared then the jcc that is linked into PyLucene is not
running shared mode and the test failure you're seeing is due to that.
It's easy enough to rebuild PyLucene with --shared.
Up to you !
Andi..

Post by marco turchi
Best
Marco

and

Post by marco turchi
the output is true, in my opinion, showing that the shared mode is

enabled

Post by marco turchi
when running tests. Is this that you were mentioning in your email?

When you built PyLucene did you include --shared on the jcc invocation command line ?
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco
On Wed, Jan 4, 2017 at 10:59 AM, Petrus Hyvönen <

your

Post by Petrus HyvÃ¶nen
patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered

the

Post by Thomas Koch
patches needed to JCC and PyLucene for use/review on the list - but

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus Hyvönen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

marco turchi

2017-01-05 16:57:20 UTC

Done! all testes passed!

thanks a lot!
Marco

Post by marco turchi
Perfect!!!
For now, I keep the version as it is. I'll try later to install jcc with
--shared flag, because I'm not sure if the patch for the setuptools
requires root access.

Your JCC install is fine. It's PyLucene that needs to be rebuilt by adding
a --shared arg to its jcc invocation command line in its Makefile. No
setuptools patching necessary.
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco

Post by marco turchi
No I didn't.
I have run the codes in sample and they work. For my project the
functionalities in the samples are enough. If necessary I can recompile

jcc

Post by marco turchi
with --shared. What do you suggest?

If you don't use --shared then the jcc that is linked into PyLucene is

not

Post by Andi Vajda
running shared mode and the test failure you're seeing is due to that.
It's easy enough to rebuild PyLucene with --shared.
Up to you !
Andi..

Post by marco turchi
Best
Marco

and

Post by marco turchi
the output is true, in my opinion, showing that the shared mode is

enabled

Post by marco turchi
when running tests. Is this that you were mentioning in your email?

When you built PyLucene did you include --shared on the jcc invocation
command line ?
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco
On Wed, Jan 4, 2017 at 10:59 AM, Petrus HyvÃ¶nen <

your

Post by Petrus HyvÃ¶nen
patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered

the

Post by Thomas Koch
patches needed to JCC and PyLucene for use/review on the list - but

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus HyvÃ¶nen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

Andi Vajda

2017-01-05 17:03:00 UTC

Post by marco turchi
Done! all testes passed!

Excellent !

Post by marco turchi
thanks a lot!
Marco

Post by marco turchi
Perfect!!!
For now, I keep the version as it is. I'll try later to install jcc with
--shared flag, because I'm not sure if the patch for the setuptools
requires root access.

Your JCC install is fine. It's PyLucene that needs to be rebuilt by adding
a --shared arg to its jcc invocation command line in its Makefile. No
setuptools patching necessary.
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco

Post by marco turchi
No I didn't.
I have run the codes in sample and they work. For my project the
functionalities in the samples are enough. If necessary I can recompile

jcc

Post by marco turchi
with --shared. What do you suggest?

If you don't use --shared then the jcc that is linked into PyLucene is

not

Post by Andi Vajda
running shared mode and the test failure you're seeing is due to that.
It's easy enough to rebuild PyLucene with --shared.
Up to you !
Andi..

Post by marco turchi
Best
Marco

and

Post by marco turchi
the output is true, in my opinion, showing that the shared mode is

enabled

Post by marco turchi
when running tests. Is this that you were mentioning in your email?

When you built PyLucene did you include --shared on the jcc invocation
command line ?
Andi..

Post by marco turchi
Thanks a lot for your help!
Marco
On Wed, Jan 4, 2017 at 10:59 AM, Petrus Hyvönen <

your

Post by Petrus HyvÃ¶nen
patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Post by marco turchi
NameError: global name 'WindowsError' is not defined

Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered

the

Post by Thomas Koch
patches needed to JCC and PyLucene for use/review on the list - but

didn't

Post by Thomas Koch
get any feedback so far.
org/msg02167.html <https://www.mail-archive.com/
Regards,
Thomas

--
_____________________________________________
Petrus Hyvönen, Uppsala, Sweden
Mobile Phone/SMS:+46 73 803 19 00

Thomas Koch

2017-01-06 09:07:52 UTC

Post by Petrus HyvÃ¶nen
Dear Thomas,
I would be very interested in a python 3 port of JCC. I am not a very skilled developer, looked at starting a development based on the old python-3 version but it's beyond my current skills.
I would be happy to help and test and review the JCC patches, I think your patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Dear Petrus,
the "patch" is available in this gist
https://gist.github.com/tommykoch/8b3cec18d6f3650d05d235c46ba3b055 <https://gist.github.com/tommykoch/8b3cec18d6f3650d05d235c46ba3b055>

It consists of 3 files:
- one build script (build.py)
- a patch file for jcc (pylucene_3_6-jcc-P3.patch)
- a patch file for pylucene3.6 (pylucene_3_6-P3.patch)

in our environment the 3 files are located in
<root>/tools/build/pylucene
and the output (i.e. patched Pylucene36) is created in
<root>/build

if you create this structure you may simply run
$ cd tools/build/pylucene
$ build.py build

which will checkout Pylucene36 from SVN (in <root>/build), apply the patch files and build JCC and PyLucene.
The Script is intended to run on Linux (or MacOS) - a diff binary and svn installation is required.

also easy_install(3) is needed to install the generated egg files -
besides the usual requirements for a PyLucene build (python3.4+, Java, gcc ...)

IMPORTANT NOTES:
- the patch is "one way" i.e. the patched Pylucene36 won't work with Py27 anymore!
- the patch is for the old (and outdated) PyLucene36 version only - it has not yet been tested/ported to newer versions of JCC/Pylucene
- the patched JCC has only been used/tested with PyLucene - not with any other Java library

A review of a core developer of JCC/Pylucene would also be good - Iâll ask Andi what he thinks about this suggestion.

best regards,

Thomas

Thomas Koch

2017-01-12 09:54:27 UTC

Dear Petrus,
thanks for your feedback - good to know that itâs useful for you application. And at least one more user who asks for Python3 Support in JCC ,-)

best regards,
Thomas
â

Post by Petrus HyvÃ¶nen
Dear Thomas,
Thanks for the patch files, I am not using pylucene so I have not tried that part, I am wrapping another java library (orekit). I have now built a JCC on your JCC patch applied on JCCv2.22. The test suite and application I have for my project runs fine with this patch under python 3, on mac, linux and windows, I saw no deviations in behavior of the library compared to v2.7. This is by far not a very deep test suite, but very good news.
With Best Regards
/Petrus

Post by Petrus HyvÃ¶nen
Dear Thomas,
I would be very interested in a python 3 port of JCC. I am not a very skilled developer, looked at starting a development based on the old python-3 version but it's beyond my current skills.
I would be happy to help and test and review the JCC patches, I think your patches would be a valuable contribution to JCC.
With Best Regards
/Petrus

Dear Petrus,
the "patch" is available in this gist
https://gist.github.com/tommykoch/8b3cec18d6f3650d05d235c46ba3b055 <https://gist.github.com/tommykoch/8b3cec18d6f3650d05d235c46ba3b055>
- one build script (build.py)
- a patch file for jcc (pylucene_3_6-jcc-P3.patch)
- a patch file for pylucene3.6 (pylucene_3_6-P3.patch)
in our environment the 3 files are located in
<root>/tools/build/pylucene
and the output (i.e. patched Pylucene36) is created in
<root>/build
if you create this structure you may simply run
$ cd tools/build/pylucene
$ build.py build
which will checkout Pylucene36 from SVN (in <root>/build), apply the patch files and build JCC and PyLucene.
The Script is intended to run on Linux (or MacOS) - a diff binary and svn installation is required.
also easy_install(3) is needed to install the generated egg files -
besides the usual requirements for a PyLucene build (python3.4+, Java, gcc ...)
- the patch is "one way" i.e. the patched Pylucene36 won't work with Py27 anymore!
- the patch is for the old (and outdated) PyLucene36 version only - it has not yet been tested/ported to newer versions of JCC/Pylucene
- the patched JCC has only been used/tested with PyLucene - not with any other Java library
A review of a core developer of JCC/Pylucene would also be good - Iâll ask Andi what he thinks about this suggestion.
best regards,
Thomas

Andi Vajda

2017-01-04 18:41:18 UTC

Post by marco turchi
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)

I've downloaded current version
pylucene-6.2.0-src.tar.gz 2016-09-18 09:04 30M
and was able to build JCC and PyLucene on MacOS X10.11 (64-bit, Python 2.7.10 , Java 1.8.0)
'make test' runs without any issues.
NameError: global name 'WindowsError' is not defined
I assume that one test fails to cleanup the store directory and then runs in the undefined 'WindowsError' in test_Python.py
shutil.rmtree(self.STORE_DIR)
# maybe leaking file handles in closing stores
# does not affect other tests
pass
Please look for a directory named 'testrepo‘ , then check it's file permissions and remove it (maybe you did ran the tests as root initially?) - finally run the test again.
https://issues.apache.org/jira/browse/PYLUCENE <https://issues.apache.org/jira/browse/PYLUCENE>
Note that 'WindowsError' is only available on Windows platform.

Post by marco turchi
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile gcc)

Indeed, re-reading this thread, I remember now. There is no patch attached and the tone of the contribution offer is a little off putting. It comes across more as a one time abandon-ware contribution as something with authors standing behind ready to respond to code review comments. I have a similar python 3 jcc patch sitting in an svn branch that could be revived. I've stated in the past that I intended to do so but lacked time. Interest in a Python 3 jcc has been scant so I haven't put much priority into this task.

Andi..

Post by Thomas Koch
Regards,
Thomas
--

Post by marco turchi
Dear Andi,
following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
----------------------------------------------------------------------
File "test/test_PythonException.py", line 34, in testThroughLayerException
qp.parse("foo bar")
JavaError: <super: <class 'JavaError'>, <JavaError object>>
java.lang.RuntimeException: TestException
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery_quoted(Native
Method)
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.getFieldQuery(Unknown
Source
at
org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(QueryParser.java:585)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:198)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:187)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:111)
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...
I have checked on the mailing list but I have not found solutions that work
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60
do you have an idea of what it is not working?
Thanks a lot in advance for your help!
Marco

Thomas Koch

2017-01-06 09:34:32 UTC

Dear Andi,

Iâve just sent the link to the public gist with the patch to Petrus and this list. As mentioned by Oliver weâd be more than happy if a core developer of JCC/PyLucene could review the patch and decide what to do with it. It has been developed without intimate knowledge of JCC with the goal to make PyLucene(36) usable with Python3. It may have some issues or need improvements (also cf. "IMPORTANT NOTES" in my last email about current limitations of the patch). Thatâs where export review (and effort) is needed.

For the future of course a port to newer versions of JCC/PyLucene would be more than valuable. I think what Oliver wanted to express is that we donât have that much deep know how of JCC and can thus can only provide initial efforts and contributions, but for production/release ready code an export review is still needed. Also we havenât watched the development of newer versions of PyLucene as weâre still stuck with PyLucene36.

I hope you didnât get this wrong! We all appreciate the existence of JCC/PyLucene and especially all the effort youâve put into this.

However, I fear that Python 3 support is a must-have for a Python tool or library that's available today:
- Python3 is here to stay! (py3.6 has just been released)
- Most of the popular Python packages do meanwhile provide Python3 support - cf. http://py3readiness.org <http://py3readiness.org/>
- Python2 support will end by 2020 (sounds far away but isn't - cf. https://pythonclock.org <https://pythonclock.org/> )

There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision. Without a transparent roadmap and ongoing development (i.e. new releases, Python3 support etc.) the usage of JCC/PyLucene is most likely unattractive for developers who start a new project and this is where the user base shrinks and further contributions are stalled (somehow a chicken-egg-problem).

I'm not sure how far the ASF may help here, but I've read that the Python Software Foundation occasionally funds projects to port libraries that are widely used but don't have enough of a community to do a port.
cf. https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions <https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions>

So if some funding is required to get this going ...

best regards,

Thomas
â

Post by Thomas Koch
Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered the patches needed to JCC and PyLucene for use/review on the list - but didn't get any feedback so far.

Jan Høydahl

2017-01-06 11:32:47 UTC

Hi,

Post by Thomas Koch
I hope you didnât get this wrong! We all appreciate the existence of JCC/PyLucene and especially all the effort youâve put into this.

PyLucene is driven by its own community, and user involvement and contributions is a must.
The (sub)project will survive only to the extent that its current users invest in it.

Post by Thomas Koch
So if some funding is required to get this going âŠ

For an ASF Open Source Project, the only thing that is required to get going is user/developer
involvement and teamwork. While Andi started the project due to needs at the time, and became
a committer, he is no longer an active user, so perhaps time has come for other users to step ut and take
responsibility.

How âfundingâ would look like in the Python3 case is not so much sending money to the ASF,
but more for individual companies like your own, to sponsor (through developer time) the major
work on the patch, and driving it through to completion. Hopefully other users will contribute along
the way too.

You will of course need help from experienced developers, but the ideal situation is that after
a couple of such patches that get committed, you (or the developer working on the code) will be nominated
as committer and can continue developing PyLucene without the need for Andi or any other one individual.

Post by Thomas Koch
There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision

The discussion sparked some new development and a release, which is a success. So the decission I guess is to keep PyLucene alive and try to strengthen the community.
As long as the project continues to produce releases, it is (somewhat) alive.
If on the other hand another year or two goes by without another release, Iâm sure the PMC will take action again.

--
Jan HÃžydahl, search solution architect
Cominvent AS - www.cominvent.com

Post by Thomas Koch
Dear Andi,
Iâve just sent the link to the public gist with the patch to Petrus and this list. As mentioned by Oliver weâd be more than happy if a core developer of JCC/PyLucene could review the patch and decide what to do with it. It has been developed without intimate knowledge of JCC with the goal to make PyLucene(36) usable with Python3. It may have some issues or need improvements (also cf. "IMPORTANT NOTES" in my last email about current limitations of the patch). Thatâs where export review (and effort) is needed.
For the future of course a port to newer versions of JCC/PyLucene would be more than valuable. I think what Oliver wanted to express is that we donât have that much deep know how of JCC and can thus can only provide initial efforts and contributions, but for production/release ready code an export review is still needed. Also we havenât watched the development of newer versions of PyLucene as weâre still stuck with PyLucene36.
I hope you didnât get this wrong! We all appreciate the existence of JCC/PyLucene and especially all the effort youâve put into this.
- Python3 is here to stay! (py3.6 has just been released)
- Most of the popular Python packages do meanwhile provide Python3 support - cf. http://py3readiness.org <http://py3readiness.org/>
- Python2 support will end by 2020 (sounds far away but isn't - cf. https://pythonclock.org <https://pythonclock.org/> )
There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision. Without a transparent roadmap and ongoing development (i.e. new releases, Python3 support etc.) the usage of JCC/PyLucene is most likely unattractive for developers who start a new project and this is where the user base shrinks and further contributions are stalled (somehow a chicken-egg-problem).
I'm not sure how far the ASF may help here, but I've read that the Python Software Foundation occasionally funds projects to port libraries that are widely used but don't have enough of a community to do a port.
cf. https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions <https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions>
So if some funding is required to get this going ...
best regards,
Thomas
â

Andi Vajda

2017-01-06 21:02:16 UTC

I hope you didn?t get this wrong! We all appreciate the existence of
JCC/PyLucene and especially all the effort you?ve put into this.

PyLucene is driven by its own community, and user involvement and contributions is a must.
The (sub)project will survive only to the extent that its current users invest in it.

So if some funding is required to get this going ?

For an ASF Open Source Project, the only thing that is required to get going is user/developer
involvement and teamwork. While Andi started the project due to needs at the time, and became
a committer, he is no longer an active user, so perhaps time has come for other users to step ut and take
responsibility.
How ?funding? would look like in the Python3 case is not so much sending money to the ASF,
but more for individual companies like your own, to sponsor (through developer time) the major
work on the patch, and driving it through to completion. Hopefully other users will contribute along
the way too.
You will of course need help from experienced developers, but the ideal situation is that after
a couple of such patches that get committed, you (or the developer working on the code) will be nominated
as committer and can continue developing PyLucene without the need for Andi or any other one individual.

There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision

I intend to produce a PyLucene 6.4 release once Lucene 6.4 is done.
It's been a few months now...

Andi..

Post by Jan HÃ¸ydahl
--
Jan HÃžydahl, search solution architect
Cominvent AS - www.cominvent.com

Dear Andi,
I?ve just sent the link to the public gist with the patch to Petrus and this list. As mentioned by Oliver we?d be more than happy if a core developer of JCC/PyLucene could review the patch and decide what to do with it. It has been developed without intimate knowledge of JCC with the goal to make PyLucene(36) usable with Python3. It may have some issues or need improvements (also cf. "IMPORTANT NOTES" in my last email about current limitations of the patch). That?s where export review (and effort) is needed.
For the future of course a port to newer versions of JCC/PyLucene would be more than valuable. I think what Oliver wanted to express is that we don?t have that much deep know how of JCC and can thus can only provide initial efforts and contributions, but for production/release ready code an export review is still needed. Also we haven?t watched the development of newer versions of PyLucene as we?re still stuck with PyLucene36.
I hope you didn?t get this wrong! We all appreciate the existence of JCC/PyLucene and especially all the effort you?ve put into this.
- Python3 is here to stay! (py3.6 has just been released)
- Most of the popular Python packages do meanwhile provide Python3 support - cf. http://py3readiness.org <http://py3readiness.org/>
- Python2 support will end by 2020 (sounds far away but isn't - cf. https://pythonclock.org <https://pythonclock.org/> )
There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision. Without a transparent roadmap and ongoing development (i.e. new releases, Python3 support etc.) the usage of JCC/PyLucene is most likely unattractive for developers who start a new project and this is where the user base shrinks and further contributions are stalled (somehow a chicken-egg-problem).
I'm not sure how far the ASF may help here, but I've read that the Python Software Foundation occasionally funds projects to port libraries that are widely used but don't have enough of a community to do a port.
cf. https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions <https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions>
So if some funding is required to get this going ...
best regards,
Thomas
?

Andi Vajda

2017-01-26 23:03:47 UTC

I hope you didn?t get this wrong! We all appreciate the existence of
JCC/PyLucene and especially all the effort you?ve put into this.

PyLucene is driven by its own community, and user involvement and
contributions is a must.
The (sub)project will survive only to the extent that its current users invest in it.

So if some funding is required to get this going ?

For an ASF Open Source Project, the only thing that is required to get
going is user/developer
involvement and teamwork. While Andi started the project due to needs at
the time, and became
a committer, he is no longer an active user, so perhaps time has come for
other users to step ut and take
responsibility.
How ?funding? would look like in the Python3 case is not so much sending
money to the ASF,
but more for individual companies like your own, to sponsor (through
developer time) the major
work on the patch, and driving it through to completion. Hopefully other
users will contribute along
the way too.
You will of course need help from experienced developers, but the ideal
situation is that after
a couple of such patches that get committed, you (or the developer working
on the code) will be nominated
as committer and can continue developing PyLucene without the need for Andi
or any other one individual.

There has been some discussions about the future of PyLucene on this list
but I still didn't see any conclusion/decision

The discussion sparked some new development and a release, which is a
success. So the decission I guess is to keep PyLucene alive and try to
strengthen the community.
As long as the project continues to produce releases, it is (somewhat) alive.
If on the other hand another year or two goes by without another release,
I?m sure the PMC will take action again.

I intend to produce a PyLucene 6.4 release once Lucene 6.4 is done.
It's been a few months now...

Lucene 6.4.0 was released on Monday. I did a quick test build of PyLucene
with that code and it required no changes at all. Given that a Lucene 6.4.1
bugfix release is around the corner, I'll wait until it is released to send
a PyLucene 6.4.1 release candidate for vote.

Andi..

Post by Andi Vajda
Andi..

Post by Jan HÃ¸ydahl
--
Jan HC8ydahl, search solution architect
Cominvent AS - www.cominvent.com

Dear Andi,
I?ve just sent the link to the public gist with the patch to Petrus and
this list. As mentioned by Oliver we?d be more than happy if a core
developer of JCC/PyLucene could review the patch and decide what to do
with it. It has been developed without intimate knowledge of JCC with the
goal to make PyLucene(36) usable with Python3. It may have some issues or
need improvements (also cf. "IMPORTANT NOTES" in my last email about
current limitations of the patch). That?s where export review (and effort)
is needed.
For the future of course a port to newer versions of JCC/PyLucene would be
more than valuable. I think what Oliver wanted to express is that we don?t
have that much deep know how of JCC and can thus can only provide initial
efforts and contributions, but for production/release ready code an export
review is still needed. Also we haven?t watched the development of newer
versions of PyLucene as we?re still stuck with PyLucene36.
I hope you didn?t get this wrong! We all appreciate the existence of
JCC/PyLucene and especially all the effort you?ve put into this.
However, I fear that Python 3 support is a must-have for a Python tool or
- Python3 is here to stay! (py3.6 has just been released)
- Most of the popular Python packages do meanwhile provide Python3 support
- cf. http://py3readiness.org <http://py3readiness.org/>
- Python2 support will end by 2020 (sounds far away but isn't - cf.
https://pythonclock.org <https://pythonclock.org/> )
There has been some discussions about the future of PyLucene on this list
but I still didn't see any conclusion/decision. Without a transparent
roadmap and ongoing development (i.e. new releases, Python3 support etc.)
the usage of JCC/PyLucene is most likely unattractive for developers who
start a new project and this is where the user base shrinks and further
contributions are stalled (somehow a chicken-egg-problem).
I'm not sure how far the ASF may help here, but I've read that the Python
Software Foundation occasionally funds projects to port libraries that are
widely used but don't have enough of a community to do a port.
cf.
https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions
<https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions>
So if some funding is required to get this going ...
best regards,
Thomas
?

Post by Thomas Koch
Note that PyLucene currently lacks official Python3 support!
We've done a port of PyLucene 3.6 (!) to support Python3 and offered the
patches needed to JCC and PyLucene for use/review on the list - but
didn't get any feedback so far.
cf.

Indeed, re-reading this thread, I remember now. There is no patch
attached and the tone of the contribution offer is a little off putting.
It comes across more as a one time abandon-ware contribution as something
with authors standing behind ready to respond to code review comments. I
have a similar python 3 jcc patch sitting in an svn branch that could be
revived. I've stated in the past that I intended to do so but lacked
time. Interest in a Python 3 jcc has been scant so I haven't put much
priority into this task.
Andi..

Thomas Koch

2017-01-12 10:22:05 UTC

Dear Jan,
thanks for your explanations.

Post by Jan HÃ¸ydahl
PyLucene is driven by its own community, and user involvement and contributions is a must.

Iâm quite familiar with community contributions - have already submitted code and feedback, tested Pylucene releases, did provide support to other users and voted several times for new releases.

Post by Jan HÃ¸ydahl
The (sub)project will survive only to the extent that its current users invest in it.

so is this different to how the main (Java)Lucene project works? just curious âŠ

Post by Jan HÃ¸ydahl
For an ASF Open Source Project, the only thing that is required to get going is user/developer
involvement and teamwork. While Andi started the project due to needs at the time, and became
a committer, he is no longer an active user, so perhaps time has come for other users to step ut and take
responsibility.

Thatâs first time I hear this (bad) news. So if Andi is no longer an active user - who is the maintainer of PyLucene/JCC?

Post by Jan HÃ¸ydahl
How âfundingâ would look like in the Python3 case is not so much sending money to the ASF,
but more for individual companies like your own, to sponsor (through developer time) the major
work on the patch, and driving it through to completion. Hopefully other users will contribute along
the way too.

We developed and provided the patch (which already took some time) for review and further adaption and really hope now for other users to step in. This is our current position for several reasons an after internal discussions which I cannot disclose here. Sorry. On the other hand, if weâd be the only user(s) interested/willing to push Python3 support (and Pylucene/JCC as a whole) then this project could not survive anyway I fear.

best regards,

Thomas
â

Post by Thomas Koch
I hope you didnât get this wrong! We all appreciate the existence of JCC/PyLucene and especially all the effort youâve put into this.

PyLucene is driven by its own community, and user involvement and contributions is a must.
The (sub)project will survive only to the extent that its current users invest in it.

Post by Thomas Koch
So if some funding is required to get this going âŠ

For an ASF Open Source Project, the only thing that is required to get going is user/developer
involvement and teamwork. While Andi started the project due to needs at the time, and became
a committer, he is no longer an active user, so perhaps time has come for other users to step ut and take
responsibility.
How âfundingâ would look like in the Python3 case is not so much sending money to the ASF,
but more for individual companies like your own, to sponsor (through developer time) the major
work on the patch, and driving it through to completion. Hopefully other users will contribute along
the way too.
You will of course need help from experienced developers, but the ideal situation is that after
a couple of such patches that get committed, you (or the developer working on the code) will be nominated
as committer and can continue developing PyLucene without the need for Andi or any other one individual.

Post by Thomas Koch
There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision

Jan Høydahl

2017-01-12 11:47:18 UTC

Post by Jan HÃ¸ydahl
The (sub)project will survive only to the extent that its current users invest in it.

so is this different to how the main (Java)Lucene project works? just curious …

This is the case for all ASF projects and sub projects. If a project is unable to produce
new releases, usually due to too few developers, or loss of interest, or lack of users,
the project will end up in the Attic. For a TLP it would be the Board intervening, while
for sub projects I believe it is the TLPs responsibility to make sure things are healthy.

That’s first time I hear this (bad) news. So if Andi is no longer an active user - who is the maintainer of PyLucene/JCC?

I may have misunderstood myself, but it is quite normal for a commiter’s availability to vary over the years,
and for that reason it would be a big benefit for PyLucene to have at least two other active devs voted in
as committers. As I understand it, PyLucene does not require much effort to keep up to date, except when
there are major changes such as Py3 or porting tests etc.

I’m not a user myself, so I’m a bit on the sideline here, being on the PMC.

We developed and provided the patch (which already took some time) for review and further adaption and really hope now for other users to step in. This is our current position for several reasons an after internal discussions which I cannot disclose here. Sorry. On the other hand, if we’d be the only user(s) interested/willing to push Python3 support (and Pylucene/JCC as a whole) then this project could not survive anyway I fear.

Absolutely, you have done “your share” and more, and that is what such a project needs, and right
now PyLucene needs more than one guy who can jump in and make Py3 happen.
I guess what I was trying to get across is that if Andi will forever be the one guy everyone depend on
to get the smallest contribution landed in the codebase, it will not be sustainable going forward, and
probably not ideal for Andi nor the other users.

So what would it take to bring one other developer up to a “pro" level, and are there anyone who have such a desire?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Andi Vajda

2017-01-12 16:37:45 UTC

Post by Jan HÃ¸ydahl

Post by Jan HÃ¸ydahl
The (sub)project will survive only to the extent that its current users invest in it.

so is this different to how the main (Java)Lucene project works? just curious …

That’s first time I hear this (bad) news. So if Andi is no longer an active user - who is the maintainer of PyLucene/JCC?

I may have misunderstood myself, but it is quite normal for a commiter’s availability to vary over the years,
and for that reason it would be a big benefit for PyLucene to have at least two other active devs voted in
as committers. As I understand it, PyLucene does not require much effort to keep up to date, except when
there are major changes such as Py3 or porting tests etc.
I’m not a user myself, so I’m a bit on the sideline here, being on the PMC.

We developed and provided the patch (which already took some time) for review and further adaption and really hope now for other users to step in. This is our current position for several reasons an after internal discussions which I cannot disclose here. Sorry. On the other hand, if we’d be the only user(s) interested/willing to push Python3 support (and Pylucene/JCC as a whole) then this project could not survive anyway I fear.

Absolutely, you have done “your share” and more, and that is what such a project needs, and right
now PyLucene needs more than one guy who can jump in and make Py3 happen.
I guess what I was trying to get across is that if Andi will forever be the one guy everyone depend on
to get the smallest contribution landed in the codebase, it will not be sustainable going forward, and
probably not ideal for Andi nor the other users.
So what would it take to bring one other developer up to a “pro" level, and are there anyone who have such a desire?

Just as on the Lucene project: repeated sizable contributions, an understanding on how projects function in the Apache community, a commitment to work within it and a successful committership vote.

If someone were to donate time to make the JCC 3 patch work with trunk and submitted that and showed interest in the project in general, over time, not just doing a one time code drop, they'd be well qualified to become a committer.

Andi..

Post by Jan HÃ¸ydahl
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Andi Vajda

2017-01-12 16:15:44 UTC

Post by Thomas Koch
Dear Jan,
thanks for your explanations.

Post by Jan HÃ¸ydahl
PyLucene is driven by its own community, and user involvement and contributions is a must.

I’m quite familiar with community contributions - have already submitted code and feedback, tested Pylucene releases, did provide support to other users and voted several times for new releases.

Post by Jan HÃ¸ydahl
The (sub)project will survive only to the extent that its current users invest in it.

so is this different to how the main (Java)Lucene project works? just curious …

That’s first time I hear this (bad) news. So if Andi is no longer an active user - who is the maintainer of PyLucene/JCC?

I am still the maintainer of this project. I am no longer using Lucene or PyLucene on a daily basis since 2011. I'm still happy to continue maintaining the project, though.

Andi..

Post by Jan HÃ¸ydahl
How “funding” would look like in the Python3 case is not so much sending money to the ASF,
but more for individual companies like your own, to sponsor (through developer time) the major
work on the patch, and driving it through to completion. Hopefully other users will contribute along
the way too.

We developed and provided the patch (which already took some time) for review and further adaption and really hope now for other users to step in. This is our current position for several reasons an after internal discussions which I cannot disclose here. Sorry. On the other hand, if we’d be the only user(s) interested/willing to push Python3 support (and Pylucene/JCC as a whole) then this project could not survive anyway I fear.
best regards,
Thomas
—

I hope you didn’t get this wrong! We all appreciate the existence of JCC/PyLucene and especially all the effort you’ve put into this.

PyLucene is driven by its own community, and user involvement and contributions is a must.
The (sub)project will survive only to the extent that its current users invest in it.

So if some funding is required to get this going …

For an ASF Open Source Project, the only thing that is required to get going is user/developer
involvement and teamwork. While Andi started the project due to needs at the time, and became
a committer, he is no longer an active user, so perhaps time has come for other users to step ut and take
responsibility.
How “funding” would look like in the Python3 case is not so much sending money to the ASF,
but more for individual companies like your own, to sponsor (through developer time) the major
work on the patch, and driving it through to completion. Hopefully other users will contribute along
the way too.
You will of course need help from experienced developers, but the ideal situation is that after
a couple of such patches that get committed, you (or the developer working on the code) will be nominated
as committer and can continue developing PyLucene without the need for Andi or any other one individual.

There has been some discussions about the future of PyLucene on this list but I still didn't see any conclusion/decision

Andi Vajda

2017-01-06 21:01:13 UTC

Hi Thomas,

I?ve just sent the link to the public gist with the patch to Petrus and
this list. As mentioned by Oliver we?d be more than happy if a core
developer of JCC/PyLucene could review the patch and decide what to do
with it. It has been developed without intimate knowledge of JCC with the
goal to make PyLucene(36) usable with Python3. It may have some issues or
need improvements (also cf. "IMPORTANT NOTES" in my last email about
current limitations of the patch). That?s where export review (and effort)
is needed.
For the future of course a port to newer versions of JCC/PyLucene would be
more than valuable. I think what Oliver wanted to express is that we don?t
have that much deep know how of JCC and can thus can only provide initial
efforts and contributions, but for production/release ready code an export
review is still needed. Also we haven?t watched the development of newer
versions of PyLucene as we?re still stuck with PyLucene36.
I hope you didn?t get this wrong! We all appreciate the existence of
JCC/PyLucene and especially all the effort you?ve put into this.
- Python3 is here to stay! (py3.6 has just been released)
- Most of the popular Python packages do meanwhile provide Python3 support - cf. http://py3readiness.org <http://py3readiness.org/>
- Python2 support will end by 2020 (sounds far away but isn't - cf. https://pythonclock.org <https://pythonclock.org/> )
There has been some discussions about the future of PyLucene on this list
but I still didn't see any conclusion/decision. Without a transparent
roadmap and ongoing development (i.e. new releases, Python3 support etc.)
the usage of JCC/PyLucene is most likely unattractive for developers who
start a new project and this is where the user base shrinks and further
contributions are stalled (somehow a chicken-egg-problem).
I'm not sure how far the ASF may help here, but I've read that the Python Software Foundation occasionally funds projects to port libraries that are widely used but don't have enough of a community to do a port.
cf. https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions <https://developers.slashdot.org/story/13/08/25/2115204/interviews-guido-van-rossum-answers-your-questions>
So if some funding is required to get this going ...

I now took a look at the python 3 patches you sent a link to in an earlier
message and here is the gist of my thoughts:
- Moving the Python 3 is desirable but what about Python 2 support today
in 2017 ? I have no desire to support both for PyLucene manually. If,
somehow, there can be two versions of JCC, one for Python 2, one for
Python 3 and the PyLucene tests can be 2to3'd automatically, then the
Python 3 support idea looks more attractive already. Supporting two
versions of JCC is fine until 2020.

- The JCC patches look very reasonable but should be updated to the latest
Python 3. In particular, the internal Python 3 string representation was
changed again after 3.2 (?) and has clever optimizations possible based
on the internal byte size of characters chosen by Python (internally)
for each string, based on the range of the characters used in the string.
This makes it possible to often just copy chars from Python to Java.
I just did a rewrite for this in PyICU (another long
term project of mine, https://github.com/ovalhub/pyicu/) and the Python 3
string story got much cleaner post 3.2 (at least more
understandable). Lots of bugs with long unicode chars (forgot the proper
term, sorry) got fixed along the way (emoticon support, yay).

So, if you're prepared to fund this effort, it might be best to hire
back the contractor who did the JCC Python 3 port originally and have
him/her refresh it for the latest JCC on trunk (not too many changes
happened in the past few years) and to the use the Python internal string
APIs that appeared post Python 3.2. The ones in use in the patch are
deprecated already. I love it that we'd then shed _all_ backwards
compatibility baggage in JCC going forward in Python 3.x, x >= 6.

If you get the JCC/Python3 patches into a shape where I can apply them to
trunk without trouble and using the latest CPython string APIs:
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUCS4
and related (PyUnicode_KIND, etc...)
then there is a good chance that PyLucene/JCC would be fully supported
with Python 3.x, x >= 6.

- The PyLucene patches should probably be redone so that they can be
automated with 2to3. If we get JCC in shape, I can take care of the rest.

Thank you for the work done so far, it's looking really good but it needs to
be refreshed to JCC/trunk and latest Python 3 to minimize work on my side.

Andi..

best regards,
Thomas
?

Thomas Koch

2017-01-12 10:37:58 UTC

Dear Andi,

many thanks for your review of the patch and helpful comments.

As mentioned in my previous mail to the list Iâm afraid to let you know that we currently cannot put more effort into this task. This may change in the future of course.

However, with âfundingâ my idea was to look for open source funding such as provided by the Python Software Foundation (e.g.):
https://www.python.org/psf/grants

Thatâs probably worth a try - especially because the PSF is well known, has some sponsors and also receives donations by companies using Python and Python related projects on regular basisâŠ

Post by Andi Vajda
Thank you for the work done so far, it's looking really good but it needs to
be refreshed to JCC/trunk and latest Python 3 to minimize work on my side.

Ok. Maybe ASF could contact PSF and ask for such a grant that could be uses to

Post by Andi Vajda
hire back the contractor who did the JCC Python 3 port originally and have
him/her refresh it for the latest JCC on trunk

? (If he/she is willing to do so âŠ thatâs another question).

Just thinking about ways to get this forward. Iâm afraid I cannot provide more support currently ,-(

best regards,

Thomas
â

Post by Andi Vajda
- Moving the Python 3 is desirable but what about Python 2 support today
in 2017 ? I have no desire to support both for PyLucene manually. If,
somehow, there can be two versions of JCC, one for Python 2, one for
Python 3 and the PyLucene tests can be 2to3'd automatically, then the
Python 3 support idea looks more attractive already. Supporting two
versions of JCC is fine until 2020.
- The JCC patches look very reasonable but should be updated to the latest
Python 3. In particular, the internal Python 3 string representation was
changed again after 3.2 (?) and has clever optimizations possible based
on the internal byte size of characters chosen by Python (internally)
for each string, based on the range of the characters used in the string.
This makes it possible to often just copy chars from Python to Java.
I just did a rewrite for this in PyICU (another long
term project of mine, https://github.com/ovalhub/pyicu/ <https://github.com/ovalhub/pyicu/>) and the Python 3
string story got much cleaner post 3.2 (at least more
understandable). Lots of bugs with long unicode chars (forgot the proper
term, sorry) got fixed along the way (emoticon support, yay).
So, if you're prepared to fund this effort, it might be best to hire
back the contractor who did the JCC Python 3 port originally and have
him/her refresh it for the latest JCC on trunk (not too many changes
happened in the past few years) and to the use the Python internal string
APIs that appeared post Python 3.2. The ones in use in the patch are
deprecated already. I love it that we'd then shed _all_ backwards
compatibility baggage in JCC going forward in Python 3.x, x >= 6.
If you get the JCC/Python3 patches into a shape where I can apply them to
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUCS4 <https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUCS4>
and related (PyUnicode_KIND, etc...)
then there is a good chance that PyLucene/JCC would be fully supported
with Python 3.x, x >= 6.
- The PyLucene patches should probably be redone so that they can be
automated with 2to3. If we get JCC in shape, I can take care of the rest.
Thank you for the work done so far, it's looking really good but it needs to
be refreshed to JCC/trunk and latest Python 3 to minimize work on my side.
Andi..

Andi Vajda

2017-01-12 16:31:20 UTC

Post by marco turchi
Dear Andi,
many thanks for your review of the patch and helpful comments.
As mentioned in my previous mail to the list Iâm afraid to let you know that we currently cannot put more effort into this task. This may change in the future of course.

That's ok. This is also how I feel.

Post by marco turchi
https://www.python.org/psf/grants
Thatâs probably worth a try - especially because the PSF is well known, has some sponsors and also receives donations by companies using Python and Python related projects on regular basisâŠ

For me, it's not a problem of money but of time.
Whether I hire someone to do the work or do it myself, the same amount of time has to be spent dealing with the issue. Hiring someone and finding money takes time too. Probably more than integrating your patch, actually.

The way open source works is usually by donation of time, not money or code. This is why your code donation is going to sit there until someone spends the _time_ to integrate it.