JCC SimilarityBase

[Peter Brown]

Hi all,

I was just wondering if anybody else has run into a similar issue before.
I’m trying to create a wrapper class for the org.apache.lucene.search.similarities.SimilarityBase class within PyLucene (similar to the existing wrapper for PythonClassicSimilarity).

However, upon re-building and re-installing PyLucene, when trying to use the wrapper class it fails with the following exception:

java.lang.UnsatisfiedLinkError: org.apache.pylucene.search.similarities.PythonSimilarityBase.score(Lorg/apache/lucene/search/similarities/BasicStats;FF)F

Upon examination of the wrap0*.cpp files, and comparing with PythonClassicSimilarity, I found that the score method is not being generated. E.g.

static PyMethodDef t_PythonSimilarityBase__methods_[] = {
DECLARE_METHOD(t_PythonSimilarityBase, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonSimilarityBase, pythonExtension, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};

and

JNINativeMethod methods[] = {
{ "pythonDecRef", "()V", (void *) t_PythonSimilarityBase_pythonDecRef0 },
{ "toString", "()Ljava/lang/String;", (void *) t_PythonSimilarityBase_toString1 },
};

But it is present in the header file enum

class PythonSimilarityBase : public ::org::apache::lucene::search::similarities::SimilarityBase {
public:
enum {
mid_init$_7353aa4eb69a6f70,
mid_finalize_7353aa4eb69a6f70,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_toString_5df7257cf25bf439,
mid_score_a6c8466c4349b376,
max_mid
};

But not in the header file methods.
PythonSimilarityBase();

void finalize() const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
::java::lang::String toString() const;


Compared to
static PyMethodDef t_PythonClassicSimilarity__methods_[] = {
DECLARE_METHOD(t_PythonClassicSimilarity, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, coord, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idf, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idfExplain, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, lengthNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, pythonExtension, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, queryNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, sloppyFreq, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, tf, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};

and

JNINativeMethod methods[] = {
{ "coord", "(II)F", (void *) t_PythonClassicSimilarity_coord0 },
{ "idf", "(JJ)F", (void *) t_PythonClassicSimilarity_idf1 },
{ "idfExplain", "(Lorg/apache/lucene/search/CollectionStatistics;[Lorg/apache/lucene/search/TermStatistics;)Lorg/apache/lucene/search/Explanation;", (void *) t_PythonClassicSimilarity_idfExplain2 },
{ "lengthNorm", "(Lorg/apache/lucene/index/FieldInvertState;)F", (void *) t_PythonClassicSimilarity_lengthNorm3 },
{ "pythonDecRef", "()V", (void *) t_PythonClassicSimilarity_pythonDecRef4 },
{ "queryNorm", "(F)F", (void *) t_PythonClassicSimilarity_queryNorm5 },
{ "sloppyFreq", "(I)F", (void *) t_PythonClassicSimilarity_sloppyFreq6 },
{ "tf", "(F)F", (void *) t_PythonClassicSimilarity_tf7 },
};


enum {
mid_init$_7353aa4eb69a6f70,
mid_coord_78a3b70e2a43b9c7,
mid_finalize_7353aa4eb69a6f70,
mid_idf_0af4b58f982c0dd1,
mid_idfExplain_08e4e4733921c6f1,
mid_lengthNorm_a7d67c4cd2e3b6a2,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_queryNorm_eecff9f314592ce9,
mid_sloppyFreq_6b1faadec3ed3278,
mid_tf_eecff9f314592ce9,
max_mid
};

and

jfloat coord(jint, jint) const;
void finalize() const;
jfloat idf(jlong, jlong) const;
::org::apache::lucene::search::Explanation idfExplain(const ::org::apache::lucene::search::CollectionStatistics &, const JArray< ::org::apache::lucene::search::TermStatistics > &) const;
jfloat lengthNorm(const ::org::apache::lucene::index::FieldInvertState &) const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
jfloat queryNorm(jfloat) const;
jfloat sloppyFreq(jint) const;
jfloat tf(jfloat) const;

Here is the code for PythonSimilarityBase.java (stored in the same directory as PythonClassicSimilarity) which compiles without issues:
package org.apache.pylucene.search.similarities;

import org.apache.lucene.search.similarities.BasicStats;
import org.apache.lucene.search.similarities.SimilarityBase;



public class PythonSimilarityBase extends SimilarityBase {

private long pythonObject;

public PythonSimilarityBase()
{
}

public void pythonExtension(long pythonObject)
{
this.pythonObject = pythonObject;
}
public long pythonExtension()
{
return this.pythonObject;
}

public void finalize()
throws Throwable
{
pythonDecRef();
}

public native void pythonDecRef();


@Override
protected native float score(BasicStats stats, float freq, float docLen);

@Override
public native String toString();
}


I’ve also tried variations of adding the BasicStats class, the Similarity$SimWeight class and the whole similarities package into the JCC command, for example
JCC=$(PYTHON) -m jcc.__main__ --shared --arch x86_64 --package org.apache.lucene.search.similarities
(Trying to follow advice in https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21>)


My feeling is that it has something to do with that BasicStats object



If anyone is able to assist it would be greatly appreciated!!

Thanks,
Peter Brown
School of Information and Communication Technology,
Griffith University - Gold Coast Campus

[Andi Vajda]

Post by Peter Brown
Hi all,
I was just wondering if anybody else has run into a similar issue before.
I?m trying to create a wrapper class for the org.apache.lucene.search.similarities.SimilarityBase class within PyLucene (similar to the existing wrapper for PythonClassicSimilarity).
java.lang.UnsatisfiedLinkError: org.apache.pylucene.search.similarities.PythonSimilarityBase.score(Lorg/apache/lucene/search/similarities/BasicStats;FF)F
Upon examination of the wrap0*.cpp files, and comparing with PythonClassicSimilarity, I found that the score method is not being generated. E.g.
static PyMethodDef t_PythonSimilarityBase__methods_[] = {
DECLARE_METHOD(t_PythonSimilarityBase, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonSimilarityBase, pythonExtension, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};
and
JNINativeMethod methods[] = {
{ "pythonDecRef", "()V", (void *) t_PythonSimilarityBase_pythonDecRef0 },
{ "toString", "()Ljava/lang/String;", (void *) t_PythonSimilarityBase_toString1 },
};
But it is present in the header file enum?
class PythonSimilarityBase : public ::org::apache::lucene::search::similarities::SimilarityBase {
enum {
mid_init$_7353aa4eb69a6f70,
mid_finalize_7353aa4eb69a6f70,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_toString_5df7257cf25bf439,
mid_score_a6c8466c4349b376,
max_mid
};
But not in the header file methods.
PythonSimilarityBase();
void finalize() const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
::java::lang::String toString() const;
Compared to
static PyMethodDef t_PythonClassicSimilarity__methods_[] = {
DECLARE_METHOD(t_PythonClassicSimilarity, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, coord, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idf, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idfExplain, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, lengthNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, pythonExtension, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, queryNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, sloppyFreq, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, tf, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};
and
JNINativeMethod methods[] = {
{ "coord", "(II)F", (void *) t_PythonClassicSimilarity_coord0 },
{ "idf", "(JJ)F", (void *) t_PythonClassicSimilarity_idf1 },
{ "idfExplain", "(Lorg/apache/lucene/search/CollectionStatistics;[Lorg/apache/lucene/search/TermStatistics;)Lorg/apache/lucene/search/Explanation;", (void *) t_PythonClassicSimilarity_idfExplain2 },
{ "lengthNorm", "(Lorg/apache/lucene/index/FieldInvertState;)F", (void *) t_PythonClassicSimilarity_lengthNorm3 },
{ "pythonDecRef", "()V", (void *) t_PythonClassicSimilarity_pythonDecRef4 },
{ "queryNorm", "(F)F", (void *) t_PythonClassicSimilarity_queryNorm5 },
{ "sloppyFreq", "(I)F", (void *) t_PythonClassicSimilarity_sloppyFreq6 },
{ "tf", "(F)F", (void *) t_PythonClassicSimilarity_tf7 },
};
enum {
mid_init$_7353aa4eb69a6f70,
mid_coord_78a3b70e2a43b9c7,
mid_finalize_7353aa4eb69a6f70,
mid_idf_0af4b58f982c0dd1,
mid_idfExplain_08e4e4733921c6f1,
mid_lengthNorm_a7d67c4cd2e3b6a2,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_queryNorm_eecff9f314592ce9,
mid_sloppyFreq_6b1faadec3ed3278,
mid_tf_eecff9f314592ce9,
max_mid
};
and
jfloat coord(jint, jint) const;
void finalize() const;
jfloat idf(jlong, jlong) const;
::org::apache::lucene::search::Explanation idfExplain(const ::org::apache::lucene::search::CollectionStatistics &, const JArray< ::org::apache::lucene::search::TermStatistics > &) const;
jfloat lengthNorm(const ::org::apache::lucene::index::FieldInvertState &) const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
jfloat queryNorm(jfloat) const;
jfloat sloppyFreq(jint) const;
jfloat tf(jfloat) const;
package org.apache.pylucene.search.similarities;
import org.apache.lucene.search.similarities.BasicStats;
import org.apache.lucene.search.similarities.SimilarityBase;
public class PythonSimilarityBase extends SimilarityBase {
private long pythonObject;
public PythonSimilarityBase()
{
}
public void pythonExtension(long pythonObject)
{
this.pythonObject = pythonObject;
}
public long pythonExtension()
{
return this.pythonObject;
}
public void finalize()
throws Throwable
{
pythonDecRef();
}
public native void pythonDecRef();
@Override
protected native float score(BasicStats stats, float freq, float docLen);
@Override
public native String toString();
}
I?ve also tried variations of adding the BasicStats class, the Similarity$SimWeight class and the whole similarities package into the JCC command, for example
JCC=$(PYTHON) -m jcc.__main__ --shared --arch x86_64 --package org.apache.lucene.search.similarities
(Trying to follow advice in https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21>)
My feeling is that it has something to do with that BasicStats object?
If anyone is able to assist it would be greatly appreciated!!

The 'score' method you're trying to get wrapped is marked 'protected' in
the source code you showed above.

By default, JCC only wraps public methods and fields.

You can either make it public or force a wrapper for it by explicitely
listing it on the JCC command line.

At line 326 in the pylucene Makefile you can see two such non public things
being force-wrapped already via listing:
org.apache.lucene.index.IndexWriter:getReader \
org.apache.lucene.analysis.Tokenizer:input \
Just add your score method to the list there...

Andi..

Post by Peter Brown
Thanks,
Peter Brown
School of Information and Communication Technology,
Griffith University - Gold Coast Campus

[Peter Brown]

Andi, you are a champion! :)
Thanks so much for your timely and helpful response.

Unfortunately, I had no luck adding the score function to the makefile.
However, I was able to resolve the issue by defining the method as public.


Peter Brown - s2623892
School of Information and Communication Technology,
Griffith University - Gold Coast Campus

Post by Peter Brown
Hi all,
I was just wondering if anybody else has run into a similar issue before.
I?m trying to create a wrapper class for the org.apache.lucene.search.similarities.SimilarityBase class within PyLucene (similar to the existing wrapper for PythonClassicSimilarity).
java.lang.UnsatisfiedLinkError: org.apache.pylucene.search.similarities.PythonSimilarityBase.score(Lorg/apache/lucene/search/similarities/BasicStats;FF)F
Upon examination of the wrap0*.cpp files, and comparing with PythonClassicSimilarity, I found that the score method is not being generated. E.g.
static PyMethodDef t_PythonSimilarityBase__methods_[] = {
DECLARE_METHOD(t_PythonSimilarityBase, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonSimilarityBase, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonSimilarityBase, pythonExtension, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};
and
JNINativeMethod methods[] = {
{ "pythonDecRef", "()V", (void *) t_PythonSimilarityBase_pythonDecRef0 },
{ "toString", "()Ljava/lang/String;", (void *) t_PythonSimilarityBase_toString1 },
};
But it is present in the header file enum?
class PythonSimilarityBase : public ::org::apache::lucene::search::similarities::SimilarityBase {
enum {
mid_init$_7353aa4eb69a6f70,
mid_finalize_7353aa4eb69a6f70,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_toString_5df7257cf25bf439,
mid_score_a6c8466c4349b376,
max_mid
};
But not in the header file methods.
PythonSimilarityBase();
void finalize() const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
::java::lang::String toString() const;
Compared to
static PyMethodDef t_PythonClassicSimilarity__methods_[] = {
DECLARE_METHOD(t_PythonClassicSimilarity, cast_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, instance_, METH_O | METH_CLASS),
DECLARE_METHOD(t_PythonClassicSimilarity, coord, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, finalize, METH_NOARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idf, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, idfExplain, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, lengthNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, pythonExtension, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, queryNorm, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, sloppyFreq, METH_VARARGS),
DECLARE_METHOD(t_PythonClassicSimilarity, tf, METH_VARARGS),
{ NULL, NULL, 0, NULL }
};
and
JNINativeMethod methods[] = {
{ "coord", "(II)F", (void *) t_PythonClassicSimilarity_coord0 },
{ "idf", "(JJ)F", (void *) t_PythonClassicSimilarity_idf1 },
{ "idfExplain", "(Lorg/apache/lucene/search/CollectionStatistics;[Lorg/apache/lucene/search/TermStatistics;)Lorg/apache/lucene/search/Explanation;", (void *) t_PythonClassicSimilarity_idfExplain2 },
{ "lengthNorm", "(Lorg/apache/lucene/index/FieldInvertState;)F", (void *) t_PythonClassicSimilarity_lengthNorm3 },
{ "pythonDecRef", "()V", (void *) t_PythonClassicSimilarity_pythonDecRef4 },
{ "queryNorm", "(F)F", (void *) t_PythonClassicSimilarity_queryNorm5 },
{ "sloppyFreq", "(I)F", (void *) t_PythonClassicSimilarity_sloppyFreq6 },
{ "tf", "(F)F", (void *) t_PythonClassicSimilarity_tf7 },
};
enum {
mid_init$_7353aa4eb69a6f70,
mid_coord_78a3b70e2a43b9c7,
mid_finalize_7353aa4eb69a6f70,
mid_idf_0af4b58f982c0dd1,
mid_idfExplain_08e4e4733921c6f1,
mid_lengthNorm_a7d67c4cd2e3b6a2,
mid_pythonDecRef_7353aa4eb69a6f70,
mid_pythonExtension_aae05472994e7446,
mid_pythonExtension_f41cd3dc7c7d5d78,
mid_queryNorm_eecff9f314592ce9,
mid_sloppyFreq_6b1faadec3ed3278,
mid_tf_eecff9f314592ce9,
max_mid
};
and
jfloat coord(jint, jint) const;
void finalize() const;
jfloat idf(jlong, jlong) const;
::org::apache::lucene::search::Explanation idfExplain(const ::org::apache::lucene::search::CollectionStatistics &, const JArray< ::org::apache::lucene::search::TermStatistics > &) const;
jfloat lengthNorm(const ::org::apache::lucene::index::FieldInvertState &) const;
void pythonDecRef() const;
jlong pythonExtension() const;
void pythonExtension(jlong) const;
jfloat queryNorm(jfloat) const;
jfloat sloppyFreq(jint) const;
jfloat tf(jfloat) const;
package org.apache.pylucene.search.similarities;
import org.apache.lucene.search.similarities.BasicStats;
import org.apache.lucene.search.similarities.SimilarityBase;
public class PythonSimilarityBase extends SimilarityBase {
private long pythonObject;
public PythonSimilarityBase()
{
}
public void pythonExtension(long pythonObject)
{
this.pythonObject = pythonObject;
}
public long pythonExtension()
{
return this.pythonObject;
}
public void finalize()
throws Throwable
{
pythonDecRef();
}
public native void pythonDecRef();
@Override
protected native float score(BasicStats stats, float freq, float docLen);
@Override
public native String toString();
}
I?ve also tried variations of adding the BasicStats class, the Similarity$SimWeight class and the whole similarities package into the JCC command, for example
JCC=$(PYTHON) -m jcc.__main__ --shared --arch x86_64 --package org.apache.lucene.search.similarities
(Trying to follow advice in https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21> <https://issues.apache.org/jira/browse/PYLUCENE-21 <https://issues.apache.org/jira/browse/PYLUCENE-21>>)
My feeling is that it has something to do with that BasicStats object?
If anyone is able to assist it would be greatly appreciated!!

The 'score' method you're trying to get wrapped is marked 'protected' in the source code you showed above.
By default, JCC only wraps public methods and fields.
You can either make it public or force a wrapper for it by explicitely listing it on the JCC command line.
org.apache.lucene.index.IndexWriter:getReader \
org.apache.lucene.analysis.Tokenizer:input \
Just add your score method to the list there...
Andi..

Post by Peter Brown
Thanks,
Peter Brown
School of Information and Communication Technology,
Griffith University - Gold Coast Campus