Can't seem to get cosine metric working.

[rn123]

hdbscan==0.8.2
Can't seem to get cosine metric working:

hdbscan.hdbscan_.DistanceMetric.get_metric('cosine')
<hdbscan.dist_metrics.ArccosDistance at 0x7f83b22a8c08>
So the metric is there, but:

clusterer = hdbscan.HDBSCAN(min_cluster_size=5, metric='cosine')
labels = clusterer.fit_predict(np.array(data_d2v))

yields:

ValueError: Unrecognized metric 'cosine'

[lmcinnes]

That is definitely a problem. I'll look into it as soon as I get a chance. In the meantime you can try 'arccos' as a metric, which may work better.

[rn123]

Same issue with 'arccos', thought it was a link to 'cos'.

[lmcinnes]

It seems that the issue is that we are instantiating ball/kd-trees from sklearn, which does not support such distance metrics (yet). If you have a small enough dataset then adding the keyword argument algorithm='generic' will get you the results you desire. In the meantime I'll see if I can push the relevant metrics upstream to sklearn.

[oxymor0n]

So I'm also trying to use cosine similarity as my distance metric. Here's what I've tried so far:

• Straight up using metric='cosine' doesn't work at all because it's not implemented in sklearn.neighbors.DistanceMetric

• Using algorithm='generic', metric='cosine' or precomputing the cosine distances and then using metric='precomputed' make the .fit method throw

   ValueError: Buffer dtype mismatch, expected 'double_t' but got 'float'`
  

  Casting the input matrix to double dtype solves the bug but the clustering run requires a lot of memory.

[lmcinnes]

There remain some issues with this -- it's still being worked on. Apologies. In the meantime it is worth noting that if you normalize your vectors then euclidean distance is equal to arccos distance and all is well, so that's the right way to proceed for now.

…

On Wed, Mar 8, 2017 at 12:41 PM, William Tran ***@***.***> wrote: So I'm also trying to use cosine similarity as my distance metric. Here's what I've tried so far: - Straight up using metric='cosine' doesn't work at all because it's not implemented in sklearn.neighbors.DistanceMetric <http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html> - Using algorithm='generic', metric='cosine' make the .fit method throw ValueError: Buffer dtype mismatch, expected 'double_t' but got 'float' - Precomputing the cosine similarity and then using metric='precomputed' throws the same error [image: screen shot 2017-03-08 at 12 30 17 pm] <https://cloud.githubusercontent.com/assets/2345378/23715732/9b9809ee-03fb-11e7-9ecd-0c2a2cf0a91e.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALaKBVOMKBGB5NqenBPQ0uNNTJEk6BlOks5rjuhagaJpZM4KjDT-> .

[oxymor0n]

Thanks for the hint, I totally missed that part! :D Is there anything I could do to help?

[lmcinnes]

If you can figure out the "right" way to make it work I would appreciate it. There are a few approaches and but they each have some drawbacks. I would be happy to know if there is a "better" approach. Pull requests are even better. On Wed, Mar 8, 2017 at 2:40 PM, William Tran <notifications@github.com> wrote:

…

Thanks for the hint, I totally missed that part! :D Is there anything I could do to help? From: Leland McInnes ***@***.***> Reply-To: scikit-learn-contrib/hdbscan ***@***.***> Date: Wednesday, March 8, 2017 at 2:37 PM To: scikit-learn-contrib/hdbscan ***@***.***> Cc: William Tran ***@***.***>, Comment < ***@***.***> Subject: Re: [scikit-learn-contrib/hdbscan] Can't seem to get cosine metric working. (#69) There remain some issues with this -- it's still being worked on. Apologies. In the meantime it is worth noting that if you normalize your vectors then euclidean distance is equal to arccos distance and all is well, so that's the right way to proceed for now. On Wed, Mar 8, 2017 at 12:41 PM, William Tran ***@***.***> wrote: > So I'm also trying to use cosine similarity as my distance metric. Here's > what I've tried so far: > > - > > Straight up using metric='cosine' doesn't work at all because it's not > implemented in sklearn.neighbors.DistanceMetric > <http://scikit-learn.org/stable/modules/generated/sklearn.neighbors. DistanceMetric.html> > - > > Using algorithm='generic', metric='cosine' make the .fit method throw ValueError: > Buffer dtype mismatch, expected 'double_t' but got 'float' > - > > Precomputing the cosine similarity and then using metric='precomputed' > throws the same error > > [image: screen shot 2017-03-08 at 12 30 17 pm] > <https://cloud.githubusercontent.com/assets/2345378/23715732/9b9809ee- 03fb-11e7-9ecd-0c2a2cf0a91e.png> > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/scikit-learn-contrib/hdbscan/issues/ 69#issuecomment-285112451>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ ALaKBVOMKBGB5NqenBPQ0uNNTJEk6BlOks5rjuhagaJpZM4KjDT-> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://github.com/ scikit-learn-contrib/hdbscan#69#issuecomment-285145306>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ ACPJonte1GIcknVapmpiDNg5qLaNi4z0ks5rjwOJgaJpZM4KjDT->. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALaKBZDkrf3avatOISdRwOEQIvgWZLwcks5rjwQfgaJpZM4KjDT-> .

[oxymor0n]

What ideas have you explored/worked on?

[lmcinnes]

One can make custom kd-trees and ball-trees that support angular distance (I have done this, but not folded it in -- I don't want to support essentially duplicates of code from sklearn). One can just make a normalized copy of the data and then run everything with euclidean distance (easy, but some other results will potentially look weird/not match with input data, for instance cluster exemplars etc., and the prediction will break). One can require distance matrices, and then fix up the bug to ensure everything gets recast as floats before going into Cython. On Wed, Mar 8, 2017 at 2:42 PM, William Tran <notifications@github.com> wrote:

…

What ideas have you explored/worked on? From: Leland McInnes ***@***.***> Reply-To: scikit-learn-contrib/hdbscan ***@***.***> Date: Wednesday, March 8, 2017 at 2:41 PM To: scikit-learn-contrib/hdbscan ***@***.***> Cc: William Tran ***@***.***>, Comment < ***@***.***> Subject: Re: [scikit-learn-contrib/hdbscan] Can't seem to get cosine metric working. (#69) If you can figure out the "right" way to make it work I would appreciate it. There are a few approaches and but they each have some drawbacks. I would be happy to know if there is a "better" approach. Pull requests are even better. On Wed, Mar 8, 2017 at 2:40 PM, William Tran ***@***.***> wrote: > Thanks for the hint, I totally missed that part! :D > > Is there anything I could do to help? > > From: Leland McInnes ***@***.***> > Reply-To: scikit-learn-contrib/hdbscan ***@***.***> > Date: Wednesday, March 8, 2017 at 2:37 PM > To: scikit-learn-contrib/hdbscan ***@***.***> > Cc: William Tran ***@***.***>, Comment < > ***@***.***> > Subject: Re: [scikit-learn-contrib/hdbscan] Can't seem to get cosine > metric working. (#69) > > There remain some issues with this -- it's still being worked on. > Apologies. In the meantime it is worth noting that if you normalize your > vectors then euclidean distance is equal to arccos distance and all is > well, so that's the right way to proceed for now. > > On Wed, Mar 8, 2017 at 12:41 PM, William Tran ***@***.***> > wrote: > > > So I'm also trying to use cosine similarity as my distance metric. > Here's > > what I've tried so far: > > > > - > > > > Straight up using metric='cosine' doesn't work at all because it's not > > implemented in sklearn.neighbors.DistanceMetric > > <http://scikit-learn.org/stable/modules/generated/sklearn.neighbors. > DistanceMetric.html> > > - > > > > Using algorithm='generic', metric='cosine' make the .fit method throw > ValueError: > > Buffer dtype mismatch, expected 'double_t' but got 'float' > > - > > > > Precomputing the cosine similarity and then using metric='precomputed' > > throws the same error > > > > [image: screen shot 2017-03-08 at 12 30 17 pm] > > <https://cloud.githubusercontent.com/assets/2345378/23715732/9b9809ee- > 03fb-11e7-9ecd-0c2a2cf0a91e.png> > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > <https://github.com/scikit-learn-contrib/hdbscan/issues/ > 69#issuecomment-285112451>, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/ > ALaKBVOMKBGB5NqenBPQ0uNNTJEk6BlOks5rjuhagaJpZM4KjDT-> > > . > > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub<https://github.com/ > scikit-learn-contrib/hdbscan#69#issuecomment-285145306>, or mute > the thread<https://github.com/notifications/unsubscribe-auth/ > ACPJonte1GIcknVapmpiDNg5qLaNi4z0ks5rjwOJgaJpZM4KjDT->. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/scikit-learn-contrib/hdbscan/issues/ 69#issuecomment-285145993>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ ALaKBZDkrf3avatOISdRwOEQIvgWZLwcks5rjwQfgaJpZM4KjDT-> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://github.com/ scikit-learn-contrib/hdbscan#69#issuecomment-285146345>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ ACPJohEU0ipIDGB2KmNqIzRQtkLlqe7Hks5rjwRkgaJpZM4KjDT->. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALaKBc1fFwXUBeoFtVjhXsyZgwcYDfLGks5rjwS-gaJpZM4KjDT-> .

[oxymor0n]

Personally I feel that merging your custom trees into sklearn proper will be the best approach, but I don't know enough about the specific implementation differences that could block the merge to comment further on that.

For the normalized copy approach, if the different vector values is a problem, maybe we could keep a scaling vector so we can recover the unscaled input data for operations that require it?

As for the Cython bug, I double checked my code today and it seems casting the input matrix to np.double actually works, so that could be another way to do it. The downside of that is that the cast cause memory usage to go through the roof (maxed out my 16GB when I clustered a (25000, 300) input with algorithm='generic', metric='cosine')

[lmcinnes]

I understand, but that is simply one of the issues with computing an all-pairs distance matrix. I'll see if I can work out something else reasonable.

…

On Thu, Mar 9, 2017 at 11:54 AM, William Tran ***@***.***> wrote: Personally I feel that merging your custom trees into sklearn proper will be the best approach, but I don't know enough about the specific implementation differences and whether any of them is blocking that merge to comment further on that. For the normalized copy approach, if the different vector values is a problem, maybe we could keep a scaling vector so we can recover the unscaled input data for operations that require it? As for the Cython bug, I double checked my code today and it seems casting the input matrix to np.double actually works, so that could be another way to do it. The downside of that is that the cast cause memory usage to go through the roof (maxed out my 16GB when I clustered a (25000, 300) input with algorithm='generic', metric='cosine') — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALaKBcSkkWcfb9aqItXjtmiMCpXTVNt-ks5rkC7JgaJpZM4KjDT-> .

[oxymor0n]

@lmcinnes sorry which "issue" out of the three do you mean?

[lmcinnes]

The casting to doubles -- in general pairwise distance matrices are expensive and not a good way to go if you can help it. I should make that easier however.

…

On Thu, Mar 9, 2017 at 1:36 PM, William Tran ***@***.***> wrote: @lmcinnes <https://github.com/lmcinnes> sorry which "issue" out of the three do you mean? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALaKBbaEOFiVxhxT0u2IbUrjYm2xzMn2ks5rkEaxgaJpZM4KjDT-> .

[zyrikby]

This seems working for me:

pw_distance = pairwise_distances(df[col_names], metric="cosine", n_jobs=-1)
clusterer = HDBSCAN(min_cluster_size=min_cluster_size, metric='precomputed')
clusterer.fit(pw_distance)