比特派官方app|sklearn kmeans

作者：比特派官方app

2024-03-09 22:23:22

sklearn.cluster.KMeans — scikit-learn 1.4.1 documentation

Install

User Guide

API

Examples

Community

Getting Started

Tutorial

What's new

Glossary

Development

FAQ

Support

Related packages

Roadmap

Governance

About us

GitHub

Other Versions and Download

Getting Started

Tutorial

What's new

Glossary

Development

FAQ

Support

Related packages

Roadmap

Governance

About us

GitHub

Other Versions and Download

Toggle Menu

PrevUp

scikit-learn 1.4.1

Other versions

Please cite us if you use the software.

sklearn.cluster.KMeans

KMeans

KMeans.fit

KMeans.fit_predict

KMeans.fit_transform

KMeans.get_feature_names_out

KMeans.get_metadata_routing

KMeans.get_params

KMeans.predict

KMeans.score

KMeans.set_fit_request

KMeans.set_output

KMeans.set_params

KMeans.set_predict_request

KMeans.set_score_request

KMeans.transform

Examples using sklearn.cluster.KMeans

sklearn.cluster.KMeans¶

class sklearn.cluster.KMeans(n_clusters=8, *, init='k-means++', n_init='auto', max_iter=300, tol=0.0001, verbose=0, random_state=None, copy_x=True, algorithm='lloyd')[source]¶

K-Means clustering.

SKlearn里面的K-means使用详解_sklearn kmeans-CSDN博客

SKlearn里面的K-means使用详解

最新推荐文章于 2024-03-08 10:44:47 发布

ASS-ASH

最新推荐文章于 2024-03-08 10:44:47 发布

阅读量2.8w

154

点赞数

分类专栏：

无监督学习

文章标签：

sklearn

人工智能

机器学习

原文链接：https://www.cnblogs.com/pinard/p/6169370.html

版权

无监督学习

专栏收录该内容

15 篇文章

4 订阅

订阅专栏

　在K-Means聚类算法原理中，我们对K-Means的原理做了总结，本文我们就来讨论用scikit-learn来学习K-Means聚类。重点讲述如何选择合适的k值。

1. K-Means类概述

　　　　在scikit-learn中，包括两个K-Means的算法，一个是传统的K-Means算法，对应的类是KMeans。另一个是基于采样的Mini Batch K-Means算法，对应的类是MiniBatchKMeans。一般来说，使用K-Means的算法调参是比较简单的。

　　　　用KMeans类的话，一般要注意的仅仅就是k值的选择，即参数n_clusters；如果是用MiniBatchKMeans的话，也仅仅多了需要注意调参的参数batch_size，即我们的Mini Batch的大小。

　　　　当然KMeans类和MiniBatchKMeans类可以选择的参数还有不少，但是大多不需要怎么去调参。下面我们就看看KMeans类和MiniBatchKMeans类的一些主要参数。

2. KMeans类主要参数

　　　　KMeans类的主要参数有：

　　　　1) n_clusters: 即我们的k值，一般需要多试一些值以获得较好的聚类效果。k值好坏的评估标准在下面会讲。

　　　　2）max_iter：最大的迭代次数，一般如果是凸数据集的话可以不管这个值，如果数据集不是凸的，可能很难收敛，此时可以指定最大的迭代次数让算法可以及时退出循环。

　　　　3）n_init：用不同的初始化质心运行算法的次数。由于K-Means是结果受初始值影响的局部最优的迭代算法，因此需要多跑几次以选择一个较好的聚类效果，默认是10，一般不需要改。如果你的k值较大，则可以适当增大这个值。

　　　　4）init：即初始值选择的方式，可以为完全随机选择'random',优化过的'k-means++'或者自己指定初始化的k个质心。一般建议使用默认的'k-means++'。

　　　　5）algorithm：有“auto”, “full” or “elkan”三种选择。"full"就是我们传统的K-Means算法， “elkan”是我们原理篇讲的elkan K-Means算法。默认的"auto"则会根据数据值是否是稀疏的，来决定如何选择"full"和“elkan”。一般数据是稠密的，那么就是 “elkan”，否则就是"full"。一般来说建议直接用默认的"auto"

3. MiniBatchKMeans类主要参数

　　　　MiniBatchKMeans类的主要参数比KMeans类稍多，主要有：

　　　　1) n_clusters: 即我们的k值，和KMeans类的n_clusters意义一样。

　　　　2）max_iter：最大的迭代次数，和KMeans类的max_iter意义一样。

　　　　3）n_init：用不同的初始化质心运行算法的次数。这里和KMeans类意义稍有不同，KMeans类里的n_init是用同样的训练集数据来跑不同的初始化质心从而运行算法。而MiniBatchKMeans类的n_init则是每次用不一样的采样数据集来跑不同的初始化质心运行算法。

　　　　4）batch_size：即用来跑Mini Batch KMeans算法的采样集的大小，默认是100.如果发现数据集的类别较多或者噪音点较多，需要增加这个值以达到较好的聚类效果。

　　　　5）init：即初始值选择的方式，和KMeans类的init意义一样。

　　　　6）init_size: 用来做质心初始值候选的样本个数，默认是batch_size的3倍，一般用默认值就可以了。

　　　　7）reassignment_ratio: 某个类别质心被重新赋值的最大次数比例，这个和max_iter一样是为了控制算法运行时间的。这个比例是占样本总数的比例，乘以样本总数就得到了每个类别质心可以重新赋值的次数。如果取值较高的话算法收敛时间可能会增加，尤其是那些暂时拥有样本数较少的质心。默认是0.01。如果数据量不是超大的话，比如1w以下，建议使用默认值。如果数据量超过1w，类别又比较多，可能需要适当减少这个比例值。具体要根据训练集来决定。

　　　　8）max_no_improvement：即连续多少个Mini Batch没有改善聚类效果的话，就停止算法，和reassignment_ratio， max_iter一样是为了控制算法运行时间的。默认是10.一般用默认值就足够了。

4. K值的评估标准

　　　　不像监督学习的分类问题和回归问题，我们的无监督聚类没有样本输出，也就没有比较直接的聚类评估方法。但是我们可以从簇内的稠密程度和簇间的离散程度来评估聚类的效果。常见的方法有轮廓系数Silhouette Coefficient和Calinski-Harabasz Index。个人比较喜欢Calinski-Harabasz Index，这个计算简单直接，得到的Calinski-Harabasz分数值ss越大则聚类效果越好。

　　　　Calinski-Harabasz分数值ss的数学计算公式是：

s(k)=tr(Bk)tr(Wk)m−kk−1s(k)=tr(Bk)tr(Wk)m−kk−1

　　　　其中m为训练集样本数，k为类别数。BkBk为类别之间的协方差矩阵，WkWk为类别内部数据的协方差矩阵。trtr为矩阵的迹。

　　　　也就是说，类别内部数据的协方差越小越好，类别之间的协方差越大越好，这样的Calinski-Harabasz分数会高。在scikit-learn中， Calinski-Harabasz Index对应的方法是metrics.calinski_harabaz_score.

5. K-Means应用实例

　　　　下面用一个实例来讲解用KMeans类和MiniBatchKMeans类来聚类。我们观察在不同的k值下Calinski-Harabasz分数。

　　　　完整的代码参见我的github: https://github.com/ljpzzz/machinelearning/blob/master/classic-machine-learning/kmeans_cluster.ipynb

　　　　首先我们随机创建一些二维数据作为训练集，选择二维特征数据，主要是方便可视化。代码如下：

import numpy as np

import matplotlib.pyplot as plt

%matplotlib inline

from sklearn.datasets.samples_generator import make_blobs

# X为样本特征，Y为样本簇类别，共1000个样本，每个样本2个特征，共4个簇，簇中心在[-1,-1], [0,0],[1,1], [2,2]，簇方差分别为[0.4, 0.2, 0.2]

X, y = make_blobs(n_samples=1000, n_features=2, centers=[[-1,-1], [0,0], [1,1], [2,2]], cluster_std=[0.4, 0.2, 0.2, 0.2],

random_state =9)

plt.scatter(X[:, 0], X[:, 1], marker='o')

plt.show()

　　　　从输出图可以我们看看我们创建的数据如下：

　　　现在我们来用K-Means聚类方法来做聚类，首先选择k=2，代码如下：

from sklearn.cluster import KMeans

y_pred = KMeans(n_clusters=2, random_state=9).fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=y_pred)

plt.show()

　　　　k=2聚类的效果图输出如下：

　　　　现在我们来看看我们用Calinski-Harabasz Index评估的聚类分数:

from sklearn import metrics

metrics.calinski_harabaz_score(X, y_pred)

　　　　输出如下：

3116.1706763322227

　　　　现在k=3来看看聚类效果，代码如下：

from sklearn.cluster import KMeans

y_pred = KMeans(n_clusters=3, random_state=9).fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=y_pred)

plt.show()　　

　　　　k=3的聚类的效果图输出如下：

　　　　现在我们来看看我们用Calinski-Harabaz Index评估的k=3时候聚类分数:

metrics.calinski_harabaz_score(X, y_pred)

　　　　输出如下：

2931.625030199556

　　　　可见此时k=3的聚类分数比k=2还差。

　　　　现在我们看看k=4时候的聚类效果：

from sklearn.cluster import KMeans

y_pred = KMeans(n_clusters=4, random_state=9).fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=y_pred)

plt.show()

　　　　k=4的聚类的效果图输出如下：

　　　　现在我们来看看我们用Calinski-Harabasz Index评估的k=4时候聚类分数:

metrics.calinski_harabaz_score(X, y_pred)

　　　　输出如下：

5924.050613480169

　　　　可见k=4的聚类分数比k=2和k=3都要高，这也符合我们的预期，我们的随机数据集也就是4个簇。当特征维度大于2，我们无法直接可视化聚类效果来肉眼观察时，用Calinski-Harabaz Index评估是一个很实用的方法。

　　　　现在我们再看看用MiniBatchKMeans的效果，我们将batch size设置为200. 由于我们的4个簇都是凸的，所以其实batch size的值只要不是非常的小，对聚类的效果影响不大。

for index, k in enumerate((2,3,4,5)):

plt.subplot(2,2,index+1)

y_pred = MiniBatchKMeans(n_clusters=k, batch_size = 200, random_state=9).fit_predict(X)

score= metrics.calinski_harabaz_score(X, y_pred)

plt.scatter(X[:, 0], X[:, 1], c=y_pred)

plt.text(.99, .01, ('k=%d, score: %.2f' % (k,score)),

transform=plt.gca().transAxes, size=10,

horizontalalignment='right')

plt.show()

　　　对于k=2,3,4,5对应的输出图为：

　　　　可见使用MiniBatchKMeans的聚类效果也不错，当然由于使用Mini Batch的原因，同样是k=4最优，KMeans类的Calinski-Harabasz Index分数为5924.05,而MiniBatchKMeans的分数稍微低一些，为5921.45。这个差异损耗并不大。

用scikit-learn学习K-Means聚类 - 刘建平Pinard - 博客园

优惠劵

ASS-ASH

关注

踩

154

觉得还不错?

一键收藏

知道了

SKlearn里面的K-means使用详解

　在K-Means聚类算法原理中，我们对K-Means的原理做了总结，本文我们就来讨论用scikit-learn来学习K-Means聚类。重点讲述如何选择合适的k值。1. K-Means类概述　　　　在scikit-learn中，包括两个K-Means的算法，一个是传统的K-Means算法，对应的类是KMeans。另一个是基于采样的Mini Batch K-Means算法，对应的类是MiniBatchKMeans。一般来说，使用K-Means的算法调参是比较简单的。　　　　用KMeans类的话，

复制链接

扫一扫

专栏目录

调用sklearn库的K-Means聚类分析实例

01-26

#class sklearn.cluster.KMeans(n_clusters=8, init=’k-means++’, n_init=10, max_iter=300, tol=0.0001, precompute_distances=’auto’, verbose=0, random_state=None, copy_x=True, n_jobs=1, algorithm=’auto’)

#参数：

#（1）对于K均值聚类，我们需要给定类别的个数n_cluster，默认值为8；

#（2）max_iter为迭代的次数，这里设置最大迭代次数为300；

#（3）n_init设为10意味着进行10次随机初始化，选择效果最好的一种来作为模型；

#（4）init=’k-means++’ 会由程序自动寻找合适的n_clusters；

#（5）tol：float形，默认值= 1e-4，与inertia结合来确定收敛条件；

#（6）n_jobs：指定计算所用的进程数；

#（7）verbose 参数设定打印求解过程的程度，值越大，细节打印越多；

#（8）copy_x：布尔型，默认值=True。当我们precomputing distances时，将数据中心化会得到更准确的结果。如果把此参数值设为True，则原始数据不会被改变。如果是False，则会直接在原始数据上做修改并在函数返回值时将其还原。但是在计算过程中由于有对数据均值的加减运算，所以数据返回后，原始数据和计算前可能会有细小差别。

#属性：

#（1）cluster_centers_：向量，[n_clusters, n_features]

# Coordinates of cluster centers (每个簇中心的坐标？？)；

#（2）Labels_:每个点的分类；

#（3）inertia_：float，每个点到其簇的质心的距离之和。

聚类算法：K-means聚类图像分割

12-22

1 K-Means聚类

K-Means聚类是最常用的聚类算法，最初起源于信号处理，其目标是将数据点划分为K个类簇，找到每个簇的中心并使其度量最小化。该算法的最大优点是简单、便于理解，运算速度较快，缺点是只能应用于连续型数据，并且要在聚类前指定聚集的类簇数。

下面是K-Means聚类算法的分析流程，步骤如下：

第一步，确定K值，即将数据集聚集成K个类簇或小组。

第二步，从数据集中随机选择K个数据点作为质心（Centroid）或数据中心。

第三步，分别计算每个点到每个质心之间的距离，并将每个点划分到离最近质心的小组，跟定了那个质心。

第四步，当每个质心都聚集了一些点后，重新定义算法选出新的质心。

1 条评论

您还未登录，请先

后发表或查看评论

Python学习——K-means聚类

机器学习库sklearn的K-Means聚类算法的使用方法 - 知乎

机器学习库sklearn的K-Means聚类算法的使用方法 - 知乎首发于学习python网络爬虫建设智慧时空数据库切换模式写文章登录/注册机器学习库sklearn的K-Means聚类算法的使用方法华天清网络爬虫 www.GooSeeker.com 创始人，数据挖掘和数据获取社区运营1，本Notebook背景介绍之前我们已经发布过一篇有关K-means算法实验的Notebook：《Jupyter Notebook使用Python做K-Means聚类分析》，在那篇Notebook里，我们从K-Means的基础原理和实现方法开始讨论，实际上K-Means作为一个常用的算法，在众多机器学习程序库中都有现成的函数可以调用。今天这篇notebook主要演示怎样调用sklearn的K-Means函数。我们先简单回顾一下上一篇notebook的内容，罗列如下：1.什么是K-means聚类算法2.K-means聚类算法应用场景3.K-means聚类算法步骤4.K-means不适合的数据集5.准备测试数据6.基于python原生代码做K-Means聚类分析实验7.使用matplotlib进行可视化输出面对这么多内容，有同学反馈给我说，他只想使用K-Means做一些社会科学计算，不想费脑筋搞明白K-Means是怎么实现的。好吧，调用机器学习库中的函数是最合适的，只要按照要求准备好样本数据，调用一个函数就把问题解决了。那么，我们今天就另发布一个使用机器学习库sklearn的k-means聚类算法的Notebook。1.1 sklearn库简介转载知乎文章《sklearn库主要模块功能简介》的介绍如下：sklearn，全称scikit-learn，是python中的机器学习库，建立在numpy、scipy、matplotlib等数据科学包的基础之上，涵盖了机器学习中的样例数据、数据预处理、模型验证、特征选择、分类、回归、聚类、降维等几乎所有环节，功能十分强大，目前sklearn版本是0.23。与深度学习库存在pytorch、TensorFlow等多种框架可选不同，sklearn是python中传统机器学习的首选库，不存在其他竞争者。1.2 基本原理K-means是无监督学习的代表。主要目的是聚类，聚类的依据就是样本之间的距离。比如要分为K类。步骤是：1. 随机选取K个点。2. 计算每个点到K个质心的距离，分成K个簇。3. 计算K个簇样本的平均值作新的质心4. 循环2、35. 位置不变，距离完成2，第三方库本notebook使用了sklearn库做k-means算法实验。如果未安装，请先使用下面的命令安装sklearnm库，再运行实验本notebook：pip install -i https://pypi.tuna.tsinghua.edu.cn/simple sklearn #国内安装使用清华的源，速度快3，本notebook所做的测试基于测试数据和sklearn官网的例子，在Jupyter Notebook中使用Python做K-Means算法实验。4，引入sklearn库导入sklearn下的K-means模块。sklearn，全称scikit-learn，是python中的机器学习库，建立在numpy、scipy、matplotlib等数据科学包的基础之上，涵盖了机器学习中的样例数据、数据预处理、模型验证、特征选择、分类、回归、聚类、降维等几乎所有环节，功能十分强大，目前sklearn版本是0.23。# coding:utf-8

from sklearn.cluster import KMeans5，引入matplotlib库matplotlib是一款命令式、较底层、可定制性强、图表资源丰富、简单易用、出版质量级别的python 2D绘图库。matplotlib算是python绘图的元老级库，类似编程语言里的C语言。很多其它的python绘图库是基于matplotlib开发的，比如seaborn、ggplot、plotnine、holoviews、basemap等。matplotlib可用于python脚本、python shell、jupyter notebook、web等。最适合来运行matplotlib绘图的工具是jupyter notebook，本Notebook也是基于该工具做可视化实验：交互式操作，在浏览器上运行代码，能直接显示运行结果和图表，import matplotlib.pyplot as plt6，生成模拟数据参看官网网页Generated Datasets，sklearn提供了一些方法，可以生成测试用数据集，生成过程中可以控制多个参数，便于验证算法。参看《sklearn中的make_blobs()函数详解》。下面我们生成一个测试用数据集，含有500个样本，每个样本2个特征。那么函数返回的结果数据有：X：是一个数组，存储生成的样本，格式是[n_samples, n_features]，详见下面的测试用数据集的查看结果y：是一个数组，里面是整数，表示每个样本所属的类别，格式是[n_samples]下面我们生成一个含有2个特征的样本数据，两个特征便于在平面上显示出来，两个特征对应画图坐标轴x和yfrom sklearn.datasets import make_blobs

import matplotlib.pyplot as plt

X, y = make_blobs(n_samples=500, # 500个样本

n_features=2, # 每个样本2个特征

centers=4, # 4个中心

random_state=1 #控制随机性

)查看一下X和y两个数组的格式X输出结果：array([[-6.92324165e+00, -1.06695320e+01], [-8.63062033e+00, -7.13940564e+00], [-9.63048069e+00, -2.72044935e+00], ....... [-8.96014913e+00, -8.06349899e+00], [-7.66603898e+00, -7.59715459e+00], [-6.46534407e+00, -2.85544633e+00]])y输出结果：array([2, 2, 1, 0, 3, 0, 3, 3, 1, 3, 2, 2, 3, 0, 3, 2, 1, 2, 0, 3, 1, 1, 3, 0, 3, 3, 0, 0, 1, 3, 2, 0, 3, 2, 3, 2, 1, 1, 2, 1, 3, 1, 0, 3, 3, 2, 1, 3, 0, 0, 0, 1, 1, 3, 2, 1, 1, 1, 1, 3, 0, 0, 1, 3, 0, 3, ...... 1, 0, 0, 2, 1, 3, 0, 3, 3, 2, 2, 3, 2, 1, 1, 2, 1, 2, 1, 0, 2, 0, 1, 3, 0, 1, 3, 0, 2, 3, 0, 0, 1, 3, 1, 3, 2, 0, 2, 3, 0, 2, 2, 2, 1, 0, 3, 2, 3, 3, 1, 1, 2, 3, 3, 3, 3, 3, 3, 2, 3, 1, 2, 3, 0, 3, 0, 3, 1, 3, 0, 0, 0, 1, 3, 1, 2, 1, 0, 3, 2, 0, 2, 0, 2, 3, 0, 0, 2, 1, 3, 2, 1, 1, 1, 2, 3, 0, 1, 3, 2, 2, 2, 3])7，使用matplotlib画图对于上面已生成的模拟数据，使用matplotlib.pyplot画出图像color = ['red', 'pink','orange','gray']

fig, axi1=plt.subplots(1)

for i in range(4):

axi1.scatter(X[y==i, 0], X[y==i,1],

marker='o',

s=8,

c=color[i]

)

plt.show()8，K-means聚类之一：k=3使用sklearn的KMeans模块进行聚类分析，可以设置要聚几类。此处k设置为3from sklearn.cluster import KMeans

n_clusters=3

cluster = KMeans(n_clusters=n_clusters,random_state=0).fit(X)9，查看聚类后的质心k=3的情况下，会有3个质心centroid=cluster.cluster_centers_

centroid输出结果： array([[-8.09286791, -3.50997357], [-1.54234022, 4.43517599], [-7.0877462 , -8.08923534]])10，使用matplotlib画图使用matplotlib.pyplot画出聚类后的图像y_pred = cluster.labels_#获取训练后对象的每个样本的标签

centtrod = cluster.cluster_centers_

color=['red','pink','orange','gray']

fig, axi1=plt.subplots(1)

for i in range(n_clusters):

axi1.scatter(X[y_pred==i, 0], X[y_pred==i, 1],

marker='o',

s=8,

c=color[i])

axi1.scatter(centroid[:,0],centroid[:,1],marker='x',s=100,c='black')11，K-means聚类之一：k=4使用sklearn的KMeans模块进行聚类分析此处k设置为4n_clusters=4

cluster2 = KMeans(n_clusters=n_clusters,random_state=0).fit(X)12，查看聚类后的质心k=4的情况下，会有4个质心centroid=cluster2.cluster_centers_

centroid 输出结果：array([[-10.00969056, -3.84944007], [ -1.54234022, 4.43517599], [ -6.08459039, -3.17305983], [ -7.09306648, -8.10994454]])13，使用matplotlib画图使用matplotlib.pyplot画出聚类后的图像y_pred = cluster2.labels_#获取训练后对象的每个样本的标签

centtrod = cluster2.cluster_centers_

color=['red','pink','orange','gray']

fig, axi1=plt.subplots(1)

for i in range(n_clusters):

axi1.scatter(X[y_pred==i, 0], X[y_pred==i, 1],

marker='o',

s=8,

c=color[i])

axi1.scatter(centroid[:,0],centroid[:,1],marker='x',s=100,c='black')14，总结本文使用生成的样本数据集，如果是真实的样本数据，那么设法存成上述格式的X数组，然后交给Kmeans模型的fit()函数计算即可。我们将在后续的notebook讲解怎样利用Kmeans模型针对真实数据做计算。发布于 2021-10-28 10:00深度学习（Deep Learning）sklearn聚类算法赞同 211 条评论分享喜欢收藏申请转载文章被以下专栏收录学习python网络爬虫建设智慧时空数据库一起探一条经济有效的开发和使用Python网络爬虫

sklearn.cluster.k_means — scikit-learn 1.4.1 documentation

Install

User Guide

API

Examples

Community

Getting Started

Tutorial

What's new

Glossary

Development

FAQ

Support

Related packages

Roadmap

Governance

About us

GitHub

Other Versions and Download

Getting Started

Tutorial

What's new

Glossary

Development

FAQ

Support

Related packages

Roadmap

Governance

About us

GitHub

Other Versions and Download

Toggle Menu

PrevUp

scikit-learn 1.4.1

Other versions

Please cite us if you use the software.

sklearn.cluster.k_means

k_means

sklearn.cluster.k_means¶

sklearn.cluster.k_means(X, n_clusters, *, sample_weight=None, init='k-means++', n_init='auto', max_iter=300, verbose=False, tol=0.0001, random_state=None, copy_x=True, algorithm='lloyd', return_n_iter=False)[source]¶

Perform K-means clustering algorithm.

2.3. Clustering — scikit-learn 1.4.1 documentation

Install

User Guide

API

Examples

Community

Getting Started

Tutorial

What's new

Glossary

Development

FAQ

Support

Related packages

Roadmap

Governance

About us

GitHub

Other Versions and Download

Getting Started

Tutorial

What's new

Glossary

Development

FAQ

Support

Related packages

Roadmap

Governance

About us

GitHub

Other Versions and Download

Toggle Menu

PrevUp

scikit-learn 1.4.1

Other versions

Please cite us if you use the software.

2.3. Clustering

2.3.1. Overview of clustering methods

2.3.2. K-means

2.3.2.1. Low-level parallelism

2.3.2.2. Mini Batch K-Means

2.3.3. Affinity Propagation

2.3.4. Mean Shift

2.3.5. Spectral clustering

2.3.5.1. Different label assignment strategies

2.3.5.2. Spectral Clustering Graphs

2.3.6. Hierarchical clustering

2.3.6.1. Different linkage type: Ward, complete, average, and single linkage

2.3.6.2. Visualization of cluster hierarchy

2.3.6.3. Adding connectivity constraints

2.3.6.4. Varying the metric

2.3.6.5. Bisecting K-Means

2.3.7. DBSCAN

2.3.8. HDBSCAN

2.3.8.1. Mutual Reachability Graph

2.3.8.2. Hierarchical Clustering

2.3.9. OPTICS

2.3.10. BIRCH

2.3.11. Clustering performance evaluation

2.3.11.1. Rand index

2.3.11.1.1. Advantages

2.3.11.1.2. Drawbacks

2.3.11.1.3. Mathematical formulation

2.3.11.2. Mutual Information based scores

2.3.11.2.1. Advantages

2.3.11.2.2. Drawbacks

2.3.11.2.3. Mathematical formulation

2.3.11.3. Homogeneity, completeness and V-measure

2.3.11.3.1. Advantages

2.3.11.3.2. Drawbacks

2.3.11.3.3. Mathematical formulation

2.3.11.4. Fowlkes-Mallows scores

2.3.11.4.1. Advantages

2.3.11.4.2. Drawbacks

2.3.11.5. Silhouette Coefficient

2.3.11.5.1. Advantages

2.3.11.5.2. Drawbacks

2.3.11.6. Calinski-Harabasz Index

2.3.11.6.1. Advantages

2.3.11.6.2. Drawbacks

2.3.11.6.3. Mathematical formulation

2.3.11.7. Davies-Bouldin Index

2.3.11.7.1. Advantages

2.3.11.7.2. Drawbacks

2.3.11.7.3. Mathematical formulation

2.3.11.8. Contingency Matrix

2.3.11.8.1. Advantages

2.3.11.8.2. Drawbacks

2.3.11.9. Pair Confusion Matrix

2.3. Clustering¶

Clustering of

unlabeled data can be performed with the module sklearn.cluster.

Each clustering algorithm comes in two variants: a class, that implements

the fit method to learn the clusters on train data, and a function,

that, given train data, returns an array of integer labels corresponding

to the different clusters. For the class, the labels over the training

data can be found in the labels_ attribute.

Input data

One important thing to note is that the algorithms implemented in

this module can take different kinds of matrix as input. All the

methods accept standard data matrices of shape (n_samples, n_features).

These can be obtained from the classes in the sklearn.feature_extraction

module. For AffinityPropagation, SpectralClustering

and DBSCAN one can also input similarity matrices of shape

(n_samples, n_samples). These can be obtained from the functions

in the sklearn.metrics.pairwise module.

2.3.1. Overview of clustering methods¶

A comparison of the clustering algorithms in scikit-learn¶

Method name

Parameters

Scalability

Usecase

Geometry (metric used)

K-Means

number of clusters

Very large n_samples, medium n_clusters with

MiniBatch code

General-purpose, even cluster size, flat geometry,

not too many clusters, inductive

Distances between points

Affinity propagation

damping, sample preference

Not scalable with n_samples

Many clusters, uneven cluster size, non-flat geometry, inductive

Graph distance (e.g. nearest-neighbor graph)

Mean-shift

bandwidth

Not scalable with n_samples

Many clusters, uneven cluster size, non-flat geometry, inductive

Distances between points

Spectral clustering

number of clusters

Medium n_samples, small n_clusters

Few clusters, even cluster size, non-flat geometry, transductive

Graph distance (e.g. nearest-neighbor graph)

Ward hierarchical clustering

number of clusters or distance threshold

Large n_samples and n_clusters

Many clusters, possibly connectivity constraints, transductive

Distances between points

Agglomerative clustering

number of clusters or distance threshold, linkage type, distance

Large n_samples and n_clusters

Many clusters, possibly connectivity constraints, non Euclidean

distances, transductive

Any pairwise distance

DBSCAN

neighborhood size

Very large n_samples, medium n_clusters

Non-flat geometry, uneven cluster sizes, outlier removal,

transductive

Distances between nearest points

HDBSCAN

minimum cluster membership, minimum point neighbors

large n_samples, medium n_clusters

Non-flat geometry, uneven cluster sizes, outlier removal,

transductive, hierarchical, variable cluster density

Distances between nearest points

OPTICS

minimum cluster membership

Very large n_samples, large n_clusters

Non-flat geometry, uneven cluster sizes, variable cluster density,

outlier removal, transductive

Distances between points

Gaussian mixtures

many

Not scalable

Flat geometry, good for density estimation, inductive

Mahalanobis distances to centers

BIRCH

branching factor, threshold, optional global clusterer.

Large n_clusters and n_samples

Large dataset, outlier removal, data reduction, inductive

Euclidean distance between points

Bisecting K-Means

number of clusters

Very large n_samples, medium n_clusters

General-purpose, even cluster size, flat geometry,

no empty clusters, inductive, hierarchical

Distances between points

Non-flat geometry clustering is useful when the clusters have a specific

shape, i.e. a non-flat manifold, and the standard euclidean distance is

not the right metric. This case arises in the two top rows of the figure

above.

Gaussian mixture models, useful for clustering, are described in

another chapter of the documentation dedicated to

mixture models. KMeans can be seen as a special case of Gaussian mixture

model with equal covariance per component.

Transductive clustering methods (in contrast to

inductive clustering methods) are not designed to be applied to new,

unseen data.

2.3.2. K-means¶

The KMeans algorithm clusters data by trying to separate samples in n

groups of equal variance, minimizing a criterion known as the inertia or

within-cluster sum-of-squares (see below). This algorithm requires the number

of clusters to be specified. It scales well to large numbers of samples and has

been used across a large range of application areas in many different fields.

The k-means algorithm divides a set of \(N\) samples \(X\) into

\(K\) disjoint clusters \(C\), each described by the mean \(\mu_j\)

of the samples in the cluster. The means are commonly called the cluster

“centroids”; note that they are not, in general, points from \(X\),

although they live in the same space.

The K-means algorithm aims to choose centroids that minimise the inertia,

or within-cluster sum-of-squares criterion:

\[\sum_{i=0}^{n}\min_{\mu_j \in C}(||x_i - \mu_j||^2)\]

Inertia can be recognized as a measure of how internally coherent clusters are.

It suffers from various drawbacks:

Inertia makes the assumption that clusters are convex and isotropic,

which is not always the case. It responds poorly to elongated clusters,

or manifolds with irregular shapes.

Inertia is not a normalized metric: we just know that lower values are

better and zero is optimal. But in very high-dimensional spaces, Euclidean

distances tend to become inflated

(this is an instance of the so-called “curse of dimensionality”).

Running a dimensionality reduction algorithm such as Principal component analysis (PCA) prior to

k-means clustering can alleviate this problem and speed up the

computations.

For more detailed descriptions of the issues shown above and how to address them,

refer to the examples Demonstration of k-means assumptions

and Selecting the number of clusters with silhouette analysis on KMeans clustering.

K-means is often referred to as Lloyd’s algorithm. In basic terms, the

algorithm has three steps. The first step chooses the initial centroids, with

the most basic method being to choose \(k\) samples from the dataset

\(X\). After initialization, K-means consists of looping between the

two other steps. The first step assigns each sample to its nearest centroid.

The second step creates new centroids by taking the mean value of all of the

samples assigned to each previous centroid. The difference between the old

and the new centroids are computed and the algorithm repeats these last two

steps until this value is less than a threshold. In other words, it repeats

until the centroids do not move significantly.

K-means is equivalent to the expectation-maximization algorithm

with a small, all-equal, diagonal covariance matrix.

The algorithm can also be understood through the concept of Voronoi diagrams. First the Voronoi diagram of

the points is calculated using the current centroids. Each segment in the

Voronoi diagram becomes a separate cluster. Secondly, the centroids are updated

to the mean of each segment. The algorithm then repeats this until a stopping

criterion is fulfilled. Usually, the algorithm stops when the relative decrease

in the objective function between iterations is less than the given tolerance

value. This is not the case in this implementation: iteration stops when

centroids move less than the tolerance.

Given enough time, K-means will always converge, however this may be to a local

minimum. This is highly dependent on the initialization of the centroids.

As a result, the computation is often done several times, with different

initializations of the centroids. One method to help address this issue is the

k-means++ initialization scheme, which has been implemented in scikit-learn

(use the init='k-means++' parameter). This initializes the centroids to be

(generally) distant from each other, leading to probably better results than

random initialization, as shown in the reference. For a detailed example of

comaparing different initialization schemes, refer to

A demo of K-Means clustering on the handwritten digits data.

K-means++ can also be called independently to select seeds for other

clustering algorithms, see sklearn.cluster.kmeans_plusplus for details

and example usage.

The algorithm supports sample weights, which can be given by a parameter

sample_weight. This allows to assign more weight to some samples when

computing cluster centers and values of inertia. For example, assigning a

weight of 2 to a sample is equivalent to adding a duplicate of that sample

to the dataset \(X\).

K-means can be used for vector quantization. This is achieved using the

transform method of a trained model of KMeans. For an example of

performing vector quantization on an image refer to

Color Quantization using K-Means.

Examples:

K-means Clustering: Example usage of

KMeans using the iris dataset

Clustering text documents using k-means: Document clustering

using KMeans and MiniBatchKMeans based on sparse data

2.3.2.1. Low-level parallelism¶

KMeans benefits from OpenMP based parallelism through Cython. Small

chunks of data (256 samples) are processed in parallel, which in addition

yields a low memory footprint. For more details on how to control the number of

threads, please refer to our Parallelism notes.

Examples:

Demonstration of k-means assumptions: Demonstrating when

k-means performs intuitively and when it does not

A demo of K-Means clustering on the handwritten digits data: Clustering handwritten digits

References:

“k-means++: The advantages of careful seeding”

Arthur, David, and Sergei Vassilvitskii,

Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete

algorithms, Society for Industrial and Applied Mathematics (2007)

2.3.2.2. Mini Batch K-Means¶

The MiniBatchKMeans is a variant of the KMeans algorithm

which uses mini-batches to reduce the computation time, while still attempting

to optimise the same objective function. Mini-batches are subsets of the input

data, randomly sampled in each training iteration. These mini-batches

drastically reduce the amount of computation required to converge to a local

solution. In contrast to other algorithms that reduce the convergence time of

k-means, mini-batch k-means produces results that are generally only slightly

worse than the standard algorithm.

The algorithm iterates between two major steps, similar to vanilla k-means.

In the first step, \(b\) samples are drawn randomly from the dataset, to form

a mini-batch. These are then assigned to the nearest centroid. In the second

step, the centroids are updated. In contrast to k-means, this is done on a

per-sample basis. For each sample in the mini-batch, the assigned centroid

is updated by taking the streaming average of the sample and all previous

samples assigned to that centroid. This has the effect of decreasing the

rate of change for a centroid over time. These steps are performed until

convergence or a predetermined number of iterations is reached.

MiniBatchKMeans converges faster than KMeans, but the quality

of the results is reduced. In practice this difference in quality can be quite

small, as shown in the example and cited reference.

Examples:

Comparison of the K-Means and MiniBatchKMeans clustering algorithms: Comparison of

KMeans and MiniBatchKMeans

Clustering text documents using k-means: Document clustering

using KMeans and MiniBatchKMeans based on sparse data

Online learning of a dictionary of parts of faces

References:

“Web Scale K-Means clustering”

D. Sculley, Proceedings of the 19th international conference on World

wide web (2010)

2.3.3. Affinity Propagation¶

AffinityPropagation creates clusters by sending messages between

pairs of samples until convergence. A dataset is then described using a small

number of exemplars, which are identified as those most representative of other

samples. The messages sent between pairs represent the suitability for one

sample to be the exemplar of the other, which is updated in response to the

values from other pairs. This updating happens iteratively until convergence,

at which point the final exemplars are chosen, and hence the final clustering

is given.

Affinity Propagation can be interesting as it chooses the number of

clusters based on the data provided. For this purpose, the two important

parameters are the preference, which controls how many exemplars are

used, and the damping factor which damps the responsibility and

availability messages to avoid numerical oscillations when updating these

messages.

The main drawback of Affinity Propagation is its complexity. The

algorithm has a time complexity of the order \(O(N^2 T)\), where \(N\)

is the number of samples and \(T\) is the number of iterations until

convergence. Further, the memory complexity is of the order

\(O(N^2)\) if a dense similarity matrix is used, but reducible if a

sparse similarity matrix is used. This makes Affinity Propagation most

appropriate for small to medium sized datasets.

Examples:

Demo of affinity propagation clustering algorithm: Affinity

Propagation on a synthetic 2D datasets with 3 classes.

Visualizing the stock market structure Affinity Propagation on

Financial time series to find groups of companies

Algorithm description:

The messages sent between points belong to one of two categories. The first is

the responsibility \(r(i, k)\),

which is the accumulated evidence that sample \(k\)

should be the exemplar for sample \(i\).

The second is the availability \(a(i, k)\)

which is the accumulated evidence that sample \(i\)

should choose sample \(k\) to be its exemplar,

and considers the values for all other samples that \(k\) should

be an exemplar. In this way, exemplars are chosen by samples if they are (1)

similar enough to many samples and (2) chosen by many samples to be

representative of themselves.

More formally, the responsibility of a sample \(k\)

to be the exemplar of sample \(i\) is given by:

\[r(i, k) \leftarrow s(i, k) - max [ a(i, k') + s(i, k') \forall k' \neq k ]\]

Where \(s(i, k)\) is the similarity between samples \(i\) and \(k\).

The availability of sample \(k\)

to be the exemplar of sample \(i\) is given by:

\[a(i, k) \leftarrow min [0, r(k, k) + \sum_{i'~s.t.~i' \notin \{i, k\}}{r(i', k)}]\]

To begin with, all values for \(r\) and \(a\) are set to zero,

and the calculation of each iterates until convergence.

As discussed above, in order to avoid numerical oscillations when updating the

messages, the damping factor \(\lambda\) is introduced to iteration process:

\[r_{t+1}(i, k) = \lambda\cdot r_{t}(i, k) + (1-\lambda)\cdot r_{t+1}(i, k)\]

\[a_{t+1}(i, k) = \lambda\cdot a_{t}(i, k) + (1-\lambda)\cdot a_{t+1}(i, k)\]

where \(t\) indicates the iteration times.

2.3.4. Mean Shift¶

MeanShift clustering aims to discover blobs in a smooth density of

samples. It is a centroid based algorithm, which works by updating candidates

for centroids to be the mean of the points within a given region. These

candidates are then filtered in a post-processing stage to eliminate

near-duplicates to form the final set of centroids.

The position of centroid candidates is iteratively adjusted using a technique called hill

climbing, which finds local maxima of the estimated probability density.

Given a candidate centroid \(x\) for iteration \(t\), the candidate

is updated according to the following equation:

\[x^{t+1} = x^t + m(x^t)\]

Where \(m\) is the mean shift vector that is computed for each

centroid that points towards a region of the maximum increase in the density of points.

To compute \(m\) we define \(N(x)\) as the neighborhood of samples within

a given distance around \(x\). Then \(m\) is computed using the following

equation, effectively updating a centroid to be the mean of the samples within

its neighborhood:

\[m(x) = \frac{1}{|N(x)|} \sum_{x_j \in N(x)}x_j - x\]

In general, the equation for \(m\) depends on a kernel used for density estimation.

The generic formula is:

\[m(x) = \frac{\sum_{x_j \in N(x)}K(x_j - x)x_j}{\sum_{x_j \in N(x)}K(x_j - x)} - x\]

In our implementation, \(K(x)\) is equal to 1 if \(x\) is small enough and is

equal to 0 otherwise. Effectively \(K(y - x)\) indicates whether \(y\) is in

the neighborhood of \(x\).

The algorithm automatically sets the number of clusters, instead of relying on a

parameter bandwidth, which dictates the size of the region to search through.

This parameter can be set manually, but can be estimated using the provided

estimate_bandwidth function, which is called if the bandwidth is not set.

The algorithm is not highly scalable, as it requires multiple nearest neighbor

searches during the execution of the algorithm. The algorithm is guaranteed to

converge, however the algorithm will stop iterating when the change in centroids

is small.

Labelling a new sample is performed by finding the nearest centroid for a

given sample.

Examples:

A demo of the mean-shift clustering algorithm: Mean Shift clustering

on a synthetic 2D datasets with 3 classes.

References:

“Mean shift: A robust approach toward feature space analysis”

D. Comaniciu and P. Meer, IEEE Transactions on Pattern Analysis and Machine Intelligence (2002)

2.3.5. Spectral clustering¶

SpectralClustering performs a low-dimension embedding of the

affinity matrix between samples, followed by clustering, e.g., by KMeans,

of the components of the eigenvectors in the low dimensional space.

It is especially computationally efficient if the affinity matrix is sparse

and the amg solver is used for the eigenvalue problem (Note, the amg solver

requires that the pyamg module is installed.)

The present version of SpectralClustering requires the number of clusters

to be specified in advance. It works well for a small number of clusters,

but is not advised for many clusters.

For two clusters, SpectralClustering solves a convex relaxation of the

normalized cuts

problem on the similarity graph: cutting the graph in two so that the weight of

the edges cut is small compared to the weights of the edges inside each

cluster. This criteria is especially interesting when working on images, where

graph vertices are pixels, and weights of the edges of the similarity graph are

computed using a function of a gradient of the image.

Warning

Transforming distance to well-behaved similarities

Note that if the values of your similarity matrix are not well

distributed, e.g. with negative values or with a distance matrix

rather than a similarity, the spectral problem will be singular and

the problem not solvable. In which case it is advised to apply a

transformation to the entries of the matrix. For instance, in the

case of a signed distance matrix, is common to apply a heat kernel:

similarity = np.exp(-beta * distance / distance.std())

See the examples for such an application.

Examples:

Spectral clustering for image segmentation: Segmenting objects

from a noisy background using spectral clustering.

Segmenting the picture of greek coins in regions: Spectral clustering

to split the image of coins in regions.

2.3.5.1. Different label assignment strategies¶

Different label assignment strategies can be used, corresponding to the

assign_labels parameter of SpectralClustering.

"kmeans" strategy can match finer details, but can be unstable.

In particular, unless you control the random_state, it may not be

reproducible from run-to-run, as it depends on random initialization.

The alternative "discretize" strategy is 100% reproducible, but tends

to create parcels of fairly even and geometrical shape.

The recently added "cluster_qr" option is a deterministic alternative that

tends to create the visually best partitioning on the example application

below.

assign_labels="kmeans"

assign_labels="discretize"

assign_labels="cluster_qr"

References:

“Multiclass spectral clustering”

Stella X. Yu, Jianbo Shi, 2003

“Simple, direct, and efficient multi-way spectral clustering”

Anil Damle, Victor Minden, Lexing Ying, 2019

2.3.5.2. Spectral Clustering Graphs¶

Spectral Clustering can also be used to partition graphs via their spectral

embeddings. In this case, the affinity matrix is the adjacency matrix of the

graph, and SpectralClustering is initialized with affinity='precomputed':

>>> from sklearn.cluster import SpectralClustering

>>> sc = SpectralClustering(3, affinity='precomputed', n_init=100,

... assign_labels='discretize')

>>> sc.fit_predict(adjacency_matrix)

References:

“A Tutorial on Spectral Clustering”

Ulrike von Luxburg, 2007

“Normalized cuts and image segmentation”

Jianbo Shi, Jitendra Malik, 2000

“A Random Walks View of Spectral Segmentation”

Marina Meila, Jianbo Shi, 2001

“On Spectral Clustering: Analysis and an algorithm”

Andrew Y. Ng, Michael I. Jordan, Yair Weiss, 2001

“Preconditioned Spectral Clustering for Stochastic

Block Partition Streaming Graph Challenge”

David Zhuzhunashvili, Andrew Knyazev

2.3.6. Hierarchical clustering¶

Hierarchical clustering is a general family of clustering algorithms that

build nested clusters by merging or splitting them successively. This

hierarchy of clusters is represented as a tree (or dendrogram). The root of the

tree is the unique cluster that gathers all the samples, the leaves being the

clusters with only one sample. See the Wikipedia page for more details.

The AgglomerativeClustering object performs a hierarchical clustering

using a bottom up approach: each observation starts in its own cluster, and

clusters are successively merged together. The linkage criteria determines the

metric used for the merge strategy:

Ward minimizes the sum of squared differences within all clusters. It is a

variance-minimizing approach and in this sense is similar to the k-means

objective function but tackled with an agglomerative hierarchical

approach.

Maximum or complete linkage minimizes the maximum distance between

observations of pairs of clusters.

Average linkage minimizes the average of the distances between all

observations of pairs of clusters.

Single linkage minimizes the distance between the closest

observations of pairs of clusters.

AgglomerativeClustering can also scale to large number of samples

when it is used jointly with a connectivity matrix, but is computationally

expensive when no connectivity constraints are added between samples: it

considers at each step all the possible merges.

FeatureAgglomeration

The FeatureAgglomeration uses agglomerative clustering to

group together features that look very similar, thus decreasing the

number of features. It is a dimensionality reduction tool, see

Unsupervised dimensionality reduction.

2.3.6.1. Different linkage type: Ward, complete, average, and single linkage¶

AgglomerativeClustering supports Ward, single, average, and complete

linkage strategies.

Agglomerative cluster has a “rich get richer” behavior that leads to

uneven cluster sizes. In this regard, single linkage is the worst

strategy, and Ward gives the most regular sizes. However, the affinity

(or distance used in clustering) cannot be varied with Ward, thus for non

Euclidean metrics, average linkage is a good alternative. Single linkage,

while not robust to noisy data, can be computed very efficiently and can

therefore be useful to provide hierarchical clustering of larger datasets.

Single linkage can also perform well on non-globular data.

Examples:

Various Agglomerative Clustering on a 2D embedding of digits: exploration of the

different linkage strategies in a real dataset.

2.3.6.2. Visualization of cluster hierarchy¶

It’s possible to visualize the tree representing the hierarchical merging of clusters

as a dendrogram. Visual inspection can often be useful for understanding the structure

of the data, though more so in the case of small sample sizes.

2.3.6.3. Adding connectivity constraints¶

An interesting aspect of AgglomerativeClustering is that

connectivity constraints can be added to this algorithm (only adjacent

clusters can be merged together), through a connectivity matrix that defines

for each sample the neighboring samples following a given structure of the

data. For instance, in the swiss-roll example below, the connectivity

constraints forbid the merging of points that are not adjacent on the swiss

roll, and thus avoid forming clusters that extend across overlapping folds of

the roll.

These constraint are useful to impose a certain local structure, but they

also make the algorithm faster, especially when the number of the samples

is high.

The connectivity constraints are imposed via an connectivity matrix: a

scipy sparse matrix that has elements only at the intersection of a row

and a column with indices of the dataset that should be connected. This

matrix can be constructed from a-priori information: for instance, you

may wish to cluster web pages by only merging pages with a link pointing

from one to another. It can also be learned from the data, for instance

using sklearn.neighbors.kneighbors_graph to restrict

merging to nearest neighbors as in this example, or

using sklearn.feature_extraction.image.grid_to_graph to

enable only merging of neighboring pixels on an image, as in the

coin example.

Examples:

A demo of structured Ward hierarchical clustering on an image of coins: Ward clustering

to split the image of coins in regions.

Hierarchical clustering: structured vs unstructured ward: Example of

Ward algorithm on a swiss-roll, comparison of structured approaches

versus unstructured approaches.

Feature agglomeration vs. univariate selection:

Example of dimensionality reduction with feature agglomeration based on

Ward hierarchical clustering.

Agglomerative clustering with and without structure

Warning

Connectivity constraints with single, average and complete linkage

Connectivity constraints and single, complete or average linkage can enhance

the ‘rich getting richer’ aspect of agglomerative clustering,

particularly so if they are built with

sklearn.neighbors.kneighbors_graph. In the limit of a small

number of clusters, they tend to give a few macroscopically occupied

clusters and almost empty ones. (see the discussion in

Agglomerative clustering with and without structure).

Single linkage is the most brittle linkage option with regard to this issue.

2.3.6.4. Varying the metric¶

Single, average and complete linkage can be used with a variety of distances (or

affinities), in particular Euclidean distance (l2), Manhattan distance

(or Cityblock, or l1), cosine distance, or any precomputed affinity

matrix.

l1 distance is often good for sparse features, or sparse noise: i.e.

many of the features are zero, as in text mining using occurrences of

rare words.

cosine distance is interesting because it is invariant to global

scalings of the signal.

The guidelines for choosing a metric is to use one that maximizes the

distance between samples in different classes, and minimizes that within

each class.

Examples:

Agglomerative clustering with different metrics

2.3.6.5. Bisecting K-Means¶

The BisectingKMeans is an iterative variant of KMeans, using

divisive hierarchical clustering. Instead of creating all centroids at once, centroids

are picked progressively based on a previous clustering: a cluster is split into two

new clusters repeatedly until the target number of clusters is reached.

BisectingKMeans is more efficient than KMeans when the number of

clusters is large since it only works on a subset of the data at each bisection

while KMeans always works on the entire dataset.

Although BisectingKMeans can’t benefit from the advantages of the "k-means++"

initialization by design, it will still produce comparable results than

KMeans(init="k-means++") in terms of inertia at cheaper computational costs, and will

likely produce better results than KMeans with a random initialization.

This variant is more efficient to agglomerative clustering if the number of clusters is

small compared to the number of data points.

This variant also does not produce empty clusters.

There exist two strategies for selecting the cluster to split:

bisecting_strategy="largest_cluster" selects the cluster having the most points

bisecting_strategy="biggest_inertia" selects the cluster with biggest inertia

(cluster with biggest Sum of Squared Errors within)

Picking by largest amount of data points in most cases produces result as

accurate as picking by inertia and is faster (especially for larger amount of data

points, where calculating error may be costly).

Picking by largest amount of data points will also likely produce clusters of similar

sizes while KMeans is known to produce clusters of different sizes.

Difference between Bisecting K-Means and regular K-Means can be seen on example

Bisecting K-Means and Regular K-Means Performance Comparison.

While the regular K-Means algorithm tends to create non-related clusters,

clusters from Bisecting K-Means are well ordered and create quite a visible hierarchy.

References:

“A Comparison of Document Clustering Techniques”

Michael Steinbach, George Karypis and Vipin Kumar,

Department of Computer Science and Egineering, University of Minnesota

(June 2000)

“Performance Analysis of K-Means and Bisecting K-Means Algorithms in Weblog Data”

K.Abirami and Dr.P.Mayilvahanan,

International Journal of Emerging Technologies in Engineering Research (IJETER)

Volume 4, Issue 8, (August 2016)

“Bisecting K-means Algorithm Based on K-valued Self-determining

and Clustering Center Optimization”

Jian Di, Xinyue Gou

School of Control and Computer Engineering,North China Electric Power University,

Baoding, Hebei, China (August 2017)

2.3.7. DBSCAN¶

The DBSCAN algorithm views clusters as areas of high density

separated by areas of low density. Due to this rather generic view, clusters

found by DBSCAN can be any shape, as opposed to k-means which assumes that

clusters are convex shaped. The central component to the DBSCAN is the concept

of core samples, which are samples that are in areas of high density. A

cluster is therefore a set of core samples, each close to each other

(measured by some distance measure)

and a set of non-core samples that are close to a core sample (but are not

themselves core samples). There are two parameters to the algorithm,

min_samples and eps,

which define formally what we mean when we say dense.

Higher min_samples or lower eps

indicate higher density necessary to form a cluster.

More formally, we define a core sample as being a sample in the dataset such

that there exist min_samples other samples within a distance of

eps, which are defined as neighbors of the core sample. This tells

us that the core sample is in a dense area of the vector space. A cluster

is a set of core samples that can be built by recursively taking a core

sample, finding all of its neighbors that are core samples, finding all of

their neighbors that are core samples, and so on. A cluster also has a

set of non-core samples, which are samples that are neighbors of a core sample

in the cluster but are not themselves core samples. Intuitively, these samples

are on the fringes of a cluster.

Any core sample is part of a cluster, by definition. Any sample that is not a

core sample, and is at least eps in distance from any core sample, is

considered an outlier by the algorithm.

While the parameter min_samples primarily controls how tolerant the

algorithm is towards noise (on noisy and large data sets it may be desirable

to increase this parameter), the parameter eps is crucial to choose

appropriately for the data set and distance function and usually cannot be

left at the default value. It controls the local neighborhood of the points.

When chosen too small, most data will not be clustered at all (and labeled

as -1 for “noise”). When chosen too large, it causes close clusters to

be merged into one cluster, and eventually the entire data set to be returned

as a single cluster. Some heuristics for choosing this parameter have been

discussed in the literature, for example based on a knee in the nearest neighbor

distances plot (as discussed in the references below).

In the figure below, the color indicates cluster membership, with large circles

indicating core samples found by the algorithm. Smaller circles are non-core

samples that are still part of a cluster. Moreover, the outliers are indicated

by black points below.

Examples:

Demo of DBSCAN clustering algorithm

Implementation

The DBSCAN algorithm is deterministic, always generating the same clusters

when given the same data in the same order. However, the results can differ when

data is provided in a different order. First, even though the core samples

will always be assigned to the same clusters, the labels of those clusters

will depend on the order in which those samples are encountered in the data.

Second and more importantly, the clusters to which non-core samples are assigned

can differ depending on the data order. This would happen when a non-core sample

has a distance lower than eps to two core samples in different clusters. By the

triangular inequality, those two core samples must be more distant than

eps from each other, or they would be in the same cluster. The non-core

sample is assigned to whichever cluster is generated first in a pass

through the data, and so the results will depend on the data ordering.

The current implementation uses ball trees and kd-trees

to determine the neighborhood of points,

which avoids calculating the full distance matrix

(as was done in scikit-learn versions before 0.14).

The possibility to use custom metrics is retained;

for details, see NearestNeighbors.

Memory consumption for large sample sizes

This implementation is by default not memory efficient because it constructs

a full pairwise similarity matrix in the case where kd-trees or ball-trees cannot

be used (e.g., with sparse matrices). This matrix will consume \(n^2\) floats.

A couple of mechanisms for getting around this are:

Use OPTICS clustering in conjunction with the

extract_dbscan method. OPTICS clustering also calculates the full

pairwise matrix, but only keeps one row in memory at a time (memory

complexity n).

A sparse radius neighborhood graph (where missing entries are presumed to

be out of eps) can be precomputed in a memory-efficient way and dbscan

can be run over this with metric='precomputed'. See

sklearn.neighbors.NearestNeighbors.radius_neighbors_graph.

The dataset can be compressed, either by removing exact duplicates if

these occur in your data, or by using BIRCH. Then you only have a

relatively small number of representatives for a large number of points.

You can then provide a sample_weight when fitting DBSCAN.

References:

“A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases

with Noise”

Ester, M., H. P. Kriegel, J. Sander, and X. Xu,

In Proceedings of the 2nd International Conference on Knowledge Discovery

and Data Mining, Portland, OR, AAAI Press, pp. 226–231. 1996

“DBSCAN revisited, revisited: why and how you should (still) use DBSCAN.”

Schubert, E., Sander, J., Ester, M., Kriegel, H. P., & Xu, X. (2017).

In ACM Transactions on Database Systems (TODS), 42(3), 19.

2.3.8. HDBSCAN¶

The HDBSCAN algorithm can be seen as an extension of DBSCAN

and OPTICS. Specifically, DBSCAN assumes that the clustering

criterion (i.e. density requirement) is globally homogeneous.

In other words, DBSCAN may struggle to successfully capture clusters

with different densities.

HDBSCAN alleviates this assumption and explores all possible density

scales by building an alternative representation of the clustering problem.

Note

This implementation is adapted from the original implementation of HDBSCAN,

scikit-learn-contrib/hdbscan based on [LJ2017].

2.3.8.1. Mutual Reachability Graph¶

HDBSCAN first defines \(d_c(x_p)\), the core distance of a sample \(x_p\), as the

distance to its min_samples th-nearest neighbor, counting itself. For example,

if min_samples=5 and \(x_*\) is the 5th-nearest neighbor of \(x_p\)

then the core distance is:

\[d_c(x_p)=d(x_p, x_*).\]

Next it defines \(d_m(x_p, x_q)\), the mutual reachability distance of two points

\(x_p, x_q\), as:

\[d_m(x_p, x_q) = \max\{d_c(x_p), d_c(x_q), d(x_p, x_q)\}\]

These two notions allow us to construct the mutual reachability graph

\(G_{ms}\) defined for a fixed choice of min_samples by associating each

sample \(x_p\) with a vertex of the graph, and thus edges between points

\(x_p, x_q\) are the mutual reachability distance \(d_m(x_p, x_q)\)

between them. We may build subsets of this graph, denoted as

\(G_{ms,\varepsilon}\), by removing any edges with value greater than \(\varepsilon\):

from the original graph. Any points whose core distance is less than \(\varepsilon\):

are at this staged marked as noise. The remaining points are then clustered by

finding the connected components of this trimmed graph.

Note

Taking the connected components of a trimmed graph \(G_{ms,\varepsilon}\) is

equivalent to running DBSCAN* with min_samples and \(\varepsilon\). DBSCAN* is a

slightly modified version of DBSCAN mentioned in [CM2013].

2.3.8.2. Hierarchical Clustering¶

HDBSCAN can be seen as an algorithm which performs DBSCAN* clustering across all

values of \(\varepsilon\). As mentioned prior, this is equivalent to finding the connected

components of the mutual reachability graphs for all values of \(\varepsilon\). To do this

efficiently, HDBSCAN first extracts a minimum spanning tree (MST) from the fully

-connected mutual reachability graph, then greedily cuts the edges with highest

weight. An outline of the HDBSCAN algorithm is as follows:

Extract the MST of \(G_{ms}\).

Extend the MST by adding a “self edge” for each vertex, with weight equal

to the core distance of the underlying sample.

Initialize a single cluster and label for the MST.

Remove the edge with the greatest weight from the MST (ties are

removed simultaneously).

Assign cluster labels to the connected components which contain the

end points of the now-removed edge. If the component does not have at least

one edge it is instead assigned a “null” label marking it as noise.

Repeat 4-5 until there are no more connected components.

HDBSCAN is therefore able to obtain all possible partitions achievable by

DBSCAN* for a fixed choice of min_samples in a hierarchical fashion.

Indeed, this allows HDBSCAN to perform clustering across multiple densities

and as such it no longer needs \(\varepsilon\) to be given as a hyperparameter. Instead

it relies solely on the choice of min_samples, which tends to be a more robust

hyperparameter.

HDBSCAN can be smoothed with an additional hyperparameter min_cluster_size

which specifies that during the hierarchical clustering, components with fewer

than minimum_cluster_size many samples are considered noise. In practice, one

can set minimum_cluster_size = min_samples to couple the parameters and

simplify the hyperparameter space.

References:

[CM2013]

Campello, R.J.G.B., Moulavi, D., Sander, J. (2013). Density-Based Clustering

Based on Hierarchical Density Estimates. In: Pei, J., Tseng, V.S., Cao, L.,

Motoda, H., Xu, G. (eds) Advances in Knowledge Discovery and Data Mining.

PAKDD 2013. Lecture Notes in Computer Science(), vol 7819. Springer, Berlin,

Heidelberg.

Density-Based Clustering Based on Hierarchical Density Estimates

[LJ2017]

L. McInnes and J. Healy, (2017). Accelerated Hierarchical Density Based

Clustering. In: IEEE International Conference on Data Mining Workshops (ICDMW),

2017, pp. 33-42.

Accelerated Hierarchical Density Based Clustering

2.3.9. OPTICS¶

The OPTICS algorithm shares many similarities with the DBSCAN

algorithm, and can be considered a generalization of DBSCAN that relaxes the

eps requirement from a single value to a value range. The key difference

between DBSCAN and OPTICS is that the OPTICS algorithm builds a reachability

graph, which assigns each sample both a reachability_ distance, and a spot

within the cluster ordering_ attribute; these two attributes are assigned

when the model is fitted, and are used to determine cluster membership. If

OPTICS is run with the default value of inf set for max_eps, then DBSCAN

style cluster extraction can be performed repeatedly in linear time for any

given eps value using the cluster_optics_dbscan method. Setting

max_eps to a lower value will result in shorter run times, and can be

thought of as the maximum neighborhood radius from each point to find other

potential reachable points.

The reachability distances generated by OPTICS allow for variable density

extraction of clusters within a single data set. As shown in the above plot,

combining reachability distances and data set ordering_ produces a

reachability plot, where point density is represented on the Y-axis, and

points are ordered such that nearby points are adjacent. ‘Cutting’ the

reachability plot at a single value produces DBSCAN like results; all points

above the ‘cut’ are classified as noise, and each time that there is a break

when reading from left to right signifies a new cluster. The default cluster

extraction with OPTICS looks at the steep slopes within the graph to find

clusters, and the user can define what counts as a steep slope using the

parameter xi. There are also other possibilities for analysis on the graph

itself, such as generating hierarchical representations of the data through

reachability-plot dendrograms, and the hierarchy of clusters detected by the

algorithm can be accessed through the cluster_hierarchy_ parameter. The

plot above has been color-coded so that cluster colors in planar space match

the linear segment clusters of the reachability plot. Note that the blue and

red clusters are adjacent in the reachability plot, and can be hierarchically

represented as children of a larger parent cluster.

Examples:

Demo of OPTICS clustering algorithm

Comparison with DBSCAN

The results from OPTICS cluster_optics_dbscan method and DBSCAN are

very similar, but not always identical; specifically, labeling of periphery

and noise points. This is in part because the first samples of each dense

area processed by OPTICS have a large reachability value while being close

to other points in their area, and will thus sometimes be marked as noise

rather than periphery. This affects adjacent points when they are

considered as candidates for being marked as either periphery or noise.

Note that for any single value of eps, DBSCAN will tend to have a

shorter run time than OPTICS; however, for repeated runs at varying eps

values, a single run of OPTICS may require less cumulative runtime than

DBSCAN. It is also important to note that OPTICS’ output is close to

DBSCAN’s only if eps and max_eps are close.

Computational Complexity

Spatial indexing trees are used to avoid calculating the full distance

matrix, and allow for efficient memory usage on large sets of samples.

Different distance metrics can be supplied via the metric keyword.

For large datasets, similar (but not identical) results can be obtained via

HDBSCAN. The HDBSCAN implementation is

multithreaded, and has better algorithmic runtime complexity than OPTICS,

at the cost of worse memory scaling. For extremely large datasets that

exhaust system memory using HDBSCAN, OPTICS will maintain \(n\) (as opposed

to \(n^2\)) memory scaling; however, tuning of the max_eps parameter

will likely need to be used to give a solution in a reasonable amount of

wall time.

References:

“OPTICS: ordering points to identify the clustering structure.”

Ankerst, Mihael, Markus M. Breunig, Hans-Peter Kriegel, and Jörg Sander.

In ACM Sigmod Record, vol. 28, no. 2, pp. 49-60. ACM, 1999.

2.3.10. BIRCH¶

The Birch builds a tree called the Clustering Feature Tree (CFT)

for the given data. The data is essentially lossy compressed to a set of

Clustering Feature nodes (CF Nodes). The CF Nodes have a number of

subclusters called Clustering Feature subclusters (CF Subclusters)

and these CF Subclusters located in the non-terminal CF Nodes

can have CF Nodes as children.

The CF Subclusters hold the necessary information for clustering which prevents

the need to hold the entire input data in memory. This information includes:

Number of samples in a subcluster.

Linear Sum - An n-dimensional vector holding the sum of all samples

Squared Sum - Sum of the squared L2 norm of all samples.

Centroids - To avoid recalculation linear sum / n_samples.

Squared norm of the centroids.

The BIRCH algorithm has two parameters, the threshold and the branching factor.

The branching factor limits the number of subclusters in a node and the

threshold limits the distance between the entering sample and the existing

subclusters.

This algorithm can be viewed as an instance or data reduction method,

since it reduces the input data to a set of subclusters which are obtained directly

from the leaves of the CFT. This reduced data can be further processed by feeding

it into a global clusterer. This global clusterer can be set by n_clusters.

If n_clusters is set to None, the subclusters from the leaves are directly

read off, otherwise a global clustering step labels these subclusters into global

clusters (labels) and the samples are mapped to the global label of the nearest subcluster.

Algorithm description:

A new sample is inserted into the root of the CF Tree which is a CF Node.

It is then merged with the subcluster of the root, that has the smallest

radius after merging, constrained by the threshold and branching factor conditions.

If the subcluster has any child node, then this is done repeatedly till it reaches

a leaf. After finding the nearest subcluster in the leaf, the properties of this

subcluster and the parent subclusters are recursively updated.

If the radius of the subcluster obtained by merging the new sample and the

nearest subcluster is greater than the square of the threshold and if the

number of subclusters is greater than the branching factor, then a space is temporarily

allocated to this new sample. The two farthest subclusters are taken and

the subclusters are divided into two groups on the basis of the distance

between these subclusters.

If this split node has a parent subcluster and there is room

for a new subcluster, then the parent is split into two. If there is no room,

then this node is again split into two and the process is continued

recursively, till it reaches the root.

BIRCH or MiniBatchKMeans?

BIRCH does not scale very well to high dimensional data. As a rule of thumb if

n_features is greater than twenty, it is generally better to use MiniBatchKMeans.

If the number of instances of data needs to be reduced, or if one wants a

large number of subclusters either as a preprocessing step or otherwise,

BIRCH is more useful than MiniBatchKMeans.

How to use partial_fit?

To avoid the computation of global clustering, for every call of partial_fit

the user is advised

To set n_clusters=None initially

Train all data by multiple calls to partial_fit.

Set n_clusters to a required value using

brc.set_params(n_clusters=n_clusters).

Call partial_fit finally with no arguments, i.e. brc.partial_fit()

which performs the global clustering.

References:

Tian Zhang, Raghu Ramakrishnan, Maron Livny

BIRCH: An efficient data clustering method for large databases.

https://www.cs.sfu.ca/CourseCentral/459/han/papers/zhang96.pdf

Roberto Perdisci

JBirch - Java implementation of BIRCH clustering algorithm

https://code.google.com/archive/p/jbirch

2.3.11. Clustering performance evaluation¶

Evaluating the performance of a clustering algorithm is not as trivial as

counting the number of errors or the precision and recall of a supervised

classification algorithm. In particular any evaluation metric should not

take the absolute values of the cluster labels into account but rather

if this clustering define separations of the data similar to some ground

truth set of classes or satisfying some assumption such that members

belong to the same class are more similar than members of different

classes according to some similarity metric.

2.3.11.1. Rand index¶

Given the knowledge of the ground truth class assignments

labels_true and our clustering algorithm assignments of the same

samples labels_pred, the (adjusted or unadjusted) Rand index

is a function that measures the similarity of the two assignments,

ignoring permutations:

>>> from sklearn import metrics

>>> labels_true = [0, 0, 0, 1, 1, 1]

>>> labels_pred = [0, 0, 1, 1, 2, 2]

>>> metrics.rand_score(labels_true, labels_pred)

0.66...

The Rand index does not ensure to obtain a value close to 0.0 for a

random labelling. The adjusted Rand index corrects for chance and

will give such a baseline.

>>> metrics.adjusted_rand_score(labels_true, labels_pred)

0.24...

As with all clustering metrics, one can permute 0 and 1 in the predicted

labels, rename 2 to 3, and get the same score:

>>> labels_pred = [1, 1, 0, 0, 3, 3]

>>> metrics.rand_score(labels_true, labels_pred)

0.66...

>>> metrics.adjusted_rand_score(labels_true, labels_pred)

0.24...

Furthermore, both rand_score adjusted_rand_score are

symmetric: swapping the argument does not change the scores. They can

thus be used as consensus measures:

>>> metrics.rand_score(labels_pred, labels_true)

0.66...

>>> metrics.adjusted_rand_score(labels_pred, labels_true)

0.24...

Perfect labeling is scored 1.0:

>>> labels_pred = labels_true[:]

>>> metrics.rand_score(labels_true, labels_pred)

1.0

>>> metrics.adjusted_rand_score(labels_true, labels_pred)

1.0

Poorly agreeing labels (e.g. independent labelings) have lower scores,

and for the adjusted Rand index the score will be negative or close to

zero. However, for the unadjusted Rand index the score, while lower,

will not necessarily be close to zero.:

>>> labels_true = [0, 0, 0, 0, 0, 0, 1, 1]

>>> labels_pred = [0, 1, 2, 3, 4, 5, 5, 6]

>>> metrics.rand_score(labels_true, labels_pred)

0.39...

>>> metrics.adjusted_rand_score(labels_true, labels_pred)

-0.07...

2.3.11.1.1. Advantages¶

Interpretability: The unadjusted Rand index is proportional

to the number of sample pairs whose labels are the same in both

labels_pred and labels_true, or are different in both.

Random (uniform) label assignments have an adjusted Rand index

score close to 0.0 for any value of n_clusters and

n_samples (which is not the case for the unadjusted Rand index

or the V-measure for instance).

Bounded range: Lower values indicate different labelings,

similar clusterings have a high (adjusted or unadjusted) Rand index,

1.0 is the perfect match score. The score range is [0, 1] for the

unadjusted Rand index and [-1, 1] for the adjusted Rand index.

No assumption is made on the cluster structure: The (adjusted or

unadjusted) Rand index can be used to compare all kinds of

clustering algorithms, and can be used to compare clustering

algorithms such as k-means which assumes isotropic blob shapes with

results of spectral clustering algorithms which can find cluster

with “folded” shapes.

2.3.11.1.2. Drawbacks¶

Contrary to inertia, the (adjusted or unadjusted) Rand index

requires knowledge of the ground truth classes which is almost

never available in practice or requires manual assignment by human

annotators (as in the supervised learning setting).

However (adjusted or unadjusted) Rand index can also be useful in a

purely unsupervised setting as a building block for a Consensus

Index that can be used for clustering model selection (TODO).

The unadjusted Rand index is often close to 1.0 even if the

clusterings themselves differ significantly. This can be understood

when interpreting the Rand index as the accuracy of element pair

labeling resulting from the clusterings: In practice there often is

a majority of element pairs that are assigned the different pair

label under both the predicted and the ground truth clustering

resulting in a high proportion of pair labels that agree, which

leads subsequently to a high score.

Examples:

Adjustment for chance in clustering performance evaluation:

Analysis of the impact of the dataset size on the value of

clustering measures for random assignments.

2.3.11.1.3. Mathematical formulation¶

If C is a ground truth class assignment and K the clustering, let us

define \(a\) and \(b\) as:

\(a\), the number of pairs of elements that are in the same set

in C and in the same set in K

\(b\), the number of pairs of elements that are in different sets

in C and in different sets in K

The unadjusted Rand index is then given by:

\[\text{RI} = \frac{a + b}{C_2^{n_{samples}}}\]

where \(C_2^{n_{samples}}\) is the total number of possible pairs

in the dataset. It does not matter if the calculation is performed on

ordered pairs or unordered pairs as long as the calculation is

performed consistently.

However, the Rand index does not guarantee that random label assignments

will get a value close to zero (esp. if the number of clusters is in

the same order of magnitude as the number of samples).

To counter this effect we can discount the expected RI \(E[\text{RI}]\) of

random labelings by defining the adjusted Rand index as follows:

\[\text{ARI} = \frac{\text{RI} - E[\text{RI}]}{\max(\text{RI}) - E[\text{RI}]}\]

References

Comparing Partitions

L. Hubert and P. Arabie, Journal of Classification 1985

Properties of the Hubert-Arabie adjusted Rand index

D. Steinley, Psychological Methods 2004

Wikipedia entry for the Rand index

Wikipedia entry for the adjusted Rand index

2.3.11.2. Mutual Information based scores¶

Given the knowledge of the ground truth class assignments labels_true and

our clustering algorithm assignments of the same samples labels_pred, the

Mutual Information is a function that measures the agreement of the two

assignments, ignoring permutations. Two different normalized versions of this

measure are available, Normalized Mutual Information (NMI) and Adjusted

Mutual Information (AMI). NMI is often used in the literature, while AMI was

proposed more recently and is normalized against chance:

>>> from sklearn import metrics

>>> labels_true = [0, 0, 0, 1, 1, 1]

>>> labels_pred = [0, 0, 1, 1, 2, 2]

>>> metrics.adjusted_mutual_info_score(labels_true, labels_pred)

0.22504...

One can permute 0 and 1 in the predicted labels, rename 2 to 3 and get

the same score:

>>> labels_pred = [1, 1, 0, 0, 3, 3]

>>> metrics.adjusted_mutual_info_score(labels_true, labels_pred)

0.22504...

All, mutual_info_score, adjusted_mutual_info_score and

normalized_mutual_info_score are symmetric: swapping the argument does

not change the score. Thus they can be used as a consensus measure:

>>> metrics.adjusted_mutual_info_score(labels_pred, labels_true)

0.22504...

Perfect labeling is scored 1.0:

>>> labels_pred = labels_true[:]

>>> metrics.adjusted_mutual_info_score(labels_true, labels_pred)

1.0

>>> metrics.normalized_mutual_info_score(labels_true, labels_pred)

1.0

This is not true for mutual_info_score, which is therefore harder to judge:

>>> metrics.mutual_info_score(labels_true, labels_pred)

0.69...

Bad (e.g. independent labelings) have non-positive scores:

>>> labels_true = [0, 1, 2, 0, 3, 4, 5, 1]

>>> labels_pred = [1, 1, 0, 0, 2, 2, 2, 2]

>>> metrics.adjusted_mutual_info_score(labels_true, labels_pred)

-0.10526...

2.3.11.2.1. Advantages¶

Random (uniform) label assignments have a AMI score close to 0.0

for any value of n_clusters and n_samples (which is not the

case for raw Mutual Information or the V-measure for instance).

Upper bound of 1: Values close to zero indicate two label

assignments that are largely independent, while values close to one

indicate significant agreement. Further, an AMI of exactly 1 indicates

that the two label assignments are equal (with or without permutation).

2.3.11.2.2. Drawbacks¶

Contrary to inertia, MI-based measures require the knowledge

of the ground truth classes while almost never available in practice or

requires manual assignment by human annotators (as in the supervised learning

setting).

However MI-based measures can also be useful in purely unsupervised setting as a

building block for a Consensus Index that can be used for clustering

model selection.

NMI and MI are not adjusted against chance.

Examples:

Adjustment for chance in clustering performance evaluation: Analysis of

the impact of the dataset size on the value of clustering measures

for random assignments. This example also includes the Adjusted Rand

Index.

2.3.11.2.3. Mathematical formulation¶

Assume two label assignments (of the same N objects), \(U\) and \(V\).

Their entropy is the amount of uncertainty for a partition set, defined by:

\[H(U) = - \sum_{i=1}^{|U|}P(i)\log(P(i))\]

where \(P(i) = |U_i| / N\) is the probability that an object picked at

random from \(U\) falls into class \(U_i\). Likewise for \(V\):

\[H(V) = - \sum_{j=1}^{|V|}P'(j)\log(P'(j))\]

With \(P'(j) = |V_j| / N\). The mutual information (MI) between \(U\)

and \(V\) is calculated by:

\[\text{MI}(U, V) = \sum_{i=1}^{|U|}\sum_{j=1}^{|V|}P(i, j)\log\left(\frac{P(i,j)}{P(i)P'(j)}\right)\]

where \(P(i, j) = |U_i \cap V_j| / N\) is the probability that an object

picked at random falls into both classes \(U_i\) and \(V_j\).

It also can be expressed in set cardinality formulation:

\[\text{MI}(U, V) = \sum_{i=1}^{|U|} \sum_{j=1}^{|V|} \frac{|U_i \cap V_j|}{N}\log\left(\frac{N|U_i \cap V_j|}{|U_i||V_j|}\right)\]

The normalized mutual information is defined as

\[\text{NMI}(U, V) = \frac{\text{MI}(U, V)}{\text{mean}(H(U), H(V))}\]

This value of the mutual information and also the normalized variant is not

adjusted for chance and will tend to increase as the number of different labels

(clusters) increases, regardless of the actual amount of “mutual information”

between the label assignments.

The expected value for the mutual information can be calculated using the

following equation [VEB2009]. In this equation,

\(a_i = |U_i|\) (the number of elements in \(U_i\)) and

\(b_j = |V_j|\) (the number of elements in \(V_j\)).

\[E[\text{MI}(U,V)]=\sum_{i=1}^{|U|} \sum_{j=1}^{|V|} \sum_{n_{ij}=(a_i+b_j-N)^+

}^{\min(a_i, b_j)} \frac{n_{ij}}{N}\log \left( \frac{ N.n_{ij}}{a_i b_j}\right)

\frac{a_i!b_j!(N-a_i)!(N-b_j)!}{N!n_{ij}!(a_i-n_{ij})!(b_j-n_{ij})!

(N-a_i-b_j+n_{ij})!}\]

Using the expected value, the adjusted mutual information can then be

calculated using a similar form to that of the adjusted Rand index:

\[\text{AMI} = \frac{\text{MI} - E[\text{MI}]}{\text{mean}(H(U), H(V)) - E[\text{MI}]}\]

For normalized mutual information and adjusted mutual information, the normalizing

value is typically some generalized mean of the entropies of each clustering.

Various generalized means exist, and no firm rules exist for preferring one over the

others. The decision is largely a field-by-field basis; for instance, in community

detection, the arithmetic mean is most common. Each

normalizing method provides “qualitatively similar behaviours” [YAT2016]. In our

implementation, this is controlled by the average_method parameter.

Vinh et al. (2010) named variants of NMI and AMI by their averaging method [VEB2010]. Their

‘sqrt’ and ‘sum’ averages are the geometric and arithmetic means; we use these

more broadly common names.

References

Strehl, Alexander, and Joydeep Ghosh (2002). “Cluster ensembles – a

knowledge reuse framework for combining multiple partitions”. Journal of

Machine Learning Research 3: 583–617.

doi:10.1162/153244303321897735.

Wikipedia entry for the (normalized) Mutual Information

Wikipedia entry for the Adjusted Mutual Information

[VEB2009]

Vinh, Epps, and Bailey, (2009). “Information theoretic measures

for clusterings comparison”. Proceedings of the 26th Annual International

Conference on Machine Learning - ICML ‘09.

doi:10.1145/1553374.1553511.

ISBN 9781605585161.

[VEB2010]

Vinh, Epps, and Bailey, (2010). “Information Theoretic Measures for

Clusterings Comparison: Variants, Properties, Normalization and

Correction for Chance”. JMLR

[YAT2016]

Yang, Algesheimer, and Tessone, (2016). “A comparative analysis of

community

detection algorithms on artificial networks”. Scientific Reports 6: 30750.

doi:10.1038/srep30750.

2.3.11.3. Homogeneity, completeness and V-measure¶

Given the knowledge of the ground truth class assignments of the samples,

it is possible to define some intuitive metric using conditional entropy

analysis.

In particular Rosenberg and Hirschberg (2007) define the following two

desirable objectives for any cluster assignment:

homogeneity: each cluster contains only members of a single class.

completeness: all members of a given class are assigned to the same

cluster.

We can turn those concept as scores homogeneity_score and

completeness_score. Both are bounded below by 0.0 and above by

1.0 (higher is better):

>>> from sklearn import metrics

>>> labels_true = [0, 0, 0, 1, 1, 1]

>>> labels_pred = [0, 0, 1, 1, 2, 2]

>>> metrics.homogeneity_score(labels_true, labels_pred)

0.66...

>>> metrics.completeness_score(labels_true, labels_pred)

0.42...

Their harmonic mean called V-measure is computed by

v_measure_score:

>>> metrics.v_measure_score(labels_true, labels_pred)

0.51...

This function’s formula is as follows:

\[v = \frac{(1 + \beta) \times \text{homogeneity} \times \text{completeness}}{(\beta \times \text{homogeneity} + \text{completeness})}\]

beta defaults to a value of 1.0, but for using a value less than 1 for beta:

>>> metrics.v_measure_score(labels_true, labels_pred, beta=0.6)

0.54...

more weight will be attributed to homogeneity, and using a value greater than 1:

>>> metrics.v_measure_score(labels_true, labels_pred, beta=1.8)

0.48...

more weight will be attributed to completeness.

The V-measure is actually equivalent to the mutual information (NMI)

discussed above, with the aggregation function being the arithmetic mean [B2011].

Homogeneity, completeness and V-measure can be computed at once using

homogeneity_completeness_v_measure as follows:

>>> metrics.homogeneity_completeness_v_measure(labels_true, labels_pred)

(0.66..., 0.42..., 0.51...)

The following clustering assignment is slightly better, since it is

homogeneous but not complete:

>>> labels_pred = [0, 0, 0, 1, 2, 2]

>>> metrics.homogeneity_completeness_v_measure(labels_true, labels_pred)

(1.0, 0.68..., 0.81...)

Note

v_measure_score is symmetric: it can be used to evaluate

the agreement of two independent assignments on the same dataset.

This is not the case for completeness_score and

homogeneity_score: both are bound by the relationship:

homogeneity_score(a, b) == completeness_score(b, a)

2.3.11.3.1. Advantages¶

Bounded scores: 0.0 is as bad as it can be, 1.0 is a perfect score.

Intuitive interpretation: clustering with bad V-measure can be

qualitatively analyzed in terms of homogeneity and completeness

to better feel what ‘kind’ of mistakes is done by the assignment.

No assumption is made on the cluster structure: can be used

to compare clustering algorithms such as k-means which assumes isotropic

blob shapes with results of spectral clustering algorithms which can

find cluster with “folded” shapes.

2.3.11.3.2. Drawbacks¶

The previously introduced metrics are not normalized with regards to

random labeling: this means that depending on the number of samples,

clusters and ground truth classes, a completely random labeling will

not always yield the same values for homogeneity, completeness and

hence v-measure. In particular random labeling won’t yield zero

scores especially when the number of clusters is large.

This problem can safely be ignored when the number of samples is more

than a thousand and the number of clusters is less than 10. For

smaller sample sizes or larger number of clusters it is safer to use

an adjusted index such as the Adjusted Rand Index (ARI).

These metrics require the knowledge of the ground truth classes while

almost never available in practice or requires manual assignment by

human annotators (as in the supervised learning setting).

Examples:

Adjustment for chance in clustering performance evaluation: Analysis of

the impact of the dataset size on the value of clustering measures

for random assignments.

2.3.11.3.3. Mathematical formulation¶

Homogeneity and completeness scores are formally given by:

\[h = 1 - \frac{H(C|K)}{H(C)}\]

\[c = 1 - \frac{H(K|C)}{H(K)}\]

where \(H(C|K)\) is the conditional entropy of the classes given

the cluster assignments and is given by:

\[H(C|K) = - \sum_{c=1}^{|C|} \sum_{k=1}^{|K|} \frac{n_{c,k}}{n}

\cdot \log\left(\frac{n_{c,k}}{n_k}\right)\]

and \(H(C)\) is the entropy of the classes and is given by:

\[H(C) = - \sum_{c=1}^{|C|} \frac{n_c}{n} \cdot \log\left(\frac{n_c}{n}\right)\]

with \(n\) the total number of samples, \(n_c\) and \(n_k\)

the number of samples respectively belonging to class \(c\) and

cluster \(k\), and finally \(n_{c,k}\) the number of samples

from class \(c\) assigned to cluster \(k\).

The conditional entropy of clusters given class \(H(K|C)\) and the

entropy of clusters \(H(K)\) are defined in a symmetric manner.

Rosenberg and Hirschberg further define V-measure as the harmonic

mean of homogeneity and completeness:

\[v = 2 \cdot \frac{h \cdot c}{h + c}\]

References

V-Measure: A conditional entropy-based external cluster evaluation

measure

Andrew Rosenberg and Julia Hirschberg, 2007

[B2011]

Identification and Characterization of Events in Social Media, Hila

Becker, PhD Thesis.

2.3.11.4. Fowlkes-Mallows scores¶

The Fowlkes-Mallows index (sklearn.metrics.fowlkes_mallows_score) can be

used when the ground truth class assignments of the samples is known. The

Fowlkes-Mallows score FMI is defined as the geometric mean of the

pairwise precision and recall:

\[\text{FMI} = \frac{\text{TP}}{\sqrt{(\text{TP} + \text{FP}) (\text{TP} + \text{FN})}}\]

Where TP is the number of True Positive (i.e. the number of pair

of points that belong to the same clusters in both the true labels and the

predicted labels), FP is the number of False Positive (i.e. the number

of pair of points that belong to the same clusters in the true labels and not

in the predicted labels) and FN is the number of False Negative (i.e. the

number of pair of points that belongs in the same clusters in the predicted

labels and not in the true labels).

The score ranges from 0 to 1. A high value indicates a good similarity

between two clusters.

>>> from sklearn import metrics

>>> labels_true = [0, 0, 0, 1, 1, 1]

>>> labels_pred = [0, 0, 1, 1, 2, 2]

>>> metrics.fowlkes_mallows_score(labels_true, labels_pred)

0.47140...

One can permute 0 and 1 in the predicted labels, rename 2 to 3 and get

the same score:

>>> labels_pred = [1, 1, 0, 0, 3, 3]

>>> metrics.fowlkes_mallows_score(labels_true, labels_pred)

0.47140...

Perfect labeling is scored 1.0:

>>> labels_pred = labels_true[:]

>>> metrics.fowlkes_mallows_score(labels_true, labels_pred)

1.0

Bad (e.g. independent labelings) have zero scores:

>>> labels_true = [0, 1, 2, 0, 3, 4, 5, 1]

>>> labels_pred = [1, 1, 0, 0, 2, 2, 2, 2]

>>> metrics.fowlkes_mallows_score(labels_true, labels_pred)

0.0

2.3.11.4.1. Advantages¶

Random (uniform) label assignments have a FMI score close to 0.0

for any value of n_clusters and n_samples (which is not the

case for raw Mutual Information or the V-measure for instance).

Upper-bounded at 1: Values close to zero indicate two label

assignments that are largely independent, while values close to one

indicate significant agreement. Further, values of exactly 0 indicate

purely independent label assignments and a FMI of exactly 1 indicates

that the two label assignments are equal (with or without permutation).

No assumption is made on the cluster structure: can be used

to compare clustering algorithms such as k-means which assumes isotropic

blob shapes with results of spectral clustering algorithms which can

find cluster with “folded” shapes.

2.3.11.4.2. Drawbacks¶

Contrary to inertia, FMI-based measures require the knowledge

of the ground truth classes while almost never available in practice or

requires manual assignment by human annotators (as in the supervised learning

setting).

References

E. B. Fowkles and C. L. Mallows, 1983. “A method for comparing two

hierarchical clusterings”. Journal of the American Statistical Association.

https://www.tandfonline.com/doi/abs/10.1080/01621459.1983.10478008

Wikipedia entry for the Fowlkes-Mallows Index

2.3.11.5. Silhouette Coefficient¶

If the ground truth labels are not known, evaluation must be performed using

the model itself. The Silhouette Coefficient

(sklearn.metrics.silhouette_score)

is an example of such an evaluation, where a

higher Silhouette Coefficient score relates to a model with better defined

clusters. The Silhouette Coefficient is defined for each sample and is composed

of two scores:

a: The mean distance between a sample and all other points in the same

class.

b: The mean distance between a sample and all other points in the next

nearest cluster.

The Silhouette Coefficient s for a single sample is then given as:

\[s = \frac{b - a}{max(a, b)}\]

The Silhouette Coefficient for a set of samples is given as the mean of the

Silhouette Coefficient for each sample.

>>> from sklearn import metrics

>>> from sklearn.metrics import pairwise_distances

>>> from sklearn import datasets

>>> X, y = datasets.load_iris(return_X_y=True)

In normal usage, the Silhouette Coefficient is applied to the results of a

cluster analysis.

>>> import numpy as np

>>> from sklearn.cluster import KMeans

>>> kmeans_model = KMeans(n_clusters=3, random_state=1).fit(X)

>>> labels = kmeans_model.labels_

>>> metrics.silhouette_score(X, labels, metric='euclidean')

0.55...

References

Peter J. Rousseeuw (1987). “Silhouettes: a Graphical Aid to the

Interpretation and Validation of Cluster Analysis”

. Computational and Applied Mathematics 20: 53–65.

2.3.11.5.1. Advantages¶

The score is bounded between -1 for incorrect clustering and +1 for highly

dense clustering. Scores around zero indicate overlapping clusters.

The score is higher when clusters are dense and well separated, which relates

to a standard concept of a cluster.

2.3.11.5.2. Drawbacks¶

The Silhouette Coefficient is generally higher for convex clusters than other

concepts of clusters, such as density based clusters like those obtained

through DBSCAN.

Examples:

Selecting the number of clusters with silhouette analysis on KMeans clustering : In this example

the silhouette analysis is used to choose an optimal value for n_clusters.

2.3.11.6. Calinski-Harabasz Index¶

If the ground truth labels are not known, the Calinski-Harabasz index

(sklearn.metrics.calinski_harabasz_score) - also known as the Variance

Ratio Criterion - can be used to evaluate the model, where a higher

Calinski-Harabasz score relates to a model with better defined clusters.

The index is the ratio of the sum of between-clusters dispersion and of

within-cluster dispersion for all clusters (where dispersion is defined as the

sum of distances squared):

>>> from sklearn import metrics

>>> from sklearn.metrics import pairwise_distances

>>> from sklearn import datasets

>>> X, y = datasets.load_iris(return_X_y=True)

In normal usage, the Calinski-Harabasz index is applied to the results of a

cluster analysis:

>>> import numpy as np

>>> from sklearn.cluster import KMeans

>>> kmeans_model = KMeans(n_clusters=3, random_state=1).fit(X)

>>> labels = kmeans_model.labels_

>>> metrics.calinski_harabasz_score(X, labels)

561.59...

2.3.11.6.1. Advantages¶

The score is higher when clusters are dense and well separated, which relates

to a standard concept of a cluster.

The score is fast to compute.

2.3.11.6.2. Drawbacks¶

The Calinski-Harabasz index is generally higher for convex clusters than other

concepts of clusters, such as density based clusters like those obtained

through DBSCAN.

2.3.11.6.3. Mathematical formulation¶

For a set of data \(E\) of size \(n_E\) which has been clustered into

\(k\) clusters, the Calinski-Harabasz score \(s\) is defined as the

ratio of the between-clusters dispersion mean and the within-cluster dispersion:

\[s = \frac{\mathrm{tr}(B_k)}{\mathrm{tr}(W_k)} \times \frac{n_E - k}{k - 1}\]

where \(\mathrm{tr}(B_k)\) is trace of the between group dispersion matrix

and \(\mathrm{tr}(W_k)\) is the trace of the within-cluster dispersion

matrix defined by:

\[W_k = \sum_{q=1}^k \sum_{x \in C_q} (x - c_q) (x - c_q)^T\]

\[B_k = \sum_{q=1}^k n_q (c_q - c_E) (c_q - c_E)^T\]

with \(C_q\) the set of points in cluster \(q\), \(c_q\) the center

of cluster \(q\), \(c_E\) the center of \(E\), and \(n_q\) the

number of points in cluster \(q\).

References

Caliński, T., & Harabasz, J. (1974).

“A Dendrite Method for Cluster Analysis”.

Communications in Statistics-theory and Methods 3: 1-27.

2.3.11.7. Davies-Bouldin Index¶

If the ground truth labels are not known, the Davies-Bouldin index

(sklearn.metrics.davies_bouldin_score) can be used to evaluate the

model, where a lower Davies-Bouldin index relates to a model with better

separation between the clusters.

This index signifies the average ‘similarity’ between clusters, where the

similarity is a measure that compares the distance between clusters with the

size of the clusters themselves.

Zero is the lowest possible score. Values closer to zero indicate a better

partition.

In normal usage, the Davies-Bouldin index is applied to the results of a

cluster analysis as follows:

>>> from sklearn import datasets

>>> iris = datasets.load_iris()

>>> X = iris.data

>>> from sklearn.cluster import KMeans

>>> from sklearn.metrics import davies_bouldin_score

>>> kmeans = KMeans(n_clusters=3, random_state=1).fit(X)

>>> labels = kmeans.labels_

>>> davies_bouldin_score(X, labels)

0.666...

2.3.11.7.1. Advantages¶

The computation of Davies-Bouldin is simpler than that of Silhouette scores.

The index is solely based on quantities and features inherent to the dataset

as its computation only uses point-wise distances.

2.3.11.7.2. Drawbacks¶

The Davies-Boulding index is generally higher for convex clusters than other

concepts of clusters, such as density based clusters like those obtained from

DBSCAN.

The usage of centroid distance limits the distance metric to Euclidean space.

2.3.11.7.3. Mathematical formulation¶

The index is defined as the average similarity between each cluster \(C_i\)

for \(i=1, ..., k\) and its most similar one \(C_j\). In the context of

this index, similarity is defined as a measure \(R_{ij}\) that trades off:

\(s_i\), the average distance between each point of cluster \(i\) and

the centroid of that cluster – also know as cluster diameter.

\(d_{ij}\), the distance between cluster centroids \(i\) and \(j\).

A simple choice to construct \(R_{ij}\) so that it is nonnegative and

symmetric is:

\[R_{ij} = \frac{s_i + s_j}{d_{ij}}\]

Then the Davies-Bouldin index is defined as:

\[DB = \frac{1}{k} \sum_{i=1}^k \max_{i \neq j} R_{ij}\]

References

Davies, David L.; Bouldin, Donald W. (1979).

“A Cluster Separation Measure”

IEEE Transactions on Pattern Analysis and Machine Intelligence.

PAMI-1 (2): 224-227.

Halkidi, Maria; Batistakis, Yannis; Vazirgiannis, Michalis (2001).

“On Clustering Validation Techniques”

Journal of Intelligent Information Systems, 17(2-3), 107-145.

Wikipedia entry for Davies-Bouldin index.

2.3.11.8. Contingency Matrix¶

Contingency matrix (sklearn.metrics.cluster.contingency_matrix)

reports the intersection cardinality for every true/predicted cluster pair.

The contingency matrix provides sufficient statistics for all clustering

metrics where the samples are independent and identically distributed and

one doesn’t need to account for some instances not being clustered.

Here is an example:

>>> from sklearn.metrics.cluster import contingency_matrix

>>> x = ["a", "a", "a", "b", "b", "b"]

>>> y = [0, 0, 1, 1, 2, 2]

>>> contingency_matrix(x, y)

array([[2, 1, 0],

[0, 1, 2]])

The first row of output array indicates that there are three samples whose

true cluster is “a”. Of them, two are in predicted cluster 0, one is in 1,

and none is in 2. And the second row indicates that there are three samples

whose true cluster is “b”. Of them, none is in predicted cluster 0, one is in

1 and two are in 2.

A confusion matrix for classification is a square

contingency matrix where the order of rows and columns correspond to a list

of classes.

2.3.11.8.1. Advantages¶

Allows to examine the spread of each true cluster across predicted

clusters and vice versa.

The contingency table calculated is typically utilized in the calculation

of a similarity statistic (like the others listed in this document) between

the two clusterings.

2.3.11.8.2. Drawbacks¶

Contingency matrix is easy to interpret for a small number of clusters, but

becomes very hard to interpret for a large number of clusters.

It doesn’t give a single metric to use as an objective for clustering

optimisation.

References

Wikipedia entry for contingency matrix

2.3.11.9. Pair Confusion Matrix¶

The pair confusion matrix

(sklearn.metrics.cluster.pair_confusion_matrix) is a 2x2

similarity matrix

\[\begin{split}C = \left[\begin{matrix}

C_{00} & C_{01} \\

C_{10} & C_{11}

\end{matrix}\right]\end{split}\]

between two clusterings computed by considering all pairs of samples and

counting pairs that are assigned into the same or into different clusters

under the true and predicted clusterings.

It has the following entries:

\(C_{00}\) : number of pairs with both clusterings having the samples

not clustered together

\(C_{10}\) : number of pairs with the true label clustering having the

samples clustered together but the other clustering not having the samples

clustered together

\(C_{01}\) : number of pairs with the true label clustering not having

the samples clustered together but the other clustering having the samples

clustered together

\(C_{11}\) : number of pairs with both clusterings having the samples

clustered together

Considering a pair of samples that is clustered together a positive pair,

then as in binary classification the count of true negatives is

\(C_{00}\), false negatives is \(C_{10}\), true positives is

\(C_{11}\) and false positives is \(C_{01}\).

Perfectly matching labelings have all non-zero entries on the

diagonal regardless of actual label values:

>>> from sklearn.metrics.cluster import pair_confusion_matrix

>>> pair_confusion_matrix([0, 0, 1, 1], [0, 0, 1, 1])

array([[8, 0],

[0, 4]])

>>> pair_confusion_matrix([0, 0, 1, 1], [1, 1, 0, 0])

array([[8, 0],

[0, 4]])

Labelings that assign all classes members to the same clusters

are complete but may not always be pure, hence penalized, and

have some off-diagonal non-zero entries:

>>> pair_confusion_matrix([0, 0, 1, 2], [0, 0, 1, 1])

array([[8, 2],

[0, 2]])

The matrix is not symmetric:

>>> pair_confusion_matrix([0, 0, 1, 1], [0, 0, 1, 2])

array([[8, 0],

[2, 2]])

If classes members are completely split across different clusters, the

assignment is totally incomplete, hence the matrix has all zero

diagonal entries:

>>> pair_confusion_matrix([0, 0, 0, 0], [0, 1, 2, 3])

array([[ 0, 0],

[12, 0]])

References

“Comparing Partitions”

L. Hubert and P. Arabie, Journal of Classification 1985

Show this page source

sklearn.cluster.KMeans-scikit-learn中文社区

安装

用户指南

API

案例

入门

教程

更新日志

词汇表

常见问题

交流群

Toggle Menu

CDA数据科学研究院提供翻译支持

sklearn.cluster.KMeans

sklearn.cluster.KMeans¶

class sklearn.cluster.KMeans(n_clusters=8, *, init='k-means++', n_init=10, max_iter=300, tol=0.0001, precompute_distances='deprecated', verbose=0, random_state=None, copy_x=True, n_jobs='deprecated', algorithm='auto')

[源码]

K-均值聚类

在用户指南中阅读更多内容。

参数

方法

n_clusters

int, default=8要形成的簇数以及要生成的质心数。

init

{‘k-means++’, ‘random’, ndarray, callable}, default=’k-means++’初始化方法：‘k-means++’：明智地选择初始聚类中心进行k均值聚类，加快收敛速度.有关详细信息，请参阅k_init中的Notes部分。‘random’：从初始质心的数据中随机选择n_clusters观测(行)。如果一个ndarray被传递，它应该是形状的(n_clusters, n_features)，并给出初始中心。如果传递了一个可调用函数，它应该接受参数X、n_clusters和一个随机状态，并返回一个初始化。

max_iter

int, default=300相对容忍度与Frobenius范数，连续两次迭代之间的聚类中心的差异声明收敛。不建议将其设置为tol=0，因为由于舍入错误，可能永远不会声明收敛。用一个很小的数字代替。

precompute_distances

{‘auto’, True, False}, default=’auto’预计算距离(速度更快，但占用更多内存)。‘auto’：如果n个样本n_samples * n_clusters > 1200万，不预先计算距离。这相当于使用双精度的每个作业大约100 MB的开销。True :总是预先计算距离False：永远不要预先计算距离。自0.23版本起被弃用：在0.22版本中不推荐使用‘precompute_distances’，并将在0.25中删除。它没有效果。

verbose

int, default=0详细模式

random_state

int, RandomState instance, default=None确定用于质心初始化的随机数生成。使用整数使随机性确定。见Glossary。

copy_x

bool, default=True当预计算距离时，首先对数据进行中心化比较精确.如果copy_x为True(默认值)，则不会修改原始数据。如果为False，则对原始数据进行修改，并在函数返回之前将其放回，但可以通过减去和添加数据均值来引入较小的数值差异。注意，如果原始数据不是C-contiguous的，即使copy_x为False，也会复制。如果原始数据稀疏，但不采用CSR格式，即使copy_x为false，也会复制。

n_jobs

int, default=None用于计算的OpenMP线程数。并行性是在主cython循环上按样本顺序进行的，该循环将每个样本分配到其最近的中心。None或-1意味着使用所有处理器。从0.23版n_jobs开始不推荐使用*：从0.23版*开始不推荐使用，并将在0.25版中删除。

algorithm

{“auto”, “full”, “elkan”}, default=”auto”表示K-means要使用的算法。经典的EM式算法是“full”的。通过三角不等式，对于具有定义良好的簇的数据，“elkan”变化更为有效。但是，由于分配了一个额外的形状数组(n_samples, n_clusters)，所以内存更多。目前，“auto”(为向后兼容而保留)选择了“Elkan”，但它将来可能会改变，以获得更好的启发式。*在版本0.18中更改：*添加了Elkan算法

属性

说明

cluster_centers_

ndarray of shape (n_clusters, n_features)簇中心坐标。如果算法在完全收敛之前停止(请参阅tol和max_iter)，这些将与labels_比配。

labels_

ndarray of shape (n_samples,)每一点的标签

inertia_

float样本到其最近的聚类中心的平方距离之和。

n_iter_

int运行的迭代次数

另见

MiniBatchKMeans

替代性在线实施，使用迷你批次对中心位置进行增量更新。对于大规模学习（例如n_samples> 10k），MiniBatchKMeans可能比默认的批处理实现要快得多。

注

k-均值问题采用Lloyd’s算法或Elkan’s算法求解。

平均复杂度为O(KnT)，n为样本数，T为迭代次数。

最坏的情形复杂度由O(n^(k+2/p))给出，n=n_samples，p=n_features。 (D. Arthur and S. Vassilvitskii, ‘How slow is the k-means method?’ SoCG2006)

在实践中，k均值算法是非常快速的(可用的最快的聚类算法之一)，但它是在局部收敛极小。这就是为什么多次重新启动它是有用的。

如果算法在完全收敛之前停止(因为tol或max_iter)， labels_和cluster_centers将不一致，也就是说，cluster_centers不会是每个簇中各点的均值。此外，估计器将在最后一次迭代后重新分配 labels_，使labels_与训练集上的预测一致。

示例

>>> from sklearn.cluster import KMeans>>> import numpy as np>>> X = np.array([[1, 2], [1, 4], [1, 0],... [10, 2], [10, 4], [10, 0]])>>> kmeans = KMeans(n_clusters=2, random_state=0).fit(X)>>> kmeans.labels_array([1, 1, 1, 0, 0, 0], dtype=int32)>>> kmeans.predict([[0, 0], [12, 3]])array([1, 0], dtype=int32)>>> kmeans.cluster_centers_array([[10., 2.], [ 1., 2.]])

方法

说明

fit(self, X[, y, sample_weight])

计算k-均值聚类

fit_predict(self, X[, y, sample_weight])

计算聚类中心并预测每个样本的聚类索引

fit_transform(self, X[, y, sample_weight])

计算聚类并将X变换成簇距离空间

get_params(self[, deep])

获取此估计器的参数

predict(self, X[, sample_weight])

预测X中每个样本所属的最接近的聚类

set_params(self, **params)

设置此估计器的参数

transform(self, X)

将X转换为簇距离空间

__init__(self, n_clusters=8, *, init='k-means++', n_init=10, max_iter=300, tol=0.0001, precompute_distances='deprecated', verbose=0, random_state=None, copy_x=True, n_jobs='deprecated', algorithm='auto')

[源码]

初始化self。请参阅help(type(self))以获得准确的说明。

fit(self, X, y=None, sample_weight=None)

[源码]

计算k-均值聚类

参数

说明

{array-like, sparse matrix} of shape (n_samples, n_features), or (n_samples, n_samples) 要聚类的训练实例。必须注意的是，数据将转换为C顺序，如果给定的数据不是C-连续的，这将导致内存副本。如果一个稀疏矩阵被传递，如果它不是CSR格式，它将被复制。

Ignored 未使用，在此按约定呈现为API一致性。

sample_weight

array-like of shape (n_samples,), default=NoneX中每个观测值的权重，如果没有，则所有观测值都被赋予相同的权重。

返回值

说明

self

拟合的估计器

fit_predict(self, X, y=None, sample_weight=None)

[源码]

计算聚类中心并预测每个样本的聚类索引

方便方法；等价于调用fit(X)后再predict(X)。

参数

说明

{array-like, sparse matrix} of shape (n_samples, n_features) 要转换的新数据

Ignored 未使用，在此按约定呈现为API一致性。

sample_weight

array-like of shape (n_samples,), default=None X中每个观测值的权重，如果没有，则所有观测值都被赋予相同的权重。

返回值

说明

labels

ndarray of shape (n_samples,)每个样本所属的聚类索引

fit_transform(self, X, y=None, sample_weight=None)

[源码]

计算聚类并将X变换成簇距离空间

等效于 fit(X).transform(X)，但更有效地实现。

参数

说明

{array-like, sparse matrix} of shape (n_samples, n_features) 要转换的新数据

Ignored 未使用，在此按约定呈现为API一致性。

sample_weight

array-like of shape (n_samples,), default=None X中每个观测值的权重，如果没有，则所有观测值都被赋予相同的权重。

返回值

说明

labels

array of shape (n_samples, n_clusters) X在新空间中的变换

get_params(self, deep=True)

[源码]

获取此估计器的参数

参数

说明

deep

bool, default=True如果为True，则将返回此估计器的参数和所包含的作为估计量的子对象。

返回值

说明

params

mapping of string to any映射到其值的参数名称

predict(self, X, sample_weight=None)

[源码]

预测X中每一个样本所属的最接近的聚类。

在矢量量化文献中，cluster_centers_被称为代码簿， predict返回的每个值都是代码簿中最接近的代码的索引。

参数

说明

{array-like, sparse matrix} of shape (n_samples, n_features) 要转换的新数据

sample_weight

array-like of shape (n_samples,), default=None X中每个观测值的权重，如果没有，则所有观测值都被赋予相同的权重。

返回值

说明

labels

ndarray of shape (n_samples,) 每个样本所属的聚类索引

score(self, X, y=None, sample_weight=None)

[源码]

K-均值目标上X值的相反

参数

说明

{array-like, sparse matrix} of shape (n_samples, n_features) 要转换的新数据

Ignored 未使用，在此按约定呈现为API一致性。

sample_weight

array-like of shape (n_samples,), default=None X中每个观测值的权重，如果没有，则所有观测值都被赋予相同的权重。

返回值

说明

score

float 与K-均值目标上的X值相反.

set_params(self, **params)

[源码]

设置此估计器的参数

该方法适用于简单估计器以及嵌套对象(例如pipelines)。后者具有表单的 __参数，这样就可以更新嵌套对象的每个组件。

参数

说明

**params

dict估计器参数

返回值

说明书

self

object估计器实例

transform(self, X)

[源码]

将X转换成簇距空间。

在新的空间中，每个维度都是到簇中心的距离。请注意，即使X是稀疏的，转换返回的数组通常也是密集的。

参数

说明

{array-like, sparse matrix} of shape (n_samples, n_features) 要转换的新数据

返回值

说明

labels

array of shape (n_samples, n_clusters) X在新空间中的变换

sklearn.cluster.KMeans使用示例¶

k-均值假设的证明

矢量量化实例

K-means聚类

基于K均值的颜色量化

k均值初始化影响的实证评价

K-Means和MiniBatchKMeans聚类算法的比较

手写数字数据上K均值聚类的一个演示

scikit-learn 0.23中的发布要点

基于KMeans聚类分析使用silhouette选择聚类的数目

使用k-means聚类文本文档

sklearn（六）-K-Means k均值聚类算法 - 知乎

sklearn（六）-K-Means k均值聚类算法 - 知乎首发于数据分析切换模式写文章登录/注册sklearn（六）-K-Means k均值聚类算法菠萝的骑行小屋爱骑车的数据人K-Means是什么k均值聚类算法（k-means clustering algorithm）是一种迭代求解的聚类分析算法，将数据集中某些方面相似的数据进行分组组织的过程，聚类通过发现这种内在结构的技术，而k均值是聚类算法中最著名的算法，无监督学习，步骤为：预将数据集分为k组（k有用户指定），随机选择k个对象作为初始的聚类中心，然后计算每个对象与各个种子类聚中心之间的距离，把每个对象分配给距离它最近的聚类中心。聚类中心以及分配给它们的对象就代表一个聚类。每分配一个样本，聚类的聚类中心会根据聚类中现有的对象被重新计算。这个过程将不断重复知道满足某个终止条件。聚类条件可以分两种：1.没有或者设置的最小数目的对象被重新分配给不同的聚类；2.没有或者设置的最小数目的聚类中心再发生变化。在K-Means基础之上，有一些优化变体方法：初始化优化K-Means++，距离计算优化elkan K-Means，大数据情况下的优化Mini Batch K-Means对于给定的数据集，按照数据集之间的距离大小，将数据集划分为k个族。让族内的点尽量的紧密的连在一起，而让族间的距离尽量的大。初始化方法通常使用的初始化方法有Forgy和随机划分(Random Partition)方法 [9] 。Forgy方法随机地从数据集中选择k个观测作为初始的均值点；而随机划分方法则随机地为每一观测指定聚类，然后运行“更新(Update)”步骤,即计算随机分配的各聚类的图心，作为初始的均值点。Forgy方法易于使得初始均值点散开，随机划分方法则把均值点都放到靠近数据集中心的地方。参考Hamerly et al的文章 [9] ，可知随机划分方法一般更适用于k-调和均值和模糊k-均值算法。对于期望-最大化(EM)算法和标准k-均值算法，Forgy方法作为初始化方法的表现会更好一些。这是一个启发式算法，无法保证收敛到全局最优解，并且聚类的结果会依赖于初始的聚类。又因为算法的运行速度通常很快，所以一般都以不同的起始状态运行多次来得到更好的结果。不过，在最差的情况下，k-均值算法会收敛地特别慢：尤其是已经证明了存在这一的点集（甚至在2维空间中），使得k-均值算法收敛的时间达到指数级。好在在现实中，这样的点集几乎不会出现：因为k-均值算法的平滑运行时间是多项式时间的。优缺点各是什么缺点：聚类数目k是一个输入参数。选择不恰当的k值可能会导致糟糕的聚类结构。解决办法：进行特征检查（通过交叉验证）来决定数据集的聚类数目2. 收敛到局部最优解，可能导致“反直观”的错误结果。3. 不能和任意的距离函数一起使用，不能处理非数值数据4. 不是凸的数据集比较难收敛5. 对迭代方法，得到的结果只是局部最优6. 对噪音和异常点比较的敏感7. 如果隐含的数据类别不平衡，则聚类效果不佳优点：原理比较简单；实现也很容易，收敛速度快；聚类效果较优；算法的可解释度比较强；主要需要调参仅仅是族数k为什么出现（为什么需要这个技术）解决什么问题应用：在巨大的数据集上，也非常容易部署实施在市场划分，机器视觉，地质统计学，天文学和农业得到成功应用经常作为其他算法的预处理步骤。K-Means初始化优化K-Means++K-Means++的对于初始化质心的优化策略也很简单，如下：　　　　a) 从输入的数据点集合中随机选择一个点作为第一个聚类中心μ1μ1　　　　b) 对于数据集中的每一个点xixi，计算它与已选择的聚类中心中最近聚类中心的距离D(xi)=argmin||xi−μr||22r=1,2,...kselectedD(xi)=argmin||xi−μr||22r=1,2,...kselected　　　　c) 选择一个新的数据点作为新的聚类中心，选择的原则是：D(x)D(x)较大的点，被选取作为聚类中心的概率较大　　　　d) 重复b和c直到选择出k个聚类质心　　　　e) 利用这k个质心来作为初始化质心去运行标准的K-Means算法K-Means距离计算优化elkan K-Means在传统的k算法中，每轮迭代时，需要计算所有的样本点到所有的质心的距离，因此存在优化的空间，减少不必要的计算。elkan K-Means利用了两边之和大于等于第三边,以及两边之差小于第三边的三角形性质，来减少距离的计算。大样本优化Mini Batch K-Means在具有数据量非常大情况下，使用该算法，该算法只使用数据集中的一部分做传统的k算法，避免数据集过大导致的计算难题，速度大大加快，待久就是精确度是会有一些降低的。一般来说，这个降低的幅度在可接受的范围之内。这个用来训练的量，占比原始数据多少，一般通过无放回的随机采样得到的。通过采用不同的随机采样集来得到聚类族，选择其中最优的聚类族。K-Means与KNNk是无监督学习的聚类算法，没有样本输出，有明显的训练过程knn是监督学习的分类算法，有对应样本输出，基本不需要训练两个算法都包含一个过程，找出和某一个最近的点，两者都利用了最近邻的思想聚类与分类的区别分类：类别是已知的，通过对已知分类的数据进行训练和学习，找到这些不同类的特征，再对未分类的数据进行分类。属于监督学习。聚类：事先不知道数据会分为几类，通过聚类分析将数据聚合成几个群体。聚类不需要对数据进行训练和学习。属于无监督学习。关于监督学习和无监督学习，这里给一个简单的介绍：是否有监督，就看输入数据是否有标签，输入数据有标签，则为有监督学习，否则为无监督学习。sklearn k-Means 导包：from sklearn.cluster import KMeans传参详解max_iter: 最大的迭代次数,如果是凸数据集可以不管这个值,如果不是凸的,可能很难收敛,此时需要指定最大的迭代次数让算法可以及时退出循环.新建完对象以后，常用方法包括：fit(X)：该函数对数据x进行聚类predict ：使用该函数对新数据类别的预测cluster_centers_：使用该函数获取聚类中心lables_：获取训练数据所属的类别inertia_：获取每个点到聚类中心的距离和。kmeans.fit_predict(data) X = np.array([[1, 2], [1, 4], [1, 0], [4, 2], [4, 4], [4, 0]])#此处要进行np的import import numpy as np

kmeans = KMeans(n_clusters=2, random_state=0)#新建KMeans对象，并传入参数

kmeans.fit(X)#进行训练

print(kmeans.labels_)

print(kmeans.predict([[0, 0], [4, 4]])) # 预测

print(kmeans.cluster_centers_)标准化# sklearn.preprocessing import StandardScaler 使用

# Standardization 标准化:将特征数据的分布调整为标准正太分布,也叫高斯分布,也就是使得数据的均值为0(所有数据之和除以数据点的个数),方差为1(表示数据集中数据点的离散程度).

# 标准化的原因在于如果有些特征的方差过大,则会主导目标函数从而使参数估计其无法正确地区学习其他特征.

# 标准化的过程分为两步:去均值的中心化(均值变为0),方差的规模化(方差变为1)

std = StandardScaler()

data = std.fit_transform(data[["field1", "field2", "field3", "field4", "field5", "field6", 'field7']])

# 将标准化后的数据转换为原始数据。

std.inverse_transform() k-means 寻找最佳k值.import pandas as pd

from sklearn.cluster import KMeans

import matplotlib.pyplot as plt

df_features = pd.read_csv(r'C:\预处理后数据.csv',encoding='gbk') # 读入数据

'利用SSE选择k'

SSE = [] # 存放每次结果的误差平方和

for k in range(1,9):

estimator = KMeans(n_clusters=k) # 构造聚类器

estimator.fit(df_features[['R','F','M']])

SSE.append(estimator.inertia_)

X = range(1,9)

plt.xlabel('k')

plt.ylabel('SSE')

plt.plot(X,SSE,'o-')

plt.show()相关理论:K-means聚类最优k值的选取_qq_15738501的博客-CSDN博客手肘法?的核心指标是sse(sum of the squared errors 误差平方和)核心思想是:随着聚类数k的增大,样本划分会更加精细,每个簇的聚合程度会逐渐提高,那么误差平方和sse自然会逐渐变小.ordata

from sklearn.model_selection import train_test_split, GridSearchCV

param_test1 = {'n_clusters': np.arange(2, 25, 1)}

gsearch1 = GridSearchCV(estimator=KMeans(init='k-means++', random_state=42), param_grid=param_test1, cv=5)

gsearch1.fit(data)

score_list = -pd.DataFrame(gsearch1.cv_results_)['mean_test_score']

print('n_clusters 值------------ k值')

print(gsearch1.best_params_, gsearch1.best_score_)其他方法：1）fit_predict（X）：先对X进行训练并预测X中每个实例的类，等于先调用fit（X）后调用predict（X），返回X的每个类；

2）transform（X）：将X进行转换，转换为K列的矩阵，其中每行为一个实例，每个实例包含K个数值（K为传入的类数量），第i列为这个实例到第K个聚类中心的距离；

3）fit_transform（X）：类似（1），先进行fit之后进行transform；

4）score（X）：输入样本（这里的样本不是训练样本，而是其他传入的测试样本）到他们的类中心距离和，然后取负数（取负数是因为距离越大值越小）。参考：编辑于 2020-09-16 11:40K-理论sklearn赞同 9添加评论分享喜欢收藏申请转载文章被以下专栏收录数据分析统计学理论，数据采集，清洗，集成，可视化，挖

sklearn中的K-means算法 - 知乎

sklearn中的K-means算法 - 知乎切换模式写文章登录/注册sklearn中的K-means算法憧憬不负现在，不惧将来。共勉。sklearn中的K-means算法目录：1 传统K-means聚类2 非线性边界聚类3 预测结果与真实标签的匹配4 聚类结果的混淆矩阵参考文章：K-means算法实现：文章介绍了k-means算法的基本原理和scikit中封装的kmeans库的基本参数的含义K-means源码解读：这篇文章解读了scikit中kmeans的部分源码本例的notebook笔记文件：git仓库首先导入必须的库：from matplotlib import pyplot as plt

from sklearn.metrics import accuracy_score

import numpy as np

import seaborn as sns; sns.set()

%matplotlib inline1 传统K-means聚类构造数据集from sklearn.datasets.samples_generator import make_blobs

X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)

plt.scatter(X[:,0], X[:, 1], s=50)from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=4)

kmeans.fit(X)

y_kmeans = kmeans.predict(X)绘制聚类结果，画出聚类中心plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50, cmap='viridis')

centers = kmeans.cluster_centers_

plt.scatter(centers[:,0], centers[:, 1], c='black', s=80, marker='x')2 非线性边界聚类对于非线性边界的kmeans聚类的介绍，查阅于《python数据科学手册》P410构造数据from sklearn.datasets import make_moons

X, y = make_moons(200, noise=0.05, random_state=0)传统kmeans聚类失败的情况labels = KMeans(n_clusters=2, random_state=0).fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=labels, s=50, cmap='viridis')应用核方法，将数据投影到更高纬的空间，变成线性可分from sklearn.cluster import SpectralClustering

model = SpectralClustering(n_clusters=2, affinity='nearest_neighbors', assign_labels='kmeans')

labels = model.fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=labels, s=50, cmap='viridis')3 预测结果与真实标签的匹配手写数字识别例子from sklearn.datasets import load_digits

digits = load_digits()进行聚类kmeans = KMeans(n_clusters=10, random_state=0)

clusters = kmeans.fit_predict(digits.data)

kmeans.cluster_centers_.shape

(10, 64)可以将这些族中心点看做是具有代表性的数字fig, ax = plt.subplots(2, 5, figsize=(8, 3))

centers = kmeans.cluster_centers_.reshape(10, 8, 8)

for axi, center in zip(ax.flat, centers):

axi.set(xticks=[], yticks=[])

axi.imshow(center, interpolation='nearest', cmap=plt.cm.binary)进行众数匹配from scipy.stats import mode

labels = np.zeros_like(clusters)

for i in range(10):

#得到聚类结果第i类的 True Flase 类型的index矩阵

mask = (clusters ==i)

#根据index矩阵，找出这些target中的众数，作为真实的label

labels[mask] = mode(digits.target[mask])[0]有了真实的指标，可以进行准确度计算accuracy_score(digits.target, labels)

0.79354479688369514 聚类结果的混淆矩阵from sklearn.metrics import confusion_matrix

mat = confusion_matrix(digits.target, labels)

np.fill_diagonal(mat, 0)

sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False,

xticklabels=digits.target_names,

yticklabels=digits.target_names)

plt.xlabel('true label')

plt.ylabel('predicted label')编辑于 2019-12-11 11:36聚类聚类算法sklearn赞同 12添加评论分享喜欢收藏申请

Definitive Guide to K-Means Clustering with Scikit-Learn

Definitive Guide to K-Means Clustering with Scikit-LearnSA LogotypeArticlesLearnWork with UsSign inSign upPythonJavaScriptJavaHomeArticlesDefinitive Guide to K-Means Clustering with Scikit-LearnCássia SampaioIntroduction

K-Means clustering is one of the most widely used unsupervised machine learning algorithms that form clusters of data based on the similarity between data instances.

In this guide, we will first take a look at a simple example to understand how the K-Means algorithm works before implementing it using Scikit-Learn. Then, we'll discuss how to determine the number of clusters (Ks) in K-Means, and also cover distance metrics, variance, and K-Means pros and cons.

Motivation

Imagine the following situation. One day, when walking around the neighborhood, you noticed there were 10 convenience stores and started to wonder which stores were similar - closer to each other in proximity. While searching for ways to answer that question, you've come across an interesting approach that divides the stores into groups based on their coordinates on a map.

For instance, if one store was located 5 km West and 3 km North - you'd assign (5, 3) coordinates to it, and represent it in a graph. Let's plot this first point to visualize what's happening:

import matplotlib.pyplot as plt

plt.title("Store With Coordinates (5, 3)")

plt.scatter(x=5, y=3)

This is just the first point, so we can get an idea of how we can represent a store. Say we already have 10 coordinates to the 10 stores collected. After organizing them in a numpy array, we can also plot their locations:

import numpy as np

points = np.array([[5, 3], [10, 15], [15, 12], [24, 10], [30, 45], [85, 70], [71, 80], [60, 78], [55, 52],[80, 91]])

xs = points[:,0] # Selects all xs from the array

ys = points[:,1] # Selects all ys from the array

plt.title("10 Stores Coordinates")

plt.scatter(x=xs, y=ys)

How to Manually Implement K-Means Algorithm

Now we can look at the 10 stores on a graph, and the main problem is to find is there a way they could be divided into different groups based on proximity? Just by taking a quick look at the graph, we'll probably notice two groups of stores - one is the lower points to the bottom-left, and the other one is the upper-right points. Perhaps, we can even differentiate those two points in the middle as a separate group - therefore creating three different groups.

In this section, we'll go over the process of manually clustering points - dividing them into the given number of groups. That way, we'll essentially carefully go over all steps of the K-Means clustering algorithm. By the end of this section, you'll gain both an intuitive and practical understanding of all steps performed during the K-Means clustering. After that, we'll delegate it to Scikit-Learn.

What would be the best way of determining if there are two or three groups of points? One simple way would be to simply choose one number of groups - for instance, two - and then try to group points based on that choice.

Let's say we have decided there are two groups of our stores (points). Now, we need to find a way to understand which points belong to which group. This could be done by choosing one point to represent group 1 and one to represent group 2. Those points will be used as a reference when measuring the distance from all other points to each group.

In that manner, say point (5, 3) ends up belonging to group 1, and point (79, 60) to group 2. When trying to assign a new point (6, 3) to groups, we need to measure its distance to those two points. In the case of the point (6, 3) is closer to the (5, 3), therefore it belongs to the group represented by that point - group 1. This way, we can easily group all points into corresponding groups.

In this example, besides determining the number of groups (clusters) - we are also choosing some points to be a reference of distance for new points of each group.

That is the general idea to understand similarities between our stores. Let's put it into practice - we can first choose the two reference points at random. The reference point of group 1 will be (5, 3) and the reference point of group 2 will be (10, 15). We can select both points of our numpy array by [0] and [1] indexes and store them in g1 (group 1) and g2 (group 2) variables:

g1 = points[0]

g2 = points[1]

After doing this, we need to calculate the distance from all other points to those reference points. This raises an important question - how to measure that distance. We can essentially use any distance measure, but, for the purpose of this guide, let's use Euclidean Distance_.

Advice: If you want learn more more about Euclidean distance, you can read our "Calculating Euclidean Distances with NumPy" guide.

It can be useful to know that Euclidean distance measure is based on Pythagoras' theorem:

c^2 = a^2 + b^2

When adapted to points in a plane - (a1, b1) and (a2, b2), the previous formula becomes:

c^2 = (a2-a1)^2 + (b2-b1)^2

The distance will be the square root of c, so we can also write the formula as:

euclidean_{dist} = \sqrt[2][(a2 - a1)^2 + (b2 - b1) ^2)]

Note: You can also generalize the Euclidean distance formula for multi-dimensional points. For example, in a three-dimensional space, points have three coordinates - our formula reflects that in the following way:

euclidean_{dist} = \sqrt[2][(a2 - a1)^2 + (b2 - b1) ^2 + (c2 - c1) ^2)]

The same principle is followed no matter the number of dimensions of the space we are operating in.

So far, we have picked the points to represent groups, and we know how to calculate distances. Now, let's put the distances and groups together by assigning each of our collected store points to a group.

To better visualize that, we will declare three lists. The first one to store points of the first group - points_in_g1. The second one to store points from the group 2 - points_in_g2, and the last one - group, to label the points as either 1 (belongs to group 1) or 2 (belongs to group 2):

points_in_g1 = []

points_in_g2 = []

group = []

We can now iterate through our points and calculate the Euclidean distance between them and each of our group references. Each point will be closer to one of two groups - based on which group is closest, we'll assign each point to the corresponding list, while also adding 1 or 2 to the group list:

for p in points:

x1, y1 = p[0], p[1]

euclidean_distance_g1 = np.sqrt((g1[0] - x1)**2 + (g1[1] - y1)**2)

euclidean_distance_g2 = np.sqrt((g2[0] - x1)**2 + (g2[1] - y1)**2)

if euclidean_distance_g1 < euclidean_distance_g2:

points_in_g1.append(p)

group.append('1')

else:

points_in_g2.append(p)

group.append('2')

Let's look at the results of this iteration to see what happened:

print(f'points_in_g1:{points_in_g1}\n \

\npoints_in_g2:{points_in_g2}\n \

\ngroup:{group}')

Which results in:

points_in_g1:[array([5, 3])]

points_in_g2:[array([10, 15]), array([15, 12]),

array([24, 10]), array([30, 45]),

array([85, 70]), array([71, 80]),

array([60, 78]), array([55, 52]),

array([80, 91])]

group:[1, 2, 2, 2, 2, 2, 2, 2, 2, 2]

We can also plot the clustering result, with different colors based on the assigned groups, using Seaborn's scatterplot() with the group as a hue argument:

import seaborn as sns

sns.scatterplot(x=points[:, 0], y=points[:, 1], hue=group)

It's clearly visible that only our first point is assigned to group 1, and all other points were assigned to group 2. That result differs from what we had envisioned in the beginning. Considering the difference between our results and our initial expectations - is there a way we could change that? It seems there is!

One approach is to repeat the process and choose different points to be the references of the groups. This will change our results, hopefully, more in line with what we've envisioned in the beginning. This second time, we could choose them not at random as we previously did, but by getting a mean of all our already grouped points. That way, those new points could be positioned in the middle of corresponding groups.

For instance, if the second group had only points (10, 15), (30, 45). The new central point would be (10 + 30)/2 and (15+45)/2 - which is equal to (20, 30).

Since we have put our results in lists, we can convert them first to numpy arrays, select their xs, ys and then obtain the mean:

g1_center = [np.array(points_in_g1)[:, 0].mean(), np.array(points_in_g1)[:, 1].mean()]

g2_center = [np.array(points_in_g2)[:, 0].mean(), np.array(points_in_g2)[:, 1].mean()]

g1_center, g2_center

Advice: Try to use numpy and NumPy arrays as much as possible. They are optimized for better performance and simplify many linear algebra operations. Whenever you are trying to solve some linear algebra problem, you should definitely take a look at the numpy documentation to check if there is any numpy method designed to solve your problem. The chance is that there is!

To help repeat the process with our new center points, let's transform our previous code into a function, execute it and see if there were any changes in how the points are grouped:

def assigns_points_to_two_groups(g1_center, g2_center):

points_in_g1 = []

points_in_g2 = []

group = []

for p in points:

x1, y1 = p[0], p[1]

euclidean_distance_g1 = np.sqrt((g1_center[0] - x1)**2 + (g1_center[1] - y1)**2)

euclidean_distance_g2 = np.sqrt((g2_center[0] - x1)**2 + (g2_center[1] - y1)**2)

if euclidean_distance_g1 < euclidean_distance_g2:

points_in_g1.append(p)

group.append(1)

else:

points_in_g2.append(p)

group.append(2)

return points_in_g1, points_in_g2, group

Note: If you notice you keep repeating the same code over and over again, you should wrap that code into a separate function. It is considered a best practice to organize code into functions, especially because they facilitate testing. It is easier to test an isolated piece of code than a full code without any functions.

Let's call the function and store its results in points_in_g1, points_in_g2, and group variables:

points_in_g1, points_in_g2, group = assigns_points_to_two_groups(g1_center, g2_center)

points_in_g1, points_in_g2, group

And also plot the scatter plot with the colored points to visualize the groups division:

sns.scatterplot(x=points[:, 0], y=points[:, 1], hue=group)

It seems the clustering of our points is getting better. But still, there are two points in the middle of the graph that could be assigned to either group when considering their proximity to both groups. The algorithm we've developed so far assigns both of those points to the second group.

This means we can probably repeat the process once more by taking the means of the Xs and Ys, creating two new central points (centroids) to our groups and re-assigning them based on distance.

Let's also create a function to update the centroids. The whole process now can be reduced to multiple calls of that function:

def updates_centroids(points_in_g1, points_in_g2):

g1_center = np.array(points_in_g1)[:, 0].mean(), np.array(points_in_g1)[:, 1].mean()

g2_center = np.array(points_in_g2)[:, 0].mean(), np.array(points_in_g2)[:, 1].mean()

return g1_center, g2_center

g1_center, g2_center = updates_centroids(points_in_g1, points_in_g2)

points_in_g1, points_in_g2, group = assigns_points_to_two_groups(g1_center, g2_center)

sns.scatterplot(x=points[:, 0], y=points[:, 1], hue=group)

Notice that after this third iteration, each one of the points now belong to different clusters. It seems the results are getting better - let's do it once again. Now going to the fourth iteration of our method:

g1_center, g2_center = updates_centroids(points_in_g1, points_in_g2)

points_in_g1, points_in_g2, group = assigns_points_to_two_groups(g1_center, g2_center)

sns.scatterplot(x=points[:, 0], y=points[:, 1], hue=group)

This fourth time we got the same result as the previous one. So it seems our points won't change groups anymore, our result has reached some kind of stability - it has got to an unchangeable state, or converged. Besides that, we have exactly the same result as we had envisioned for the 2 groups. We can also see if this reached division makes sense.

Let's just quickly recap what we've done so far. We've divided our 10 stores geographically into two sections - ones in the lower southwest regions and others in the northeast. It can be interesting to gather more data besides what we already have - revenue, the daily number of customers, and many more. That way we can conduct a richer analysis and possibly generate more interesting results.

Clustering studies like this can be conducted when an already established brand wants to pick an area to open a new store. In that case, there are many more variables taken into consideration besides location.

What Does All This Have To Do With K-Means Algorithm?

While following these steps you might have wondered what they have to do with the K-Means algorithm. The process we've conducted so far is the K-Means algorithm. In short, we've determined the number of groups/clusters, randomly chosen initial points, and updated centroids in each iteration until clusters converged. We've basically performed the entire algorithm by hand - carefully conducting each step.

The K in K-Means comes from the number of clusters that need to be set prior to starting the iteration process. In our case K = 2. This characteristic is sometimes seen as negative considering there are other clustering methods, such as Hierarchical Clustering, which don't need to have a fixed number of clusters beforehand.

Due to its use of means, K-means also becomes sensitive to outliers and extreme values - they enhance the variability and make it harder for our centroids to play their part. So, be conscious of the need to perform extreme values and outlier analysis before conducting a clustering using the K-Means algorithm.

Also, notice that our points were segmented in straight parts, there aren't curves when creating the clusters. That can also be a disadvantage of the K-Means algorithm.

Note: When you need it to be more flexible and adaptable to ellipses and other shapes, try using a generalized K-means Gaussian Mixture model. This model can adapt to elliptical segmentation clusters.

K-Means also has many advantages! It performs well on large datasets which can become difficult to handle if you are using some types of hierarchical clustering algorithms. It also guarantees convergence, and can easily generalize and adapt. Besides that, it is probably the most used clustering algorithm.

Now that we've gone over all the steps performed in the K-Means algorithm, and understood all its pros and cons, we can finally implement K-Means using the Scikit-Learn library.

How to Implement K-Means Algorithm Using Scikit-Learn

To double check our result, let's do this process again, but now using 3 lines of code with sklearn:

from sklearn.cluster import KMeans

# The random_state needs to be the same number to get reproducible results

kmeans = KMeans(n_clusters=2, random_state=42)

kmeans.fit(points)

kmeans.labels_

Here, the labels are the same as our previous groups. Let's just quickly plot the result:

sns.scatterplot(x = points[:,0], y = points[:,1], hue=kmeans.labels_)

The resulting plot is the same as the one from the previous section.

Free eBook: Git EssentialsCheck out our hands-on, practical guide to learning Git, with best-practices, industry-accepted standards, and included cheat sheet. Stop Googling Git commands and actually learn it!Download the eBook

Note: Just looking at how we've performed the K-Means algorithm using Scikit-Learn might give you the impression that this is a no-brainer and that you don't need to worry too much about it. Just 3 lines of code perform all the steps we've discussed in the previous section when we've gone over the K-Means algorithm step-by-step. But, the devil is in the details in this case! If you don't understand all the steps and limitations of the algorithm, you'll most likely face the situation where the K-Means algorithm gives you results you were not expecting.

With Scikit-Learn, you can also initialize K-Means for faster convergence by setting the init='k-means++' argument. In broader terms, K-Means++ still chooses the k initial cluster centers at random following a uniform distribution. Then, each subsequent cluster center is chosen from the remaining data points not by calculating only a distance measure - but by using probability. Using the probability speeds up the algorithm and it's helpful when dealing with very large datasets.

Advice: You can learn more about K-Means++ details by reading the "K-Means++: The Advantages of Careful Seeding" paper, proposed in 2007 by David Arthur and Sergei Vassilvitskii.

The Elbow Method - Choosing the Best Number of Groups

So far, so good! We've clustered 10 stores based on the Euclidean distance between points and centroids. But what about those two points in the middle of the graph that are a little harder to cluster? Couldn't they form a separate group as well? Did we actually make a mistake by choosing K=2 groups? Maybe we actually had K=3 groups? We could even have more than three groups and not be aware of it.

The question being asked here is how to determine the number of groups (K) in K-Means. To answer that question, we need to understand if there would be a "better" cluster for a different value of K.

The naive way of finding that out is by clustering points with different values of K, so, for K=2, K=3, K=4, and so on:

for number_of_clusters in range(1, 11):

kmeans = KMeans(n_clusters = number_of_clusters, random_state = 42)

kmeans.fit(points)

But, clustering points for different Ks alone won't be enough to understand if we've chosen the ideal value for K. We need a way to evaluate the clustering quality for each K we've chosen.

Manually Calculating the Within Cluster Sum of Squares (WCSS)

Here is the ideal place to introduce a measure of how much our clustered points are close to each other. It essentially describes how much variance we have inside a single cluster. This measure is called Within Cluster Sum of Squares, or WCSS for short. The smaller the WCSS is, the closer our points are, therefore we have a more well-formed cluster. The WCSS formula can be used for any number of clusters:

WCSS = \sum(Pi_1 - Centroid_1)^2 + \cdots + \sum(Pi_n - Centroid_n)^2

Note: In this guide, we are using the Euclidean distance to obtain the centroids, but other distance measures, such as Manhattan, could also be used.

Now we can assume we've opted to have two clusters and try to implement the WCSS to understand better what the WCSS is and how to use it. As the formula states, we need to sum up the squared differences between all cluster points and centroids. So, if our first point from the first group is (5, 3) and our last centroid (after convergence) of the first group is (16.8, 17.0), the WCSS will be:

WCSS = \sum((5,3) - (16.8, 17.0))^2

WCSS = \sum((5-16.8) + (3-17.0))^2

WCSS = \sum((-11.8) + (-14.0))^2

WCSS = \sum((-25.8))^2

WCSS = 335.24

This example illustrates how we calculate the WCSS for the one point from the cluster. But the cluster usually contains more than one point, and we need to take all of them into consideration when calculating the WCSS. We'll do that by defining a function that receives a cluster of points and centroids, and returns the sum of squares:

def sum_of_squares(cluster, centroid):

squares = []

for p in cluster:

squares.append((p - centroid)**2)

ss = np.array(squares).sum()

return ss

Now we can get the sum of squares for each cluster:

g1 = sum_of_squares(points_in_g1, g1_center)

g2 = sum_of_squares(points_in_g2, g2_center)

And sum up the results to obtain the total WCSS:

g1 + g2

This results in:

2964.3999999999996

So, in our case, when K is equal to 2, the total WCSS is 2964.39. Now, we can switch Ks and calculate the WCSS for all of them. That way, we can get an insight into what K we should choose to make our clustering perform the best.

Calculating WCSS Using Scikit-Learn

Fortunately, we don't need to manually calculate the WCSS for each K. After performing the K-Means clustering for the given number of clusters, we can obtain its WCSS by using the inertia_ attribute. Now, we can go back to our K-Means for loop, use it to switch the number of clusters, and list corresponding WCSS values:

wcss = []

for number_of_clusters in range(1, 11):

kmeans = KMeans(n_clusters = number_of_clusters, random_state = 42)

kmeans.fit(points)

wcss.append(kmeans.inertia_)

wcss

Notice that the second value in the list, is exactly the same we've calculated before for K=2:

[18272.9, # For k=1

2964.3999999999996, # For k=2

1198.75, # For k=3

861.75,

570.5,

337.5,

175.83333333333334,

79.5,

17.0,

0.0]

To visualize those results, let's plot our Ks along with the WCSS values:

ks = [1, 2, 3, 4, 5 , 6 , 7 , 8, 9, 10]

plt.plot(ks, wcss)

There is an interruption on a plot when x = 2, a low point in the line, and an even lower one when x = 3. Notice that it reminds us of the shape of an elbow. By plotting the Ks along with the WCSS, we are using the Elbow Method to choose the number of Ks. And the chosen K is exactly the lowest elbow point, so, it would be 3 instead of 2, in our case:

ks = [1, 2, 3, 4, 5 , 6 , 7 , 8, 9, 10]

plt.plot(ks, wcss);

plt.axvline(3, linestyle='--', color='r')

We can run the K-Means cluster algorithm again, to see how our data would look like with three clusters:

kmeans = KMeans(n_clusters=3, random_state=42)

kmeans.fit(points)

sns.scatterplot(x = points[:,0], y = points[:,1], hue=kmeans.labels_)

We were already happy with two clusters, but according to the elbow method, three clusters would be a better fit for our data. In this case, we would have three kinds of stores instead of two. Before using the elbow method, we thought about southwest and northeast clusters of stores, now we also have stores in the center. Maybe that could be a good location to open another store since it would have less competition nearby.

Alternative Cluster Quality Measures

There are also other measures that can be used when evaluating cluster quality:

Silhouette Score - analyzes not only the distance between intra-cluster points but also between clusters themselves

Between Clusters Sum of Squares (BCSS) - metric complementary to the WCSS

Sum of Squares Error (SSE)

Maximum Radius - measures the largest distance from a point to its centroid

Average Radius - the sum of the largest distance from a point to its centroid divided by the number of clusters.

It's recommended to experiment and get to know each of them since depending on the problem, some of the alternatives can be more applicable than the most widely used metrics (WCSS and Silhouette Score).

In the end, as with many data science algorithms, we want to reduce the variance inside each cluster and maximize the variance between different clusters. So we have more defined and separable clusters.

Applying K-Means on Another Dataset

Let's use what we have learned on another dataset. This time, we will try to find groups of similar wines.

Note: You can download the dataset here.

We begin by importing pandas to read the wine-clustering CSV (Comma-Separated Values) file into a Dataframe structure:

import pandas as pd

df = pd.read_csv('wine-clustering.csv')

After loading it, let's take a peek at the first five records of data with the head() method:

df.head()

This results in:

Alcohol Malic_Acid Ash Ash_Alcanity Magnesium Total_Phenols Flavonoids Nonflavanoid_Phenols Proanthocyanidins Color_Intensity Hue OD280 Proline

0 14.23 1.71 2.43 15.6 127 2.80 3.06 0.28 2.29 5.64 1.04 3.92 1065

1 13.20 1.78 2.14 11.2 100 2.65 2.76 0.26 1.28 4.38 1.05 3.40 1050

2 13.16 2.36 2.67 18.6 101 2.80 3.24 0.30 2.81 5.68 1.03 3.17 1185

3 14.37 1.95 2.50 16.8 113 3.85 3.49 0.24 2.18 7.80 0.86 3.45 1480

4 13.24 2.59 2.87 21.0 118 2.80 2.69 0.39 1.82 4.32 1.04 2.93 735

We have many measurements of substances present in wines. Here, we also won't need to transform categorical columns because all of them are numerical. Now, let's take a look at the descriptive statistics with the describe() method:

df.describe().T # T is for transposing the table

The describe table:

count mean std min 25% 50% 75% max

Alcohol 178.0 13.000618 0.811827 11.03 12.3625 13.050 13.6775 14.83

Malic_Acid 178.0 2.336348 1.117146 0.74 1.6025 1.865 3.0825 5.80

Ash 178.0 2.366517 0.274344 1.36 2.2100 2.360 2.5575 3.23

Ash_Alcanity 178.0 19.494944 3.339564 10.60 17.2000 19.500 21.5000 30.00

Magnesium 178.0 99.741573 14.282484 70.00 88.0000 98.000 107.0000 162.00

Total_Phenols 178.0 2.295112 0.625851 0.98 1.7425 2.355 2.8000 3.88

Flavonoids 178.0 2.029270 0.998859 0.34 1.2050 2.135 2.8750 5.08

Nonflavanoid_Phenols 178.0 0.361854 0.124453 0.13 0.2700 0.340 0.4375 0.66

Proanthocyanidins 178.0 1.590899 0.572359 0.41 1.2500 1.555 1.9500 3.58

Color_Intensity 178.0 5.058090 2.318286 1.28 3.2200 4.690 6.2000 13.00

Hue 178.0 0.957449 0.228572 0.48 0.7825 0.965 1.1200 1.71

OD280 178.0 2.611685 0.709990 1.27 1.9375 2.780 3.1700 4.00

Proline 178.0 746.893258 314.907474 278.00 500.500 673.500 985.0000 1680.00

By looking at the table it is clear that there is some variability in the data - for some columns such as Alcohol there is more, and for others, such as Malic_Acid, less. Now we can check if there are any null, or NaN values in our dataset:

df.info()

RangeIndex: 178 entries, 0 to 177

Data columns (total 13 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 Alcohol 178 non-null float64

1 Malic_Acid 178 non-null float64

2 Ash 178 non-null float64

3 Ash_Alcanity 178 non-null float64

4 Magnesium 178 non-null int64

5 Total_Phenols 178 non-null float64

6 Flavonoids 178 non-null float64

7 Nonflavanoid_Phenols 178 non-null float64

8 Proanthocyanidins 178 non-null float64

9 Color_Intensity 178 non-null float64

10 Hue 178 non-null float64

11 OD280 178 non-null float64

12 Proline 178 non-null int64

dtypes: float64(11), int64(2)

memory usage: 18.2 KB

There's no need to drop or input data, considering there aren't empty values in the dataset. We can use a Seaborn pairplot() to see the data distribution and to check if the dataset forms pairs of columns that can be interesting for clustering:

sns.pairplot(df)

By looking at the pair plot, two columns seem promising for clustering purposes - Alcohol and OD280 (which is a method for determining the protein concentration in wines). It seems that there are 3 distinct clusters on plots combining two of them.

There are other columns that seem to be in correlation as well. Most notably Alcohol and Total_Phenols, and Alcohol and Flavonoids. They have great linear relationships that can be observed in the pair plot.

Since our focus is clustering with K-Means, let's choose one pair of columns, say Alcohol and OD280, and test the elbow method for this dataset.

Note: When using more columns of the dataset, there will be a need for either plotting in 3 dimensions or reducing the data to principal components (use of PCA). This is a valid, and more common approach, just make sure to choose the principal components based on how much they explain and keep in mind that when reducing the data dimensions, there is some information loss - so the plot is an approximation of the real data, not how it really is.

Let's plot the scatter plot with those two columns set to be its axis to take a closer look at the points we want to divide into groups:

sns.scatterplot(data=df, x='OD280', y='Alcohol')

Now we can define our columns and use the elbow method to determine the number of clusters. We will also initiate the algorithm with kmeans++ just to make sure it converges more quickly:

values = df[['OD280', 'Alcohol']]

wcss_wine = []

for i in range(1, 11):

kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 42)

kmeans.fit(values)

wcss_wine.append(kmeans.inertia_)

We have calculated the WCSS, so we can plot the results:

clusters_wine = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

plt.plot(clusters_wine, wcss_wine)

plt.axvline(3, linestyle='--', color='r')

According to the elbow method we should have 3 clusters here. For the final step, let's cluster our points into 3 clusters and plot the those clusters identified by colors:

kmeans_wine = KMeans(n_clusters=3, random_state=42)

kmeans_wine.fit(values)

sns.scatterplot(x = values['OD280'], y = values['Alcohol'], hue=kmeans_wine.labels_)

We can see clusters 0, 1, and 2 in the graph. Based on our analysis, group 0 has wines with higher protein content and lower alcohol, group 1 has wines with higher alcohol content and low protein, and group 2 has both high protein and high alcohol in its wines.

This is a very interesting dataset and I encourage you to go further into the analysis by clustering the data after normalization and PCA - also by interpreting the results and finding new connections.

Conclusion

K-Means clustering is a simple yet very effective unsupervised machine learning algorithm for data clustering. It clusters data based on the Euclidean distance between data points. K-Means clustering algorithm has many uses for grouping text documents, images, videos, and much more.

# python# machine learning# scikit-learn# algorithms# data science# pandas# matplotlib# seabornLast Updated: November 17th, 2023Was this article helpful?You might also like...Get Feature Importances for Random Forest with Python and Scikit-LearnDefinitive Guide to Hierarchical Clustering with Python and Scikit-LearnDefinitive Guide to Logistic Regression in PythonSeaborn Boxplot - Tutorial and ExamplesImprove your dev skills!Get tutorials, guides, and dev jobs in your inbox.Email addressSign UpNo spam ever. Unsubscribe at any time. Read our Privacy Policy.Cássia SampaioAuthorData Scientist, Research Software Engineer, and teacher. Cassia is passionate about transformative processes in data, technology and life. She is graduated in Philosophy and Information Systems, with a Strictu Sensu Master's Degree in the field of Foundations Of Mathematics.

David LandupEditorDimitrije StamenicEditorIn this articleIntroductionMotivationHow to Manually Implement K-Means AlgorithmWhat Does All This Have To Do With K-Means Algorithm?How to Implement K-Means Algorithm Using Scikit-LearnThe Elbow Method - Choosing the Best Number of GroupsManually Calculating the Within Cluster Sum of Squares (WCSS)Calculating WCSS Using Scikit-LearnAlternative Cluster Quality MeasuresApplying K-Means on Another DatasetConclusionProjectBank Note Fraud Detection with SVMs in Python with Scikit-Learn# python# machine learning# scikit-learn# data scienceCan you tell the difference between a real and a fraud bank note? Probably! Can you do it for 1000 bank notes? Probably! But it...DetailsCourseData Visualization in Python with Matplotlib and Pandas# python# pandas# matplotlibData Visualization in Python with Matplotlib and Pandas is a course designed to take absolute beginners to Pandas and Matplotlib, with basic Python knowledge, and...David LandupDetailsTwitterGitHubFacebook© 2013-2024 Stack Abuse. All rights reserved.AboutDisclosurePrivacyTermsDo not share my Personal Informati

Just a moment...

a moment...Enable JavaScript and cookies to conti

Follow

比特派官方app|sklearn kmeans

比特派官方app|sklearn kmeans

sklearn.cluster.KMeans — scikit-learn 1.4.1 documentation

SKlearn里面的K-means使用详解_sklearn kmeans-CSDN博客

机器学习库sklearn的K-Means聚类算法的使用方法 - 知乎

sklearn.cluster.k_means — scikit-learn 1.4.1 documentation

2.3. Clustering — scikit-learn 1.4.1 documentation

sklearn.cluster.KMeans-scikit-learn中文社区

sklearn（六）-K-Means k均值聚类算法 - 知乎

sklearn中的K-means算法 - 知乎

Definitive Guide to K-Means Clustering with Scikit-Learn

Just a moment...

最近的新闻

您可能喜欢的文章

imtoken安卓版下载app|xch价格

bitpie下载地址|墨西哥父子

私钥怎么导入比特派

比特派被盗怎么找回

比特派钱包怎么查收

比特派怎么有病毒

怎么充值比特派钱包