We propose two methods based on machine learning algorithms to estimate the level of spectral overlapping in a specific optical channel, without information of adjacent channels in Nyquist-wavelength division multiplexing (WDM) systems. The first method uses the fuzzy c-means (FCM) clustering algorithm to relate the membership degrees of the FCM matrix with the level of spectral overlapping in frames of 10 k symbols that relied on the k-nearest neighbors algorithm. The second method uses the density-based spatial clustering of application with noise algorithm to identify the level of spectral overlapping based on the number of symbols classified as noise (outliers) as well as the number of extra clusters found in a constellation diagram, resulting in an overlapping index. Both methods were experimentally verified in a 3  ×  16 Gbaud 16-QAM Nyquist-WDM system with different channel spacing. Knowing a priori the level of OSNR, results showed accuracy percentages up to 91% and up to 100% by the first proposed method in a multiclass and a binary classification, respectively. Moreover, the second method can achieve a percentage estimation up to 100% when optical channels are overlapped more than 12.5%. Thereby, both methods could be implemented in monitoring tools for incoming gridless optical transmission systems.