Review history

Summary and contributions

Graph-based Collaborative Filtering (CF) is used in recommendation systems but faces issues with data sparsity and negative sampling. This paper introduces a new method, SCONE, which uses stochastic sampling for contrastive views and hard negative samples to address these challenges. This sampling method is built on score-based generative models and produces varied views and hard-to-find negative samples. The experiments on 6 datasets demonstrate the accuracy and robustness of the proposed method.

Strong points

(1) This work generalizes the contrastive learning (CL) view and negative sampling (NS) in the CF model into a sampling problem, which is a novel idea.
(2) This paper introduces score-based generative models (SGMs) into the CF, which is a relatively new idea.
(3) The work analyzes and compares the existing issues of existing CL and NS methods, which is very clear.

Weak points

(1) In the introduction section, the description of the method should give a high-level overview of the motivation and emphasize why it can solve the issues faced by CL and NS, and should not write the method in too much detail. For example, the author spends a lot of paragraphs introducing specific details of the method in lines 164-193, which is relatively redundant.
(2) The limitations of the CL-based and NS-based method are repeatedly emphasized in the paper.
CL: Perturbing the structure of the graph may cause the potential loss of interaction or topological information.
NS: Synthetic negative samples are usually in a predetermined region, which may lead to limited diversity.
The work presented in this paper does not highlight why both limitations can be addressed simultaneously.
(3) some related works on CL and NS in CF are missing:
- On the Theories Behind Hard Negative Sampling for Recommendation. WWW’2023
- Adap-: Adaptively Modulating Embedding Magnitude for Recommendation. WWW’2023
- Incorporating Bias-aware Margins into Contrastive Loss for Collaborative Filtering. NeurIPS’2022
(4) Figure 9 shows that the proposed method is very sensitive to the weight w, so it is difficult to adjust the value range of w in practice.
(5) The paper does not provide a discussion of the complexity analysis or model size.

Detailed evaluation

Although introducing SGMs into CL and NS-base CF models is a relatively new idea, this paper should focus on explaining why introducing SGMs can effectively solve the limitations of existing methods.
For example, the author should give some theoretical or experimental explanations.

Summary and contributions

The paper introduces a duffision method to generate the hard negative samples for collaborative filtering.

Strong points

1. The paper is well written, and it provides very details about the negative sampling in the collaborative.

2. The authors use diffusion algorithm gor negative sampling, which is also interesting. I haven't seen similar work in the area of collaborative filtering in my mind.

3. The authors provide a detailed experiments to validate the proposed algorithm.

Weak points

1. The novelty of this work is not high.

2. The complexity is not discussed, and the improvement of this work is mild.

3. Some simple baselines are not included.

Detailed evaluation

In this paper, the auhtors introduced SCONE, a novel approach to address the challenges of data sparsity and negative sampling in graph-based CF models. The SCONE proposes the concept of synthesizing samples, which had not been explored before in recommender systems. By employing a stochastic sampling process, SCONE generates user-item-specific augmented views for contrastive learning.

Here are my concerns:
1. The novelty of this work is not high. The paper is a mixture of Graph Neural network, diffusion, and contrastive learning. Its mainly adopt the generative algorithm to generate the negative samples. The novelty is limited.

2. The complexity is not discussed, and the improvement of this work is mild.
1) Compare to the mixup techniques like MixGCF, I think the entire process in this work is expensive. The authors may need to provide complexity analysis to make the algorithm more convincing.
2) The improvement of this work is mild. As shown in table 2, the relative improvement is < 2% in all dataset, except douban (2.79%). It may raise a question that what is the main benefit of use complex diffusion algorithm to generate negative samples.

3. Some simple baselines are not included. Some related work but much cheaper and simpler algorithms are not compare in the experiment. For example: Disentangled Negative Sampling for Collaborative Filtering, WSDM 2023

Overall, the duffusion algorithm for negative sampling is interested. However, its performance and complexity may question the reader that what is the benefits of adopting the proposed work.

Summary and contributions

The paper proposes to use a generative model to find contrastive views that will be used to train a collaborative filtering model based on LightGCN with a contrastive loss. The diffusion model is applied on the final embeddings E learned by LightGCN.

Strong points

The idea of using diffusion to create contrastive views seems interesting. However my experience is that diffusion models do not work very well on recommendations so the samples obtained may not be very good. This was not discussed in the paper though.

The paper reports the optimal values for a selection of hyperparameters on each dataset as well as two search spaces. There is source code.

The analysis looks for the model robustness at different sparsity levels and includes ablation studies, hyperparamete sensitivity and efficiency.

Weak points

I don't think the paper explains the intuition behind the "positive injection". Indeed to me seems a contradiction to find a hard negative by adding positive embeddings. I understand the goal is to be at the boundary between positive and negative, but still I am missing why this should be a good idea. The fact that the results show it works is not enough, because there are so many other factors that can affect the results that are completely uncorrelated to how good the model is (for example inconsistent basleline optimization).

- "To make a fair comparison with previous studies, we follow hyperparameters settings in the original papers." This is an example of bad practice that will result in an inconsistently optimized set of baselines which, as several reproducibility papers have shown, essentially invalidates any result reported in the paper. Indeed the reported baselines did not use the same train-test split and datasets. It is unfortunate because now we cannot derive any conclusion from the experimental section.

Detailed evaluation

The introduction section is very long and covers details that are usually left for the "related works" section. I suggest the authors consider to simplify the introduction focusing on a high level motivation and description of the contribution, moving the details in section 2.

"It is not feasible to directly apply the denoising process of the diffusion model, which generates a new sample x̂(0) from a noisy sample x(T), to contrastive views generation, because it cannot maintain the specific characteristics of users and items."
The statement is not justified and should be better explained.

Summary and contributions

This paper proposes a novel approach called SCONE (Stochastic Sampling for Contrastive Views and Hard Negative Samples in Graph-based Collaborative Filtering) to overcome challenges faced by graph-based collaborative filtering (CF) models, resulting in significant improvements in recommendation accuracy and robustness. The paper introduces a score-based generative model that leverages contrastive learning and negative sampling to learn user and item embeddings. The proposed approach also addresses user sparsity and item popularity issues through user-item specific stochastic sampling. The paper provides extensive experimental results on benchmark datasets, demonstrating the superiority of SCONE over various state-of-the-art models. The contributions of this paper include a novel approach to graph-based CF, a comprehensive evaluation of the proposed approach, and significant improvements in recommendation accuracy and robustness.

Strong points

1) SCONE's Novelty: The paper introduces SCONE, which combines various techniques to tackle CF challenges, notably addressing user sparsity and item popularity.

2) Evidence: Through experiments on six datasets, SCONE consistently outperforms top models, highlighting its effectiveness.

3) Clarity: The paper details SCONE's components and experimental methods for easy comprehension and replication.

Weak points

1) Despite extensive testing on six datasets, the paper doesn't compare its approach to state-of-the-art models like NCF and DeepFM.

2) The paper touches on privacy and bias in recommendation systems but lacks a thorough discussion on their treatment in the proposed method.

3) While scalability of the approach is mentioned, there's no empirical evidence provided to back this claim.

4) The paper details results but omits specifics on the experimental setup, such as hyperparameter selection and data preprocessing.

Detailed evaluation

Overall, the paper presents some contributions to the field of CF by proposing a novel approach that addresses prevalent problems in CF and provides significant improvements in recommendation accuracy and robustness. The paper provides clear and detailed explanations of the proposed approach and extensive experimental results that demonstrate its effectiveness and robustness. However, the paper could be improved by comparing the proposed approach with some state-of-the-art models, providing a more in-depth discussion on privacy and information bias issues, providing a detailed analysis of the scalability of the proposed approach, and providing a more detailed explanation of the experimental setup.

Relevance to SIGIR

4 (good)

Originality of the work

2 (poor)

Originality of the work (comments)

The motivation behind the proposed approach is not clear. While it appears to aim at combining contrasting views and hard negative samples, the necessity and advantages of this combination are not clear.

Technical soundness

2 (poor)

Technical soundness (comments)

In comparing with baselines, the enhancements achieved by the proposed SCONE appear marginal, particularly on datasets such as Amazon CDs and Yelp2018. The authors need to conduct a significance test to ascertain the statistical significance of these improvements.

Quality of presentation

4 (good)

Quality of presentation (comments)

This work is well-written.

Related work

4 (good)

Perceived significance or potential impact of the contribution

3 (fair)

Strengths

- The authors generate samples for each specific purpose via stochastic sampling.
- The authors summarize two directions' works: contrastive learning and hard negative sampling in CF.
- The authors summarize two directions' works: contrastive learning and hard negative sampling in CF.

- Extensive experiments on six real-world datasets.

Weaknesses

- The motivation behind the proposed approach is not clear. While it appears to aim at combining contrasting views and hard negative samples, the necessity and advantages of this combination are not clear.

- In comparing with baselines, the enhancements achieved by the proposed SCONE appear marginal, particularly on datasets such as Amazon CDs and Yelp2018. The authors need to conduct a significance test to ascertain the statistical significance of these improvements.

- The authors highlight data sparsity and negative sampling as prevalent issues. However, it's important to note that negative sampling is a strategy or function rather than a problem in itself. Clarification on this distinction would improve the precision of the discussion.

Detailed comments to authors

Please refer to the weaknesses.

Relevance to SIGIR

5 (excellent)

Originality of the work

3 (fair)

Originality of the work (comments)

1. The authors claim that the combination of contrastive learning and negative sampling is novel. However, SGL and SimGCL mentioned in the paper also combine the two views.
2. The main modules of the model including LightGCN and SGM are proposed by existing work.
3. The methods for contrastive learning and negative sampling may be novel. However, the authors fail to articulate the advantages of their methods over existing methods.

Technical soundness

3 (fair)

Technical soundness (comments)

Advantages:
1. The authors provide the code and data.
2. The experiments are adequate. The only regret is that the authors do not replace their contrastive learning and negative sampling methods with other existing methods to verify the validity of their proposed methods.

Disadvantages:
1. Although the problem in recommendation systems is well known, the author should give the definition of the problem in the paper.
2. Table 4 claims that SGL and SimGCL belong to the methods using contrastive learning. However, Table 2 claims that the two methods belong to the methods using contrastive views and negative samples.
3. Table 1 claim that SGM belongs to the domain of recommendation, but SGM is proposed to solve the image generation problem.

Quality of presentation

3 (fair)

Quality of presentation (comments)

I can understand the ideas and methods proposed by the authors. However, there are many inconsistencies in the paper, and the authors need to revise it carefully to achieve rigor.

Related work

4 (good)

Perceived significance or potential impact of the contribution

4 (good)

Strengths

1. The idea of combining the contrastive learning and negative sampling methods is novel.
2. The authors provide the code and data.
3. The experiments are adequate. The only regret is that the authors do not replace their contrastive learning and negative sampling methods with other existing methods to verify the validity of their proposed methods.

Weaknesses

1. The authors claim that the combination of contrastive learning and negative sampling is novel. However, SGL and SimGCL mentioned in the paper also combine the two views.
2. The main modules of the model including LightGCN and SGM are proposed by existing work.
3. The methods for contrastive learning and negative sampling may be novel. However, the authors fail to articulate the advantages of their methods over existing methods.
4. Although the problem in recommendation systems is well known, the author should give the definition of the problem in the paper.
5. There are many inconsistencies in the paper, and the authors need to revise it carefully to achieve rigor. For example,
1) Table 4 claims that SGL and SimGCL belong to the methods using contrastive learning. However, Table 2 claims that the two methods belong to the methods using contrastive views and negative samples.
2) Table 1 claim that SGM belongs to the domain of recommendation, but SGM is proposed to solve the image generation problem.

Detailed comments to authors

The authors provide a novel idea of combining the contrastive learning and negative sampling methods. Please refer to the following for detailed evaluation.

Advantages:
1. The idea of combining the contrastive learning and negative sampling methods is novel.
2. The authors provide the code and data.
3. The experiments are adequate. The only regret is that the authors do not replace their contrastive learning and negative sampling methods with other existing methods to verify the validity of their proposed methods.

Concerns:
1. The authors claim that the combination of contrastive learning and negative sampling is novel. However, SGL and SimGCL mentioned in the paper also combine the two views.
2. The main modules of the model including LightGCN and SGM are proposed by existing work.
3. The methods for contrastive learning and negative sampling may be novel. However, the authors fail to articulate the advantages of their methods over existing methods.
4. Although the problem in recommendation systems is well known, the author should give the definition of the problem in the paper.
5. There are many inconsistencies in the paper, and the authors need to revise it carefully to achieve rigor. For example,
1) Table 4 claims that SGL and SimGCL belong to the methods using contrastive learning. However, Table 2 claims that the two methods belong to the methods using contrastive views and negative samples.
2) Table 1 claim that SGM belongs to the domain of recommendation, but SGM is proposed to solve the image generation problem.

Relevance to SIGIR

4 (good)

Originality of the work

4 (good)

Originality of the work (comments)

This work proposes a novel recommendation method that leverages diffusion model to generate negative samples and contrastive views.

Technical soundness

2 (poor)

Technical soundness (comments)

This work is not well-motivated. The experimental design could be enhanced.

Quality of presentation

4 (good)

Related work

2 (poor)

Perceived significance or potential impact of the contribution

4 (good)

Strengths

1. This work studies on an interesting problem.
2. This work proposes a novel recommendation method that leverages diffusion model to generate negative samples and contrastive views.
3. The paper is well-writing.

Weaknesses

1. This work is not well-motivated.
2. The experimental design could be enhanced.
3. This work misses a significant amount of relevant literature.

Detailed comments to authors

While this work studies an important problem and is potential, it has some limitations:

1. My major concern lies in the motivation behind the proposed method. The authors propose the use of a diffusion model for negative sampling and contrastive learning, yet the rationale behind this choice remains vague. What specific advantages does the diffusion model offer for these applications? A more thorough analysis is needed to support their approach. Specifically, in the discussion on contrastive learning, the paper focuses solely on GSL and SimGCL, overlooking various advanced strategies such as [2][42][a1]. The limitations of existing methods and the benefits of the proposed approach are not adequately explained. Moreover, employing a diffusion model for data augmentation is not novel, as evidenced by [a5]. For negative sampling, the superiority of the diffusion approach over other strategies is not convincingly demonstrated. The arguement that covering the entire embedding space would yield better performance is questionable. If it were true, it means that randomly sampling over the entire space would be a best choice. So, why would diffusion be more effective than DNS and MixGCF that aim to sample hard negative negatives to accelerate learning? Besides, The paper also fails to consider other important and advanced sampling strategies like [a2][a3].

2. The experiments are insufficient:
2.1 The compared methods are outdated. More SOTA methods on contrastive learning and negative sampling should be considered, like [2][42][a1][a2][a3].
2.2 Given that the proposed method integrates contrastive learning with negative sampling, it is crucial to compare it against a simple combination of existing contrastive learning and negative sampling methods (e.g., SimGCL+MixGCF). Such a comparison would provide stronger evidence of the proposed diffusion-based strategy's effectiveness.
2.3 In the ablation study, SCONE-W/o NS should be benchmarked against existing contrastive-learning-based methods. Similarly, SCONE-w/o CL should be compared with existing negative samplers. Only through such comparisons can the effectiveness of the proposed method be attributed to the novel diffusion model rather than the combined application of contrastive learning and negative sampling. Unfortunately, from figure 7 and table 4, it seems that SCONE-w/o NS underperforms compared to SimGCL, casting doubt on the diffusion-based strategy's efficacy.
2.4 The authors assert that their method can generate diverse recommendations, yet they only evaluate the uniformity of the embeddings and not the actual diversity of the recommendation results.

3. This work misses some important related work. A few are listed here:

[a1] WWW'23: Automated Self-Supervised Learning for Recommendation [a2] CIKM'23: Batch-Mix Negative Sampling for Learning Recommendation Retrievers [a3] WWW'22: Learning Recommenders for Implicit Feedback with Importance Resampling [a4]NIPS'24: Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion [a5] CIKM'23: Diffusion augmentation for sequential recommendation

Some minor ones:

4. It would be better to envision some future directions.
5.It would be better to include a section of related work that comprehensive reviews recent developments in recommendation samplers, contrastive-based recommendation methods, and diffusion-based recommendation methods (like [a4][a5]).

In summary, while the application of a diffusion model in negative sampling and contrastive learning is promising, the current work has some serious limitations in terms of motivation, experiments and related work. Consequently, I suggest a reject.

Relevance to SIGIR

5 (excellent)

Originality of the work

4 (good)

Originality of the work (comments)

The paper introduces a novel approach to generative views for contrastive learning and sample hard negative samples via score-based generative models.

Technical soundness

3 (fair)

Technical soundness (comments)

The method is overall sound. Experiments are conducted on six public datasets, which is good. However, there are issues in the evaluation: (1) In section 5.1.3, it seems that most hyperparameters for baseline methods are fixed. It's unknown whether the baselines have been well tuned for fair comparison. (2)Only@20 metrics are reported. Author(s) are suggested to report more metrics (e.g., @5, @10, or @50) to make the evaluation comprehensive. (3) 1 cannot draw the observation about "SCONE works better for cold-start users" from Figure 4. The improvements are relatively similar across different user groups.

Quality of presentation

3 (fair)

Related work

4 (good)

Perceived significance or potential impact of the contribution

4 (good)

Strengths

1. This paper studies an important problem, i.e., contrastive learning for collaborative filtering models.
2. The idea of "generating views via leveraging the reverse process of score-based generative models" is interesting.
3. Experiments are conducted on six public datasets.
4. Code is available during the reviewing phase, which is good for reproduction.

Weaknesses

1. Motivation is unclear. The idea and the proposed view generation method are interesting and novel. However, it's unclear why we need such method or what challenges does this method solve. In the introduction, the author(s) claim that a motivation is about thinking "can we integrate two approaches into a single unified framework?". Yes it's great that SCONE can do that, but is there any additional benefit compared to simply combining a contrastive learning method (like SGL) with DNS?

2. Statements are not well supported by experiments. 2-1. "Good performance", the proposed method uses both CL and hard negative sampling. However, all the baselines meatares baceaaortanteebaraanaae cesalanea canar caneponesone ese mot arnul totaocooiecasellne use either CL or hard negative sampling, resulting in unfair comparison. It's not painful to simply combine GCL methods like SGL or SimGCL with DNS. 2-2. "Sparsity", I cannot draw the observation about "SCONE works better for cold-start users" from Figure 4. The improvements are relatively similar across different user groups. 2-3. "Diversity", it's not so clear why higher uniformity leads to more diverse recommendation results. More evaluations (e.g., directly reporting diversity metrics) are required.

3. Evaluation issues. soms at ms bupararanotar for pacline merhade are fyed des urknou uhasher ahe 3-1. In section 5.1.3, it seems that most hyperparameters for baseline methods are fixed. It's unknown whether the baselines have been well tuned for fair comparison. 3-2. Only @20 metrics are reported. Author(s) are suggested to report more metrics (e.g., @5, @10, or @50) to make the evaluation comprehensive.

Detailed comments to authors

Please kindly refer to "Strengths" and "Weaknesses".

Relevance to CIKM

5 (excellent)

Originality of the Work

4 (good)

Technical Soundness

4 (good)

Quality of Presentation

4 (good)

Impact of Ideas or Results

4 (good)

Reproducibility of Methods

4 (good)

Detailed Comments to the Author(s)

Strengths:
1. The idea of using generative models to solve the problems of views generation in contrastive learning and hard negative sampling is very innovative, bringing new insights and considerations to the recommendation systems community.
2. The paper is well-organized, with a clear structure and easily understandable content.
3. The authors conducted thorough experiments, demonstrating improvements across six benchmark datasets. Additionally, they analyzed the uniformity of the representations, showing that the proposed method not only mitigates popularity bias but also enhances the personalization of recommendations.

Weaknesses:
1. Although the authors conducted experiments on multiple benchmark datasets, the performance improvements in Recall@20 and NDCG@20 are limited.
2. The selection of baseline methods for negative sampling does not include the most advanced negative sampling methods, such as DINS[1] and DENS[2]. Would the recommendation performance be better if the positive injection method of DINS were used?

[1] Xi Wu, Liangwei Yang, Jibing Gong, Chao Zhou, Tianyu Lin, Xiaolong Liu, and Philip S. Yu. 2023. Dimension Independent Mixup for Hard Negative Sample in Collaborative Filtering. CIKM '23
[2] Riwei Lai, Li Chen, Yuhan Zhao, Rui Chen, and Qilong Han. 2023. Disentangled Negative Sampling for Collaborative Filtering. WSDM '23

Summary to support your recommendation

The authors observe a similarity between different views in contrastive learning and the sampling of hard negative examples. To address this, they introduce generative models into recommendation systems to generate different views for contrastive learning, while also generating hard negative samples based on specific user-item interaction.
This is a novel approach, and the idea is innovative, providing some inspiration for the development of the recommendation systems community.

Relevance to CIKM

5 (excellent)

Originality of the Work

5 (excellent)

Technical Soundness

5 (excellent)

Quality of Presentation

5 (excellent)

Impact of Ideas or Results

5 (excellent)

Reproducibility of Methods

5 (excellent)

Detailed Comments to the Author(s)

This paper proposes to integrate contrastive learning and hard negative samples into a single unified framework. The topic is interesting and the paper is well-written. However, the improvements are knd of limited even and no significant test results are provided. The paper doesn't compare their method to SGM-based recommendation methods.

Summary to support your recommendation

Please refer to Detailed Comments.

Relevance to CIKM

4 (good)

Originality of the Work

4 (good)

Technical Soundness

4 (good)

Quality of Presentation

4 (good)

Impact of Ideas or Results

3 (fair)

Reproducibility of Methods

4 (good)

Detailed Comments to the Author(s)

Summary:

This paper presents SCONE, a novel approach for generating contrastive views and hard negative samples in GNN based CF methods using score-based generative models (SGMs). SCONE uses a LightGCN model as the base recommender system and incorporates an SGM to generate contrastive views for self-supervised learning and hard negative samples for improved training using a co-trained SGM model on the node embeddings. The SGM is trained on the final embeddings from LightGCN and uses a stochastic sampling process to generate personalized views/samples for each user-item pair. For contrastive learning, two views are generated: a perturbed embedding and a generated embedding from the perturbed one. Hard negative samples are created by injecting positive sample information during the SGM sampling process making it harder than a randomly sampled negative. Experiments on 6 benchmark datasets show SCONE outperforms state-of-the-art baselines in most cases, especially on sparse datasets.

Strengths:

* The paper is well presented and easy to read and follow.
* Experiments are extensive with in-depth ablations to understand the impact of different components and different hyperparameters. Includes runtime comparisons to address computational overhead concerns.
* Novel application of SGMs to recommender systems for both contrastive learning and negative sampling.

Weaknesses:

* Since the method does both InfoNCE and BPR, no baseline with combination of these methods is presented. In other words, no ablation is shown to show the value of using the generative model for selecting the negatives.
* Considerably slower than GCL but does not provide much benefits over that. Fig 5: no discussion on worse perf on high degree items than MixGCF: the claim is that SCONE provides both hard negatives and CV but the behavior is very similar to CV only.
* Only LightGCN as backbone
* No analysis of the generative model is included, how good is that model trained at various locations in training and how does it impact the model training
* Fig4 there is not clarity that SCONE does better than GCL in sparse users

Summary to support your recommendation

Overall, SCONE presents an innovative and effective approach for improving recommender systems through generative modeling. But the current experiments and results do not effectively demonstrate the effectiveness of the proposed method. I believe more extensive experiments are needed to demonstrate the effectiveness of the proposed approach. Even with the limitations, I think the idea is interesting and worth discussion.