Review history

Summary

This paper proposes a Beyond Self-Attention framework for the sequential recommendation, which leverages the Fourier transform to solve the oversmoothing issue in common Transformer-based sequential recommendation methods. The proposed method uses inductive bias via Fourier transform with self-attention, and introduces a frequency rescaler to capture users' high-frequency and low-frequency behavioral patterns. Extensive experiments on six datasets demonstrate the effectiveness of the proposed method.

Strengths and weaknesses

Strong points:
1. The method description is very clear.
1. The paper is well written and easy to follow.
2. Extensive experiments are conducted by comparing with 7 baselines on six datasets.

Weak points:
1. As shown in Figure 3 of this paper, the oversmoothing issue tends to arise in deep layers, particularly when there are more than 12 layers. Could oversmoothing be mitigated in normal recommendation systems by simply reducing the number of layers? The author ought to explore this topic.
2. It's suggested that the authors conduct experiments to verify the effectiveness of the proposed method at lower layers. For instance, designing experiments to reveal the correlation between the recommendation performance, the number of layers, and the proposed method.

Questions for the authors

Please refer to the ‘weak points’.

Summary

This paper studies the oversmoothing problem and proposes a new model called BSARec for sequential recommendation. In particular, the authors claim that self-attention is by design a low-pass filter and thus cuases oversmoothing features. To solve this problem, BSARec comes with BSA layers of rescaled filter matrices and additional inductive bias, which uses FFT to divide features into low/high frequencies and recale the frequencies. In this sense, BSARec is an improved version of the combination of SASRec and FMLP-Rec. The experiments demonstrate improvements of the proposed method on multiple datasets.

Strengths and weaknesses

Pros:
(1) Interesting idea of transformers being low-pass filter, the paper also comes with some theoretical analysis w.r.t. this issue
(2) Based on the issue, the authors design BSA components to rescale low and high frequency features
(3) Experiments are performed on multiple datasets with certain improvements compared to baseline methods

Cons:
(1) The baseline methods have lower metric results as in the original paper or reported in other papers (e.g., SASRec, FMLP-Rec)
(2) Due to the above issue, more details on dataset statistics / experiemen setting / implementation details may help justify the performance of BSARec

Questions for the authors

Please clarify the issues in the mentioned cons, thanks!

One minor issue is the choice of 99 negative samples, which differs from either 100 negative items in the previous papers (i.e., SASRec & BERT4Rec) or ranking against all items.

Summary

This paper proposes a novel problem in sequential recommendation, i.e. the over-smoothing problem due to the low-pass filtering nature of self-attention. This paper firstly unveil the low-pass filtering intrinsic of self-attention in transformer-based model. They also propose a novel beyond self-attention layer for sequential recommendation to leverage the inductive bias.

Strengths and weaknesses

Strength:
1 Formally define the low-pass filtering nature of self-attention in sequential recommendation.
2 Proposing a new beyond self-attention layer.
3 The experiments are comprehensive.

Weakness:
1. some of the definition of terms in the paper is not clear. Such as the \(A_{IB}\) and HFC.
2. The intuition of using high-pass filtering in sequential recommendation is not well explained with intuitive examples, which reads like a simple combination of high-pass filtering extension from FMLPRec.

Questions for the authors

1 Where is the definition of A_{IB} in eq (5)? I didn't find in the main paper.
2 Why this high-pass filtering performs significantly better in LastFM? Does the data have specific patterns for high-pass filtering?
3 Though this paper argues the contrastive learning is unnecessary, will it perform better if we consider the eq 5 as data augmentation and train with contrastive learning?

Summary

This paper proposes a modified transformer-based sequential recommendation model (BSARec), aiming at addressing the oversmoothing problem existing in the self-attention mechanism from the perspective of the frequency domain. BSARec tries to fuse low-frequency and high-frequency information by Fourier transform to model users’ long-term and short-term interests. Experimental results show that the proposed method outperforms state-of-the-art sequential recommendation models.

Strengths and weaknesses

Strengths:
(1) The paper is well-motivated. The idea of using the Fourier transform to solve the oversmoothing problem is very clear.
(2) The structure of this paper is reasonable and the content of the experiment is complete.
Weaknesses:
(1) In this paper, low-frequency and high-frequency values are used to characterize the long-term and short-term interests of users, respectively. However, some abrupt items that a user has interacted with may also be noise (such as clicking by mistake). This may introduce noise by modeling high-frequency information in a non-learning way.
(2) The technical innovation is limited, and the model combines some existing methods.

Questions for the authors

Graph Fourier Transform (Section 2.3) does not have a strong relationship with the topic of this paper, so why spend a long length to introduce it?