## Reviewer 2 ### Summary This paper proposes a Beyond Self-Attention framework for the sequential recommendation, which leverages the Fourier transform to solve the oversmoothing issue in common Transformer-based sequential recommendation methods. The proposed method uses inductive bias via Fourier transform with self-attention, and introduces a frequency rescaler to capture users' high-frequency and low-frequency behavioral patterns. Extensive experiments on six datasets demonstrate the effectiveness of the proposed method. ### Strengths and weaknesses Strong points: 1. The method description is very clear. 1. The paper is well written and easy to follow. 2. Extensive experiments are conducted by comparing with 7 baselines on six datasets. Weak points: 1. As shown in Figure 3 of this paper, the oversmoothing issue tends to arise in deep layers, particularly when there are more than 12 layers. Could oversmoothing be mitigated in normal recommendation systems by simply reducing the number of layers? The author ought to explore this topic. 2. It's suggested that the authors conduct experiments to verify the effectiveness of the proposed method at lower layers. For instance, designing experiments to reveal the correlation between the recommendation performance, the number of layers, and the proposed method. **Questions for the authors:** Please refer to the ‘weak points’. **Reproducibility:** Yes ### Overall evaluation Weak accept: Technically solid paper where reasons to accept, e.g., good novelty, outweigh reasons to reject, e.g., fair quality. ### Confidence Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation. ## Reviewer 3 ### Summary This paper studies the oversmoothing problem and proposes a new model called BSARec for sequential recommendation. In particular, the authors claim that self-attention is by design a low-pass filter and thus cuases oversmoothing features. To solve this problem, BSARec comes with BSA layers of rescaled filter matrices and additional inductive bias, which uses FFT to divide features into low/high frequencies and recale the frequencies. In this sense, BSARec is an improved version of the combination of SASRec and FMLP-Rec. The experiments demonstrate improvements of the proposed method on multiple datasets. ### Strengths and weaknesses Pros: (1) Interesting idea of transformers being low-pass filter, the paper also comes with some theoretical analysis w.r.t. this issue (2) Based on the issue, the authors design BSA components to rescale low and high frequency features (3) Experiments are performed on multiple datasets with certain improvements compared to baseline methods Cons: (1) The baseline methods have lower metric results as in the original paper or reported in other papers (e.g., SASRec, FMLP-Rec) (2) Due to the above issue, more details on dataset statistics / experiemen setting / implementation details may help justify the performance of BSARec ### Questions for the authors Please clarify the issues in the mentioned cons, thanks! One minor issue is the choice of 99 negative samples, which differs from either 100 negative items in the previous papers (i.e., SASRec & BERT4Rec) or ranking against all items. **Reproducibility:** Yes ### Overall evaluation Weak accept: Technically solid paper where reasons to accept, e.g., good novelty, outweigh reasons to reject, e.g., fair quality. ### Confidence Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation. ## Reviewer 4 ### Summary This paper proposes a novel problem in sequential recommendation, i.e. the over-smoothing problem due to the low-pass filtering nature of self-attention. This paper firstly unveil the low-pass filtering intrinsic of self-attention in transformer-based model. They also propose a novel beyond self-attention layer for sequential recommendation to leverage the inductive bias. ### Strengths and weaknesses Strength: 1 Formally define the low-pass filtering nature of self-attention in sequential recommendation. 2 Proposing a new beyond self-attention layer. 3 The experiments are comprehensive. Weakness: 1. some of the definition of terms in the paper is not clear. Such as the $A_{IB}$ and HFC. 2. The intuition of using high-pass filtering in sequential recommendation is not well explained with intuitive examples, which reads like a simple combination of high-pass filtering extension from FMLPRec. ### Questions for the authors 1 Where is the definition of A_{IB} in eq (5)? I didn't find in the main paper. 2 Why this high-pass filtering performs significantly better in LastFM? Does the data have specific patterns for high-pass filtering? 3 Though this paper argues the contrastive learning is unnecessary, will it perform better if we consider the eq 5 as data augmentation and train with contrastive learning? **Reproducibility:** Yes ### Overall evaluation Weak accept: Technically solid paper where reasons to accept, e.g., good novelty, outweigh reasons to reject, e.g., fair quality. ### Confidence Very confident. I have checked all points of the paper carefully. I am certain I did not miss any aspects that could otherwise have impacted my evaluation. ## Reviewer 5 ### Summary This paper proposes a modified transformer-based sequential recommendation model (BSARec), aiming at addressing the oversmoothing problem existing in the self-attention mechanism from the perspective of the frequency domain. BSARec tries to fuse low-frequency and high-frequency information by Fourier transform to model users’ long-term and short-term interests. Experimental results show that the proposed method outperforms state-of-the-art sequential recommendation models. ### Strengths and weaknesses Strengths: (1) The paper is well-motivated. The idea of using the Fourier transform to solve the oversmoothing problem is very clear. (2) The structure of this paper is reasonable and the content of the experiment is complete. Weaknesses: (1) In this paper, low-frequency and high-frequency values are used to characterize the long-term and short-term interests of users, respectively. However, some abrupt items that a user has interacted with may also be noise (such as clicking by mistake). This may introduce noise by modeling high-frequency information in a non-learning way. (2) The technical innovation is limited, and the model combines some existing methods. ### Questions for the authors Graph Fourier Transform (Section 2.3) does not have a strong relationship with the topic of this paper, so why spend a long length to introduce it? **Reproducibility:** Yes ### Overall evaluation Accept: Technically solid paper, with high impact on at least one sub-area of AI or modest-to-high impact on more than one area of AI, with good to excellent quality, reproducibility, and if applicable, resources, and no unaddressed ethical considerations. ### Confidence Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions. ## Author response ### Author response General Response We appreciate the insightful comments and positive feedback from all reviewers. We actively accepted the comments and conducted extra experiments. We hope to add these experiments to the appendix of the accepted paper in the future. Response to Reviewer #2 Thanks for your valuable comments. Here are our responses: Q1: Could oversmoothing be mitigated in normal recommendation systems by reducing the number of layers? R1: In the context of sequential recommendation, models usually consist of two layers. Although more layers worsen oversmoothing, the issue doesn’t vanish with fewer layers. In Fig. 3, BSARec shows lower cosine similarity, even in early layers, due to its higher passage of high-frequency components, as shown in Fig. 2 (b). Q2: The correlation between the performance and L. R2: Table 9 shows the NDCG@20 by varying the number of layers L on Beauty. As the number of layers increases, the performance decreases. However, BSARec, capturing high-frequency components, shows consistent performance even as more layers are added. Additionally, BSARec outperforms other models across all numbers of layers. Table 9: Sensitivity to L on Beauty (L: SASRec / DuoRec / BSARec) 1: 0.0382 / 0.0648 / 0.0690 2: 0.0356 / 0.0671 / 0.0703 4: 0.0354 / 0.0668 / 0.0681 8: 0.0276 / 0.0651 / 0.0665 16: 0.0258 / 0.0572 / 0.0645 Response to Reviewer #3 Thanks for your valuable comments. Here are our responses: Q1: Why are the results different from the original paper? R1: The difference lies in the evaluation method. While SASRec and FMLPRec use only 99 randomly sampled negative items for evaluation, we utilized the evaluation method of DuoRec and FEARec, which employ the entire items in dataset for a fair comparison. The evaluation results are in Table 10, following the settings of SASRec and FMLPRec. Q2: More details on dataset statistics, experimental settings, and implementation details. R2: We have provided detailed information on dataset statistics, experimental setup, and implementation details in our Appendix. Table 10: Performance comparison on 99 negative sampling (SASRec / FMLPRec / BSARec) Beauty: HR@5 0.3512 / 0.3922 / 0.4312; HR@10 0.4434 / 0.4914 / 0.5225; NDCG@5 0.2628 / 0.2964 / 0.3379; NDCG@10 0.2926 / 0.3284 / 0.3673; MRR 0.2637 / 0.2949 / 0.3350 Sports: HR@5 0.3480 / 0.3781 / 0.4133; HR@10 0.4717 / 0.4997 / 0.5303; NDCG@5 0.2492 / 0.2739 / 0.3102; NDCG@10 0.2891 / 0.3131 / 0.3479; MRR 0.2520 / 0.2742 / 0.3089 Toys: HR@5 0.3594 / 0.3867 / 0.4224; HR@10 0.4566 / 0.4852 / 0.5180; NDCG@5 0.2726 / 0.2926 / 0.3351; NDCG@10 0.3040 / 0.3244 / 0.3659; MRR 0.2746 / 0.2917 / 0.3349 Yelp: HR@5 0.5553 / 0.6058 / 0.6447; HR@10 0.7406 / 0.7707 / 0.7848; NDCG@5 0.3902 / 0.4337 / 0.4824; NDCG@10 0.4504 / 0.4873 / 0.5280; MRR 0.3748 / 0.4114 / 0.4587 LastFM: HR@5 0.2716 / 0.2853 / 0.3752; HR@10 0.3972 / 0.4138 / 0.5028; NDCG@5 0.1871 / 0.1975 / 0.2634; NDCG@10 0.2276 / 0.2394 / 0.3045; MRR 0.1976 / 0.2081 / 0.2636 ML-1M: HR@5 0.6874 / 0.6763 / 0.7023; HR@10 0.7904 / 0.7858 / 0.7978; NDCG@5 0.5308 / 0.5212 / 0.5646; NDCG@10 0.5642 / 0.5568 / 0.5955; MRR 0.5020 / 0.4941 / 0.5406 Response to Reviewer #4 Thanks sincerely for your feedback. Here are our responses: Q1: The definition of $A_{IB}$ in Eq. (5). R1: We apologize for the confusion. The definition of $A_{IB}$ in Eq. (5) is defined in Eq. (7). The revised paper will provide this definition to make it easier to find. Q2: Why this high-pass filtering excels in LastFM? R2: LastFM consists of user listening records, where high-pass filtering can effectively capture short-term changes in user preferences, such as switching between different music genres or exploring new artists. We report a case study in Fig. 7 (b) where the role of high-pass filter leads to a large performance improvement in LastFM. Q3: Can extending Eq. (5) to contrastive learning (CL) enhance performance? R3: We did not consider CL in our model design. Due to time constraints, we couldn’t experiment with CL, though we acknowledge its potential for performance improvement. We appreciate the suggestion and consider it for future research. One concern with CL is the potential increase in cosine similarity in the early layers, as observed in cases like DuoRec, likely due to enhanced embedding uniformity. Response to Reviewer #5 Thanks for your insightful review. Here are our responses: Q1: Concerns that high-frequency data may contain noise R1: We recognize that interaction noise can vary among users. Therefore, in Equation (7), we introduce a learnable vector parameter, $\beta$, to adapt to high-frequency variations. Q2: Why introduce the graph Fourier transform? R2: The introduction of graph Fourier transform in Section 2.3 was intended to provide a background for the subsequent discussion on oversmoothing and Fourier Transform applied to sequential recommendation. We agree that the length dedicated to this topic may have caused confusion, and we will revise the paper to provide a more concise explanation. ## Meta-review ### Reasons to accept Three reviewers agree that the paper should be weak accepted and one reviewer bid to accept the paper. All of them agree that the proposed BSARec model is quite novel for sequential recommendation. **Reasons to reject:** NA