Review history

Relevance to CIKM

3 (fair)

Originality of the Work

4 (good)

Technical Soundness

3 (fair)

Quality of Presentation

3 (fair)

Impact of Ideas or Results

3 (fair)

Adequacy of Citations

3 (fair)

Reproducibility of Methods

3 (fair)

Comments to the Author(s)

This paper proposed a new graph neural network for collaborative filtering based on neural ODE. In particular, it proposed the learnable-time ODE-based collaborative filtering method. As such, it can learn optimal architecture for the network and the ODE solutions are suitable for the desired task.

Overall, the idea looks novel (I am not very familiar with this domain so that I am not quite sure whether it is novel enough. ) and the experimental results show some improvement.

Q1. For ODE, there is a common concern for the stability of its solution. How is the stability of the proposed method?

Q2. How is the computational performance of the proposed method? Is it faster than traditional methods?

Summary

The idea looks novel. But I am not quite sure whether it is novel enough because I am not familiar with this domain.

Relevance to CIKM

5 (excellent)

Originality of the Work

4 (good)

Technical Soundness

5 (excellent)

Quality of Presentation

3 (fair)

Impact of Ideas or Results

4 (good)

Adequacy of Citations

4 (good)

Reproducibility of Methods

4 (good)

Comments to the Author(s)

This paper provides a novel ordinary differential equation-based solution to the classic collaborative filtering problem in recommendation systems. It considers the linear GCN as neural ordinary differential equations. The authors provide sufficient theoretical analysis and present a closed-form solution for the proposed problem. Experiments are done to compare LT-OCF to various baseline methods on three datasets.

Strong points:
(1) The paper explores using neural ordinary differential equations to improve the collaborative filtering in recommender systems, which has many real-world applications.
(2) The paper adopts a novel idea that the architecture is trained instead of manually specified, which seems to produce competitive performance compared to the state-of-the-art.
(3) The authors find out that the dense connections are a performance booster for the proposed framework.

Weak points:
(1) The problem formulation and the reason to use ordinary differential equations are not well-motived. See detailed evaluation (1).
(2) The experiment results are not convincing enough. See detailed evaluation (2) - (4).
(3) The paper can benefit from a better presentation. See detailed evaluation (5) - (7).

Detailed evaluations:
1. Although collaborative filtering is a very important research topic, it is unclear from the paper that why neural differential equations are used. In the introduction, the paper claims that the user-product interaction bipartite graphs are mostly sparse. Is the LT-OCF framework able to address this issue? How does the proposed solution exploit the bipartite graph structure?
2. The authors need to conduct some experiments without dense connections to evaluate the effectiveness of their model in a standalone fashion. Dense connections as a performance booster have been investigated in the literature. Please accredit the following work in the following paper. Wenfeng Liu, Maoguo Gong, Zedong Tang, A.K. Qin, Kai Sheng, Mingliang Xu, Locality preserving dense graph convolutional networks with graph context-aware node representations, Neural Networks, Volume 143, 2021, Pages 108-120, ISSN 0893-6080,https://doi.org/10.1016/j.neunet.2021.05.031. (https://www.sciencedirect.com/science/article/pii/S0893608021002276)

3. The experiments could be more convincing if more than three datasets were used.
4. Please discuss how the hyperparameters are chosen in the experiments. Is a grid search involved to find the best parameter combination? How do we set the hyper-parameters on a new dataset?
5. Please give more details of the training process for the learnable-time in Section 3.2.1. How are the time points to construct the layer combination learned? What loss function is chosen in this process?
6. The paper is inundated with formulas, and the authors need to do a better job explaining them. For example, please better explain why BPR loss is used in equation (7).
7. The captions are too long, e.g., Figure 2, and Figure 4. In table 3, please mention in the capture what performance metric is being reported.

Summary

This paper studies an interesting problem, and the authors provide sufficient theoretical analysis and present a closed-form solution for the proposed problem. In my opinion, this paper is a candidate for acceptance.

Relevance to CIKM

4 (good)

Originality of the Work

4 (good)

Technical Soundness

4 (good)

Quality of Presentation

4 (good)

Impact of Ideas or Results

3 (fair)

Adequacy of Citations

3 (fair)

Reproducibility of Methods

3 (fair)

Comments to the Author(s)

The paper propose the method of Learnable-Time ODE-based Collaborative Filtering (LT-OCF) in this paper.

There are several minor comments to improve this manuscript:

1) The related work part in section 2 is not very well organized. You listed a lot of citations, but what are their advantages and disadvantages, and how is your proposed method relate to/different from these prior work?

2) The method used for comparative experiments is a long time ago, and comparison results with other most advanced methods should be provided to show the effect of this method. methods should be provided to show the effect of this method.

3) The selected reference is too old. It is recommended to use the journal for nearly 3 years.

Summary

The writing is clear for the most part and the experimental design was described in detail with many figures, which is good.