Review historyHow Bridging Dynamic Factor Models and Neural Controlled Differential Equations for Nowcasting GDP got from its first submission to CIKM 2024: 5 submissions. Scores use each venue's own scale (hover a score for the reviewer's confidence; 3→5 = changed after the rebuttal). ← CV of Failures Aug 2023CIKM 2023mean 0.3, rejected Dec 2023AAAI 2024rejected Feb 2024TheWebConf 2024mean 0.4, rejected May 2024KDD 2024mean 2.8, rejected Jul 2024CIKM 2024mean 0.5, accepted About the full reviews on this page. Reviews that are already public on OpenReview are only linked. For the other rounds I share the reviews my paper received myself, under three rules: only for papers that were eventually accepted, only conference reviews (never journal reviews), and only as received — reviewers appear under the anonymous IDs or numbers the venue gave them, and I do not know who they are. The venue rules I checked are quoted with each round; none of them prohibits authors from sharing the reviews of their own submission. CIKM 2023 ✗ rejectedsubmitted 2023-05-27-111mean 0.3
reviews not public — shared by the authors download as .md
What the reviewers raised
What changed for the next submission
CIKM is an ACM conference and does not publish reviews. ACM’s Peer Review Policy FAQ keeps reviewer identities and review content confidential, but states that “authors are permitted to use and share the text of the reviews that they receive”. Full reviewsReviewer 1: rating -1 (weak reject)Overall evaluation -1 (weak reject) Strengths(1) The authors clearly describe three research questions they want to address, and conduct the experiments with them. The research questions are meaningful and common in the real-world application. Weaknesses(1) The task used for evaluation is somewhat singular. Its empirical study was barely conducted on macroeconomic data from South Korea. This method should be able to applied to other similar issues facing the same problems (as illustrated in the 3 research questions). However, this work did not show this, even did not show whether it could be applied to other regions. Thus, cannot convincingly show its great impact to the real-world. (2) It lacks of description of the selection of those macroeconomic indicators. This is very important in economic research. In this study, I think the report time of those macroeconomic indicators are more critical than the time labels, as some macroeconomic indicators might also have delay in time of publication like the GDP. Nowcasting should guarantee only use those data accessible before the current time. (3) 1 have doubts about the science and applicability of using cubic spline interpolation in the method. Which point, and how many points are involved for the cubic spline interpolation, should be justified. This fitting method is sensitive to the data points. If all data points in train/dev/test set are used for that, there might be an information leakage from the future time step. If only the data points from the previous time step are involved here, it might be better to provide some details on how to process the first few intervals. Because it makes little sense to fit a cubic spline to fewer than five points. Overall evaluation (comments)Based on the Weaknesses (1) and concerns in Weakness (2) and (3), I tend to weak reject this paper. I think this paper could be improved. Reviewer 2: rating 1 (weak accept)Overall evaluation 1 (weak accept) Strengths1. The problem studied in this paper is interesting. Weaknesses1. It would be better if more datasets are used for experiments. Overall evaluation (comments)The paper introduces a new model called NCDENow, which combines Nonlinear Continuous Dynamic Equations (NCDEs) with Dynamic Factor Models (DFMs) to improve the accuracy of nowcasting GDP growth. The proposed method addresses challenges such as sudden drops and irregular or missing data more effectively. The researchers conducted experiments to evaluate the performance of NCDENow in different scenarios, considering factors like missing data rate and volatility of economic data. The results showed that NCDENow outperformed various versions of Recurrent Neural Networks (RNNs) and DFMs in nowcasting GDP. Pros: 1. The problem studied in this paper is interesting. 2. The proposed method is reasonable. 3. The experimental results seem promising. Cons: 1. It would be better if more datasets are used for experiments. Reviewer 3: rating 1 (weak accept)Overall evaluation 1 (weak accept) StrengthsThe model is well explained. The experimental setting is clear and the results are promising. Weaknessessome minor issues with the choice of the dataset and the lack of additional datasets to compare with. Overall evaluation (comments)In this paper, the authors introduce a novel framework for nowcasting GDP, making the following contributions: As far as my knowledge is concerned, the model seems well explained. The experimental setting is clear and the results are promising. I think that there might be some minor issues with the choice of the dataset and the lack of additional datasets to compare with. - Can you share the dataset you collected from Economics Statistics System? It would be important for both reproducibility and baseline studies. - Independently from the country, could you add in your experiment a dataset already used by other papers, such as https:// onlinelibrary.wiley.com/doi/full/10.1002/jae.2845, to see how your approach works on datasets and methods optimized on that particular dataset? - The improvements shown in Table 2 and Table 3 are very likely significant from a statistical significance point of view. What about the computational complexity of your algorithms compared to the baselines? Meta-review and decisionAAAI 2024 ✗ rejectedsubmitted 2023-08-15Weak RejectReject
reviews not public — shared by the authors download as .md
What the reviewers raised
AAAI-24 did not publish reviews. Its ethical guidelines for reviewers ask reviewers to keep the papers and their reviews confidential; nothing restricts authors from sharing the reviews of their own submission. Full reviewsReviewer 3: rating Weak Reject, confidence Quite confidentReproducibility Yes, Overall evaluation Weak Reject, Confidence Quite confident SummaryThis paper proposes a method combining NCDEs with DFMs to nowcast GDP. According to experiments on real-life datasets, the model performs well on sudden drops and irregular or missing data. Strengths and weaknessesThe paper is well written, and contributes a novel framework combining NCDEs and DFMs for nowcasting GDP, which enables better nowcasting of irregular time series to solve the real-world problems that irregular times and lagged factors often interfere with GDP nowcasting. And the proposed scheme outperforms the baselines and can be applied to predict various economic indicators to help make better decisions, which has significant practical value in real life. However, there exist several problems, which must be solved before it is considered for publication. Questions for the authors1.There is a model called DFM-NCDEs in the baselines of your experiment. What’s the difference between it and your proposed model as they both seem to be a fusion of DFM and NCDEs? Reviewer 5: rating Reject, confidence Somewhat confidentReproducibility Yes, Overall evaluation Reject, Confidence Somewhat confident SummaryThis paper proposes a novel method for GDP nowcasting by combining NCDEs and DFMs. The key idea is to use DFMs for estimating latent factors and NCDEs for capturing temporal dependencies and idiosyncratic disturbances. Experiments on real world GDP datasets validate the effectiveness of the proposed method compared with other baselines. Extensive analysis on the experiment results were carried out thereafter. Strengths and weaknesses- Strength - Novelty: poor - Quality: fair - Clarity: poor - Significance: poor Questions for the authorsMore explanations are needed for experiment results: Meta-review and decisionTheWebConf 2024 ✗ rejectedsubmitted 2023-10-31110-22mean 0.4
reviews not public — shared by the authors download as .md
What the reviewers raised
Full reviewsReviewer 1: rating 1 (weak accept)Overall evaluation 1 (weak accept) Overall evaluation (comments)Summary: Strong Points: Weak Points: Reviewer 2: rating 1 (weak accept)Overall evaluation 1 (weak accept) Overall evaluation (comments)The paper introduces a novel GDP nowcasting model that combines dynamic factor models and neural-controlled differential The paper introduces a novel GDP nowcasting model that combines dynamic factor models and neural-controlled differential equations. An experimental evaluation demonstrates its superior performance over six baselines, and the ability to provide effective insights into economic conditions for informed policy decisions. The paper reads well and the problem is relevant. The main limitation is the application to only two datasets. Since GDP and economic data are usually provided as open data from governments and academics, I would have expected an experimental analysis considering more countries/macro-areas. Reviewer 3: rating 0 (borderline paper)Overall evaluation 0 (borderline paper) Overall evaluation (comments)This paper proposes a new framework called NCDENow for nowcasting Gross Domestic Product (GDP) growth. In this work, the authors investigate the traditional DFM model and try to fix its issues such as i) the lack of capturing economic uncertainties such as sudden recessions or booms; and ii) the limitation of capturing irregular dynamics from data of mixed frequency. To overcome the issues, authors proposed NCDENow that combines the strengths of Dynamic Factor Model (DFM) in extracting latent factors and Neural Controlled Differential Equations (NCDE) in handling irregular time series data. This paper claims that NCDENow outperforms existing methods in terms of nowcasting accuracy, especially in scenarios involving missing data and volatile economic conditions. Strengths of this work: Weaknesses of this work: Reviewer 4: rating -2 (reject)Overall evaluation -2 (reject) Overall evaluation (comments)This paper presents an innovative approach to predicting GDP and demonstrates commendable accuracy in forecasting the GDP of two specific countries. However, despite these strengths, the paper falls short in various critical aspects, leading to the overall recommendation of rejection. The strengths of the paper: It introduces a new method for GDP prediction. The authors successfully implement this methodology to forecast the GDP of two countries, showcasing promising performance. The innovative approach and the accurate prediction of GDP for these countries highlight the potential of the proposed method in economic forecasting. However, several weaknesses hinder the value to the WebConf audience. Firstly, the experiment section appears excessively restricted, failing to cater to the diverse interests of the audience. The paper should encompass broader applications such as predicting other time series in Economical / Financial domain. At the minimum, I suggest predicting GDP for multiple countries, particularly those in developing regions. Secondly, the paper's method description section lacks relevance beyond the realm of GDP prediction. It fails to discuss the potential implications or adaptability of the proposed method to other time series prediction tasks. By extending the discussion to encompass its applicability in diverse forecasting domains, the paper could significantly increase its value to a broader to encompass its applicability in diverse forecasting domains, the paper could significantly increase its value to a broader audience. Lastly, the absence of a rigorous analysis regarding the impact of individual modules within the proposed method is a notable weakness. The paper lacks ablation analysis or insights into the specific roles played by different modules. A deeper understanding of these components and their contributions to the forecasting process is crucial for comprehending the methodology's efficacy and potential enhancements. In conclusion, while the paper introduces a novel method for GDP forecasting and demonstrates promising results for two specific countries, its limitations in scope, applicability, and lack of comprehensive analysis diminish its relevance and appeal to the broader audience of the WebConf. Reviewer 5: rating 2 (accept)Overall evaluation 2 (accept) Overall evaluation (comments)The paper introduces a novel NCDENow framework for GDP forecasting, merging neural-controlled differential equations with dynamic factor models, an innovative approach particularly effective during economic uncertainties. While the methodology and experimental evaluation are thorough, showing effectiveness against baseline models, the paper could benefit from more detailed technical explanations. Its significance in economic forecasting under uncertain conditions is highlighted, with potential impact on policymaking and economic analysis, though some technical aspects, particularly NCDE integration, warrant further clarity. The paper does not delve deeply into the technical specifics of how Neural Controlled Differential Equations (NCDEs) are integrated with Dynamic Factor Models (DFMs). A more detailed explanation of the NCDEs' role and functioning within the framework would benefit readers, especially those unfamiliar with this approach. While the paper presents results using datasets from South Korea and the United Kingdom, it does not extensively address the generalizability and scalability of the proposed method to other regions or larger datasets. Understanding how the model performs across diverse economic contexts and scales is crucial for its applicability in global economic forecasting. The paper could enhance its analysis of how the proposed model handles sudden economic shocks and irregularities, such as those caused by unforeseen global events. While the COVID-19 pandemic is discussed, a broader examination of the model's robustness in various volatile scenarios would be beneficial. KDD 2024 ✗ rejectedsubmitted 2024-02-022333mean 2.8
reviews not public on OpenReview — shared by the authors download as .md
What the reviewers raised
What changed for the next submission
KDD is an ACM SIGKDD conference; its OpenReview reviews are visible only to the authors and the committee. ACM’s Peer Review Policy FAQ keeps reviewer identities and review content confidential, but states that “authors are permitted to use and share the text of the reviews that they receive”. Full reviews and discussionReviewer kQMY: rating 3Novelty 3, Presentation 3, Reproducibility 5, Overall evaluation 3 Paper category3 ReviewIn the context of GDP nowcasting, the authors proposed a new framework named NCDENow. This framework first interpolate within the irregular datasets, followed by a NCDE neural networks to predict the coefficient of hidden states. Then, the time series data were sent to Kalman filter to extract the hidden variables. Combining the coefficients and hidden variables gives the desired predicted value. The author then compared the NCDENow with existing models on South Korea and UK GDP datasets. The results showed a significant decreases in MSE and MAPE in most of the studied cases. The NCDENow also showed robustness against data missing and high efficiency with further experiments. Influence3 Technical soundness2 Impact4 SummaryIn the context of GDP nowcasting, the authors proposed a new framework named NCDENow. This framework first interpolate within the irregular datasets, followed by a NCDE neural networks to predict the coefficient of hidden states. Then, the time series data were sent to Kalman filter to extract the hidden variables. Combining the coefficients and hidden variables gives the desired predicted value. The author then compared the NCDENow with existing models on South Korea and UK GDP datasets. The results showed a significant decreases in MSE and MAPE in most of the studied cases. The NCDENow also showed robustness against data missing and high efficiency with further experiments. Paper strength1) The proposed framework of NCDENow is novel. It combines the advantages of DFM and NCDE in one framework to reach better results. Paper weakness1) The design motivation/philosophy behind the NCDENow architecture is unclear. Is this architecture a pure empirically designed one? Detailed evaluation and suggestions for authors1) A Question on the NCDENow framework. As described by the author, the "regression step" does not train directly the coefficients in this step, so it may cause some confusion if this step is termed "regression". Questions for authors for discussion1) How to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models? Confidence level1 Authors, 2024-04-12 RebuttalW1. The design motivation/philosophy behind the NCDENow architecture is unclear. Is this architecture a pure empirically designed one? Our architecture is not an empirical construct but a fusion of DFM and NCDE, drawing from a theoretical background[1,2]. The novelty of NCDENow lies in integrating DFM with NCDE's advanced handling of complex, non-linear dynamics. This integration is motivated by the recognition that traditional DFMs, while powerful for linear analysis, struggle to capture complex patterns and rapid economic changes. NCDEs, generalizing over RNNs, offer a more expressive framework. Our strategic design aims to synergize the interpretability of DFMs with the temporal modeling capabilities of NCDEs. This is not an ad-hoc combination but a thoughtful response to the limitations of existing nowcasting models, with a clear goal to enhance predictive accuracy and adaptability. Also, our NCDENow can incorporate diverse DFM methods, highlighting the extensibility of NCDENow. > [1] Cirone, Nicola Muca, et al. "Theoretical Foundations of Deep Selective State-Space Models." arXiv preprint arXiv:2402.19047 (2024). W2. The training process of NCDENow is not clear. For instance, how to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models? Please refer to our response for Q1. W3. The sizes of the dataset seemed to be insufficient. Only two small datasets were tested. I suppose that this is why only one layer of neural networks is enough to reach the best MSE and MAPE. More cases are desired. Since the release frequency of the economic indicators used in our project is monthly or quarterly, the size of the dataset is not sufficient. Therefore, in future work, we will consult with the Korean government agencies involved in this project to find a way to use high-frequency data. Authors, 2024-04-12 RebuttalS1. A Question on the NCDENow framework. As described by the author, the "regression step" does not train directly the coefficients in this step, so it may cause some confusion if this step is termed "regression". We apologize for the confusion. We will change the terminology to "fusion module" in our updated version. S2. The sizes of the dataset seemed to be insufficient. Only two small datasets were tested. I suppose that this is why only one layer of neural networks is enough to reach the best MSE and MAPE. More cases are desired. Please refer to our response for W3. Authors, 2024-04-19 CommentDear Reviewer kQMY, as the discussion period comes to a close, we would like to thank you once again for your positive assessment. Your support and comments have been invaluable to us. We remain open and eager to incorporate any further feedback or insights you might offer. Reviewer tRk4: rating 3Novelty 4, Presentation 3, Reproducibility 5, Overall evaluation 3 Paper category4 ReviewThe paper is technically dense but well-structured, presenting a clear methodology and substantial experimental validation. The originality of combining DFMs with NCDEs for nowcasting GDP represents a significant contribution to the literature, addressing limitations in capturing economic uncertainties and irregular dynamics. Nonetheless, the paper could have better contextualized its contributions within the existing body of work. While the proposed NCDENow framework shows potential in improving nowcasting accuracy and handling irregular time-series data, its significance is somewhat overshadowed by the lack of demonstrated real-world deployment or impact. Pros Cons Influence3 Technical soundness5 Impact5 SummaryThe paper presents NCDENow, a promising framework that leverages the strengths of DFMs and NCDEs for GDP nowcasting, aiming to address challenges related to economic uncertainties and data irregularities. While it demonstrates methodological innovation and improved forecasting accuracy through experimental validation, it falls short in evidencing real-world deployment and impact, limiting its significance within the applied data science context. Paper strength- The paper presents a novel approach by combining NCDEs with traditional DFM, enhancing the model's capability to handle irregular time series data and capture economic uncertainties. Paper weakness- The paper could provide a deeper theoretical foundation for the choice of integrating NCDEs with DFM, explaining why this combination is particularly suited for addressing the challenges of nowcasting GDP. Detailed evaluation and suggestions for authors- Authors are encouraged to provide more theoretical insights into the integration of NCDEs with DFM, expand on the model's applicability to other economic indicators or regions. Questions for authors for discussion- Can the authors provide additional theoretical background on the integration of NCDEs with DFM? Confidence level4 Authors, 2024-04-12 RebuttalWe sincerely thank you for the review and positive feedback. Below, we address each of your concerns and questions: --- Q1. Can the authors provide additional theoretical background on the integration of NCDEs with DFM? Below, we describe the theoretical background of the integration of NCDE and DFM. The DFM is expressed in state-space form and can be estimated using Kalman filters and multiple solution algorithms. While DFMs have been well studied on data using economic indicators, they may be insufficient to model complex patterns or rapid changes in data due to the lack of non-linear activation. A possible complement to this is the use of Neural CDEs, which can theoretically generalize not only RNNs but also state-space models [1], which have recently gained attention. Also, our NCDENow has the advantage of being an extensible framework that can be extended to other kinds of DFM methods in addition to the DFM we used. > [1] Cirone, Nicola Muca, et al. "Theoretical Foundations of Deep Selective State-Space Models." arXiv preprint arXiv:2402.19047 (2024). Authors, 2024-04-12 RebuttalQ2. Could you elaborate on the robustness of the model in scenarios with varying data availability or quality? (1) Robustness of the model with respect to data quality To check the robustness of NCDENow with respect to data quality, we performed GDP nowcasting after arbitrarily removing a portion of the economic indicators (10% and 20%). The results are shown in Table 7. Even with poor data quality, NCDENow outperformed DFM, the leading GDP nowcasting model used by many central banks around the world. (2) Robustness of the model with respect to data availability We used economic indicators from South Korea and the UK to nowcast the GDP growth rate of each country. The South Korean economic indicators were available through a web service provided by the Bank of Korea [1]. However, some of the UK economic indicators used in previous work [2], such as the PMI construction and the UK focused equity index, were unavailable due to their proprietary nature. Nevertheless, in both scenarios, NCDENow outperforms all baseline models (including DFM) in terms of MSE and MAPE, demonstrating the robustness of NCDENow with respect to data availability. In the future, we will consult with government agencies in South Korea that are participating in this project to find ways to use high-frequency data collected by financial authorities, which will allow us to conduct various experiments on quantitative data. In addition, our project is considering applying NCDENow to exchange rate forecasting to enhance its usefulness. We will provide more details on this in our response to question 5. > [1] Bank of Korea: Economic Statistics System Authors, 2024-04-12 RebuttalQ3. Could the authors detail the preprocessing steps for handling irregular time series data, particularly any data imputation or transformation techniques employed before applying the NCDE-DFM framework? (1) Data imputation The exposure estimation module uses the cubic spline method. This interpolation method imputes missing values or outliers by connecting data points with a curve between them. It has the advantage of preserving the natural dynamics of time series. (2) Data transformation The factor extraction module uses two different types of transformation methods (differencing or log differencing), depending on the nature of the economic indicator. For some indicators, such as GDP growth, we use the raw values. In our paper, Table 2 shows the transformation method for each economic indicator. - Differencing: \(\Delta D_n = D_n - D_{n-1}\) Authors, 2024-04-12 RebuttalQ4. Has any sensitivity analysis been conducted to assess the impact of key parameters on the forecasting performance of NCDENow? How sensitive is the model to changes in its architecture or hyperparameters? (1) ODE Solver In our paper, Table 9 shows the GDP nowcasting results when using the Euler method and the 4th order Runge-Kutta (RK4) method. For all datasets, the RK4 method performs better than the Euler method because it performs more precise operations. For example, the Euler method performs only one computation at each time step, while the RK4 method performs four computations and then uses the average value. The RK4 method is known to be more accurate for solving nonlinear differential equations [1]. Because it is specifically designed for various types of differential equations, it provides better accuracy than the simplest method, the Euler method. > [1] Islam, Md Amirul. "A comparative study on numerical solutions of initial value problems (IVP) for ordinary differential equations (ODE) with Euler and Runge Kutta Methods." American Journal of computational mathematics 5.03 (2015): 393-404. (2) Number of hidden layers In our paper, Table 10 shows how the performance of NCDENow varies depending on the number of hidden layers in the exposure estimation module. For the South Korea dataset, the lowest MSE was achieved when using a single layer, and for the UK dataset, the lowest MSE was obtained when using three layers. (3) Dimension of hidden layer Although we could not include it due to the page limit, we also carried out a sensitivity analysis on the dimension of the hidden layer. The results are as follows: | | Korean GDP | | UK GDP | | The South Korea dataset had the lowest MSE when the size of the hidden dimension was 128, and the UK dataset had the lowest MSE and MAPE when the size of the hidden dimension was 256. Authors, 2024-04-12 RebuttalQ5. Are there limitations or challenges in applying NCDENow that were not discussed, and how might these affect its broader applicability? (1) Nowcasting of other economic indicators We are currently considering expanding the use of the NCDENow model to nowcast the KRW-USD exchange rate. Economic experts are involved in collecting various daily or monthly economic indicators that affect the exchange rate. For example, the economic indicators relevant to the KRW-USD exchange rate are shown in the following table. | Group | Economic indicators | We are also working on structuring appropriate factor groups considering the characteristics of each economic indicator. Once this work is completed, we expect NCDENow to be used to nowcast the KRW-USD exchange rate. (2) GDP Nowcasting for other countries As mentioned in Section 5, a limitation of our study is that we focused only on South Korea and the United Kingdom. Therefore, in the future, we will extend our experiments to nowcast GDP growth for a variety of countries. In particular, we will consider datasets from countries whose economic indicators tend to be volatile. (3) Model Architecture In NCDENow, the factor extraction module based on the Kalman filter and smoother extracts factors by economic indicator group, and the exposure estimation module based on NCDE estimates factor exposures and idiosyncratic disturbances. As mentioned in Section 5, we will attempt to use the mechanisms of NCDE to model the continuous modelling of the factor estimation process within the Kalman filter and smoother. Through this work, we hope to explore the potential of the NCDENow model for GDP nowcasting. Authors, 2024-04-12 RebuttalW1. The paper could provide a deeper theoretical foundation for the choice of integrating NCDEs with DFM, explaining why this combination is particularly suited for addressing the challenges of nowcasting GDP. Please refer to our respones for Q1. W2. While the model shows promising results on the datasets used, the paper could benefit from a discussion on its applicability to other types of economic data or different geographic regions. Please refer to our respones for Q5. W3. More detailed information on model architecture, hyperparameters, and training details could enhance the reproducibility of the results. Additionally, discussing how the model performs with varying amounts of available data could provide insights into its robustness. (1) Model architecture Please refer to our respones for Q1. (2) Hyperparameter settings Please refer to our respones for Q4. (3) Amounts of data As the data collected in this project are monthly/quarterly economic indicators, the amount of time series data is not abundant. Therefore, we will consult with Korean government agencies participating in this project to find ways to use high-frequency data, and we will conduct various experiments regarding the amount of data in the future. Authors, 2024-04-19 CommentDear Reviewer tRk4, as the discussion period comes to a close, we would like to thank you once again for your positive assessment. Your support and comments have been invaluable to us. We remain open and eager to incorporate any further feedback or insights you might offer. Reviewer 5y5M: rating 3Novelty 3, Presentation 3, Reproducibility 4, Overall evaluation 3 Paper category1 ReviewThe research is interesting and meaningful. The quality is good and the manuscript has some innovations. Influence2 Technical soundness4 Impact3 SummaryDFM suffers from two issues: i) the lack of capturing economic uncertainties such as sudden recessions or booms; and ii) the limitation of capturing irregular dynamics from data of mixed frequency. To overcome the above two challenges, the research introduce a new GDP Nowcasting framework called NCDENow , which combines neural controlled differential equations (NCDEs) to DFM, which effectively handles the dynamics of irregular time series. NCDENow consists of two main parts: i) one extracts factors reflecting the economic conditions based on DFM, and ii) the other captures temporal dependencies of the impact of these factors on GDP growth based on NCDE. The research evaluates NCDENow against 6 baselines on 2 real-world GDP datasets from South Korea and United Kingdom, demonstrating its enhanced predictive capability. The empirical results favor our method, highlighting the significant potential of integrating NCDE into nowcasting models. Paper strengthThe research is meaningful and interesting. The method of combining DFM with NCDEs is novel. The experiment is solid and can better support the effectiveness of the proposed model. Paper weaknessThe proposed model is based on existing related researches, and the innovations should be further illsturated to demonistrate the contribution of the research in a clearer way. Detailed evaluation and suggestions for authorsPlease see the content in paper weakness. Questions for authors for discussionPlease see the content in paper weakness. Confidence level3 Authors, 2024-04-12 RebuttalWe sincerely thank you for the review and positive feedback. Below, we address each of your concerns and questions: W1. The proposed model is based on existing related researches, and the innovations should be further illsturated to demonistrate the contribution of the research in a clearer way. There are many other researches that can detect abnormal fluctuations in large-scale time series, the research should discuss the unique advantages of the proposed model between thoses researches. The illustration of the research could be further improved. For example, the mechanism of NCDE to detect idiosyncratic disturbances; The domain expert knowledge data for factor group information data and etc. The research is suggested to add new content to evaluate the effectivenss of the proposed model in real applications. (1) The contribution of the research GDP nowcasting has been studied based on DFM models. Our novelty is that to the best of our knowledge, no research has been done to integrate Neural CDEs, which can generalize recurrent neural networks, with DFMs, which have strengths in economic indicator data. Our contribution is that our NCDENow framework can be used in conjunction with many other variants of DFMs. This implies a promising extensibility of NCDENow, which is our strength. We would like to explain that time series abnormal detection models and NCDENow have different objectives as you mentioned. The main difference between NCDENow and time series abnormal detection models is their respective goals and applications. While time series abnormal detection models are designed to identify unusual patterns or outliers in the data that may indicate errors or new events, NCDENow was developed specifically for GDP nowcasting. Our model aims to accurately nowcast current economic conditions using a combination of DFM and NCDE, and focuses on capturing and interpreting the underlying economic dynamics rather than detecting anomalies. In addition, unlike anomaly detection models that focus on pattern recognition and identifying deviations, NCDENow leverages NCDE to model the ongoing dynamics of economic indicators, providing a deeper understanding of the temporal dependence and the impact of macroeconomic factors on GDP growth. And when we say that we are good at capturing abrupt fluctuations in Figure 1, we mean that DFM is a linear state-space model, whereas NCDE has the advantage of compensating for the non-linearity of these fluctuations. Authors, 2024-04-12 Rebuttal*The following continues from the previous rebuttal for W1.* (2) The mechanism of NCDE to detect idiosyncratic disturbances As shown in Eq. (4), the exposure estimation module in NCDENow uses an augmented NCDE to estimate factor exposures and idiosyncratic disturbances. The module uses the cubic spline method to generate continuous paths from economic indicators. From these paths, the NCDE continuously learns how economic indicators change over time and identifies and captures idiosyncratic disturbances, which are unexpected fluctuations. This mechanism effectively detects idiosyncratic disturbances in complex time series, enabling more precise GDP nowcasting. (3) The economic domain expert knowledge The dataset of economic indicators for South Korea was selected, taking into account its economic structure. In South Korea, manufacturing and imports and exports account for a large share of the economy, and we included economic indicators such as manufacturing production and shipments, and the size of exports and imports, as in previous work [1]. We have also classified the economic indicators into four blocks (global, real, soft, and labor) as in the previous work [1]: the global group includes all economic indicators; the real group includes indicators related to the real economy; the soft group includes survey data, considering the importance of soft indicators in GDP nowcasting [2]; the labor group consists of indicators related to the labor market. In our project, we are constantly analyzing the characteristics of the economic indicators in order to update the dataset. For example, we have recently been working with economic experts to correct measurement errors in the ‘Number of job openings per unemployed person’ indicator. We hope our response will address your concerns. > [1] Hyun Chang Yi, Dongkyu Choi, and Yonggun Kim. 2022. Dynamic Factor Model and Deep Learning Algorithm for GDP Nowcasting. Issue Note 28 (2022). Authors, 2024-04-19 CommentDear Reviewer 5y5M, as the discussion period comes to a close, we would like to thank you once again for your positive assessment. Your support and comments have been invaluable to us. We remain open and eager to incorporate any further feedback or insights you might offer. Reviewer bngv: rating 2Novelty 2, Presentation 3, Reproducibility 4, Overall evaluation 2 Paper category4 ReviewThe paper proposes a novel framework called NCDENow, which integrates dynamic factor models (DFM) with neural controlled differential equations (NCDE) for GDP nowcasting. It is well-written and easy to understand, and the problem of nowcasting GDP is important. However, my major concern is that the novelty of the proposed method is insufficient. Influence2 Technical soundness4 Impact3 SummaryThe paper proposes a novel framework called NCDENow, which integrates dynamic factor models (DFM) with neural controlled differential equations (NCDE) for GDP nowcasting. While the methodology demonstrates improved performance in handling irregular data and capturing economic uncertainties compared to baselines, it lacks novelty in its extension of DFM and relies heavily on NCDE for addressing irregular dynamics. Further, the paper could benefit from presenting more innovative designs built upon NCDE. Paper strength1. This paper is well-written and easy to understand. Paper weakness1. The noverty of the proposed method is limited. Detailed evaluation and suggestions for authors1. The proposed method seemly simply integrate NCDE with DFM. The extension seems relatively straightforward, lacking significant innovation beyond leveraging NCDE's capabilities. 2. The paper heavily relies on NCDE for handling irregular dynamics, without introducing substantial enhancements or innovations within the NCDE framework itself. 3. The paper lacks in-depth discussions on innovative designs or adaptations within the NCDE framework that could further distinguish NCDENow from existing approaches, potentially limiting its novelty and significance. Questions for authors for discussion1. Can you elaborate on any novel designs or adaptations within the NCDE framework that differentiate NCDENow from existing approaches? How does NCDENow leverage the unique capabilities of NCDE beyond standard applications? Confidence level2 Authors, 2024-04-12 RebuttalWe sincerely thank you for the review and feedback. Below, we address each of your concerns and questions: --- Q1. Can you elaborate on any novel designs or adaptations within the NCDE framework that differentiate NCDENow from existing approaches? How does NCDENow leverage the unique capabilities of NCDE beyond standard applications? Our novelty is the integration of the DFM with the NCDE framework. NCDENow can be seen as a unification of CDEs that generalize recurrent models. It can also be linked to linear CDEs, which can theoretically generalize state-space models[1] (e.g., S4[2], Mamba[3]), which have recently gained prominence. Thus, NCDENow encompasses traditional recurrent models and, according to [1], is more expressive than traditional recurrent models, while also leveraging the strengths of DFMs. As you know, our NCDENow is based on the NCDE framework, and we had the design goal of integrating it with DFM as a loose dependency. We plan to integrate the Kalman filter of a DFM into NCDE in our future work. The advantage of our NCDENow is that it is a framework that can use not only the DFM we used, but also other kinds of DFM methods, and our method is extensible in this respect. > [1] Cirone, Nicola Muca, et al. "Theoretical Foundations of Deep Selective State-Space Models." arXiv preprint arXiv:2402.19047 (2024). Q2. Given the comment about the reliance on NCDE for addressing irregular dynamics, are there additional enhancements or innovations within the NCDE framework that could further strengthen the proposed method's performance and novelty? Our NCDENow focuses on integrating the respective strengths of NCDE and DFM. Although we mention it as future work in the conclusion section, our goal of NCDENow in this paper is not our intention to replace the functions of NCDE with the algorithms of DFM. Instead, our NCDENow has the advantage of being able to use some kind of DFM with NCDE, which can generalize recurrent models. Authors, 2024-04-12 RebuttalQ3. How does NCDENow address potential limitations or challenges associated with the integration of DFM and NCDE? Are there specific considerations or strategies employed to ensure the synergy and effectiveness of these two components in the nowcasting process? NCDE has the advantage of effectively modeling non-linear and irregular time series data, but it may not sufficiently reflect economic theory. On the other hand, DFM has the advantage of capturing correlations between economic indicators well, but is limited by linearity and normality assumptions. Utilizing the strengths and weaknesses of each of these models to complement each other is an effective approach to improving overall prediction performance. As a strategy for this, the NCDENow framework clearly distinguishes between the roles of DFM and NCDE. DFM extracts latent factors by taking advantage of the ability to capture correlations between economic indicators, and NCDE models non-linear and irregular characteristics well to derive idiosyncratic disturbances and factor loadings. By integrating the results derived from these two modules in the regression module, the advantages of DFM and NCDE can be effectively combined to improve overall prediction performance. This design is expected to maximize the synergy effect between NCDE and DFM and enable effective GDP nowcasts that reflect both economic theory and non-linear dynamics. W1. The novelty of the proposed method is limited. Please refer to our response for Q1. W2. The proposed method does not significantly distinguish itself from NCDE. Please refer to our response for Q2. W3. Discussion on design Innovations is not sufficient. Please refer to our response for Q3. Authors, 2024-04-19 CommentDear Reviewer bngv, as the discussion period comes to a close, we would like to thank you once again for your insightful comments have been invaluable to us. We remain open and eager to incorporate any further questions or suggestions you might offer. Meta-review and decisionDecision by Program Chairs, 2024-05-16 DecisionRevise and Resubmit CIKM 2024 ✓ acceptedsubmitted 2024-05-13, decision 2024-07-161010mean 0.5
reviews not public — shared by the authors download as .md
What was discussed
CIKM is an ACM conference and does not publish reviews. ACM’s Peer Review Policy FAQ keeps reviewer identities and review content confidential, but states that “authors are permitted to use and share the text of the reviews that they receive”. Full reviewsReviewer 1: rating 1 (weak accept)Overall recommendation 1 (weak accept) Quality of Presentation4 (good) Detailed Comments to the Author(s)Strengths: Major weaknesses: Summary to support your recommendationThis study nowcasts GDP under normal and abnormal periods. The problem is important and has great social impact. The authors design NCDENow, which integrates the advantage of DFM and NCDE. NCDENow outperforms baseline models significantly on datasets from South Korea and the UK. The main concerns are regarding the experimental design and baseline comparison. Also some important justification and explanation. Another concern is the missing of deployment or system implementation section, which are encourages by the Applied Research Track. Reviewer 2: rating 0 (borderline paper)Overall recommendation 0 (borderline paper) Quality of Presentation5 (excellent) Detailed Comments to the Author(s)The author shows the model's performance in the UK and South Korea and releases the codes. Why not show more countries that are impacted by irregular events like COVID-19 as well? This would help to check if the model runs robustly. Summary to support your recommendationThe paper introduces a novel nowcasting framework, NCDENow, that combines Dynamic Factor Models (DFM) with Neural Controlled Differential Equations (NCDE) to enhance GDP prediction. This hybrid approach aims to address the shortcomings of traditional DFMs by effectively capturing the dynamics of irregular time series data, providing significant improvements in nowcasting accuracy and interpretability compared to existing methods. The method's effectiveness is validated through empirical results on real-world GDP datasets from South Korea and the United Kingdom. Reviewer 3: rating 1 (weak accept)Overall recommendation 1 (weak accept) Quality of Presentation5 (excellent) Detailed Comments to the Author(s)Strong points: Weak points: Question: Summary to support your recommendationThe paper offers a valuable contribution to the field of GDP nowcasting by identifying current challenges and proposing a novel framework that integrates Neural Controlled Differential Equations with Dynamic Factor Models. The paper is well-written, proposes initiating solution, and includes detailed numerical experiments. However, due to the limitations regarding dataset validation and the selection of baselines, I recommend a weak acceptance. Reviewer 4: rating 0 (borderline paper)Overall recommendation 0 (borderline paper) Quality of Presentation3 (fair) Detailed Comments to the Author(s)This paper proposes a novel framework for GDP nowcasting using irregularly sampled macroeconomic indicators, based on the Neural Controlled Differential Equations framework for modeling latent factors. The authors compare their approach against DFM and its combination with different flavors of sequential models. Several research questions are explored by empirical evaluation on two datasets, including prediction performance, effects of missing data and ODE solvers used. Strong points: Weak points: Summary to support your recommendationEven though the idea seems novel and the results look interesting, it seems that the model is not deployed in the real world. Meta-review and decision |
Rebuttal
We sincerely thank you for the review and positive feedback. Below, we address each of your concerns and questions:
---
Q1. How to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models?
(1) Updating neural network parameters
For training the neural network, please refer to Algorithm 1 in Section 3.5, where line 11 describes updating the parameters.
(2) Labels to train models
If the GDP growth rate we want to nowcast is at time \(N+1\), as shown in Figure 3, we use the GDP growth rate up to time \(N\) and related economic indicators up to time \(N+1\) for training. The label (ground truth) you mentioned is the GDP growth rate at time \(N+1\). This learning method is about the concept of nowcasting; please refer to Figure 2.
Q2. A question considering the results in table 8: the MAPE seems to be large (57%) even for NCDENow for the sudden drop. How to comment on this result?
We apologize for the confusion. There was an error in the calculation of the MSE and MAPE for the COVID-19 period (2020Q1, 2020Q2, 2020Q3) that caused the 2020Q3 forecast to be missing. As shown in the table below, the corrected MAPE for NCDENow decreases to 53.87%, while the MAPE for DFM increases to 99.55%. The hyperparameters are the best settings based on the MSE, which are the same as those described in Table 3.
| COVID-19 | Ground Truth | NCDENow | DFM |
|------|:-----:|:------:|:-------:|
| 2020Q1 | -1.2615067 | 0.1034569 | 0.10668433 |
| 2020Q2 | -3.1507382 | -3.3880901 | -1.14140557 |
| 2020Q3 | 2.2334244 | 1.2085018 | -0.5897278 |
| MSE | - | 0.9900 | 4.6265 |
| MAPE | - | 0.5387 | 0.9954 |
The best setting based on MAPE (learning rate: 0.01, hidden dimension: 512, # layers: 3) results in a performance of 44.53%, which you could also consider poor. However, we can see from this study that the performance can be significantly improved compared to DFM, which is used by several central banks for GDP nowcasting.
Q3. What is the experiment setting for RQ3?
The experimental setting for RQ3 is identical to that shown in Table 5.