many thanks for letting us consider your manuscript. Upon consideration of the enclosed reviews, we have several reservations and have decided not to accept the paper for publication in JOCN. I hope the enclosed remarks are helpful and will guide you in your future efforts on this work.
With the increasing popularity of JOCN, these difficult editorial
decitions are becoming increasingly frequent. Our best wishes as you
submit your paper elsewhere.
===================================================================
Reviewer 1:
--------------------------------------------------------------
The intuition that forms the basis of this paper is quite sensible: to
what extent PET and MRI strudies that purport to investigate the same
cognitiv function replicate? The author has chosen over research
articles published in 1997 and assesses (a) whether the results of a
given study are explicitly compared to previous data bearing on the
same issue and (b) whether the issue the replicability is discussed.
The author finds that there is alarming degree of variability among
results, and that the issue of replicability has not been addressed
properly.
While I share the author's concern that the issue deserves considerably more attantion, I do not find that the paper is sufficiently rich, provocative and/or analytical to warrant publication in its present form. Neither technical (acquisition-oriented) nor data-analytic or concetual/design reasons are put forth to account for the observation. Insofar as the manuscript is merely elaborated of the (already informally stated) observation (but not an analysis of the reasons leading to the alleged non-replicacbility), it will not fare well as a research article. Importantly, there are already papers in the literature which work through some of the problems associated with cognitive imaging in detail, so the burden is on this author to develop further, specific arguments that illuminate possible pitfalls for the field.
I have two concrete suggestions. First, rewrite this manuscript as a
*letter* summarizing the "replicability-is-rare observation" and
pointing to the importance of the problem. There are a number of
jounals that publish technical notes or commentaries, and this
manuscript as appropriate for such a forum. The observation could
certainly be stated effectively in a page or so. Second, the
manuscript needs extensive language editing - it would help tolet
several readers look at the paper before submission
===================================================================
Reviewer 2:
--------------------------------------------------------------
It is a reasonable goal to perform meta-analysis of the imaging data
to assess the replicability of cognitive activations across different
studies. However, the persent paper fails to accomplish this for two
reasons. First it does not actually analyze and quantify variability,
but instead merely make anecdotal references to other author's
comments (published) regarding replicability. A truly careful analysis
would be welcome. Second, the author does not consider (much) the true
variability between methods, and especially, between tasks and
statistical comparisons. Before one can evaluate the effects of task A
on activations B, C and D across studies, one must precisely define
task A. The persent paper fails in this regard and should not be
published.
===================================================================
Reviewer 3:
--------------------------------------------------------------
Review of "Replicabiliy of cognitive imaging of the cerebral cortex by
PET and fMRI: a survey of recent literature by Yehouda Harpaz
This is critical review about a very important problem in functional
neuroimaging: the replicability of the data. Unfortunately, the
present paper contributes very little to the clarification or solution
of this problem. As stated in the abstract, the review provided two
findings :(a) that data about replicability in cognitive imaging are
extremely sparse; and (b) the data availale are mostly negative, i.e.
it suggests lack of replicability. The first finding could be reported
in one paragraph, and does not justify the publication of a journal
paper. The second finding is more interesting, but it has a serious
flaw: it not based on a systematic and comprehensive meta-analysis of
the data, but on what other people said about replicability during 10
months in 1997. The conclusion of lack of replicability is based on
the following premises:
1.Most of the studies reviewed (116 out of 125) do not discuss the
issue of replicability;
2. The arguments for replicability made by the remaining papers are
not valid.
3. The few reviews that investigated replicability either have problems
(e.g. Poline et al., 1996) or show only minimal replicability (e.g.
Poepple, 1996).
The same argument is applied to individual differences in activation.
Thus the conclusion that there is a lack of replicability is not based
on original scientific research, but on the opinion of the author
about the opinions of other researchers. If the paper is summarized
into a couple of paragraphs, it could perhaps be published in another
journal as a letter to the editor.