Evaluating the Quality of Dutch Academic Legal Publications: Results from a Survey

In 2009 the Koers Committee, which evaluated the research of all Dutch law faculties, concluded that law is a discipline in transition moving, among other things, from: a national to a more international focus, from a monodisciplinary to a more multiand interdisciplinary perspective, from implicit tradition towards more methodological accountability, from individualism towards research programming, from overhead financing towards competitive financing and from laissez-faire towards (output) monitoring and control. This inevitably must have consequences for how scholarly legal research is being evaluated. In this respect the report concluded:


Introduction
In 2009 the Koers Committee, which evaluated the research of all Dutch law faculties, concluded that law is a discipline in transition moving, among other things, from: a national to a more international focus, from a monodisciplinary to a more multi-and interdisciplinary perspective, from implicit tradition towards more methodological accountability, from individualism towards research programming, from overhead financing towards competitive financing and from laissez-faire towards (output) monitoring and control. This inevitably must have consequences for how scholarly legal research is being evaluated. In this respect the report concluded: In the opinion of the Commission, the most important weakness in Dutch scholarly legal research at the moment is that there is no consensus whatsoever regarding the quality standards by which scholarly legal publications ought to be assessed. In other disciplines as well, it is often debated what separates academic research from non-academic research. However, in these other fields, scholars have usually managed to reach at least some form of consensus on the way in which manuscripts should be (peer) reviewed and how journals can be ranked. Law as a discipline has not yet reached this far. The result is that the evaluation of scholarly legal research is both relatively difficult and time and energy consuming since each evaluation committee is forced to reinvent the wheel -which in turn results in the need for justification of all decisions that the committee require to make on an ad hoc basis. 1 Law faculties in the Netherlands have in the meanwhile received the final outcomes of their 2016-2017 research assessment exercise (RAE) and the collation by the general reporter. Perhaps the most striking outcome of this assessment was that on a four-point scale ranging from: 4 = unsatisfactory, 3 = good, 2 = very good and 1 = world leading/excellent, all faculties received the grade: very good. The question is, of course, what does this score on an aggregated level tell us about, for example, the quality of law journals, legal publishers, legal research proposals submitted for funding, and so on?
We find it remarkable that the debate on the quality of academic legal research has, so far, been dominated by reports of expert committees. 2 To our knowledge, the forum of legal scholars has never been interviewed to discover how legal scholars think about the quality of legal scholarship and how it should be measured or assessed. We believe there are both practical and theoretical reasons for doing so. Kaltenbrunner & De Rijcke, for example have argued that: Utrecht Law Review | Volume 13 | Issue 3, 2017 | Special Issue: Methodology of Legal Research [W]e have found a striking diversity of ways in which Dutch legal scholars handle evaluation indicators in their administrative counting practices. Moreover, rather than a reluctant yielding to externally-imposed procedures, these practices frequently constitute proactive attempts to use research evaluation in the pursuit of particular epistemic and strategic agendas. For example, some coordinators police the boundary between scholarly and professional publications very strictly. They tend to count articles as scholarly if they are in line with a particular vision of legal scholarship, that is, research that is empirically orientated, conceptually robust, and primarily relevant to international academic audiences. However, there are diverse other strategies of using indicators that are connected to very different intellectual outlooks. 3 From a theoretical perspective these different 'intellectual outlooks' are probably related to how legal scholars view law as a science. Those who believe that legal research is supposed to serve the practice of law, will probably have a different take on, for instance, the distinction between academic and professional publications and the way these should be valued, than those who think that academic legal research is first and foremost meant to increase 'the body of knowledge' about the law. Practically, how one weighs the merits of publications may lead to strategic behaviour. In other words, if only academic legal publications count in faculty-internal evaluation protocols or in external research assessment exercises, the pressure to label as many publications as possible as academic is rather likely to exist. The same goes for refereed journals, peer reviewed books, and prize winning research proposals. Quality indicators, after a while, tend to get a life of their own. Moreover, research in the humanities has learned that the introduction of journal rankings or other quality management tools should not be imposed upon a discipline without the understanding and support of the scholarly community. 4 This is why we felt that is was time to conduct a survey with regard to how Dutch legal scholars feel about the direction in which scholarly legal research is moving, the way in which faculties, law journals, and legal publishers evaluate the quality of scholarly publications and the extent to which changes are considered necessary and, if so, in what direction these changes should go.

Questionnaire
In 2015, we designed a questionnaire, which was pretested and adjusted several times on the basis of comments by colleagues. The questionnaire consisted of five parts (demographic data, research and publications, research assessments, internal evaluation policies of law schools and research culture and future expectations). In this paper, we present those parts of the outcomes that concentrate on general trends in academic legal research in relation to other disciplines, research evaluation and legal publishing.
We decided to allow respondents to leave certain questions in the survey unanswered since some questions were more relevant to senior researchers than to PhD students (and vice versa). The consequence of this is that the number of respondents per question may vary. Nevertheless, most questions in the survey were answered. The possibility provided to make open comments in special text boxes was frequently used. These comments helped us to understand better the context of certain answers; however, because of their sheer volume we decided not to include a separate empirical analysis of these comments in the current contribution. 5

Response
The survey was distributed in February 2015 via online survey software. 2,740 scholars associated with a Dutch university law school received an invitation to participate in the survey. Ultimately, 24% of the addressees completed and returned our questionnaire. 6 Although we cannot guarantee the proportionate distribution of respondents over the law faculties, our impression is that a reasonable coverage of academic staff at Dutch law schools was accomplished. After initial publication of our findings we did not receive any indication that our outcomes were one-sided or lacking representativeness. 7 The response rate is quite evenly divided over the different faculties and is usually between 21% and 29%, the University of Amsterdam being the only exception with a response of only 13%.

Demographics, position, type of appointment
Of the respondents, 55% are male and 45% female. 8 The age ranges from 22 to 72 years, the average age is 39.9 years. The three largest groups of respondents are: professors, assistant professors and PhD students (see Table 1). 58% of the male respondents claim to have a permanent contract, while only 42% of the female respondents have a similar contract. 9 12% of the respondents used the English language version of the questionnaire. 10 The same percentage of respondents indicated that Dutch was not their native language. 11 The latter group is mainly composed of PhDs (53%) and professors (15%; n=79). In response to the question whether respondents considered themselves to be lawyers, 80% answered affirmatively. For postdocs and researchers without a PhD only 50% and 52% respectively considered themselves to be a lawyer, while teaching staff without research duties felt overwhelmingly that they were lawyers. The term 'lawyer' was not defined, though; hence these answers only indicate how respondents see themselves. We will revert to this distinction where there are interesting differences between how lawyers and non-lawyers working in Dutch law faculties perceive research quality (assessment).
6 The number of answers for each question varied. This has to be taken into account when studying the survey outcomes. One should also keep in mind that percentages are sometimes calculated from answers with a different number of respondents (the n varies). 7 We could not find an open source to verify the absolute number of scientific staff in Dutch law faculties. Koers Committee (2009)

Field of expertise and experience
Most of the respondents work in the broadly defined field of private law (for instance, civil procedure, corporate law and insolvency law), followed by legal theory and constitutional and administrative law. Most of the (self-declared) non-lawyers are to be found in the area of criminology and international public law, which apparently attracts many non-lawyers (see Table 2). * (% concerns percentage of the respondents who mentioned this field of expertise) ** (% concerns percentage of the respondents who indicated both their field of expertise and their position as lawyer/ non-lawyer) When asked whether respondents had had prior working experience (as a lawyer) outside academia after their studies, 55% of the lawyers responded affirmatively. 12 Of the non-lawyers this was only 14%. 13

The results
Hereafter, we report the main findings of our survey. Only significant differences in responses between lawyers and non-lawyers are reported.

How do scholars see themselves?
In this section, respondents were asked to classify their own research. Table 3 summarises the responses.  (Table 3). At 61.1%, comparative research is also popular which might be caused by the internationalisation of both legal practice and legal research. Legal theory and empirical legal research are almost equally popular, however, still clearly lagging behind doctrinal and comparative research. With regard to legal theory and empirical legal research, it is interesting that opinions differ much more with regard to the extent to which scholars want to label their research as such. The question is: why? The view that doctrinal legal research is in a state of crisis, as some scholars claim, 14 does not immediately seem to correspond with these statistics. Two-thirds of the respondents still identify their research as legal doctrine. However, this does not mean that doctrinal work is still as important as it was a few decades ago.
We asked respondents to indicate at which audience their publications were aimed. A considerable number of respondents claimed that their research was focused on a European and international debate with other scholars (see Table 4). Almost 75% of respondents indicated that their own research was targeted at an international scholarly legal audience, while 70% (also) concentrated on the debate with other Dutch scholars. With regard to the focus on legal practice, it is worth noticing that our respondents claim to be more focused on a debate with Dutch (55.3%) practitioners than with European and international practitioners (40.2%).
Our survey also revealed that respondents overwhelmingly characterise their publications as attempts to improve the quality of law making (77.1%). 15 Dutch legal scholars appear to see themselves more as 'legal engineers', 16 who provide a service to legal practice than as analytical social scientists or indeed, more interpretative-orientated scholars who belong to the humanities. This seems consistent with the fact that a majority of legal scholars report that they apply similar methods in their research as judges and attorneys. 17 However, this does raise questions with regard to the preference which legal scholars also seem to have (see Table 4) for a peer audience over a practitioners' audience. The data also indicate that Dutch legal scholarship is not as close to the humanities as some still think. 18 Lawyers claim to feel more comfortable answering questions about positive law than questions concerning the effectiveness and efficiency of law, how the law ought to read, or explanations as to why the law develops as it does. 19 Non-lawyers, on the other hand, feel more conversant with questions regarding the functioning and effects of the law than with questions concerning the state of the law and how and why it develops in a certain direction. 20 This makes one wonder whether there is a tension between the law reform ambitions of legal scholars and their lack of familiarity with matters of effectiveness and efficiency and with empirical legal methods. Next, we asked respondents to rank different types of publications according to their own preference (see Table 5). What follows from the answers is that respondents have a clear preference for publishing articles in international and Dutch law journals instead of writing, for example, commentaries. The difference in the extent to which lawyers and non-lawyers value handbooks versus textbooks is remarkable. Textbooks are an essential part of almost every law school curriculum both for the training of practitioners and future scholars. Therefore, it is hard to understand why textbooks are so low on the priority list of lawyers. Why would handbooks be more 'academic' than textbooks? Later editions of a handbook, especially, do not always add new insights to the body of knowledge. A modest update of a handbook might therefore not be enough to label it as an academic publication. 21 Table 5 As concerns my own publications, I attach the most (=1) and the least (=8) value to writing: It is not surprising that there is a significant preference amongst non-lawyers, such as criminologists, for contributions in international journals. This is probably due to the fact that in the social sciences and in STM (Science, Technology and Medicine) this preference is the rule rather than the exception due to how (the impact of) research is evaluated, 23 whereas in most of the other areas of legal research the opposite holds true.   Next, we asked about personal publication preferences. At first glance (see Table 6), the answers to the propositions concerning personal publications do not appear very useful because they may have elicited reactions that are socially desirable or expected (e.g. 'of course, I think carefully about my own publication strategy!') However, although many respondents claim to possess an individual publication strategy, a large number of them simultaneously say they are led by what their faculty considers important, by invitations that cross their path and by requests from editorial boards.
We wonder to what extent these answers are reconcilable. After all, having a personal publication strategy also presupposes, for example, 'the art of saying no' to requests that do not fit into that strategy. 24 It is interesting that non-lawyers appear to be led more than lawyers by what their faculty considers important. 25 They also claim to think more carefully about the language in which they publish. 26 This might be explained by the fact that publishing in international (English language) journals is more common for non-lawyers, such as criminologists. With respect to publishing, focusing on international journals has become important for Dutch legal scholars. This is probably influenced by the fact that scholars believe their own field of expertise is becoming increasingly international 27 (see Table 7 and the high rank of international journals in Table 5). 28 Hence, it is perhaps not surprising that almost 50% of the respondents show a willingness to invest more time and energy in preparing publications in international media (journals, books). At the same time, 57.5% of the respondents are more or less convinced that for the quality of their research it does not make a difference whether one publishes in national or international journals. 29 This might be explained by the fact that the quality indicators used by national and international journals are considered to be rather similar or because scholars feel that quality is enshrined in the content of publications and not in the form or the language in which they are written.

Quality of academic legal research
In this section, the results of the questionnaire primarily concern the quality indicators and standards for scholarly legal research and the way in which this quality can be recognised in publications. Respondents were asked to rank a number of indicators for quality of scholarly legal publications. Thoroughness and profundity clearly score higher among lawyers than among non-lawyers. The latter group feels that originality should be the primary trademark of scholarly legal research (Table 8). It is also noteworthy that methodological rigour scores relatively low on the list of possible quality indicators, while both groups consider convincing results and conclusions as more important than methodological rigour. The question is of course: what does this mean? Does it imply that Dutch legal scholars not only see themselves first and foremost as legal engineers (see above under 3.1) but that they also prefer 'playing for results' rather than respecting the methodological 'rules of the game' because they are primarily interested in useful outcomes? Alternatively, is there another reason why methodological rigour is considered relatively unimportant, such as a lack of methodological awareness in the discipline as such?
27 There is a strong correlation (r=-.637, n=547, p<0.001): the more respondents give weight to publishing in international journals the more they agree with the proposition that their field is becoming more international. 28 Koers Committee (2009), supra note 1 (at pp. 30-31) also signalled this. 29 These are not necessarily the same respondents. The answers to propositions, such as 'for a publication in an international journal I am prepared to invest more time and energy in my research than for a publication in Dutch' and 'with regard to quality there is no difference between a publication in a national or international journal' there is a negative correlation; r=-.336, n=620, p<0.001. The presence of theory-building 3 The presence of theory-building The way in which the use of sources is accounted for 4 The use of clear and precise language The presence of solid research methodology 5 The way in which the use of sources is accounted for * We only included respondents who ranked all items. Differences in means between lawyer/non-lawyer statistically significant at p<0,001except for variables 'research question' and 'theory building'. 30 When asked to rank potential indicators for the quality of the content of scholarly legal publications (see Table 9), lawyers and non-lawyers put the presence of a clear research question at the top of the list, while 'the way in which the use of sources is accounted for' scores much lower in both camps. The need for a clear research question as an important quality indicator is noteworthy because, ten years ago, the view that both journal articles and legal (PhD) dissertations should have a clear research question was still contested among Dutch scholars. 31 Yet, today we increasingly find the requirement for an explicit and well-formulated research question in journals' author guidelines. 32 There is also a difference between lawyers and non-lawyers with regard to the value they attach to the use of clear and precise language and the presence of a solid research methodology. For non-lawyers methodology appears to be an important indicator for quality, whereas this is less so for clear and precise language. By contrast, lawyers seem to have the exact opposite preference, which indicates that linguistic precision is still considered highly important for scholarly legal research. Last but not least, lawyers and non-lawyers attach the same weight to theory-building in their research, which does not necessarily imply that they have the same understanding of what theorising entails. Next, our respondents were questioned about general quality standards for legal publications (Table 10). What can be concluded is that the majority feels that the language in which a publication is written says little or nothing about the quality itself. Moreover, where there seems to be support (43%) for a harmonisation of the standards for PhD dissertations, there does not appear to be a strong desire among Dutch scholars to harmonise the quality standards that law journals apply throughout Europe. Only some 26% are, more or less, in favour of this, while 32% are, to a certain extent, against this. This indicates that Dutch legal scholars probably consider it too early to start talking about a European harmonisation of the standards that law journals should apply to submissions.   With regard to the proposition, 'I feel that the extent to which scholarly legal publications contribute to theory-building is often unclear', lawyers and non-lawyers answer in a similar vein (See Table 11). 33 For all other propositions, they differ. A majority of both groups, nevertheless, is of the opinion that the methodological requirements for scholarly legal publications should be made more explicit and that methodological accountability would contribute to the quality of existing legal research. Lawyers and nonlawyers clearly have different views on the contribution that scholarly legal research delivers to solving practical legal problems. Where 39% of the lawyers feel that this is the case, 36.4% of them have doubts. Non-lawyers seem more convinced about the problem-solving capacities of scholarly legal research since 44.8 % (partly) agree and only 21.9 % (partly) disagree. This is relevant since we saw earlier that, in their work, legal scholars focus quite heavily on improving the quality of law making in practice (see Section 3.1).
Utrecht Law Review | Volume 13 | Issue 3, 2017 | Special Issue: Methodology of Legal Research

Quality of journals
When asked about the relevant aspects of the quality of law journals, lawyers and non-lawyers had different priorities (Table 12). The reputation the journal has according to quantitative indicators (e.g. impact factor) Whether the entire editorial board assesses a paper or not 5 The expertise of the editorial board The reputation the journal has according to quantitative indicators (e.g. impact factor) 6 Whether the entire editorial board assesses a paper or not * We included only respondents who scored all items. Differences in means between lawyer/non-lawyer statistically significant at p<0.001. 34

34
Lawyers consider the expertise of editorial boards to be clearly more important than non-lawyers do. The latter group pays more attention to whether an editorial board assesses incoming manuscripts itself or delegates this to external referees (peer review). The most significant difference between lawyers and nonlawyers concerns the importance that is given to the reputation of a journal on the basis of quantitative indicators, such as the journal impact factor. Lawyers seem to pay hardly any attention to this. This might be explained by the fact that there are hardly any Dutch legal journals that make use of bibliometric indicators and there is no citation database for law journals. Whether an editorial board as a whole assesses publications or not is considered largely irrelevant by both groups.  Table 13 contains some sensitive issues concerning quality management. What catches the eye is that 76.4% of the respondents feel that law faculties put too much emphasis on measuring the number of publications that their staff produce. At the same time, 42.5% of the respondents claim that proper quality indicators for scholarly legal publications do not exist. This would, of course, make it hard to assess the quality of publications. The latter claim becomes even more interesting now that 53.7% of the respondents feel that double, blind peer review leads to a higher quality of publications. One wonders how that is possible without consensus over quality indicators. Moreover, there appears to be disagreement with regard to whether or not peer review takes up too much time for all parties involved. One wonders what is behind these opinions. Do legal scholars compare their personal experience with publishing for Dutch law journals with publishing for foreign (peer reviewed) journals or do they just express a normative judgment about how things ought to be? It is also hard to tell to what extent legal scholars have a clear notion about the central features, different forms (open, single blind, double blind etc.), and the general strengths and weaknesses (e.g. biases) of peer review. The same goes for the opinion of 63.2% of the respondents that the current method of quality measurement leads to undesirable strategic behaviour. Is this something respondents relate to their own research or is it more of a general notion unrelated to personal experience?

Warranting quality
What follows is the part of the questionnaire concerned with the aims that are served by research evaluation, the preferred evaluation methods (e.g. ranking, peer review, metrics) and the assessment of research by external bodies, such as the national research foundation and internal faculty procedures.  Table 14 illustrates that the possible goals of research evaluation that were specified were considered to be important. Warranting a certain minimum quality of scholarly legal publications comes out on top. 92.7% of our respondents claim to find this (very) important. This might imply that legal scholars are concerned primarily with the definition of minimum standards for scholarly research to serve as a threshold and perhaps somewhat less with defining what is cutting edge. However, 81.2% find promoting research excellence (very) important. With regard to the other goals, the scores gradually decrease with efficient and well-targeted allocation of funds at the bottom end of the scale, although 64.3% still see this as (very) important. The problem is, however, that we do not know how respondents would have reacted if they had had to make an explicit choice between the different, possible goals. This is relevant because it is, for instance, likely that ensuring a certain minimum quality of scholarly legal publications requires measures other than promoting research excellence. Differences in distributions between lawyers and non-lawyers are significant for all reported items.
We also posed questions with regard to the general ways in which research quality could be evaluated. What stands out in the answers is firstly that a large majority of Dutch legal scholars prefer a substantive assessment (reading) of their publications to citation measurement (see Table 15). No less than 91.6% of the legal scholars who answered the questionnaire (partly) agree with this and 82% of the non-lawyers. Both groups are also of the opinion that assessment of publications by a professional editorial board is not necessarily worse than peer review, but lawyers (71.9%) are much more outspoken about this than nonlawyers (40.2%). With respect to a ranking of journals, 42.3% of the lawyers are (partly) against this and only 34.6% are more or less in favour of it. For non-lawyers the order is exactly the opposite. 44.4% are in favour of such a ranking while only 20.9% are partly against it. As far as the lawyers are concerned, 32.5% somehow feel that a national research assessment exercise is a suitable means of assessing research groups, whereas almost the same number (32.3%) has no faith in such an assessment and 35.2% have no opinion about it. Non-lawyers obviously have more faith in a national research assessment with 51.1% (partly) in favour and only 22.2% (partly) against, while 26.4% have no opinion about it. Lawyers are apparently more sceptical than non-lawyers about the benefits of both rankings and national research assessment exercises, which calls for further research as to the reasons behind this. With regard to peer review (research assessment by external referees who are not part of the editorial board), non-lawyers have more faith in this type of scrutiny than lawyers (see Table 16). More than 75% of the non-lawyers (partly) agree with the proposition that law journals should always have some form of external review, while only 37.7% of the lawyers are (partly) in favour of this and 42.2% are more or less against it. It is noteworthy that lawyers have more sympathy for peer review of books. No less than 46.8% of the lawyers, more or less, agree with this, while only 34.6% are, somewhat, against this. However, nonlawyers are here again more vocal than lawyers. This is consistent with the view that non-lawyers hold stronger beliefs in research fraud prevention via peer review than lawyers do. Almost 50% of the nonlawyers have faith in peer review as a means of filtering out fraud, while only 34.8% of the lawyers take this view. For both groups, one wonders what the underlying basis is for the faith which people have in peer review as a means to prevent research fraud. The literature concerning the role of peer review in this respect reveals that it is not a particularly effective method of preventing fraud. Reviewers are usually not focused on detecting fraud and look mostly at individual manuscripts, which makes it difficult to discover suspicious patterns in the publications of the author under scrutiny. 35 From the answers to questions about publication pressure, it is clear that a lot of respondents feel that too much emphasis is put on counting the number of publications they produce (Table 17). With regard to the question whether unreasonable publication pressure is being felt, 40.9% agree, and for the question whether this focus of attention on the number of publications is detrimental to research quality, 72.7% of the respondents (partly) agree. At the same time, respondents do not seem to think that there is too much managerial interference with the number of publications they need to produce or with the nature of these publications. 41.6% (partly) disagree with this, while only 29.4% (partly) agree there is too much managerial interference. This makes one wonder: who is putting emphasis on counting publications if it is not management? The fact that almost half of the respondents (47.3%) feel that the current emphasis on research production encourages fraud makes this question especially poignant.

Research culture
Utrecht Law Review | Volume 13 | Issue 3, 2017 | Special Issue: Methodology of Legal Research In this part of the questionnaire, respondents were asked about the direction they expected the law, as a scholarly discipline, to develop (see Table 18). We also asked them in which direction they hoped their discipline would develop. 36 The answers show that, both in terms of hope and expectation, respondents feel that law will develop more closely in the direction of the social sciences. With respect to the arts and humanities the expectations of a closer relationship are low; however, the hope is significantly higher. Regarding economics and business studies the opposite is true. There, most legal scholars expect convergence; however, generally they do not really favour such convergence. Predictably, the pure sciences remain out of the picture for the near future. Unsurprisingly, lawyers and non-lawyers held different views, as Graph 1 shows. 36 We did not ask respondents whether they expect and/or hope that legal scholarship will not develop in any other direction or develop itself further as a discipline in its own right. The high percentages 'Neither agree nor disagree' together with the outcome of Table 3 (doctrinal legal research is still dominant) could nevertheless indicate that many lawyers view law as a unique discipline that will not be colonised by other disciplines so easily, at least not in the short run.

Analysis
One of the first things that comes to mind when reflecting on the outcome of the survey is that (traditional) doctrinal legal research still has a strong position in the Netherlands. Comparative legal research follows in second place. Thereafter, some distance behind, come 'legal theory', 'empirical legal research' and 'law and… research', tied in close competition for the third position. Based on these answers, it is difficult to justify the notion that traditional legal research is in a state of crisis as some have suggested. 37 It is probably also true, though, that scholarly legal research can no longer primarily be seen as a service to legal practice, as the Koers Committee has already argued. 38 Almost three-quarters of our respondents indicate that their research is primarily focused on a European and international debate with other scholars, closely followed by a debate with other national scholars and far less on a debate with practitioners. It seems somewhat contradictory that, although most respondents focus on debate with fellow academics, their main aim is to contribute to better law making. Almost 80% of the respondents indicate that they see their research primarily as a way to define concrete proposals for law reform. 39 How should we explain this? Is it a matter of giving socially desirable answers (e.g. 'the creators of this survey are apparently interested in my academic ambitions and would like to hear that I am focused on an academic debate with other scholars') or is it simply that legal scholars do not see any contradiction between debating with fellow scholars and being focused on improving the law? When asked to rank the sorts of problems which Dutch legal scholars are interested in, questions with respect to 'why the law develops as it does', score the lowest. In the literature, however, it is sometimes argued that it is precisely the focus on the 'why-questions' that distinguishes scholars from practitioners. 40 Asking why-questions brings one closer to theory-building and further away from direct problem-solving. What is also interesting is that, on the one hand, scholars claim to feel at ease with questions about the effectiveness and efficiency of the law, while, on the other hand, they are hesitant to label their research as empirical and rate methodological accountability rather low as a quality indicator. What does this mean? Do Dutch legal scholars really think that it is possible to design better working laws without empirical research? Do they believe most legal problems can be solved via common sense and practical wisdom or is it that they have difficulties in drawing the line between normative and empirical claims, which in our view require different research methods?
What matches well with the focus on a debate with other scholars is that our respondents rank commentaries and case notes relatively low and give priority to writing articles for national and international academic journals. More confusing is the fact that many respondents claim to have an individual publication strategy and carefully select what sort of publications they want to publish in, while a lot of them also admit to being led by what their faculty finds important, by invitations that cross their path and by what journals seem to be looking for. Because of this, it remains unclear whether scholars voluntarily focus on publishing in international journals or are encouraged to do so by incentives from their own faculty, driven perhaps by national evaluation policies. Regarding such policies, just think of the distinction in national research assessments between professional and academic publications. Over the past two decades or so, this has led to a devaluation of case notes and practical commentaries in national research assessments. In turn, this has probably made writing such professional publications less attractive for academics. In other words, individual preferences may also be shaped by changes in the academic work environment. Contrary to this line of argument is that lawyers, especially when compared with non-lawyers, still attach considerable value to the writing of handbooks, since these are commonly excluded from the category of academic publications. This is particularly the case for later editions of handbooks, which merely update existing materials.
What comes to the fore from the survey is that one of the reasons for focusing more on publishing in international journals is that scholars feel the debate in their discipline is becoming increasingly transnational. We do not know, however, whether this also means that legal scholars are prepared to put more time and energy into international publications than into publications written in Dutch. Unfortunately, we were unable to discover whether our respondents believe that the threshold for publishing in international journals is higher than for national journals, which could have explained why scholars are prepared to invest more time and energy in them. An alternative explanation is of course that command of a foreign language requires more effort from scholars to produce international publications.
As far as the quality of scholarly legal research is concerned, we observed that originality scores lower with lawyers than, for example, thoroughness and profundity, while methodological rigour remains even further behind. This is remarkable because both lawyers and non-lawyers view the presence of a clear and well-defined research question as an important indicator for the quality of legal publications. One of the most import features of a sound research question is, according to the literature, that it indicates what the research is going to add to the current state of the play and hence what is new and original about it. It is probably less of a surprise that non-lawyers seem to pay more attention to methodology and research design than lawyers, while the latter group sets higher standards for clear and precise language. After all, research methodology still does not play an important role in the law school curriculum, whereas argumentation and rhetoric are given centre stage in the training of how to think like a lawyer.
Regarding standards, there appears to be much support for harmonisation of the quality standards for PhD dissertations but not for the quality criteria that law journals throughout Europe apply to the submission of manuscripts. A small majority of our respondents feel that the quality standards for academic and professional publications are different but we do not learn how and where they are different. Something similar applies to the view amongst both lawyers and non-lawyers that the methodological requirements for academic legal publications should be made more explicit. Here too, we should ask ourselves: how far should this go, how should this be done and who is to be responsible for it?
As far as the quality of academic journals is concerned, non-lawyers attach more weight to peer review than lawyers. Lawyers believe that the assessment of scholarly papers by a professional editorial board is not necessarily inferior to assessment by independent external referees. They also feel that there are not enough independent experts in the Netherlands to make peer review the standard for every journal and that peer review is going to require too much time and energy for all parties involved. On the other hand, our survey shows that Dutch legal scholars prefer a substantive assessment of scholarly legal publications (reading manuscripts) to counting citations. 41 Moreover, they clearly prefer double, blind peer review to other forms of substantive review. It is not immediately clear why lawyers appear to be less sceptical about the peer review of books. Do our respondents believe that book publishers go easier on authors than do editorial boards of law journals? As far as warranting the quality of scholarly legal publications is concerned, most respondents are concerned about guaranteeing a certain academic minimum standard. Simultaneously, however, promoting research excellence also scored highly. We wonder whether these goals are fully compatible. Requiring a minimum standard for publications is closely related to indicators, such as the presence of a clear research question and an explicit methodology, whereas research excellence has much more to do with things like creativity, theorising, and drawing insights from other legal systems or other disciplines than law.
Concerning publication pressure and the risk of research fraud, a vast majority of our respondents share the view that emphasising research output is detrimental to quality and almost half of them even believe that the current focus on research production (e.g. number of publications) encourages fraud. This raises serious concerns about evaluation methods that focus on output parameters. Scholars are, however, divided about the question as to whether there is too much managerial interference with their personal research production both in terms of output and type of research. This begs the question: who is seen as responsible for the existing publication pressure? If it is not university or faculty management, who is increasing the pressure? Only more qualitative research could provide answers here. This is also the case for the alleged relationship between publication pressure and research fraud.
Utrecht Law Review | Volume 13 | Issue 3, 2017 | Special Issue: Methodology of Legal Research Finally, when legal scholars have to locate their own discipline in a broader academic landscape of disciplines, they feel more connected to the social sciences, then to the arts and humanities and they consider the least connection to be to the pure sciences. When asked about their expectations, legal scholars believe that a closer engagement with the social sciences in the near future is more likely than a movement towards the arts and humanities. With respect to economics and business, legal scholars expect a stronger connection but they do not necessarily hope that this will happen. The question is how this relates to the strong preference that lawyers have for peer review and assessment of the substance of publications versus the use of bibliometrics, such as citation scores, which are common ways of impact measurement in the social sciences.

Follow-up
It is not up to us to decide what should happen with the results of this survey. Nevertheless, it is important to take this opportunity to provide some suggestions for further debate and follow-up research. After all, Dutch law schools are still debating the distinction between professional and academic publication, the pros and cons of a journal ranking system and the benefits and burdens of peer review versus bibliometrics and have not even reached the beginning of consensus on quality indicators for academic publications. Therefore, it is important to involve the academic legal community more in the development of research evaluation. How important such involvement is, becomes clear from a recent study from Kaltenbrunner & De Rijcke, who interviewed a number of administrators (e.g. research coordinators and vice deans) in high-level positions at three Dutch universities (Leiden University, Tilburg University, Free University Amsterdam). 42 They discovered among other things that some law school administrators steer the way in which publications are counted as academic or professional. What is more important, though, is that their interviews reveal that quality indicators have not resolved debates about the quality of academic publications but merely relocated them from an academic to an administrative setting. 43 We believe this is a dangerous development, because how the quality of legal research is being determined has a direct impact on the choices that academics are going to make in their research. If case notes or textbooks, for example, will not count as academic publications in evaluation schemes, writing them will become far less attractive for academics. Not only does this raise the issue of who determines what sorts of publications are preferred, but it probably also has a far greater impact on the relationship between academia and legal practice than many people realise. The growing emphasis on publishing for international journals, for example, raises questions about how national legal practice can still benefit from independent academic research and how research evaluation relates to societal impact.
Choices concerning evaluation methods do not only have important consequences for the type of research that is going to be favoured but also for the publication system as such. A recent change in copyright law in the Netherlands, for example, has resulted in the fact that academic papers written with the support of public funding must become available to the public via open access within a reasonable time. 44 This will make it attractive for journals and publishers to label as many papers as possible as professional instead of scholarly publications. If simultaneously, however, only academic publications count in the internal evaluation protocols of law schools or in the protocol for the national research assessment exercise, scholars will have to deal with conflicting demands. Perhaps publishers will increasingly look for different business models or look elsewhere for authors of their practitioners' series.
Something similar applies to the use of terms, such as 'peer reviewed' of 'refereed'. If these labels are to be considered meaningful in research evaluations, it is important that journals and publishers have the same understanding about what these labels mean because it will have direct consequences on individual academics (e.g. do not write for non-peer reviewed journals because your publications will receive a lower Utrecht Law Review | Volume 13 | Issue 3, 2017 | Special Issue: Methodology of Legal Research ranking). The choice for certain evaluation practices are, in other words, far from neutral. These choices have a major impact on academic and professional life and may even determine in which direction the discipline of law as such is moving. Therefore decisions about research evaluation should not be left to policy makers or university managers and they should not be made without knowledge about the possible consequences. Currently, for example, law journals in the Netherlands are experimenting with different types of peer review. However, we have no evidence whether independent peer review leads to a different selection of articles than assessment by an editorial board with sufficient expertise. Before introducing peer review as the new golden standard in the field of law, it might be beneficial to look into the advantages and disadvantages of this type of research evaluation. Moreover, what seems inconsistent is that where law journals are experimenting with peer review, almost nothing is happening with respect to Dutch law books, whereas the outcome of our survey indicates that, especially in the case of books, legal scholars believe there is a need to introduce some kind of (independent) peer review in order to prevent everything that is published being referred to as academic.
Research evaluation may also have important political and economic consequences for law schools and legal scholars. The introduction of bibliometrics in the evaluation of legal research, for example, may lead to a reallocation of research funds and the same holds true for a shift in focus between the value that is attached to fundamental research as opposed to that attached to applied research. Perhaps lessons should be learned from foreign experiences, such as the experience in the UK, where there is an ongoing debate about the extent to which a national research assessment contributes to the quality of academic research and also about the possibilities of relying more on bibliometrics in research evaluations, because counting citations is much cheaper than organising peer review. At the same time, though, we should not forget that assessing the impact of academic publications is not the same as determining the quality. In that sense, citations, numbers of downloads or altmetrics (e.g. references to academic publications in popular media) are approximations of quality at best. Before taking drastic decisions about, for instance, the introduction of a ranking of journals or publishers, a shift from editorial review to (double) blind peer review or to measuring impact via citations scores, one should think through the consequences and involve those who are the subject of evaluation, namely academics, in the process.
We wonder if it would not be sensible to compare the current internal research evaluation protocols, quality standards, and credit systems of the different law schools in the Netherlands and try to find out what is the rationale behind these models. The outcomes of such a comparison could be the subject of national debate amongst the forum of legal scholars in which they are encouraged to give their own views and learn from each other's experiences. The extent to which different faculties, for instance, encourage legal scholars to develop their own research agenda and try to filter out substandard research, on the one hand, and try to encourage research excellence on the other hand, should be investigated before a harmonisation of quality standards is arrived at.
The heart of the matter, in our view, is that law as a discipline should take more of the initiative with respect to research evaluation and quality management. Instead of doing nothing or simply relying on the assessment methods that have been developed in other disciplines, we could also try to determine our own destiny, referring to and relying on the experience in neighbouring fields, when useful or necessary, accompanied by research. In this way, we could try to benefit from the fact that we lag behind other disciplines and we could learn from the mistakes that others have made before us. Moreover, in developing our own standards and quality systems, the clear advantage is that we can take the specific features of law as an academic field into account and stay close to what the scholarly legal forum itself considers to be important.