Managgiario in distance1)


Summary

Peer advice and advisory committees are the principal instruments used by the benefactor, usually called the Program Officer (PO), who handles the decision making process for research grants. His experiences have neither been described nor investigated. An attempt is made here to fill this gap. Consideration is given to the structure and operation of panels, contradictions in peer review and the role of the defense.



POs in general.

Most scientists have mixed feelings about the fellow I would like to put in focus to-day. It seems the appropriate occasion to do so, because I guess, that it is the character that Jaap Kistemaker will be glad to miss in the near future.
My subject is the program officer, PO, in a government research agency, also referred to as a research administrator. He administers or manages science at a distance. This reminds me of my old Leiden professor of philosophy, who visibly being of a certain religeous denomination, kept on doubting if such 'actio' could really exist. A PO is a person, who sits back at a desk in some office building, in most countries - but not in Holland - in the nation's Capital, far away from the bench, where scientists try to do their jobs. His function is to decide to which person, group or laboratory the always scarce funds for research will flow and to whom they will not. In The Netherlands for many centuries people did not believe very much in the wisdom of power vested in individuals. (Neither did they believe in Capitals, I might add.) Throughout the state and in many other sectors of society the formal decision taking is and has always been done by committees, boards, cabinets, or whatever names these bodies may carry.
John de Wit in the time of the Republic of the Seven United Provinces was considered by most foreign heads of state to be the nation's chief. He was not. He was the servant of the state, its true secretary. He had about five assistants to help him. But he was not allowed to pick and to appoint these men by himself.
Our queen, although entitled by the constitution to appoint and to dismiss cabinet ministers as she pleases, would never dare to dream of doing so.
In line with this good old tradition we have no program directors as they exist in the USA. We rather have program officers, who serve as secretaries to committees or boards and whose job it is to formulate the decisions drawn up by these bodies. Now I suppose you have all heard about Alvin Weinberg's allusion to the inaptness of committees to do any good. He referred to a camel as being a horse drawn by a committee. This he may have observed, but I dare say that his committee must have had a rather poor secretary.
Whether in realty the system, where the secretary only formulates the decisions, differs much as to the quality of the policy it produces, from the system, where the PO decides after having consulted an advisory group or panel, remains to be seen. Anyhow for a country in which it is virtually impossible to sack any crazy or senile tending official, I think the committee rule system to be safer. Let us keep it that way.

POs come in variety. Some can always cover a neighbouring field as well. Those are the empire builders and in extreme cases they may stir up a lot of trouble. On the other hand there are the seclusionists for whom anything new falls for this reason or another just outside their limits of responsibility. No troublemakers for sure, but in their extremes, herders of very dull and unimaginative science. The golden course is somewhere in between.
The selection of a good PO is of utmost importance for the well being of the specialty, he is dealing with. This I found a generally recognized concept in the USA, where the PO takes the decisions. In Holland, however, where the power is with the committee, this requirement is often overlooked, because he is 'only' the secretary and his function is 'just' to carry out, what the committee in its wisdom decides. This point of view is erroneous. It may lead to ineffectiveneness of committees, even of those that are made up from the best experts a nation has. Sometimes the flaw is compensated by a good chairman. But then the chairman must also assume the role of the secretary. In busy committees this means his farewell to the research front.
POs are a privileged set. They communicate daily with the best experts the country posesses. Their frame of reference is the nation's top. On the other hand they write the grant awarding or grant refusing letters. This dualistic role may lead to overestimation of the own position. Therefore, feelings of humbleness and a certain firmness are to be balanced carefully.
A good PO splits his time between desk work, contacts with the research field and contacts with the sources from which the money and directives come. He should be sensitive to new developments in science as well to new trends in the government's science requirements and expectations. It is his job to match the demands of the scientific world as to organizational matters with those of the financial superstructure. He mans the first bastion to shield the scientific community. He should always be alert to protect against the natural tendency of the government bureaucracy to uniformise, and regulate and to stifle the free opportunities that are necessary for a vigorous and healthy development of the research enterprise.
I want to emphasize once more the danger of underestimation the quality of the PO. Recently we have seen a rather unusual growth in the number of PO-type government appointments in different branches of the administration. In The Netherlands these positions were filled with young scientists arriving fresh from the universities. They often lack the necessary knowledge of the structure of the scientific community. And they try as soon as they can to set up their own advisory structure. This is apparantly a simple task, since most scientists are always on the look-out for new resources for their work. They are easily persuaded to serve on this or that committee, certainly if it is appointed by the minister. Such advisory panels usually suffer from - unintentional - serious omissions. Secondly the new body lacks coherence and experience, present in older, existing organizations. Therefore it is easily guided by its young secretary - who supposedly knows, what the minister wants - rather than the other way round, which is the underlying principle of an advisory group. The practice often leads to interesting looking new initiatives, which are well received by parliament. But for those, whose memory is slightly better than the stories in yesterday's newspaper, the outcome is often a disappointment.

A PO should always strive for balanced scientific advice. He should follow, not lead! The time to lead may come later, when he knows the field thoroughly. When he knows the options and the criticisms. Only then he becomes fit to help make decisions, which through lack of courage, or irresoluteness might otherwise not be taken. A PO, who starts out knowing what has to be done and how, should be treated with utmost distrust especially if he is young and if his main arguments can be read in any newspaper. In the USA I was repeatedly told that even POs who had been full professors in their fields and who became NSF program directors, were not to be regarded as specialists any longer. This is of course more true for people who have never, or have only incidentally reached the research front.



Beware for good 'administrators'

Although a PO is supposed to administer science, he should not be too good an administrator. This may sound odd, but there is a good reason for the statement. Often sience requires fast decisions and flexibility. These characteristics are absent in big administrative systems. Therefore the PO should maintain a healthy personal dislike for the regulatory disease. I want to give an example.
Consider the case of a famous professor, who applies for a grant. The application is received by the PO, who reads the accompanying letter. It states that the principal investigator, the PI, happens to have met a brilliant young physicist, especially suited for the job. Since the man is also considering other offers, please decide immediately! The PO smiles about the professor's naivety. He starts the advisory procedure. After some 6 months, he informs the PI gladly that the grant is awarded. And now the need for a not so good administrator becomes obvious. Because our PO soon finds out that a student of electrical engineering is appointed and the money allocated for instruments has to be used in part for travel to Yorktown Heights, whereas rather than solid expendables an amount of software has been ordered. In the first article published the only words that can be traced to having something to do with the research grant, are in the 'acknowledgement'.
If he were a real administrator, he would raise hell. If he is a good PO, he picks up the phone, has himself convinced by the PI that this was the best course of action and writes OK on all bills and letters of appointment sent up to him with big question marks from the personnel and finance departments. He knows:
  1. The PI is not mad;
  2. If he is, the next application with its attached progress report will surely be killed by the cleverly selected peers in his advisory apparatus.
An attempt to have the modifications approved by an inbetween advisory procedure might easily lead to a subsequent request to appoint a mathematician instead of the electrical engineer and to supplement the funds for equipment with a handsome amount, because now a full size computer is needed.
POs should develop a skill for such occurrences. They are there to cover up, to help rules and regulations adapt and in general to stimulate not to frustrate scientists. They are there to be fooled, to help to enable exceptions to be made. And they should - that is the code of honour of their trade - stand up jointly and defend an unfortunate colleague, if he is accused of arbitrariness, prejudice, or being an inapt administrator. That is, if he did so in order to help an odd bird of a scientist, who tried to do something in a different way.
After this plea, unfortunately based on practical rather than philisophical wisdom, I would like to dwell for some time on the role of committees, boards etc., which constitute the policy building for science and technology.



Committees

Before I raise doubts, let me state clearly: we need committees. And we need them not only for cosmetic reasons, but also for organization and for efficiency, but...

Let me again approach the problem in a general way. We need scientific advice and proper decisions. The first thing that comes to mind is: Let's have a committee. The aims and objects of this committee we may call the charge: c. A number of individuals preferably renowned for their skills in the particular field are approached with respect to the charge and appointed. The idea is, that the charge will unite them and focus them with neglect of all other ties, figure (a). But of course, this picture is hopelessly incorrect. In fact the charge constitutes but a minor and insignificant part of the lives of these individuals. Most of them knew each other long before the charge was there. They have their mutual ties and interests and they have numerous ties and charges with the surrounding community. So a more appropriate picture of the committee is figure (b). Here we see that our committee is almost non-existent. (Except for its full time secretary to whom the committee is a substantial part of his life.) Committee members have usually no interest to allienate their fellows, because there are so many common and long lasting interests. They will virtually forget about the charge. And they will act socially.
Think of this when you sit in on a committee. It will help you endure it. My description implies a lot, but not what the instigators had in mind. If things don't work out so well, it is usually the composition that is blamed. But that is mostly a mistake. Usually these panels encompass all the talent in a special field that the country has. In the physics research branch, FOM, for instance they include all leading specialists in that discipline. There could be no more qualified set of people to establish priorities. Actually that was not always the case. The mechanism described before, interferes. Most members of such committees will not speak out freely. The few that try, may easily become isolated; or the net result is a common resentment against that what was stated so bluntly. If a proposal is discussed, it turns out to be very important, who speaks first. If he praises it, the chances are low that this opinion will be averted. The same happens, when the first speaker shows himself critical. In the USA this phenomonon is called 'Senatorial Courtesy'. It is a well known feature in the study of human behaviour. We should not forget that, although we are dealing with scientists, we are first and foremost dealing with humans. Fortunately, I may say.
It is here, that the proper PO can play an important role. With due consideration for the impressive knowledge and skills of the top experts, he should manoeuvre, negotiate, persuade and persist in trying to draw out the best of what they allow to become known. If they say things in private that they are not willing to repeat in public, schemes and procedures should be developed to depersonalize data and opinions. Comments should be made debatable without revealing the source and so on. By doing so, it is possible to approach the essence of the committee idea: Top experts exchanging and freely forming opinions, while being totally devoted to nothing but the charge of the committee.



Mail review

Many research agencies around the world acquire peer advice by mail. Agencies send applications for review to independent advisors, who are supposed to fully understand the proposals. They are asked to comment and even to grade these documents, as if they were student's termpapers. This method is also used by FOM and STW in various programs. In many cases the reviewers are foreigners and the applications and the advice are written in English. This method came under discussion as a result of a COSPUP study by Cole, Cole and Simon, published in Science in 19812. They studied decisions made by NSF on 50 grant applications in solid state physics, 50 in chemical dynamics and 50 in economics. In order to check whether program directors predetermined their decisions by a biased selection of reviewers, they selected with the aid of a panel composed mainly of NAS-members a number of new referees. And they asked them to judge the proposals in the same way as the original reviewers were asked. The outcome was compared to that of the original advice. Their conclusion was that 25% of the projects would have received a reversed decision if other referees had been selected by the program director. In the sample half the proposals received an award, so a random choice would have produced a 50% reversal rate.
The article triggered a lively discussion. Some of it reached the public at large. Cole et al. emphasized in a later comment that half of the decisions were based on consensus in the reviewer community and half on the luck of the reviewer draw, which means that the current peer review system is 'decidedly superior to one based on random selection'3. But other critics maintained that even that conclusion is too weak. Humphreys e.g. noted that grant proposals received by NSF are far from a random selection of all potential research proposals, due to self-selection by the applicants. They restrain themselves, because they know that their proposals will be reviewed by their peers4. He argues that the 150 proposals studied must be closer to one another in quality - with all the risk of small number permutations - than a true random sample would be. We have evidence in FOM to support Humphreys' assumption. The variation in quality of the proposals in the working communities - where the program is systhematic reviewed by what the participants recognize to be their peers - is much less than in the 'Beleidsruimte', a general program, that is perceived as being closer to a random sample.



PO influence

Cole et al. disregard the role of the PO in the decision process. In a previous observation they found a good correlation between grant decisions and peer review ratings. So they used in the study mentioned here the average ratings of a peer group. A shift from below average to above average in their view means a shift from refusal to award. It may well be, that in some NSF-programs the directors favour the easy way to avoid arguments. And that they base their decisions on average ratings only. But that was certainly not the case in those NSF-programs with which I was familiar at that time. Among those I count the solid state physics program. Let me give two examples.
The research method of Cole et al. is too cumbersome. There is no need for a second group of reviewers. One could simply look at the consensus and dissension within the original reviewers group. Consider a proposal in theoretical physics. The prevailing criterion in that subfield is, as everyone knows, the quality of the people involved and - since the junior-staf is not yet known - predominantly the quality of the PI himself. In this case the proposal is sent to four reviewers. Three of them applaud the proposal on grounds of past performance and they rate it excellent. The , fourth, however, comments that his judgement (= about average) stems from the fact, that the PI is presently in the hospital and he will most likely never return to his lab. His confidence in the acting project leader is not very high. The NSF program director whom I met, had no hesistation about what to decide. Although the judgement of the four was certainly above average, the proposal was turned down.
An analogue situation may arise on other criteria. Take the following example of an experimental proposal. Some referees made serious objections to the allowance of a grant, because the small laboratory from which it originated, did not dispose of some pieces of equipment, which were thought to be essential for the conduct of the experiment. One referee, who rated the proposal 'excellent', remarked that he admired so much the clever way in which the group made use of precisely that kind of equipment, which was standing idle in a nearby industrial lab. The average rating by the peers was about average, not high enough at the time. But the program officer nevertheless and quite rightly, I think, awarded the grant. These occurrences are overlooked in the research of the Coles and Simon.
This interference by a PO on some of the decisions may completely upset the statistics on which the investigators based their conclusions. The FOM and STW archives are loaded with material to support this statement.
In the same archives one can find data about the judgement of 'juries'. That are groups of people that once assume the role of a PO. (In this way the organizations avoid vesting power in individuals, see above.) It would allow one to study the behaviour of different "POs", who are looking at the same proposal and the same peer advice. Let us hope that some science of science student at one time will reveal the wealth of empirical policy material which is readily available for harvesting. There is no need even to bother people with questionnaires or interviews.
In a brief scan of the archives of FOM and STW, we found so many examples, that we dare to state: POs (jury members) do not simply count the pros and cons in referee reports, neither do they just average referee's judgements. They do look at the contents of the advice and they pay due attention to the arguments.
We also state: The COSPUP study shows a serious deficiency by overlooking the role of the PO in the procedure. It is amazing that this aspect has not been observed. We estimate that it would have been cheap and easy to complete the experiment with a PO-simulation.



Applicant's defense

There is a fundamental difference between most NSF-review and that practised by FOM or STW. The latter two provide for an opportunity for the applicant to defend himself5. The defense is in some cases of definite importance. We discovered even some generalities there. One is: if the peer statements, apart from some general polite comments, are in majority sceptical, but not specific and if the defense by the PI is concise, clear, to the point and convincing, juries (series of pseudo "POs") tend to grade the proposal higher than when it was given to them without accompanying peer review and defense! In some cases the effect was astonishing. The proposal rose to the top in a certain set of competing proposals. It is amazing that there are still many government research agencies that do not offer the applicants this right of defense. Since Justinianus codified the Roman Law it has been good practice in all civilized countries to allow the defendend his say before a verdict is given6.



General conclusion

I hope I succeeded to convince you, that POs are important, both when they act as directors - the US case - and when they act as secretaries. More attention should be paid to their selection than is currently the case in some organizations in The Netherlands. At least a longer apprenticeship should be allowed for.



References

  1. Slightly revised version of a contribution to the seminar in honour of Prof.dr. J. Kistemaker's 65th birthday. Published in: Management of Science, relation to industrial and national needs, April 23, 1982, North Holland.
  2. S. Cole, J.R. Cole and G.A. Simon: Chance and consensus in peer review, Science 214(1981) 881.
  3. S. Cole et al., Science 215(1982)346/8.
  4. L.G. Humphreys, Science 215(1982)344.
  5. Recently I learnt that 'applicants defense' is now becoming more a part of the decision process in the US than it used to be. That is not too early, I would say. We were some 30 years ahead.
  6. C. le Pair, Physics Today (1976)13.