...

19Segment - Ganesha Associates

by user

on
Category:

science

10

views

Report

Comments

Transcript

19Segment - Ganesha Associates
Basic reading, writing and
informatics skills for biomedical
research
Segment 9. Research
assessment trends
18 April 2008
Copyright Ganesha Associates
2008
1
The need for assessment in science
• “Afinal das contas é tudo uma questão de
transformar conhecimento em riqueza e bemestar para a população.”
• “When I was in graduate school I thought I would
go into a room and people would slide money
through a slot in the door, I would do science,
and then slide the results back out. But as my
career has gone on, they are increasingly
opening the door and asking me what I am
doing with their money.”
18 April 2008
Copyright Ganesha Associates
2008
2
O Cientista Brasileiro é avaliado ?!
– À primeira vista, o papel do cientista parece ser tão-somente expandir
as fronteiras do conhecimento humano. Porém, o contrato social da
ciência prevê que a informação gerada pelo cientista deva ter por fim
modificar a realidade para a qual ele produziu a informação. Mas isso
nem sempre ocorre. E a avaliação é um processo que possibilita à
sociedade e aos gestores de pesquisa meios de aferir a real efetividade
do trabalho do cientista.
– O cientista, também, está sempre solicitando mais verbas ao governo
para publicar cada vez mais informação. E a avaliação, novamente, é
uma importante ferramenta para que se concedam verbas aos mais
produtivos e inventivos.
– Assim, a avaliação do cientista deve considerar o cotidiano do
pesquisador, respeitando as demandas da sociedade, que nem
sempre são apenas publicar artigos e livros que só interessarão a
outros cientistas.
– Atualmente, o desempenho dos cientistas é avaliado através das
citações por eles obtidas, por os artigos publicados só nas revistas
indexadas no Science Citation Index (SCI), ou Medline ou Scielo.
–
Taken from Pedrini, Alexandre de Gusmão, 2005
18 April 2008
Copyright Ganesha Associates
2008
3
Como o/a cientista é avaliado ?!
–
–
–
–
–
–
–
–
National programs for scientific research have targets
National wealth and well-being
‘Basic science with a purpose’
Getting funding is a highly competitive process
The scientist has an obligation to deliver results
We need management tools in science.
The quality of scientific output is one measure
Methods of scientific assessment include the analysis
of citation statistics
18 April 2008
Copyright Ganesha Associates
2008
4
Brazilian scientific output – molecular
biology and biochemistry
18 April 2008
Copyright Ganesha Associates
2008
5
18 April 2008
Copyright Ganesha Associates
2008
6
The quality of (inter)national scientific
research is at the heart of the problem
Data from a CAPES report 2005
18 April 2008
Copyright Ganesha Associates
2008
7
18 April 2008
Copyright Ganesha Associates
2008
8
18 April 2008
Copyright Ganesha Associates
2008
9
Qualis e a classificação de periódicos,
anais, revistas e journais
•
Qualis é o resultado do processo de classificação dos veículos utilizados
pelos programas de pós-graduação para a divulgação da produção
intelectual de seus docentes e alunos
•
Tal processo foi concebido pela CAPES para atender as necessidades
específicas do sistema de avaliação e baseia-se nas informações
fornecidas pelos programas e pelo Coleta de Dados
•
A classificação é feita ou coordenada pelo representante de cada área e
passa por processo anual de atualização.
• Os veículos de divulgação citados pelos programas de pósgraduação são enquadrados em categorias indicativas da
qualidade - A, B ou C e do âmbito de circulação dos mesmos local, nacional ou internacional
•
As combinações dessas categorias compõem nove alternativas indicativas
da importância do veículo utilizado, e, por inferência, do próprio trabalho
divulgado.
18 April 2008
Copyright Ganesha Associates
2008
10
18 April 2008
Copyright Ganesha Associates
2008
11
CAPES – scientific subdivisions
– Ciências Biológicas I: Biologia Geral, Botânica,
Genética, Oceanografia Biológica e Zoologia
– Ciências Biológicas II: Biofísica, Biologia Molecular,
Bioquímica, Farmacologia, Fisiologia, Morfologia,
Anatomia, Histologia e Neurociências
– Ciências Biológicas III: Microbiologia, Imunologia e
Parasitologia
– Saúde Coletiva
– Ecologia e Meio Ambiente
18 April 2008
Copyright Ganesha Associates
2008
12
Critérios de Classificação do Qualis
• Ciências Biológicas III: Microbiologia, Imunologia e
Parasitologia
• QUALIS A - Trabalhos publicados em revista com fator
de impacto (F1) igual ou maior que 1,945
• QUALIS B - Trabalhos publicados em manuscrito com
F1 entre 1 e 1,945 trabalhos publicados nas Memórias
do Instituto Oswaldo Cruz e no Brasilian Journal of
Medical and Biological Research
• QUALIS C - trabalhos publicados em revistas nacionais
ou internacionais com F1 menor que 1.
18 April 2008
Copyright Ganesha Associates
2008
13
Define: indexação
• A indexação de uma revista é o processo que confere o
indicador de qualidade necessário para que esta possa
pertencer a um ou mais bancos de dados de renome
internacional, em uma área específica do conhecimento,
como por exemplo, MEDLINE, ISI, LILACS, etc.,
segundo critérios preestabelecidos por essas
instituições.
• É um fator importante para que a produção científica
nela apresentada se torne conhecida e reconhecida
internacionalmente.
• O periódico necessita, para isto, estar de acordo com os
padrões básicos de apresentação formal e excelência
de conteúdo científico, determinados
internacionalmente.
18 April 2008
Copyright Ganesha Associates
2008
14
As fontes de ‘indexação’ em ordem de
‘qualidade’
•
•
•
•
•
•
•
•
Google
Scirus
LILACS
ScieLo
PubMed/Medline
Scopus
Thomson Web of Science
E muitos outros mais especializados
18 April 2008
Copyright Ganesha Associates
2008
15
18 April 2008
Copyright Ganesha Associates
2008
16
Journal selection process - Medline
• Scope and coverage: The journals brought to the Committee for
review will contain articles predominantly on core biomedical
subjects.
– Journals whose content is predominantly a subject peripheral or related
to biomedicine are occasionally brought to the Committee when they
have some biomedical content.
– In these cases, the Committee's advice is sought not only on the quality
of the content but also on the contribution it makes to the coverage of
the subjects in question. Generally, such journals will not be indexed if
their biomedical content is already adequately covered.
• Quality of content: Scientific merit of a journal's content is the
primary consideration in selecting journals for indexing. The validity,
importance, originality, and contribution to the coverage of the field
of the overall contents of each title are the key factors considered in
recommending a title for indexing, whatever the intended purpose
and audience.
18 April 2008
Copyright Ganesha Associates
2008
18
18 April 2008
Copyright Ganesha Associates
2008
22
18 April 2008
Copyright Ganesha Associates
2008
23
The importance of ISI Impact Factors
– What is a citation ?
• A citation for an article in a journal includes the author, title of
article, title of periodical (also referred to as the source),
volume and issue number, page number(s), and date of
publication.
• A single citation is called a reference; a group of citations is a
bibliography.
– How are Impact Factors defined ?
• It is calculated each year by the commercial publisher
Thomson for those journals which it tracks, and are published
in the Journal Citation Report.Impact
• Impact Factors have a huge, but controversial, influence on
the way published scientific research is perceived and
evaluated.
• Originally intended to provide librarians with a relative index
of importance for journals in a particular field in order to
assist collection management
18 April 2008
Copyright Ganesha Associates
2008
24
Impact Factor defined
Impact Factor 2008 =
# citations received in 2008 to articles published in 2007/2006
______________________________________________________
# number of source items published 2007 and 2006
“The Impact Factor of a journal is probably the only quantitative way of
assessing its worth and relevance to the academic community it serves.”
Editorial, Journal of Computer-assisted Learning, 2001
18 April 2008
Copyright Ganesha Associates
2008
25
Alternatives ?
• The Hirsch Index is rapidly gaining
favor as a performance
measurement tool for scientific
authors.
• A scientist has index h if h of his/her
Np papers have at least h citations
each, and the other (Np - h) papers
have no more than h citations
•
•
•
Thus, the h-index reflects both the
number of publications and the number
of citations per publication. The index is
designed to improve upon simpler
measures such as the total number of
citations or publications.
Like the IF, the index works properly
only for comparing scientists working in
the same field.
18more
April 2008
Copyright Ganesha Associates
For
information see Scopus or Wikipedia
2008
26
H-index - weaknesses
• The h-index is bounded by the total number of
publications.
• This means that scientists with a short career
are at an inherent disadvantage, regardless of
the importance of their discoveries.
• For example, had Albert Einstein died in early
1906, his h-index would be stuck at 4 or 5,
despite his being widely acknowledged as one
of the most important physicists, even
considering only his publications to that date.
18 April 2008
Copyright Ganesha Associates
2008
27
Impact Factors and Authors
Correlation between
article citation rate and
journal impact for four
authors
18 April 2008
Copyright Ganesha Associates
2008
28
Impact Factors and Journals
18 April 2008
Citation rates in 1986 or
1987 of articles published
in three biochemical
journals in 1983 or 1984,
respectively. From
Seglen, BMJ
Copyright Ganesha Associates 1997;314:497
29
2008
Impact Factors over time
18 April 2008
Copyright Ganesha Associates
2008
30
Impact Factor and area
18 April 2008
Copyright Ganesha Associates
2008
31
Impact Factors for Microbiology
18 April 2008
Copyright Ganesha Associates
2008
32
Impact Factors for Neuroscience
18 April 2008
Copyright Ganesha Associates
2008
33
Impact Factors and article type
18 April 2008
Copyright Ganesha Associates
2008
34
Impact Factor - Summary points
• Use of journal impact factors conceals the
difference in article citation rates
• Journals' impact factors are determined by
technicalities unrelated to the scientific quality of
their articles
• Journal impact factors depend on the research
field: high impact factors are likely in journals
covering large areas of basic research with a
rapidly expanding but short lived literature that
use many references per article
• Article citation rates determine the journal impact
factor, not vice versa
18 April 2008
Copyright Ganesha Associates
2008
35
The emergence of Open Access
Clockwise: Harold Varmus, Michael Eisen,
18 April
2008and David LipmanCopyright Ganesha Associates
Pat
Brown
2008
36
Scientific publishing is a very profitable
commercial business
18 April 2008
Copyright Ganesha Associates
2008
37
Even when it is not-for-profit
18 April 2008
Copyright Ganesha Associates
2008
38
Define: business model
• A business model (also called a business
design) is the mechanism by which a
business intends to generate revenue and
profits. It is a summary of how a company
plans to serve its customers. It involves
both strategy and implementation
• Wikipedia
18 April 2008
Copyright Ganesha Associates
2008
39
Some fundamental problems with
the ‘print’ business model
• Exponential growth in the amount of data
being published
• Subscription charge increases: 200% in last
ten years - pressure on library budgets,
reduction in number of subscriptions
• Publisher retention of copyright and
therefore control
• “Excessive” profits: 35%+ margins
• Restrictive online access - the bundle deal
18 April 2008
Copyright Ganesha Associates
2008
40
Key features of the ‘open access’
business model
•
•
•
•
•
Free, immediate access online
Unrestricted use
Author retains copyright
Papers are deposited in a public online archive
Author pays for publication
18 April 2008
Copyright Ganesha Associates
2008
41
Advantages of Open Access
• Maximum impact for authors
access to the largest possible audience
• New ways to access and use literature
full-text searching and mining
• Greatly expanded access to research
for scientists, educators, physicians, the public
18 April 2008
Copyright Ganesha Associates
2008
42
Old ‘print’ subscription
business model
Researcher
€
€
Gov
Funders
Institutions
Industry
Publisher
€
Pay-per-view
€
€
Library
Subscription
Reader
18 April 2008
Copyright Ganesha Associates
2008
43
New open access business
model
€
Researcher
Gov
Funders
Institutions
Industry
€
Publisher
Public
Digital
Library
Reader
18 April 2008
Copyright Ganesha Associates
2008
44
Open access a major force in Brasil
18 April 2008
Copyright Ganesha Associates
2008
45
SciELO’s role in supporting OA
18 April 2008
Copyright Ganesha Associates
2008
46
Consequences of Open Access
• A fundamental change in the business model
underlying the scientific publishing industry
• Revenue: from subscription to fee-for-service
• Payer: was ‘the librarian’, now the research
funder, usually the government, ultimately the
tax-payer
• Marketing focus: was the librarian, now the
researcher (author/reader)
• Future business driver: accountability to the tax
payer
18 April 2008
Copyright Ganesha Associates
2008
47
Some myths about Open Access
Urban myth 1: “ISI doesn’t cover Open Access journals”
•
ISI (Thomson Scientific) indexes around 200 Open Access journals (OAJs) for its Web of Science service. It
applies the same criteria for inclusion in that index to Open Access journals as it does to subscription-based titles.
Urban myth 2: “Open Access journals are not good quality”
•
By dint of being included in the Web of Science index, around 200 Open Access journals enjoy the prestige of
having an official Impact Factor, thus denoting some measure of quality to these titles. Moreover, some OAJs
enjoy particularly high impact factors: the most notable in this respect, PLoS Biology, romped straight to the top
position in the General Biology category the first year it qualified for such a calculation (journals must have been
published for at least two years before an impact factor can be calculated).
Urban myth 3: “Paying a publication fee to Open Access journals
corrupts peer review”
•
This would be very significant and deeply damaging to the integrity of the scholarly communication system were it
true. It is not. Open Access journals that charge an article-processing fee have a firewall between the peer review
process and the charge-imposition system, ensuring that authors cannot pay to subvert the peer review procedure
in a manner akin to vanity publishing.
Urban myth 4: “Open Access journal business models are unsustainable”
•
Well, they may yet prove to be, but time is passing and we are seeing a move towards Open Access publishing
not away from it, suggesting that some publishers at least are finding ways to sustain their publications even while
making their content free online..
•
Alma Swan, American Scientist, 2006
18 April 2008
Copyright Ganesha Associates
2008
48
What is Open Access not ?
• A movement – then who is the leader ?
• Free – electronic publishing, whatever the business
model, is expensive
• Robust – benefits are being mis-sold to the main payer,
and there is a real risk of loss of momentum
• A big issue for the majority of researchers “…researchers do not consider that problems in
accessing the journal literature are a significant barrier to
their work…” Publishing Research Consortium 2006
• Good news for national publishers in countries such as
Brasil
• Just about documents and open archiving
18 April 2008
Copyright Ganesha Associates
2008
49
Benefits of Open Access
• Access to full text. Better indexation and retrieval.
• Better integration between full text OA resources and other public
archives of scientific data
• The potential for the emergence of new services based upon open
access content or overlaying open access content (e.g. peer review
services, text- and data-mining tools, bibliometrics and webometrics
for better research evaluation, etc
• The potential for much wider access than the subscription publishing
system gives for doctors/nurses, teachers/students, small biotech
companies, etc
• Redistribution of publishing costs to richer nations
•18 April
Adapted
of Enhanced
to Research Findings by John Houghton
2008 from The Economic Impact
Copyright
GaneshaAccess
Associates
50
& Peter Sheehan, Centre for Strategic Economic Studies, Victoria University, Melbourne, 2006
2008
• Practical activity 9a - Research assessment
• Total duration ca. 30 minutes.
• Obtain Impact Factors for your top 5 journals and add
them to your Excel spreadsheet. Send them as an
attachment to the course organiser.
• Practical activity 9b - Why research is often wrong.
• Total duration ca. 2 hours.
• Read the New York Times article about the problems of
making strong associations between diet and disease.
• Now read the Lancet Editorial about the drug Aprotinin.
You may need to read some of the more recent literature
on aprotinin to understand the full relevance of this
second paper.
18 April 2008
Copyright Ganesha Associates
2008
57
Fly UP