The term “reference material” refers to published works that you use in the preparation of your own reports, theses, etc. Ideally, if you are going to draw conclusions based on a work, it needs to report interesting information, be factually correct, and draw defensible conclusions from its facts. If the work represents new research, that research must have been carried out using appropriately rigorous techniques. In other words, the work must be valid.
The hard question is: how do you tell whether any particular work is valid? And the unfortunate answer is: in the general case, you can’t. So, we have to rely on external factors to determine validity.
The most important external factor is the degree of peer review that preceded publication of the work. Peer review is exactly what it sounds like—-review by the peers of a work’s author to determine whether it is worthy of publication. In the absence of peer review, you can rely to a certain, limited extent on the author’s and the publisher’s reputation based on previously-published work.
In all cases you need to apply your own critical faculties in judging the validity of any work you intend to rely on.
There are many types of reference material. Here are capsule descriptions of a few of the more important ones:
Journal articles. Peer-reviewed by the journal’s editorial board. This is typically a lengthy process allowing plenty of time for reflection and revision. Journals are the “gold standard” of quality references; however, some journals are considerably more rigorous than others. There is often a considerably delay (in some cases, years) between journal submission and publication.
Conference papers. Peer reviewed to some extent by the conference’s program committee. Some conferences have a peer-review process that is every bit as rigorous as the best journals (though typically faster); you need to find out which these are in your field. A good clue is the acceptance rate of submitted papers. If it’s very low (say, below one in three) the odds are good that the conference is of very high quality. On the other hand, some conferences accept virtually all submitted papers, and some conferences don’t even review the papers before acceptance. Conference papers tend to be more recent than journal articles, but are generally considered less authoritative.
Books. Sometimes peer reviewed by technical reviewers, sometimes only by an editor, and sometimes not at all. The reputation of the author(s) and of the publishing imprint are important, as is anything you can learn about the process by which the book was produced.
Doctoral dissertations and Master’s theses. Peer reviewed by the degree-granting institution and typically one external examiner. The reputation of the institution or the student’s supervisor may be useful in predicting the quality of the work, but not always. The practical issue with these works is their sheer length—typically many, many times longer than a conference paper. If no conference or journal paper resulted from the work, you should ask yourself “why?”
Technical reports.Published by a corporation or educational institution, typically with little or no peer review. Here reputation is everything. Ask the same question as for Doctoral dissertations or Master’s theses.
Random web sites. Anyone can publish anything on the web, at any time. Heck, this document was published on the web! In the general case, web sources have the same extrinsic validity as a poster stapled to a light pole downtown. You must carefully consider the source’s reputation and processes before relying on an arbitrary web site. (Note that many journals and conference papers are now published on the web. These are still journals and conference papers and should be judged as such—here I’m talking about other sites that may or may not have any level of review or oversight.)
Marketing material. Are you kidding me? It is normally written specifically to over-claim and mislead—rely on this at your own peril!
There are two general techniques for identifying reference material: search, and following the “reference graph.”
Search is generally performed using internet-based indexes. The RMC Library has subscriptions to many of these, which are listed on the library’s indexes and databases page. Two of the most useful for Electrical and Computer Engineering are IEEE Xplore and Engineering Village. For software related sources, the ACM Portal can also be very useful.
Two other excellent search engines for scholarly work are Google Scholar and Google Book Search. Google Scholar tries to index everything scholarly, and frequently includes direct links to online copies of the articles themselves. Google Book Search allows you to search the full text of an ever-growing collection of books.
Finally, don’t forget “casual search”! That’s where you ask your course instructor, supervisor, or other colleague to suggest relevant material to read.
If you can find one relevant paper, you can start tracing the reference graph. The idea is simple in principle: the paper you are reading will cite other papers, books, etc. Some of these will appear interesting, so make note of them and track them down. These papers will in turn make reference to other papers—continue the process recursively until you’ve found everything you need.
There are a few wrinkles to be aware of.
First, as described above, reference graphs always move backwards in time, since a paper can only cite works that have already been published. Except… with electronic indexing services, reference graphs can be reversed! Many indexes (including IEEE Xplore, the ACM Portal, and Google Scholar) provide “cited by” lists for papers. So, if you find a particularly relevant paper you can turn to the index and find all the other papers that cited it, moving forwards in time.
Second, reference graphs have weird properties. For example, it is actually quite common to trace a reference graph for a narrow subject until you think you have found all the relevant papers—and then to find one more paper that opens up into an entirely new region of the reference graph that you hadn’t been aware of! This typically indicates groups of scholars working in the same problem space, but unaware of (or uninterested in) one another’s existence. One example is the overlap in research areas between computer science and management information systems.
The RMC Library has electronic subscriptions to many quality journals, including the entire IEEE and ACM collections, so you can often acquire an interesting article without leaving the dubious comfort of your swivel chair. The subscription links normally require you to be on campus for access; if you’re working off campus and want access to the Library’s electronic subscriptions, log in to the College’s virtual private network service and follow the Library link from there.
If an article is not available via an RMC Library electronic subscription, you will often be able to find a loose copy on the web somewhere using Google Scholar.
Also, don’t forget that the library still maintains extensive back issue copies of paper journals, and they even have real books made of actual dead trees on their shelves!
Don’t forget the Queen’s Library. As an RMC student you have access to the full Queen’s Library collection, including electronic subscriptions (but you can’t generally access those remotely).
Finally, when all else fails, you can also request materials via inter-library loan.
When you find a potentially-useful work, the first thing to do is to make note of its bibliographic information. If it does turn out to be useful, you’re going to need to cite it some day—-and going back to find bibliographic information later is way more of a pain than it’s worth (although getting easier with wide access to online search!)
Also, make sure that your own copy of the work, whether electronic or paper, is stored in a way that you can easily find it later. For tens or dozens of papers this isn’t a big deal, but when the number gets large, the sheer volume of stuff you have on hand can make it difficult to find anything. Have a filing system, and stick to it.
Many people like to make notes about papers as they read them. Ensure that these notes are themselves well-organized, and cross-referenced to the papers they refer to.
There are many excellent electronic tools to help you in keeping track of papers, notes, etc. Once excellent open-source tool for assisting with your research is the Zotero plugin for the Firefox web browser. I’d strongly recommend taking some time to explore this tool and see if it will fit with your personal work style. Zotero comes bundled with the Firefox Campus edition or you can install it into any copy of Firefox.
This web site is not an official publication of the Royal Military College of Canada nor of the Department of National Defence. Ce site web n’est pas une publication officielle du Collége militaire royal du Canada ni du Ministère de la défense nationale.