Empirical Research Terminology

Foundational terms in empirical research:

Measurement – the process of quantifying variables.
– Its 2 parts are qualitative and quantitative.
2 options are to select:

  1. measurements which already exist and have proven reliability and validity.
    – Measures which exist can be direct or indirect.
  2. measurements which don’t exist. In this case you have to show reliability and validity.

Variables – That which is manipulated in quantitative reseach. These are described in qualitative research.

Qualitative Research – identify (fact), describe (definition) potential variables, and attempt to prove they exist (& that they have construct validity, reliability, etc.). You actually have to persuade people that this is the case, and so you’re always already engaged in a rhetorical practice, even when you’re doing empirical research.

Quantitative Research – object is to show relationship (quality) between variables–i.e. to persuade people that some [usually causal] relationship exists. This presupposes that the variables exist, which means that you have to 1st qualitatively show this to be the case.

Methods of evaluating composition:
(Critical question is always are these direct or indirect measures?  IOW, do these methods measure the student…or the rater…? Answer to this will tell whether they’re direct or indirect measures)

– Holistic Evaluation – Give one score based on overall impression. No factorial breakdown. [low inter-rater reliability]
– Analytic Evaluation – Break down based on categories or variables
– Primary Traits – Put the object into  categories based on how well it fits into the description of that category. Will end up with several scores. (e.g. the Eng 103 descriptive grading rubric)

Types of Data:
(Type must match statistical measure to ensure validity.)

  • Nominal – Classifications which can be named.
  • Ordinal – Rank ordering. Not equidistant between points. [e.g. A, B, C, D, F…Likert scales]
  • Interval – Has equal distance between variables. [more powerful statistics are associated with these]
  • Ratio – Interval data with an absolute zero. Almost never crops up with human subjects.
    Z-Score – lets you normalize across a large population. But you have to exclude to normalize. (measure of reliability)

Reliability – measure of agreement or disagreement between raters or instruments (r = -1 – +1). Inter-rater reliability should approach +0.7 to have predictive power that what you’re describing is actually there.
(Out for a nightmarish situation: if your research is predicated on r=0.7, and you don’t achieve this, you can argue that r is a social construct, which is inherently predicated on truth by consensus, and is therefore not really a “scientific” or precise measure.)
3 types of reliability:
– Equivalency – you’re triangulating with multiple instruments and all of the  instruments are giving the same results.
Stability – do the instruments/people change over time? If yes, that’s a reliability issue.
Internal – Consistency of instruments/people. Granularity and scale.

Validity – measuring what you say you’re measuring. Reliability is a necessary condition for validity, but it’s not sufficient. You can have construct validity, without being able to measure it reliably. Likewise, you can measure something reliably without it having construct validity.

What is the difference between reliability and validity? You can reliably measure something you did not intend to measure in your study, or around which your study does not hinge.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: