I thought I would share a conversation I had this week with the kind folks at EDITS. The bottom line being that aptitudes themselves don't increase significantly with training. Rather, a person may underperform when weak literacy or numeracy skills get in the way of assessing their true ability levels. As always, feel free to comment!
Good afternoon,
I am hoping you can address this question regarding the impact of training on aptitude scores. I am a vocational evaluator based in Toronto and a colleague of mine recently asked a pertinent question as follow:
Are there any guidelines for increasing aptitudes with training? My colleague reportedly received training suggesting that suggested that all aptitudes can be increased by only 1 level (with appropriate training), and that V, N, S cannot be increased above the G.
I provide below my thoughts on this and was told there is a Dr. Lee at your office that might have more insight:
This is a complex question without a straightforward answer. Aptitudes, as I understand them, represent innate abilities that are generally stable over time, with only minimal impact from training. However, training and practice can indeed improve performance, particularly in specific areas like math or language.
Aptitude tests, such as the GATB, CAPS, CareerScope, or CFIT, typically focus on speed and accuracy rather than a broad knowledge base. For instance, Numerical aptitude tests often consist of arithmetic items and don't include higher-level math questions.
I believe that training might lead to noticeable improvements in aptitude testing at lower levels (e.g., from low to low average or to an average level). However, the impact of training is likely negligible at higher levels (e.g., from high average to high).
While I don't have specific research or literature to reference, I think it would be reasonable to consider raising a person's aptitude score if they successfully undergo related training, but I would caution against raising it beyond the average level.
I should add that I am the person in charge of professional development with CAVEWAS, the Canadian association of vocational evaluators. Your input would be of help to multiple people. Thank you in advance for your help.
Sincerely,
Francois Paradis, M.A., CVE, CCVE, ICVE
Francois,
Thanks for reaching out! My name is Andrew Tricarico and I have discussed this with Lisa, my direct supervisor. I have a master's degree in industrial-organizational psychology (where I accumulated a solid amount of testing and measurement expertise), so we both can offer some insight.
My background is in Industrial Psychology, and we typically classify job-related attributes as "knowledge," "skills," "abilities," or "other characteristics." Below are some relevant definitions as I understand them:
-
Knowledge is an attribute that was purely developed through learning (biology, physics, the English language, etc.).
-
A skill is an attribute that improves with practice (persuasion, judgment and decision making, time management, etc.).
-
An ability is a more innate quality that can't as easily be practiced (physical flexibility, deductive reasoning, reaction time, etc.).
-
Aptitude (in terms of psychology) is closest in definition to ability, but specifically concerns one's potential.
While I am not intimately familiar with the GATB, I was able to find some literature that discusses its origin and structure which may have direct bearing on the training that your colleague participated in. Scores on V and S certainly couldn't be higher than G, as V is measured by Test 4 and S is measured by Test 3. G is simply a composite score of Tests 3, 4, and 6. N is measured by Tests 2 and 6, so someone could potentially score very high on Test 2 and surpass G, but it is probably a rare occurrence (and could be impossible depending on how the scores are calculated). Perhaps some math was done in the test development process to ensure that neither V, N, nor S could not surpass the G composite score, but I do not have access to a manual.
I agree that the question of aptitude versus ability and the possibility of improving scores is a very complex question. From my perspective, I think your definition is closest to "ability." Though again, aptitude and ability are quite similar. I browsed a couple abstracts, and did find a meta-analysis discussing how education can increase general mental ability (Richie & Tucker, 2018). Kulik et al. (1984) specifically focused on participants taking practice versions of aptitude tests, finding that ability moderated the effect of practice tests on aptitude test performance. Specifically, those with high ability scores experienced higher score gains, opposite to what is proposed in your email. Of course, even if it is a meta-analysis, it's an older one and I have not read enough to offer a thorough critique. Additionally, a method of training that isn't general schooling or a practice test certainly could moderate the relationship differently… I would need to compile a true literature review to know for sure. Also yes, numerical aptitude assessments are typically focused on speed and accuracy regarding lower-level problems, because using higher-level math problems would start to lean the assessment towards a measure of high-level math knowledge rather than general math-related aptitude.
I don't understand what was meant by "it would be reasonable to consider raising a person's aptitude score, (but) would caution raising it beyond the average level," though. To me, the wording implies modifying scores on an individual basis after an assessment has been taken, which should never happen (unless you're deleting their data entirely because you know they cheated, didn't finish the assessment, or something along those lines). The only way to determine if one's aptitude (as measured by a particular assessment) has been modified is to have them take the same assessment again. Practice effects can influence the validity of these assessments for sure (particularly if assessment-specific training is received), but editing results is a guaranteed threat to validity. The CAPS (and most other aptitude assessments), are designed to take a snapshot of a participant's aptitude at a specific point in time.
A participants' score is just what the assessment gives you- it's your *interpretation* of that score that should change. If you know the participant has undergone training (and especially if that training is specific to the content of a particular test), you also know that their results *might* be a little less valid. On the other hand, especially if the previous administration of the assessment and subsequent training had been years prior, perhaps the training did manage to increase one's aptitude (or, simply, more experience was gained in general).
In short, if you're leveraging a validated psychometric instrument like the GATB or the CAPS, one shouldn't ever consider raising (or lowering) scores due to outside factors because they were not designed for post-hoc modifications. If someone who scored far below average later scored high above average on an aptitude assessment, I'd certainly look into it, but I would always be using the scores the assessment presented as the basis of my interpretation.
Overall, test batteries such as the CAPS offer a fairly accurate measure, at this point in time, of a client's ability levels. Test scores can improve through additional training, but we have no way to tell just how much improvement can be achieved.
I hope I have addressed your question! Regardless, I am happy to discuss this further over the phone or through email.
References:
Kulik, J. A., Kulik, C.-L. C., & Bangert, R. L. (1984). Effects of Practice on Aptitude and Achievement Test Scores.
American Educational Research Journal,
21(2), 435–447.
https://doi.org/10.2307/1162453
Sincerely,
Andrew
_________________________
EdITS LLC
Educational & Industrial Testing Service
Good morning Andrew and Lisa and thank you Andrew for your detailed response. I am including one of my colleague, Laura, who initiated this discussion. Laura, please feel free to jump in on this conversation. I think it is an important one and I wish our forum members to benefit from it.
Regarding my previous statement ("it would be reasonable to consider raising a person's aptitude score, (but) would caution raising it beyond the average level," ), it was not meant to say that we would raise a person's test score. Rather, we may recommend an occupation one level above a person's scores if we have reasonable expectations those scores are an under-representation of their true ability levels due to literacy/numeracy issues. This does not mean that person has increased those aptitudes but rather, we are getting a more accurate measurement of their true aptitude levels by removing said literacy/numeracy barriers.
Let's imagine for a moment a typical scenario to illustrate this: an immigrant with a bachelor's degree in engineering and previous work experience as a mechanical engineer comes to Canada and wishes to get certified as a mechanical engineering technician. He now has to complete 3 years of college studies in engineering technology to qualify for a license here. Let's say this man's English literacy skills are at about a grade 8 equivalent. Let's also assume we gave him an aptitude test and that he achieved a low average level of verbal aptitude and an average level of general learning ability. We know, from the occupational profile of a mechanical engineering technician, that a high average level of general learning ability and an average level of verbal aptitude are required. Can we say that this person is likely to fail in becoming an engineering technician in Canada? I think not. Rather, we can expect that his true aptitude levels are higher than demonstrated on testing, given his background. His scores are depressed by his lower literacy skills. In this scenario, we are not raising his aptitude scores but assume those are under-representations of his true ability levels. We may recommend a career requiring one aptitude level above his current scores, with the assumption that he successfully complete academic upgrading to bring up his literacy level to a point where he can succeed in an engineering technology program.
If we were to test this person again after completion of his training, it would be reasonable to expect that his G and V aptitude scores have increased to the required level for his occupation. So, to be clear, we did not raise his initial test scores but made an assumption that his true ability levels would be revealed once the literacy barrier is removed.
In summary, I would advise against recommending occupations above a person's aptitude scores, unless there are specific circumstances and remediations to support it.
Thank you for your time and insights!
Sincerely,
Francois Paradis, M.A., CVE, CCVE, ICVE
Francois,
I totally understand what you're getting at now! I was looking at it from the wrong angle- test bias/type II error obfuscating a true score is a massive issue.
In short, Lisa and I 100% agree with you guys here. We recommend that our CAPS examinees draw a line to the stanine above and below their given score, creating a crude "confidence interval" that is probably more representative of their true ability. Of course, language difficulties (affecting the non-language aspects of the assessment) may contaminate an observed score more than "typical error" would.
This all emphasizes the importance of making an effort to ensure that assessments are, at the very least, culture-fair. It also certainly helps to get assessments back-translated and available in various languages so these threats to validity can slowly be stomped out (but of course, doing so is quite expensive).
Regards,
Andrew Tricarico
_________________________
EdITS LLC
Educational & Industrial Testing Service
Original Message:
Sent: 08-25-2024 12:34
From: Scott T. Stipe
Subject: Aptitude Increase with Training
Assuming substantial additional training/education, I too would expect stability in aptitudes S P. and as you say, maybe slight decline in Q and motor skills K F M, but probably not GVN. Q is iffy. I would expect much higher GVN if academic deficits were addressed over time. V and N are markedly impacted by academics. If Joe hated school, never could read or do math too well and dropped out to work construction, his GVN quite likely would not have been so hot at 20 or 30, or 40. We often don't really know but sometimes see it in ASVAB scores. But if he gets additional training (in what V and N are all about) such training improves those aptitude scores
------------------------------
Scott T. Stipe, MA, CRC, CDMS, IPEC, D/ABVE
Certified Rehabilitation Counselor
Board Certified Vocational Expert
Scott Stipe & Associates, Inc.
DBA Career Directions Northwest
4110 SE Hawthorne Blvd
#188
Portland, Oregon, 97214
(503)807-2668
email: sstipecrc@gmail.com
Original Message:
Sent: 08-23-2024 16:21
From: Jeffrey Truthan
Subject: Aptitude Increase with Training
Mid-life testing I would expect more stability in aptitudes G V N S P. Maybe slight decline in Q and motor skills K F M E.
------------------------------
Jeff Truthan, MS-Rehabilitation Counseling, CVE
President - SkillTRAN LLC - https://skilltran.com
Spokane Valley, WA 99206
(800) 827-2182 (Voice & Fax)
Original Message:
Sent: 08-23-2024 14:38
From: Scott Stipe
Subject: Aptitude Increase with Training
No implication Joe was that young. He had simply dropped out of school following 9th grade. Very common scenario in WC and other injury cases with blue collar workers. Let's say he was 45-ish at time he took first aptitude test. With all other events same and with his further education, what, would none of that much matter and with another go round of aptitude testing half a dozen years later would GVN merely mirror those before?
------------------------------
Scott T. Stipe, MA, CRC, CDMS, IPEC, D/ABVE
Certified Rehabilitation Counselor
Board Certified Vocational Expert
Scott Stipe & Associates, Inc.
DBA Career Directions Northwest
4110 SE Hawthorne Blvd
#188
Portland, Oregon, 97214
(503)807-2668
email: sstipecrc@gmail.com
Original Message:
Sent: 08-23-2024 12:23
From: Jeffrey Truthan
Subject: Aptitude Increase with Training
Assuming Joe is a younger individual, implied by 9th grade education and aptitude testing back in 2018. Failed completion of high school in the typical age range implies any of a wide variety of issues likely occurring during that time that may have suppressed performance on the aptitude testing as well as completion of HS in traditional timing.
Aptitudes and interests generally stabilize by the mid-twenties. Subsequent completion of high school via GED and interest and engagement in community college training are terrific accomplishments for Joe.
Subsequent testing may well reveal an upward bump in G, V, N scores partly from additional education but likely due to better attention span and motivation at an older age. If achievement testing would be done, would certainly expect solid gains here as well.
------------------------------
Jeff Truthan, MS-Rehabilitation Counseling, CVE
President - SkillTRAN LLC - https://skilltran.com
Spokane Valley, WA 99206
(800) 827-2182 (Voice & Fax)
Original Message:
Sent: 08-23-2024 11:35
From: Scott Stipe
Subject: Aptitude Increase with Training
Agree, however, a hypothetical:
Assume Joe had 9th grade education and was administered aptitude testing back in 2018. G, V, N scores were below average, level 4.
Then assume that in 2024 he completed significant GED preparation courses primarily in math, language, reading.
Assume Joe successfully completed GED in late 2023.
Assume Joe has been taking prerequisite math, English and other courses in 2024 in order to enter a community college engineering technology program and doing well.
Assume that Joe qualifies for a program which will require that he take aptitude testing in late 2024.
Would you expect that Joe's G, V, and N scores in late 2024 will be the same as they were in 2018?
Original Message:
Sent: 8/23/2024 11:00:00 AM
From: Jeffrey Truthan
Subject: RE: Aptitude Increase with Training
I have always thought the following:
Aptitude as the potential to learn/perform
GED-RML as a measure of academic training
SVP as the demonstration of proficiency through learning and/or work experience.
------------------------------
Jeff Truthan, MS-Rehabilitation Counseling, CVE
President - SkillTRAN LLC - https://skilltran.com
Spokane Valley, WA 99206
(800) 827-2182 (Voice & Fax)
Original Message:
Sent: 08-23-2024 01:58
From: Scott Stipe
Subject: Aptitude Increase with Training
Bad decision to abandon GATB. But in reference to the question: If G is essentially similar to IQ (and IQ is essentially, as some say, little more than the ability to do well in school) and G is made up of V, N and S, and all of these are very obviously developed and practiced and burnished in school, work OJT (training), I do not think it is as simple as you make it out to be.
------------------------------
Scott T. Stipe, MA, CRC, CDMS, IPEC, D/ABVE
Certified Rehabilitation Counselor
Board Certified Vocational Expert
Scott Stipe & Associates, Inc.
DBA Career Directions Northwest
4110 SE Hawthorne Blvd
#188
Portland, Oregon, 97214
(503)807-2668
email: sstipecrc@gmail.com
Original Message:
Sent: 08-22-2024 11:45
From: Francois Paradis
Subject: Aptitude Increase with Training
Thank you. Your comments go along my thinking that training would likely have little impact on aptitude test scores, except perhaps at lower aptitude levels, as previously mentioned. I hope to get some feedback from EDITS.
------------------------------
Francois Paradis
Certified Vocational Evaluator
francois@career-options.ca
Toronto, ON Canada
Original Message:
Sent: 08-22-2024 09:34
From: Renee B. Jubrey
Subject: Aptitude Increase with Training
Jeffrey,
I love Gale's short, but sweet, and oh so true, explanation for aptitudes. As OG CVEs, you & I have probably given hundreds of thousands of aptitude tests (gotta love that GATB) and know that some people have a knack for verbal, some numerical, etc. Explaining that to the client was always rewarding as we narrowed vocational options. Thanks for your input!!
Renee B. Jubrey, MS
Certified Vocational Evaluator
Diplomate/American Board of Vocational Experts
RBJ Vocational Experts LLC
Tele (860) 707-2220
Fax (860) 651-7750
Original Message:
Sent: 8/22/2024 9:24:00 AM
From: Jeffrey Truthan
Subject: RE: Aptitude Increase with Training
Francois ... here is a great definition from the 1991 Revised Handbook for Analyzing Jobs defining many nuances of what is our traditional understanding of the concept of Aptitudes. https://skilltran.com/rhaj/rhaj9.pdf#page=4
I recall a statement from the late, great Gale Gibson - Founder of Vertek, Inc. and its various software products which are now a part of SkillTRAN. His simple phrase was "Skills are aptitudes realized." I love that simple statement because it goes to the heart of it ... it is really difficult to master any kind of skill, however acquired, if you don't have the "knack" for it ... the potential (aptitude) to learn it.
We are all created equal before a far greater authority. But we each have been gifted differently.
------------------------------
Jeff Truthan, MS-Rehabilitation Counseling, CVE
President - SkillTRAN LLC - https://skilltran.com
Spokane Valley, WA 99206
(800) 827-2182 (Voice & Fax)
Original Message:
Sent: 08-21-2024 19:21
From: Francois Paradis
Subject: Aptitude Increase with Training
Thank you. I have contacted Dr. Lee.
Sincerely,
Original Message:
Sent: 8/21/2024 6:00:00 PM
From: Michelle McBroom Weiss
Subject: RE: Aptitude Increase with Training
Great idea! Dr. Lee is always so responsive and willing to answer questions. You can reach her at lisalee@edits.net.
Edits is an exhibitor at the IARP Fall Forum and IALCP Symposium. For those attending be sure to stop by, say hello, and ask questions.
Best regards,
Michelle McBroom Weiss, MA, CRC, CCM, NCC, MSCC, ABVE/D, IPEC
5543 Edmondson Pike, Suite 128
Nashville, TN 37211
mcbroomweiss@mcbroomweiss.com
(P) 615-834-0186
(F) 615-831-5274
(C) 615-308-6395
Original Message:
Sent: 8/21/2024 3:22:00 PM
From: Cindy L. Fisher MA CRC ABVED IPEC PV
Subject: RE: Aptitude Increase with Training
Hello,
I would speak with Doctor Lee at EDITS publishing, San Diego, CA as she has much research on abilities testing and the EDITS test she created.
Original Message:
Sent: 8/21/2024 3:15:00 PM
From: Francois Paradis
Subject: Aptitude Increase with Training
Good afternoon,
I was recently asked by a colleague about guidelines for increasing aptitudes through training. They were informed that all aptitudes could be increased by one level with appropriate training and that aptitudes V, N, and S cannot increase above the G aptitude level. I am exploring this topic further and would appreciate your insights and any resources you might recommend.
My understanding is that aptitudes represent innate abilities and are generally stable over time, with minimal impact from training. However, I recognize that training and practice can improve performance, especially in specific areas like math or language. I suspect that such improvements may be more noticeable at lower levels of aptitude (e.g., from low to low average), while the impact might be negligible at higher levels (e.g., from high average to high).
What are your thoughts on this? Have you encountered any research or practical experiences that address this question?
Thank you.
------------------------------
Francois Paradis
Certified Vocational Evaluator
francois@career-options.ca
Toronto, ON Canada
------------------------------