Consistencia intra e inter evaluador de la comparación pareada en la evaluación de la competencia de innovación de estudiantes universitarios

  1. Marin-Garcia, Juan A. 1
  2. Aragonés Beltrán, Pablo
  3. García Melón, Mónica
  1. 1 ROGLE - Departamento de Organización de Empresas - Universitat Politècnica de València
Zeitschrift:
WPOM

ISSN: 1989-9068

Datum der Publikation: 2014

Ausgabe: 5

Nummer: 2

Seiten: 24-46

Art: Artikel

DOI: 10.4995/WPOM.V5I2.3220 DIALNET GOOGLE SCHOLAR lock_openDialnet editor

Andere Publikationen in: WPOM

Zusammenfassung

(Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students)The aim of this paper is to propose a critical example to assess student competencies using the paired comparison as multiple criteria decision making tool based on analytic hierarchy process (AHP). We apply it to innovation competency in a subject with few students (10 master's students, divided into two groups). With the proposed methodology it is possible to create an ordered list for each group, having a distance between students (in one dimension) that reflects the degree of learning (knowledge, skills and attitudes). We found that the scores given by an evaluator for a student are consistent with the rest of the evaluation scores given to the other students in the group. It has also been found that the scores obtained by various sources (teacher, student self-assessment and evaluation of student peers) are consistent with each other. In the future, we should check whether this model is viable with large groups and we should propose a variant that allows sorting, in a list, people from different groups or convert the relative distance between group members in an absolute measure of the degree of achieving the learning outcomes set for the course(El objetivo de esta comunicación es proponer un ejemplo crítico del uso de la comparación pareada basada en el proceso de decisión multicriterio de jerarquización analítica (AHP)  para evaluar competencias de los alumnos. Lo aplicaremos a la competencia de innovación en una asignatura con pocos alumnos (10 alumnos de máster, repartidos en dos grupos). Con la metodología propuesta es posible crear, en cada grupo, una lista ordenada desde la persona que más domina una competencia a la que menos la domina, teniendo una distancia entre ellos (en una dimensión) que refleje el grado de aprendizaje (conocimientos, habilidades y actitudes) relativo entre las personas del grupo. Hemos comprobado que las puntuaciones que da un evaluador para un alumno son consistentes con el resto de puntuaciones que da ese evaluador al resto de alumnos del grupo. También se ha comprobado que las puntuaciones obtenidas por varias fuentes (profesor, auto-evaluación del alumno y evaluación del alumno a sus compañeros) son consistentes entre sí. En el futuro, deberíamos comprobar si es viable este modelo con grupos numerosos y proponer una variante que permita ordenar, en una misma lista, a las personas de diferentes grupos o convertir la distancia relativa entre las personas del grupo en una media absoluta de grado de consecución de los resultados de aprendizaje planteados para la asignaturaPalabras clave: AHP; Evaluación de estudiantes; educación universitaria; evaluación por compañeros; auto-evaluación; evaluación del profesor; método de decisión multicriterio)

Bibliographische Referenzen

  • Albayrak, E.; Erensal, Y. C. (2004). Using analytic hierarchy process (AHP) to improve human performance: An application of multiple criteria decision making problem. Journal of Intelligent Manufacturing, Vol. 15, nº. 4, pp. 491-503. http://dx.doi.org/10.1023/B:JIMS.0000034112.00652.4c
  • Andreu Andrés, M. A.; García-Casas, M. (2014). La evaluación de la participación en equipos de trabajo universitarios (Assessment of participation in higher education team working activities). WPOM-Working Papers on Operations Management, Vol. 5, nº. 1. http://dx.doi.org/10.4995/wpom.v5i1.1758
  • Chin, S.; Pun, F.; Xu, Y.; Chan, S. F. (2002). An AHP based study of critical factors for TQM implementation in Shanghai manufacturing industries. Technovation, Vol. 22, nº. 11, p. 707. http://dx.doi.org/10.1016/S0166-4972(01)00065-7
  • Comisión Europea (1995). Libro verde de la innovación. Comisión Europea (ES/13/95/55220800.P00).
  • Conlon, P.; Hecker, K.; Sabatini, S. (2012). What should we be selecting for? A systematic approach for determining which personal characteristics to assess for during admissions. BMC Medical Education, Vol. 12, nº. 1. http://dx.doi.org/10.1186/1472-6920-12-105
  • De Beuckelaer, A.; Toonen, S.; Davidov, E. (2013). On the optimal number of scale points in graded paired comparisons. Quality and Quantity, Vol. 47, nº. 5, pp. 2869-2882. http://dx.doi.org/10.1007/s11135-012-9695-2
  • European Commission (2008). The European Qualifications Framework for Lifelong Learning (EQF). Office for Official Publications of the European Communities.
  • Fernández March, A. (2010). La evaluación orientada al aprendizaje en un modelo de formación por competencias en la educación universitaria. Revista de Docencia Universitaria, Vol. 8, nº. 1, pp. 11-34.
  • Gee, S. (1981). Technology transfer, innovation & international cometitiveness. Wiley & Sons.
  • Goffin, K.; Mitchell, R. (2010). Innovation management. Palgrave-MacMillan.
  • González Pernía, J. L.; Pe-a-Legazkue, I. (2007). Determinantes de la capacidad de innovación de los negocios emprendedores en Espa-a. Economia Industrial nº. 363, pp. 129-147.
  • Goswami, S.; Mathew, M. (2005). Definition of innovation revisited: An empirical study on Indian information technology industry. International Journal of Innovation Management, Vol. 09, nº. 03, pp. 371-383. http://dx.doi.org/10.1142/S1363919605001307
  • Hatzinger, R.; Dittrich, R. (2012). Prefmod: An R package for modeling preferences based on paired comparisons, rankings, or ratings. Journal of Statistical Software, Vol. 48, nº. 10.
  • Ikehara, K.; Toyoda, H. (2012). A course evaluation model using paired comparisons to integrate importance of rating criteria and students' ratings of teaching: Comparison of students' and teachers' evaluations. Japanese Journal of Educational Psychology, Vol. 60, nº. 1, pp. 48-59. http://dx.doi.org/10.5926/jjep.60.48
  • Ingols, C.; Shapiro, M. (2014). Concrete Steps for Assessing the Soft Skills in an MBA Program. Journal of Management Education, Vol. 38, nº. 3, pp. 412-435. http://dx.doi.org/10.1177/1052562913489029
  • Kan Ma, H.; Min, C.; Neville, A.; Eva, K. (2013). How Good Is Good? Students and Assessors' Perceptions of Qualitative Markers of Performance. Teaching and Learning in Medicine, Vol. 25, nº. 1, pp. 15-23. http://dx.doi.org/10.1080/10401334.2012.741545
  • Kasirian, M. N.; Yusuff, R. M.; Ismail, M. Y. (2010). Application of AHP and ANP in supplier selection process-a case in an automotive company. International Journal of Management Science and Engineering Management, Vol. 5, nº. 2, pp. 125-135. http://dx.doi.org/10.1080/17509653.2010.10671100
  • Khalaf, M. A. & El Mokadem, M. Y. (2011). A Systematic approach for prioritizing lean practices using AHP, EurOMA.
  • Klippel, A. F.; Petter, C. O.; Antunes, J. (2008). Management Innovation, a way for mining companies to survive in a globalized world. Utilities Policy, Vol. 16, nº. 4, pp. 332-333. http://dx.doi.org/10.1016/j.jup.2007.09.001
  • Lasnier, F. (2000). Réussir la formation par compétences. Guérin.
  • Lawson, B.; Samson, D. (2001). Developing innovation capability in organizations: A dynamic capabilities approach. International Journal of Innovation Management, Vol. 05, nº. 03, pp. 377-400. http://dx.doi.org/10.1142/S1363919601000427
  • LeBreton, J. M.; Senter, J. L. (2008). Answers to 20 Questions About Interrater Reliability and Interrater Agreement. Organizational Research Methods, Vol. 11, nº. 4, pp. 815-852. http://dx.doi.org/10.1177/1094428106296642
  • Lehto, A.; Kairisto-Mertanene, L.; Penttilä, T. (2011). Towards innovation pedagogy. A new approach to teaching and learning for universities of applied sciences. Turku University of Apllied Sciences.
  • Li, Y., Wang, L., & Dong, X. (2009). An AHP-based study on the assessment of bilingual teaching in higher education institute, in 2009 2nd International Conference on Education Technology and Training, ETT 2009; Sanya; China; 13 December 2009 through 14 December 2009; Category numberP3936; Code 79728, pp. 231-234.
  • Lind, D. S.; Rekkas, S.; Bui, V.; Lam, T.; Beierle, E.; Copeland, I. I. I. (2002). Competency-Based Student Self-Assessment on a Surgery Rotation. Journal of Surgical Research, Vol. 105, nº. 1, pp. 31-34. http://dx.doi.org/10.1006/jsre.2002.6442
  • Ljungman, A. G.; Silén, C. (2008). Examination involving students as peer examiners. Assessment & Evaluation in Higher Education, Vol. 33, nº. 3, pp. 289-300. http://dx.doi.org/10.1080/02602930701293306
  • Lohmann, J. R.; Rollins, H. A.; Hoey, J. J. (2006). Defining, developing and assessing global competence in engineers. European Journal of Engineering Education, Vol. 31, nº. 1, pp. 119-131. http://dx.doi.org/10.1080/03043790500429906
  • Lyons, R. K.; Chatman, J. A.; Joyce, C. K. (2007). Innovation in services: Corporate culture and investment banking. California Management Review, Vol. 50, nº. 1, pp. 174-191. http://dx.doi.org/10.2307/41166422
  • Marin-Garcia, J. A. (2009). Los alumnos y los profesores como evaluadores. Aplicación a la calificación de presentaciones orales. Revista Espa-ola de Pedagogía, Vol. 67, nº. 242, pp. 79-97.
  • Marin-Garcia, J. A.; Aznar-Mas, L. E.; González-Ladrón-de-Gevara, F. (2011). Innovation types and talent managment for innovation. Working Papers on Operations Management, Vol. 2, nº. 2, pp. 25-31. http://dx.doi.org/10.4995/wpom.v2i2.926
  • Marin-Garcia, J. A.; Garcia-Sabater, J. P.; Miralles, C.; Rodríguez Villalobos, A. (2008). Profile and competences of Spanish industrial engineers in the European Higher Education Area (EHEA). Journal of Industrial Engineering and Management, Vol. 1, nº. 2, pp. 269-284. http://dx.doi.org/10.3926/jiem.2008.v1n2.p269-284
  • Marin-Garcia, J. A.; Garcia-Sabater, J. P.; Perello-Marin, M. R.; Canos-Daros, L. (2009a). Proposal of skills for the bachelor degree of Industrial Engineering in the context of the new curriculum. Intangible Capital, Vol. 5, nº. 4, pp. 387-406. http://dx.doi.org/10.3926/ic.2009.v5n4.p387-406
  • Marin-Garcia, J. A.; Garcia-Sabater, J. J.; Bonavia, T. (2009b). The impact of Kaizen Events on improving the performance of automotive components' first-tier suppliers. International Journal of Automotive Technology and Management, Vol. 9, nº. 4, pp. 362-376.
  • Marin-Garcia, J. A.; Perez-Pe-alver, M. J.; Watts, F. (2013). How to assess innovation competence in services: The case of university students. Dirección y Organización nº. 50, pp. 48-62.
  • Marrin, M. L.; McIntosh, K. A.; Keane, D.; Schmuck, M. L. (2004). Use of the paired-comparison technique to determine the most valued qualities of the McMaster medical programme admissions process. Advances in Health Sciences Education, Vol. 9, nº. 2, pp. 129-135. http://dx.doi.org/10.1023/B:AHSE.0000027439.18289.00
  • Mol, M. J.; Birkinshaw, J. (2009). The sources of management innovation: When firms introduce new management practices. Journal of Business Research, Vol. 62, nº. 12, pp. 1269-1280. http://dx.doi.org/10.1016/j.jbusres.2009.01.001
  • Morley, D. D. (2009). SPSS macros for assessing the reliability and agreement of student evaluations of teaching. Assessment & Evaluation in Higher Education, Vol. 34, nº. 6, pp. 659-671. http://dx.doi.org/10.1080/02602930802474151
  • Mula, J.; Días-Madro-ero, M.; Poler, R. (2012). Configuración del Grado en Ingeniería de Organización Industrial en las universidades españolas. Dirección y Organización, Vol. 47, pp. 5-20.
  • Penttilä, T. & Kairisto-Mertanene, L. (2012). Innovation competence barometer ICB - a tool for assessing students' innovation competences as learning outcomes in higher education, in INTED2012 Conference. 5th-7th March 2012, pp. 6347-6351.
  • Perrenoud, P. (2005). La universitat entre la transmissió de coneixements i el desenvolupament de competències. El debat sobre les competències a l'ensenyament universitari, ICE UB. Documents de Docència Universitària, núm. 5., pp. 8-25.
  • Pond, K. (2007). Student Experiences of Peer Review Marking of Team Projects. International Journal of Management Education, Vol. 6, nº. 1, pp. 30-43.
  • Saaty, T. L. (1980). The Analytic Hierarchy Process. McGraw-Hill.
  • Schumpeter, J. (1934). The Theory of Economic Development. Harvard University Press.
  • Terry, R. E.; Harb, J. N.; Hecker, W. C.; Wilding, W. V. (2002). Definition of student competencies and development of an educational plan to assess student mastery level. International Journal of Engineering Education, Vol. JAM-PDF, nº. 2, pp. 225-235.
  • Tonnessen, T. (2005). Continuous innovation through company wide employee participation. TQM Magazine, Vol. 17, nº. 2, pp. 195-207. http://dx.doi.org/10.1108/09544780510583254
  • Vaccaro, I. G.; Jansen, J. J. P.; Van Den Bosch, F. A. J.; Volberda, H. W. (2012). Management Innovation and Leadership: The Moderating Role of Organizational Size. Journal of Management Studies, Vol. 49, nº. 1, pp. 28-51. http://dx.doi.org/10.1111/j.1467-6486.2010.00976.x
  • Van Overveld, K. & Verhoeff, T. (2013). Self-consistent peer ranking for assessing student work: Dealing with large populations, in CSEDU 2013 - Proceedings of the 5th International Conference on Computer Supported Education, pp. 399-404.
  • Veronese Bentes, A.; Carneiro, J.; Ferreira da Silva, J.; Kimura, H. (2012). Multidimensional assessment of organizational performance: Integrating BSC and AHP. Journal of Business Research, Vol. 65, nº. 12, pp. 1790-1799. http://dx.doi.org/10.1016/j.jbusres.2011.10.039
  • Viladrich Segués, M. C.; Doval Dieguez, E. (2011). Medición: fiabilidad y validez. Laboratori d'Estadística Aplicada i de Modelització (UAB).
  • Villa Sánchez, A.; Poblete, M. (2007). Aprendizaje basado en competencias. Una propuesta para la evaluación de las competencias genéricas. Universidad de Deusto.
  • Watts, F.; Garcia-Carbonell, A.; Andreu Andrés, M. A. (2013). Innovation competencies development: INCODE barometer and use guide. Turku University od Applied Sciences.
  • Watts, F.; Marin-Garcia, J. A.; Garcia-Carbonell, A.; Aznar-Mas, L. E. (2012). Validation of a rubric to assess innovation competence. Working Papers on Operations Management, Vol. 3, nº. 1, pp. 61-70. http://dx.doi.org/10.4995/wpom.v3i1.1159
  • Yao, J. (2010). Feature AHP method used for excellent students evaluation, in 2010 International Conference on Optics, Photonics and Energy Engineering, OPEE 2010; Wuhan; China; 10 May 2010 through 11 May 2010; Category numberCFP1033I-PRT; Code 81252, pp. 422-425.
  • Yiu, E.; Chan, K.; Mok, R. (2007). Reliability and confidence in using a paired comparison paradigm in perceptual voice quality evaluation. Clinical Linguistics and Phonetics, Vol. 21, nº. 2, pp. 129-145 http://dx.doi.org/10.1080/02699200600756355
Fuente de los datos: Dialnet