STEM critical thinking assessment for measuring students' critical thinking skills in the automotive chemistry course

: Critical thinking is one of the HOTS areas that students must master. Students can use these skills to solve contextual problems when studying automotive chemistry. The development of STEM-based instruments to measure STEM critical thinking is still rare. This study aims to develop a STEM-critical thinking assessment to measure students' critical thinking skills in the Automotive Chemistry course. This type of research is R&D which modifies the Borg and Gall model. Instruments developed in the form of questions and reasoning to determine students' critical thinking skills. The questions developed cover the dimensions of critical thinking skills, including argument analysis; answering questions for assignment determination; consider the credibility of the source; monitoring and calculating results; draw conclusions; make and evaluate decisions; define and evaluate terms; identify assumptions; and determine the action. The validation results from evaluators, learning experts and chemistry teachers show that the instrument is valid and can be used to measure students' critical thinking skills in automotive chemistry courses.


Introduction
The rapid development of science and technology in the era of society 5.0 requires graduates to be able to work in complex ways of thinking (Sulaiman & Ismail, 2020), solving various problems and challenges, and responding to them with the help of technology from the era of the industrial revolution 4.0 (Nugraha & Rahman, 2021). Especially for prospective z-generation graduates (born 1995-2001), who will compete in the world of work in the future (Plochocki, 2019). However, learning changes during the pandemic caused some student skills to not develop properly (Withing, 2021). Transformation to digital learning requires the active participation of students, so that their knowledge, skills, and creativity must be integrated to become ready-to-use graduates (Sari & Wulanda, 2019). Prospective graduates in the future are expected to have different skills, such as higher-order thinking skills (HOTS) (Chonkaew et al., 2016;Retnawati et al., 2018). These skills must be taught early so that students are used to solving real problems (Balakrishnan et al., 2016;Ichsan & Rahmayanti, 2020;Misykah & Adiansha, 2018). Therefore, learning must fully integrate these skills.
Critical thinking is part of HOTS and one of the 10 Top skills of 2025 (Withing, 2021). Critical thinking is a person's ability to analyze, evaluate, improve his thinking, use analytical thinking, evaluate problems, draw conclusions based on logic (Paulsen & Dankert, 2022), and create solutions (Paul & Elder, 2019). Critical thinking is needed to prepare graduates who are ready to compete globally. Acquiring these skills enables students to find appropriate information about the academic phenomena or problems they face and solve problems in real life (Chan, 2019).
Automotive chemistry is one of the compulsory subjects of the Ivet University Mechanical Engineering Vocational Education Study Program. This course covers the application of chemical concepts to the automotive world, such as the use of elements, compounds, and mixtures in the automotive industry; the application of the basic laws of chemistry and the 5290 mole concept in chemical stoichiometry to the automotive world; the concept of reaction rate in the catalytic converter; and the use of petroleum fractions in the automotive industry. These materials are believed to be able to help students develop critical thinking. Therefore, in lectures, students can be directed to explain examples of contextually applying material, demonstrating, and relating the concepts learned to different life experiences.
A preliminary study of odd semester automotive chemistry lectures for the 2021-2022 academic year shows that students' critical thinking skills are still weak. Students still have difficulty integrating chemical concepts in the automotive field, such as elements and compounds. Facts in the field also show that students are less able to analyze the reasons for using aluminium alloy as a material for making aluminium pistons for vehicles, with a number of reasons related to the properties of aluminum metal. This is in line with the opinion of Utami et al., (2017) that students have difficulty understanding chemical material because it is interrelated. In learning assessment, most students could only answer questions based on examples of questions that had been given by lecturers. However, when presented with open questions that require divergent and convergent skills, students have difficulty solving them. In fact, students must be able to integrate learning material with its application in real life (Kriswantoro et al., 2021). This is supported by the results of the PISA and TIMSS surveys, which place Indonesia in the bottom 10 in critical thinking. In addition, Indonesia ranks lowest in the Human Development Index, which was ranked 116th out of 189 countries in 2017 (UNDP, 2018). Therefore, critical thinking must really be developed in the learning process. Currently, many studies examine critical thinking skills. However, a good critical thinking ability measurement test is needed before the lecturer designs the right lesson.
Based on these problems, an evaluation tool is needed for valid and reliable learning assessments to measure students' critical thinking levels. This is in accordance with (Danczak, 2018). that through the assessment of learning outcomes, students are encouraged to improve their thinking skills. However, some lecturers have difficulties preparing and implementing good assessments in learning (Janati et al., 2018). Assessments for measuring critical thinking skills are still rarely developed by lecturers.
One alternative way to assess critical thinking skills is by using test instruments. The test can be objective or non-objective Shavelson et al., 2019). Previous studies have produced tests of critical thinking skills in terms of description of chemical materials (Khoirunisa & Sabekti, 2020;Kriswantoro et al., 2021), statements on science lectures (Al-Mahrooqi & Denman, 2020), and dual choice on IPA learning (Dewi & Prasetyo, 2016). Nevertheless, it is still rare to develop tests that assess students' critical thinking abilities by combining concepts from the fields of Science, Technology, Engineering, and Mathematics. In fact, the integration of the four concepts in STEM can improve students' ability to understand and solve problems related to well-learned material concepts (Hacioglu & Gulhan, 2021). Therefore, in this study, a test instrument will be developed for assessing critical thinking skills that integrate the four concepts in STEM.
The test developed in this study is a question of statement-cause that integrates the relationship of the concepts of Science, Technology, Engineering, and Mathematics to the phenomena or application of the concept of chemistry in the automotive field. The choice of forms of statements is based on the characteristics of automotive chemicals so that it can develop the ability to think critically well. Studies show that tests that combine objective and non-objective statements with reasonable statements are considered to have an advantage over other types of tests. Using this combination, students can analyze the truth, correlation, and reasoning behind the statement. In addition, students are given the critical task of critically explaining the reasons for their answers by using their own sentences. Furthermore, the choice of forms is based on the properties of the Automotive Chemistry material that are useful and potentially enhance students' critical thinking skills.
The STEM-critical thinking assessment developed in this study is in the form of chemistry questions that integrate the interrelationships of the concepts of Science, Technology, Engineering, and Mathematics in the form of cause-statement questions by including the reasons for the answers in the Automotive Chemistry course material. The instrument was developed to measure the achievement level of students' critical thinking skills. The purpose of this research is to develop a STEM-critical thinking assessment for the Automotive Chemistry course.

Method
The purpose of this research development model is to develop and validate STEM-critical thinking assessment products to measure students' critical thinking skills in the Automotive Chemistry course. This research was conducted at Universitas Ivet from January to July 2022.

Research procedure
This research was adapted from the research and development model of Borg & Gall (Borg & Gall, 1983). However, the research was only carried out up to the sixth step in the research and development of the Borg & Gall model. It is based on limited research conducted in the Mechanical Engineering Vocational Education Study Program for semester 1 students, as well as time considerations that are not expected to be possible when developing ten development steps according to Borg & Gall. The stages of this research were (1) preliminary study, (2) product planning, (3) initial product development, (4) product validation, (5) early product revision to produce the main product, and (6) limited product dissemination (Borg & Gall, 1983).
Preliminary studies consist of field studies and literature studies. A field study was conducted to find out the conditions of automotive chemistry learning and the lecturer's need for assessment of critical thinking skills. A literature study was conducted to find literature on the research topic. At the planning stage, an analysis of the competence of the Automotive Chemistry course is carried out, as well as indicators of critical thinking skills that must be mastered by students. According to Nitko & Brookhart (2011), there are three dimensions of critical thinking skills used to develop STEM-critical thinking assessment instruments: (1) analyzing arguments; (2) answer questions that indicate difficulty; (3) consider the credibility of the source; (4) observing and considering the results; (5) draw conclusions; (6) make and evaluate decisions; (7) identify and assess terms; (8) identify assumptions; and (9) determine the action.
Furthermore, based on the results of the mapping, indicator questions were prepared. Table 1 shows the details of the STEM critical thinking assessment item numbers based on indicators of critical thinking skills. The item of STEM critical thinking assessment was developed based on these indicators. STEM-critical thinking assessment for the Automotive Chemistry course made during the product development stage, following the dimension of critical thinking skills. This assessment consists of 35 questions in the form of statements and reasons questions. The expert appraisal is conducted through a questionnaire, which includes aspects of content, presentation, and language. This questionnaire was prepared using a Likert scale which has four options for answers which are changed to four scales. Learning experts, evaluation experts, and chemistry lecturers have judged the assessment. The expert appraisal recommendation will be the base for the revision of the STEM-critical thinking assessment.
The revised product was tried to 10 Mechanical Engineering Vocational Education Study Program students. Furthermore, students were asked to provide reviews and recommendations. The next stage consists of data analysis and improvement of the STEM-critical thinking assessment based on student feedback. This research finding is qualitative data and quantitative data. Qualitative data consists of the expert appraisal and students of the quality of STEM instruments for critical thinking. Quantitative data consists of product assessments carried out by the expert.
The next stage is data analysis and revision of the STEM-critical thinking assessment based on students' responses. Furthermore, this assessment was disseminated to chemistry lecturers at several universities. It is hoped that this instrument can be used as a teacher's reference for evaluating the chemistry learning process related to measuring students' critical thinking skills.

Data Analysis
Qualitative data in the form of recommendations from experts were tabulated and summarized as a guideline for product revision of STEM-critical thinking assessment.
The expert appraisal score obtained for each assessment indicator is tabulated, and the average score is calculated and then converted into interval data with a scale of four. Reference for changing the score of each component can be seen in Table 2.

Result and Discussion
The STEM-critical thinking assessment instrument has been developed by adopting the Borg and Gall (1983) development model, which consists of preliminary studies, product planning, early-stage product development, product validation, early-stage product revision to produce the main product, and limited product dissemination. Based on the results of the preliminary study, it was revealed that students sometimes experience difficulties in solving automotive chemistry questions that are contextual and applicable. Students tend to memorize material rather than understand it. In addition, the assessment tool used is not suitable for measuring students' critical thinking. The questions posed by lecturers are also rarely related to daily life. Therefore the assessment given by the teacher cannot develop convergent and divergent thinking skills. Thus, students critical thinking skills are not developed and measured properly. The next step is a literature review by collecting and analyzing various information about curriculum competencies and critical thinking skills in automotive chemistry courses. The analysis results show that Automotive Chemistry material potential to train students' critical thinking skills. The second stage is planning; mapping the possibilities and indicators of critical thinking that students have to master in accordance with the curriculum. The next step is to develop a STEM-critical thinking assessment. An example of the questions in the STEM-critical thinking assessment is given in Table 3. The toxicity of exhaust gases can be reduced by reacting CO gas and NO gas, and NO2 gas with hydrocarbons to become new compounds that are not toxic. The reaction must take place at high temperatures so that it can damage engine performance. Therefore, a catalytic converter is needed to accelerate the reaction at low temperatures The quality of the developed STEM-critical thinking assessment product is viewed from the content component (20 indicators), the presentation component (7 indicators), and the language component (4 indicators). The scores obtained for each component are then converted into grades. Comparison of the total scores of the validation results of the STEM-critical thinking assessment instrument from material experts and evaluation experts is presented in Figure 1. Based on Figure 1, the quality of the STEM-critical thinking assessment instrument for the content aspect, with a maximum score of 80, an average total score of material experts and evaluation experts is 75 with a very good category. In the presentation aspect with a maximum score of 28, an average total score of 27.33 is obtained in the very good category. As for the language component with a maximum score of 20, an average total score of 15.67 is obtained in the very good category as well. The validation of the STEM-critical thinking assessment instrument was also carried out by two chemistry lecturers as users, from Sultan Thaha Jambi University and IAIN Kudus. Lecturers' responses regarding the quality of the STEM-critical thinking assessment for each component are presented in Figure  2.  Figure 2, in the assessment of the content aspect, an average total score of 69 was obtained from the chemistry lecturer in the very good category. In the presentation component, the average total score is 25 with a very good category. As for the language aspect, an average total score of 14 was also obtained in the very good category. The expert appraisal results from both material experts, evaluation experts and chemistry lecturers show that the developed STEM-critical thinking assessment instrument is suitable for use by lecturers in the evaluation of the chemistry learning process for measuring critical thinking skills.
Student responses to STEM-critical thinking assessment instrument products were obtained using a questionnaire. The data obtained from the questionnaire are in the form of assessments and student responses to the presentation and language components. The questionnaire used contains 11 statements with yes (1) and no (0) answer choices. The average score of student responses to the quality of the STEM-critical thinking assessment instrument is 8 out of a maximum score of 11. The score obtained is in a good category. Product revisions to the STEM-critical thinking assessment instrument were carried out based on responses and input from evaluation experts, subject matter experts, chemistry lecturers, and students. An example of product revisions made is presented in Table 6. Add an image of the Accumulator and its components so students can better understand the context of the problem Adding a picture of a battery to question no.2 so that students get an idea of the parts of the Accumulator Questions in the STEM-critical thinking assessment are questions of contextual applicability of the concept of chemical matter. In automotive chemistry courses, questions are presented in an integrated manner between subject matter. Some of the questions are examples of real cases on motorbikes and we often experience them, for example the use of catalytic converters, batteries. Guruh et al. (2018) explained that asking questions related to everyday life can develop students' critical thinking. Questions are structured in such a way that initial information about the questions is presented. Information related to the concept of automotive chemistry courses is presented in the form of reading text, examples of cases/problems, pictures and graphs. This can make students more enthusiastic about processing this information and linking it to material concepts that have been previously studied to solve problems. Thus, students no longer only use conventional learning, but can use their critical thinking to answer questions related to the material being tested. The STEM-critical thinking assessment product resulting from this development can be used as an alternative assessment to measure critical thinking. This is in accordance with the research findings of Ritdamaya & Suhandi (2016) which shows that through the use of appropriate and continuous critical thinking skills assessment instruments can train and develop critical thinking skills.

Conclusion
Based on the research and development results, the following conclusions can be drawn: (a) The stages of developing the STEM-critical thinking assessment instrument were carried out, namely preliminary studies, planning, initial product development, product validation, limited trials, product revisions based on limited trial results, and limited dissemination; (b) the quality of the STEM-critical thinking assessment instrument that has been developed in terms of content, presentation and language components is included in the very good category. Thus, the STEM-critical thinking assessment instrument product that has been developed is suitable for use in learning evaluation.

Conflicts of Interest
The authors declare no conflict of interest