2017 APCSSP
Asia-Pacific Conference on Education, Social Studies and Psychology
February 7-9, 2017 at Courtyard by Marriott Seoul Times Square

Keynote Speaker

Multimedia Learning for the Better Education: Visual Attention and Semantic Integration

Research Question: It is true that for hundreds of years verbal messages such as lectures and printed lessons have been the primary means of explaining ideas to learners. Despite the power of verbal learning, we have all witnessed for the last century that learning strategies have been a lot changed by the introduction of information and communication technologies. In the age of multimedia, as the concept-based verbal learning have given way to the image-based multimedia learning, we all recognize that people learn more deeply when ideas are expressed in both words and pictures rather than in words alone. Theoretical Background: In the linguistics literature on verbal semantic integration, the N400 effect refers to the fact that unexpected words will cause very low level of negative potentials in brainwave measures around 400ms(milli-seconds) after repeated presented settings and the P600 effect refers to the fact the unexpected words will cause very high level of positive potentials around 600ms. Verbal linguists proposed two hypothetical effects on ERP responses to of verbal instructional stimuli using EEG measurement data. Let’s look at the relation between the phrase 'a type of fish' and a word 'dolphin'. Subjects are looking forward to a type of fish, say 'salmon', and when they hear something other than a fish, then their expectation makes a mis-match with this latest word, 'dolphin'. Let’s look at another case for P600. P600 effects are considered as brain wave responses which reflect the cognitive effort to process the stimulus information that is difficult to handle with. What if ‘a type of fish’ is followed by ‘starfish’ or ‘catfish’? N400 effect for the former, P600 for the latter, for sure. This paradigm using brainwave measurements has become prevalent both in the area of verbal information processing and in the area of visual information processing. Sitnikova et. al (2003) and Niel Cohn ( applied this paradigm to short video clips and comic strips. The former, video clips or dynamic images are more naturalistic in a continuous flow, while the latter, pictures or graphics in the form of static images are less naturalistic.

Methods: The cognitive model of multimedia learning compares perceptual information through different channels with the topical semantic information already in the working memory. It is expected that this semantic topic process is to collate the provisional topic with newly incoming video elements. The renewal or updating process of the provisional topic by viewers must be a complex cognitive process of integrating the results of the processing of audiovisual elements that are newly introduced with this complex of provisional topic and pre-processed audiovisual elements supporting the topic. Research literature on the video summarization indicates importance of the development of an un-obtrusive method of gathering external information of video users (Z. Lu and K. Grauman,, 2013; W. Ren and Y. Zhu,, 2008; A. Porselvi and S. Gunasundari, 2013). The analysis of event-related potentials (ERP) is such a method which extracts only a result of reaction with respect to certain stimuli from the brain waves.

Findings and Implications: We looked at the main effect of topic-relevant and their interaction effects with electrodes position with regard to N400 and P600 effects. At the statistical significant level of 0.05, we observed greater maximum potentials at the left prefrontal cortex (FP1, t = 6.930, p = 0.013), the left, right, and middle central lobes (C3, t = 4.670, p = 0.039; Cz, t = 9.818, p = 0.004; C4, t = 10.549 , p = 0.003), and the right and middle frontal-central lobes (FC4, t = 7.370, p = 0.011; FCz, t = 6.541, p = 0.016) of brain wave responses to topic-relevant shots, compared with topic-irrelevant shots. The right parietal and right temporal-parietal lobes (P8, t = 4.483, p =0.043; TP8, t = 5.326, p = 0.028) also showed P600 effects. Several contributions could be identified. First, it is indisputable to further attempt this sort of ERP analysis of the EEG data during continuous viewing session using topic-relevance ratings from still image testing. Second, the most puzzling finding was the strongest effect size of N400 and P600 at prefrontal lobe. Third, the baseline of butterfly plot of the EEG data during video viewing session was somewhat unstable, compared to the baseline figure of the butterfly plot of the EEG data during the test frame.