We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.
Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.
Ilka Huesmann, Cedric Lawida, Ina-Maria Maahs
et al.
Der Beitrag stellt die enge Verzahnung digitalitätsbezogener und sprachlicher Kompetenzen heraus, deren Ausbildung eine Grundlage für gesellschaftliche Teilhabe und für zukunftsorientiertes Lernen in allen Fächern darstellt. Hierfür werden zunächst theoretische Grundlagen zur Relevanz einer durchgängigen Sprachbildung in einer Kultur der Digitalität (Stalder 2016) ausgeführt. Anschließend wird ein Beispiel zum digitalen kollaborativen Schreiben einer Online-Schüler:innenzeitung im Deutschunterricht illustriert. Dabei wird dargelegt, wie im Kontext eines mehrsprachigkeitsorientierten sprachsensiblen Fachunterrichts unter Einsatz digitaler Sprachhilfen sowohl grundlegende bildungs- und fachsprachliche Kompetenzen als auch erweiterte digitalitätsbezogene Kompetenzen gefördert werden können.
Abstract With the rapid development of science and technology, augmented reality technology provides intelligent and application services. The research is based on imaging techniques using augmented reality technology and camera image capture. Then, it uses screen error algorithms and scale-invariant feature transformation operators to test the quality of scene spatial models. The experimental results demonstrated that the camera significantly improved the frame rate of scene model rendering and could steadily enhance rendering efficiency. For image quality and its influencing factors, binary robust invariant scalable keypoints and scale-invariant feature transformation algorithms in viewpoint changes had the highest recall of 92%. The map drawing module, Hessian matrix, and scale-invariant feature transformation algorithm in the image blurring test achieved the highest recall rate of 98%. This demonstrates the advantage of using a scale-invariant feature transformation operator to capture scene space influence and provide a more accurate spatial model reference for augmented reality technology. This enhances the functional design of the guide system.
Computational linguistics. Natural language processing, Electronic computers. Computer science
Sara Vinyals-Mirabent, Emma Rodero, Isabel Rodríguez-de-Dios
et al.
External crises create opportunities to strengthen companies’ commitment to society through CSR claims; however, consumers often perceive them as opportunistic. This study leverages the unique setting of COVID-19 to determine the power of consumer self-centered factors (i.e., concern about the crisis, personal impact, and political ideology) to predict situational skepticism towards CSR communication during a large-scale external crisis. An online survey of 1,000 consumers, analyzed using structural equation modeling, revealed that self-centered variables are key determinants to predict skepticism during external events. This effect is mediated by the inferential process of attributing motives. Their predictive power is consistent across the four CSR domains (i.e., customer, environment, employees, philanthropy). This study moves forward on the egocentric pattern projection and the attribution theories by addressing reactive CSR to external crises, a type of crisis that has been overlooked, and supports managerial decisions to mitigate CSR skepticism for potential external crises to come.
Este artículo reconstruye la trayectoria de la cinefilia del escritor Manuel Puig, desde su etapa temprana como espectador frecuente de salas cinematográficas durante el período 1930 y 1940 hasta su posterior conversión en coleccionista de videocassettes durante la década de 1980. A través del análisis de sus novelas y de su correspondencia personal, se exploran las transformaciones en su relación con el cine, enmarcadas en los cambios tecnológicos y culturales del siglo XX. La investigación se inscribe en el cruce entre la sociología de la cultura y la nueva historia del cine, con un enfoque centrado en las prácticas situadas de los espectadores. Se examinan aspectos como la dimensión económica del acceso al cine en diferentes ciudades, la experiencia de las salas como espacio colectivo, el impacto del doblaje y los subtítulos en su formación cinéfila, y la progresiva individualización del consumo audiovisual con la aparición del video. El estudio muestra cómo Puig construye una relación íntima y reflexiva con el cine, que va más allá del entretenimiento y se convierte en una fuente constante de conversación, análisis y memoria, articulando afectos, saberes y formas de archivo personal.
Zhivar Sourati, Farzan Karimi-Malekabadi, Meltem Ozcan
et al.
Language is far more than a communication tool. A wealth of information - including but not limited to the identities, psychological states, and social contexts of its users - can be gleaned through linguistic markers, and such insights are routinely leveraged across diverse fields ranging from product development and marketing to healthcare. In four studies utilizing experimental and observational methods, we demonstrate that the widespread adoption of large language models (LLMs) as writing assistants is linked to notable declines in linguistic diversity and may interfere with the societal and psychological insights language provides. We show that while the core content of texts is retained when LLMs polish and rewrite texts, not only do they homogenize writing styles, but they also alter stylistic elements in a way that selectively amplifies certain dominant characteristics or biases while suppressing others - emphasizing conformity over individuality. By varying LLMs, prompts, classifiers, and contexts, we show that these trends are robust and consistent. Our findings highlight a wide array of risks associated with linguistic homogenization, including compromised diagnostic processes and personalization efforts, the exacerbation of existing divides and barriers to equity in settings like personnel selection where language plays a critical role in assessing candidates' qualifications, communication skills, and cultural fit, and the undermining of efforts for cultural preservation.
We consider the possible role of current large language models (LLMs) in the study of human linguistic cognition. We focus on the use of such models as proxies for theories of cognition that are relatively linguistically-neutral in their representations and learning but differ from current LLMs in key ways. We illustrate this potential use of LLMs as proxies for theories of cognition in the context of two kinds of questions: (a) whether the target theory accounts for the acquisition of a given pattern from a given corpus; and (b) whether the target theory makes a given typologically-attested pattern easier to acquire than another, typologically-unattested pattern. For each of the two questions we show, building on recent literature, how current LLMs can potentially be of help, but we note that at present this help is quite limited.
Is character education a legitimate goal of higher education? Character education should aim to form people so they can live well in a world worth living in. All universities, whether faith-inspired or not, have an obligation to prepare students for life—a life worth living, a life with purpose. The Christian faith conviction that we as humans have a common telos, that there is an ultimate common good, or highest good, that is God, is central to any Catholic concept of character and flourishing in the university. The practice of the virtues, through good character, is the road to this spiritual end. Catholic Universities traditionally have many features that make them well-placed to cultivate the virtues of character in their students, particularly through the lens of a Christian anthropology. The work of the Jubilee Centre for Character and Virtues on universities is highlighted together with recent scholarly discussion of the place of character virtues in secular and Christian universities.
Philosophy of religion. Psychology of religion. Religion in relation to other subjects, Communication. Mass media
Machine reading comprehension is an essential natural language processing task, which takes into a pair of context and query and predicts the corresponding answer to query. In this project, we developed an end-to-end question answering model incorporating BERT and additional linguistic features. We conclude that the BERT base model will be improved by incorporating the features. The EM score and F1 score are improved 2.17 and 2.14 compared with BERT(base). Our best single model reaches EM score 76.55 and F1 score 79.97 in the hidden test set. Our error analysis also shows that the linguistic architecture can help model understand the context better in that it can locate answers that BERT only model predicted "No Answer" wrongly.
Esta entrevista com Philippe-Alain Michaud, curador do Centre Pompidou, professor na Universidade de Genebra, foi realizada por três membros da equipe da Revista Eco-Pós tendo a finalidade específica de ser publicada nesta primeira edição do dossiê "Visualidades". Nesse sentido, os entrevistadores se apoiaram em uma conferência ministrada por Michaud em 16 de março de 2023 na Cinemateca MAM-Rio, que versava “Sobre a tela”. O texto da conferência em si mesma, que discute a tela e suas funções simultâneas de mostrar e esconder, passando pelos mais diversos objetos de análise, é disponibilizado neste dossiê. A entrevista, feita online em 18 de agosto de 2023, esmiúça certos assuntos da conferência, mas também vai além, trazendo provocações originais de Michaud. Entre elas, a ideia de que artistas não fazem imagens, mas as desfazem; ou a sugestão de se olhar para a cultura visual bizantina para se pensar o regime de imagens digitais contemporâneo.
Dentro de los estudios gordos se deben abordar técnicas de investigación que permitan acceder a capas del discurso que van más allá de lo verbal. La técnica cualitativa de mapas corporales digitales supone una herramienta apropiada que nos acerca a les ‘cuerpes’ y permite hacer ejercicios autorreflexivos a todes les participantes del proceso de investigación. Adaptar técnicas comúnmente desarrolladas en espacios físicos a espacios online supone un reto metodológico que amplía las dimensiones de análisis al entrecruzarse y difuminarse las fronteras de lo físico y lo digital. En este artículo se explora la técnica de investigación de los mapas corporales digitales aportando una definición de la técnica. Se expone una genealogía de las técnicas similares y se reflejan los resultados que permite esta técnica obtener.
Communication. Mass media, Social sciences (General)
We propose EmoDistill, a novel speech emotion recognition (SER) framework that leverages cross-modal knowledge distillation during training to learn strong linguistic and prosodic representations of emotion from speech. During inference, our method only uses a stream of speech signals to perform unimodal SER thus reducing computation overhead and avoiding run-time transcription and prosodic feature extraction errors. During training, our method distills information at both embedding and logit levels from a pair of pre-trained Prosodic and Linguistic teachers that are fine-tuned for SER. Experiments on the IEMOCAP benchmark demonstrate that our method outperforms other unimodal and multimodal techniques by a considerable margin, and achieves state-of-the-art performance of 77.49% unweighted accuracy and 78.91% weighted accuracy. Detailed ablation studies demonstrate the impact of each component of our method.
Referring image segmentation aims to segment the image region of interest according to the given language expression, which is a typical multi-modal task. Existing methods either adopt the pixel classification-based or the learnable query-based framework for mask generation, both of which are insufficient to deal with various text-image pairs with a fix number of parametric prototypes. In this work, we propose an end-to-end framework built on transformer to perform Linguistic query-Guided mask generation, dubbed LGFormer. It views the linguistic features as query to generate a specialized prototype for arbitrary input image-text pair, thus generating more consistent segmentation results. Moreover, we design several cross-modal interaction modules (\eg, vision-language bidirectional attention module, VLBA) in both encoder and decoder to achieve better cross-modal alignment.
Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto
et al.
The rapid development of Large Language Models (LLMs) and the emergence of novel abilities with scale have necessitated the construction of holistic, diverse and challenging benchmarks such as HELM and BIG-bench. However, at the moment, most of these benchmarks focus only on performance in English and evaluations that include Southeast Asian (SEA) languages are few in number. We therefore propose BHASA, a holistic linguistic and cultural evaluation suite for LLMs in SEA languages. It comprises three components: (1) a NLP benchmark covering eight tasks across Natural Language Understanding (NLU), Generation (NLG) and Reasoning (NLR) tasks, (2) LINDSEA, a linguistic diagnostic toolkit that spans the gamut of linguistic phenomena including syntax, semantics and pragmatics, and (3) a cultural diagnostics dataset that probes for both cultural representation and sensitivity. For this preliminary effort, we implement the NLP benchmark only for Indonesian, Vietnamese, Thai and Tamil, and we only include Indonesian and Tamil for LINDSEA and the cultural diagnostics dataset. As GPT-4 is purportedly one of the best-performing multilingual LLMs at the moment, we use it as a yardstick to gauge the capabilities of LLMs in the context of SEA languages. Our initial experiments on GPT-4 with BHASA find it lacking in various aspects of linguistic capabilities, cultural representation and sensitivity in the targeted SEA languages. BHASA is a work in progress and will continue to be improved and expanded in the future. The repository for this paper can be found at: https://github.com/aisingapore/BHASA
Unifying acoustic and linguistic representation learning has become increasingly crucial to transfer the knowledge learned on the abundance of high-resource language data for low-resource speech recognition. Existing approaches simply cascade pre-trained acoustic and language models to learn the transfer from speech to text. However, how to solve the representation discrepancy of speech and text is unexplored, which hinders the utilization of acoustic and linguistic information. Moreover, previous works simply replace the embedding layer of the pre-trained language model with the acoustic features, which may cause the catastrophic forgetting problem. In this work, we introduce Wav-BERT, a cooperative acoustic and linguistic representation learning method to fuse and utilize the contextual information of speech and text. Specifically, we unify a pre-trained acoustic model (wav2vec 2.0) and a language model (BERT) into an end-to-end trainable framework. A Representation Aggregation Module is designed to aggregate acoustic and linguistic representation, and an Embedding Attention Module is introduced to incorporate acoustic information into BERT, which can effectively facilitate the cooperation of two pre-trained models and thus boost the representation learning. Extensive experiments show that our Wav-BERT significantly outperforms the existing approaches and achieves state-of-the-art performance on low-resource speech recognition.
Sophia Althammer, Mark Buckley, Sebastian Hofstätter
et al.
Domain-specific contextualized language models have demonstrated substantial effectiveness gains for domain-specific downstream tasks, like similarity matching, entity recognition or information retrieval. However successfully applying such models in highly specific language domains requires domain adaptation of the pre-trained models. In this paper we propose the empirically motivated Linguistically Informed Masking (LIM) method to focus domain-adaptative pre-training on the linguistic patterns of patents, which use a highly technical sublanguage. We quantify the relevant differences between patent, scientific and general-purpose language and demonstrate for two different language models (BERT and SciBERT) that domain adaptation with LIM leads to systematically improved representations by evaluating the performance of the domain-adapted representations of patent language on two independent downstream tasks, the IPC classification and similarity matching. We demonstrate the impact of balancing the learning from different information sources during domain adaptation for the patent domain. We make the source code as well as the domain-adaptive pre-trained patent language models publicly available at https://github.com/sophiaalthammer/patent-lim.