Hasil untuk "Information theory"

Christophe J. MacLellan, Karthik Singaravadivelan, Xin Lian et al.

We present a new theory of categorization based on an information-theoretic rational analysis. To evaluate this theory, we investigate how well it can account for key findings from classic categorization experiments conducted by Hayes-Roth and Hayes-Roth (1977), Medin and Schaffer (1978), and Smith and Minda (1998). We find that it explains the human categorization behavior at least as well (or better) than the independent cue and context models (Medin & Schaffer, 1978), the rational model of categorization (Anderson, 1991), and a hierarchical Dirichlet process model (Griffiths et al., 2007).

Charles Westphal, Stephen Hailes, Mirco Musolesi

The Information Bottleneck (IB) principle offers a compelling theoretical framework to understand how neural networks (NNs) learn. However, its practical utility has been constrained by unresolved theoretical ambiguities and significant challenges in accurate estimation. In this paper, we present a \textit{Generalized Information Bottleneck (GIB)} framework that reformulates the original IB principle through the lens of synergy, i.e., the information obtainable only through joint processing of features. We provide theoretical and empirical evidence demonstrating that synergistic functions achieve superior generalization compared to their non-synergistic counterparts. Building on these foundations we re-formulate the IB using a computable definition of synergy based on the average interaction information (II) of each feature with those remaining. We demonstrate that the original IB objective is upper bounded by our GIB in the case of perfect estimation, ensuring compatibility with existing IB theory while addressing its limitations. Our experimental results demonstrate that GIB consistently exhibits compression phases across a wide range of architectures (including those with \textit{ReLU} activations where the standard IB fails), while yielding interpretable dynamics in both CNNs and Transformers and aligning more closely with our understanding of adversarial robustness.

Nolwenn Bernard, Krisztian Balog

User simulation is a promising approach for automatically training and evaluating conversational information access agents, enabling the generation of synthetic dialogues and facilitating reproducible experiments at scale. However, the objectives of user simulation for the different uses remain loosely defined, hindering the development of effective simulators. In this work, we formally characterize the distinct objectives for user simulators: training aims to maximize behavioral similarity to real users, while evaluation focuses on the accurate prediction of real-world conversational agent performance. Through an empirical study, we demonstrate that optimizing for one objective does not necessarily lead to improved performance on the other. This finding underscores the need for tailored design considerations depending on the intended use of the simulator. By establishing clear objectives and proposing concrete measures to evaluate user simulators against those objectives, we pave the way for the development of simulators that are specifically tailored to their intended use, ultimately leading to more effective conversational agents.

Mark Whitmeyer

We study what changes to an agent's decision problem increase her value for information. We prove that information becomes more valuable if and only if the agent's reduced-form payoff in her belief becomes more convex. When the transformation corresponds to the addition of an action, the requisite increase in convexity occurs if and only if a simple geometric condition holds, which extends in a natural way to the addition of multiple actions. We apply these findings to two scenarios: a monopolistic screening problem in which the good is information and delegation with information acquisition.

Cheuk Ting Li

We resolve three long-standing open problems, namely the (algorithmic) decidability of network coding, the decidability of conditional information inequalities, and the decidability of conditional independence implication among random variables, by showing that these problems are undecidable. The proof utilizes a construction inspired by Herrmann's arguments on embedded multivalued database dependencies, a network studied by Dougherty, Freiling and Zeger, together with a novel construction to represent group automorphisms on top of the network.

Corrado Monti, Giuseppe Manco, Cigdem Aslay et al.

Modeling information cascades in a social network through the lenses of the ideological leaning of its users can help understanding phenomena such as misinformation propagation and confirmation bias, and devising techniques for mitigating their toxic effects. In this paper we propose a stochastic model to learn the ideological leaning of each user in a multidimensional ideological space, by analyzing the way politically salient content propagates. In particular, our model assumes that information propagates from one user to another if both users are interested in the topic and ideologically aligned with each other. To infer the parameters of our model, we devise a gradient-based optimization procedure maximizing the likelihood of an observed set of information cascades. Our experiments on real-world political discussions on Twitter and Reddit confirm that our model is able to learn the political stance of the social media users in a multidimensional ideological space.

Clément Luneau, Jean Barbier, Nicolas Macris

We consider generalized linear models in regimes where the number of nonzero components of the signal and accessible data points are sublinear with respect to the size of the signal. We prove a variational formula for the asymptotic mutual information per sample when the system size grows to infinity. This result allows us to derive an expression for the minimum mean-square error (MMSE) of the Bayesian estimator when the signal entries have a discrete distribution with finite support. We find that, for such signals and suitable vanishing scalings of the sparsity and sampling rate, the MMSE is nonincreasing piecewise constant. In specific instances the MMSE even displays an all-or-nothing phase transition, that is, the MMSE sharply jumps from its maximum value to zero at a critical sampling rate. The all-or-nothing phenomenon has previously been shown to occur in high-dimensional linear regression. Our analysis goes beyond the linear case and applies to learning the weights of a perceptron with general activation function in a teacher-student scenario. In particular, we discuss an all-or-nothing phenomenon for the generalization error with a sublinear set of training examples.

Jianping Cao, Senzhang Wang, Danyan Wen et al.

Currently, many intelligence systems contain the texts from multi-sources, e.g., bulletin board system (BBS) posts, tweets and news. These texts can be ``comparative'' since they may be semantically correlated and thus provide us with different perspectives toward the same topics or events. To better organize the multi-sourced texts and obtain more comprehensive knowledge, we propose to study the novel problem of Mutual Clustering on Comparative Texts (MCCT), which aims to cluster the comparative texts simultaneously and collaboratively. The MCCT problem is difficult to address because 1) comparative texts usually present different data formats and structures and thus they are hard to organize, and 2) there lacks an effective method to connect the semantically correlated comparative texts to facilitate clustering them in an unified way. To this aim, in this paper we propose a Heterogeneous Information Network-based Text clustering framework HINT. HINT first models multi-sourced texts (e.g. news and tweets) as heterogeneous information networks by introducing the shared ``anchor texts'' to connect the comparative texts. Next, two similarity matrices based on HINT as well as a transition matrix for cross-text-source knowledge transfer are constructed. Comparative texts clustering are then conducted by utilizing the constructed matrices. Finally, a mutual clustering algorithm is also proposed to further unify the separate clustering results of the comparative texts by introducing a clustering consistency constraint. We conduct extensive experimental on three tweets-news datasets, and the results demonstrate the effectiveness and robustness of the proposed method in addressing the MCCT problem.

Simon Stolarczyk, Manisha Bhardwaj, Kevin E. Bassler et al.

We consider model social networks in which information propagates directionally across layers of rational agents. Each agent makes a locally optimal estimate of the state of the world, and communicates this estimate to agents downstream. When agents receive information from the same source their estimates are correlated. We show that the resulting redundancy can lead to the loss of information about the state of the world across layers of the network, even when all agents have full knowledge of the network's structure. A simple algebraic condition identifies networks in which information loss occurs, and we show that all such networks must contain a particular network motif. We also study random networks asymptotically as the number of agents increases, and find a sharp transition in the probability of information loss at the point at which the number of agents in one layer exceeds the number in the previous layer.

Andre L. L. Aquino, Tamer S. G. Cavalcante, Eliana S. Almeida et al.

This work proposes the use of Information Theory for the characterization of vehicles behavior through their velocities. Three public data sets were used: i.Mobile Century data set collected on Highway I-880, near Union City, California; ii.Borlänge GPS data set collected in the Swedish city of Borlänge; and iii.Beijing taxicabs data set collected in Beijing, China, where each vehicle speed is stored as a time series. The Bandt-Pompe methodology combined with the Complexity-Entropy plane were used to identify different regimes and behaviors. The global velocity is compatible with a correlated noise with f^{-k} Power Spectrum with k >= 0. With this we identify traffic behaviors as, for instance, random velocities (k aprox. 0) when there is congestion, and more correlated velocities (k aprox. 3) in the presence of free traffic flow.

Steffen Limmer, Slawomir Stanczak

This paper addresses some aspects of the general problem of information transfer and distributed function computation in wireless networks. Many applications of wireless technology foresee networks of autonomous devices executing tasks that can be posed as distributed function computation. In today's wireless networks, the tasks of communication and (distributed) computation are performed separately, although an efficient network operation calls for approaches in which the information transfer is dynamically adapted to time-varying computation objectives. Thus, wireless communications and function computation must be tightly coupled and it is shown in this paper that information theory may play a crucial role in the design of efficient computation-aware wireless communication and networking strategies. This is explained in more detail by considering the problem of computing $\ell_p$-norms over multiple access channels.

Avishy Y. Carmi, Daniel Moskovich

We provide an axiomatic characterization of information fusion, on the basis of which we define an information fusion network. Our construction is reminiscent of tangle diagrams in low dimensional topology. Information fusion networks come equipped with a natural notion of equivalence. Equivalent networks `contain the same information', but differ locally. When fusing streams of information, an information fusion network may adaptively optimize itself inside its equivalence class. This provides a fault tolerance mechanism for such networks.

Charles Alexandre, Martin Bordemann, Salim Riviere et al.

In this paper we focus on a certain self-distributive multiplication on coalgebras, which leads to so-called rack bialgebra. Inspired by semi-group theory (adapting the Suschkewitsch theorem), we do some structure theory for rack bialgebras and cocommutative Hopf dialgebras. We also construct canonical rack bialgebras (some kind of enveloping algebras) for any Leibniz algebra and compare to the existing constructions. We are motivated by a differential geometric procedure which we call the Serre functor: To a pointed differentible manifold with multiplication is associated its distribution space supported in the chosen point. For Lie groups, it is well-known that this leads to the universal enveloping algebra of the Lie algebra. For Lie racks, we get rack-bialgebras, for Lie digroups, we obtain cocommutative Hopf dialgebras.

Giovanni Quattrone, Licia Capra, Pasquale De Meo et al.

Social (or folksonomic) tagging has become a very popular way to describe content within Web 2.0 websites. However, as tags are informally defined, continually changing, and ungoverned, it has often been criticised for lowering, rather than increasing, the efficiency of searching. To address this issue, a variety of approaches have been proposed that recommend users what tags to use, both when labeling and when looking for resources. These techniques work well in dense folksonomies, but they fail to do so when tag usage exhibits a power law distribution, as it often happens in real-life folksonomies. To tackle this issue, we propose an approach that induces the creation of a dense folksonomy, in a fully automatic and transparent way: when users label resources, an innovative tag similarity metric is deployed, so to enrich the chosen tag set with related tags already present in the folksonomy. The proposed metric, which represents the core of our approach, is based on the mutual reinforcement principle. Our experimental evaluation proves that the accuracy and coverage of searches guaranteed by our metric are higher than those achieved by applying classical metrics.

Abderrahim El Qadi, Driss Aboutajedine, Yassine Ennouary

In this paper we describe a mechanism to improve Information Retrieval (IR) on the web. The method is based on Formal Concepts Analysis (FCA) that it is makes semantical relations during the queries, and allows a reorganizing, in the shape of a lattice of concepts, the answers provided by a search engine. We proposed for the IR an incremental algorithm based on Galois lattice. This algorithm allows a formal clustering of the data sources, and the results which it turns over are classified by order of relevance. The control of relevance is exploited in clustering, we improved the result by using ontology in field of image processing, and reformulating the user queries which make it possible to give more relevant documents.

Pankaj Mehta, Sidhartha Goyal, Tao Long et al.

Bacteria communicate using secreted chemical signaling molecules called autoinducers in a process known as quorum sensing. The quorum-sensing network of the marine bacterium {\it Vibrio harveyi} employs three autoinducers, each known to encode distinct ecological information. Yet how cells integrate and interpret the information contained within the three autoinducer signals remains a mystery. Here, we develop a new framework for analyzing signal integration based on Information Theory and use it to analyze quorum sensing in {\it V. harveyi}. We quantify how much the cells can learn about individual autoinducers and explain the experimentally observed input-output relation of the {\it V. harveyi} quorum-sensing circuit. Our results suggest that the need to limit interference between input signals places strong constraints on the architecture of bacterial signal-integration networks, and that bacteria likely have evolved active strategies for minimizing this interference. Here we analyze two such strategies: manipulation of autoinducer production and feedback on receptor number ratios.

Ayfer Ozgur, Ramesh Johari, David Tse et al.

In analyzing the point-to-point wireless channel, insights about two qualitatively different operating regimes--bandwidth- and power-limited--have proven indispensable in the design of good communication schemes. In this paper, we propose a new scaling law formulation for wireless networks that allows us to develop a theory that is analogous to the point-to-point case. We identify fundamental operating regimes of wireless networks and derive architectural guidelines for the design of optimal schemes. Our analysis shows that in a given wireless network with arbitrary size, area, power, bandwidth, etc., there are three parameters of importance: the short-distance SNR, the long-distance SNR, and the power path loss exponent of the environment. Depending on these parameters we identify four qualitatively different regimes. One of these regimes is especially interesting since it is fundamentally a consequence of the heterogeneous nature of links in a network and does not occur in the point-to-point case; the network capacity is {\em both} power and bandwidth limited. This regime has thus far remained hidden due to the limitations of the existing formulation. Existing schemes, either multihop transmission or hierarchical cooperation, fail to achieve capacity in this regime; we propose a new hybrid scheme that achieves capacity.

Fabio G. Guerrero, Lucio A. Perez

In this paper, a tutorial software to learn Information Theory basics in a practical way is reported. The software, called IT-tutor-UV, makes use of a modern existing Spanish corpus for the modeling of the source. Both the source and the channel coding are also included in this educational tool as part of the learning experience. Entropy values of the Spanish language obtained with the IT-tutor-UV are discussed and compared to others that were previously calculated under limited conditions.

J. L. Vazquez-Bello

This paper reviews the covariant formalism of N=1, D=10 classical superparticle models. It discusses the local invariances of a number of superparticle actions and highlights the problem of finding a covariant quantization scenario. Covariant quantization has proved problematic, but it has motivated in seeking alternative approaches that avoids those found in earlier models. It also shows new covariant superparticle theories formulated in extended spaces that preserve certain canonical form in phase-space, and easy to quantize by using the Batalin-Vilkovisky procedure, as the gauge algebra of their constraints only closes on-shell. The mechanics actions describe particles moving in a superspace consisting of the usual $N=1$ superspace, together with an extra spinor or vector coordinate. A light-cone analysis shows that all these new superparticle models reproduce the physical spectrum of the N=1 super-Yang-Mills theory.

Tony T. Lee

All communication networks comprise of transmission systems and switching systems, even though they are usually treated as two separate issues. Communication channels are generally disturbed by noise from various sources. In circuit switched networks, reliable communication requires the error-tolerant transmission of bits over noisy channels. In packet switched networks, however, not only can bits be corrupted with noise, but resources along connection paths are also subject to contention. Thus, quality of service (QoS) is determined by buffer delays and packet losses. The theme of this paper is to show that transmission noise and packet contention actually have similar characteristics and can be tamed by comparable means to achieve reliable communication, and a number of analogies between switching and transmission are identified. The sampling theorem of bandlimited signals provides the cornerstone of digital communication and signal processing. Recently, the Birkhoff-von Neumann decomposition of traffic matrices has been widely applied to packet switches. With respect to the complexity reduction of packet switching, we show that the decomposition of a doubly stochastic traffic matrix plays a similar role to that of the sampling theorem in digital transmission. We conclude that packet switching systems are governed by mathematical laws that are similar to those of digital transmission systems as envisioned by Shannon in his seminal 1948 paper, A Mathematical Theory of Communication.