Hasil untuk "Architecture"

Kai Han, An Xiao, E. Wu et al.

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as"visual sentences"and present to further divide them into smaller patches (e.g., 4$\times$4) as"visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.

F. Jameson

Now in paperback, Fredric Jameson’s most wide-ranging work seeks to crystalize a definition of ”postmodernism”. Jameson’s inquiry looks at the postmodern across a wide landscape, from “high” art to “low” from market ideology to architecture, from painting to “punk” film, from video art to literature.

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados et al.

We focus on solving the univariate times series point forecasting problem using deep learning. We propose a deep neural architecture based on backward and forward residual links and a very deep stack of fully-connected layers. The architecture has a number of desirable properties, being interpretable, applicable without modification to a wide array of target domains, and fast to train. We test the proposed architecture on several well-known datasets, including M3, M4 and TOURISM competition datasets containing time series from diverse domains. We demonstrate state-of-the-art performance for two configurations of N-BEATS for all the datasets, improving forecast accuracy by 11% over a statistical benchmark and by 3% over last year's winner of the M4 competition, a domain-adjusted hand-crafted hybrid between neural network and statistical time series models. The first configuration of our model does not employ any time-series-specific components and its performance on heterogeneous datasets strongly suggests that, contrarily to received wisdom, deep learning primitives such as residual blocks are by themselves sufficient to solve a wide range of forecasting problems. Finally, we demonstrate how the proposed architecture can be augmented to provide outputs that are interpretable without considerable loss in accuracy.

Shennan Lu, Jiyao Wang, F. Chitsaz et al.

As NLM's Conserved Domain Database (CDD) enters its 20th year of operations as a publicly available resource, CDD curation staff continues to develop hierarchical classifications of widely distributed protein domain families, and to record conserved sites associated with molecular function, so that they can be mapped onto user queries in support of hypothesis-driven biomolecular research. CDD offers both an archive of pre-computed domain annotations as well as live search services for both single protein or nucleotide queries and larger sets of protein query sequences. CDD staff has continued to characterize protein families via conserved domain architectures and has built up a significant corpus of curated domain architectures in support of naming bacterial proteins in RefSeq. These architecture definitions are available via SPARCLE, the Subfamily Protein Architecture Labeling Engine. CDD can be accessed at https://www.ncbi.nlm.nih.gov/Structure/cdd/cdd.shtml.

Shikun Liu, Edward Johns, A. Davison

We propose a novel multi-task learning architecture, which allows learning of task-specific feature-level attention. Our design, the Multi-Task Attention Network (MTAN), consists of a single shared network containing a global feature pool, together with a soft-attention module for each task. These modules allow for learning of task-specific features from the global features, whilst simultaneously allowing for features to be shared across different tasks. The architecture can be trained end-to-end and can be built upon any feed-forward neural network, is simple to implement, and is parameter efficient. We evaluate our approach on a variety of datasets, across both image-to-image predictions and image classification tasks. We show that our architecture is state-of-the-art in multi-task learning compared to existing methods, and is also less sensitive to various weighting schemes in the multi-task loss function. Code is available at https://github.com/lorenmt/mtan.

Hui Li, Xiaojun Wu

In this paper, we present a novel deep learning architecture for infrared and visible images fusion problems. In contrast to conventional convolutional networks, our encoding network is combined with convolutional layers, a fusion layer, and dense block in which the output of each layer is connected to every other layer. We attempt to use this architecture to get more useful features from source images in the encoding process, and two fusion layers (fusion strategies) are designed to fuse these features. Finally, the fused image is reconstructed by a decoder. Compared with existing fusion methods, the proposed fusion method achieves the state-of-the-art performance in objective and subjective assessment.

Kuan Wang, Zhijian Liu, Yujun Lin et al.

Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. There are plenty of specialized hardware for neural networks, but little research has been done for specialized neural network optimization for a particular hardware architecture. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such as FLOPs and model size, we employ a hardware simulator to generate direct feedback signals (latency and energy) to the RL agent. Compared with conventional methods, our framework is fully automated and can specialize the quantization policy for different neural network architectures and hardware architectures. Our framework effectively reduced the latency by 1.4-1.95x and the energy consumption by 1.9x with negligible loss of accuracy compared with the fixed bitwidth (8 bits) quantization. Our framework reveals that the optimal policies on different hardware architectures (i.e., edge and cloud architectures) under different resource constraints (i.e., latency, energy and model size) are drastically different. We interpreted the implication of different quantization policies, which offer insights for both neural network architecture design and hardware architecture design.

Jonathan Shen, Ruoming Pang, Ron J. Weiss et al.

This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text. The system is composed of a recurrent sequence-to-sequence feature prediction network that maps character embeddings to mel-scale spectrograms, followed by a modified WaveNet model acting as a vocoder to synthesize time-domain waveforms from those spectrograms. Our model achieves a mean opinion score (MOS) of 4.53 comparable to a MOS of 4.58 for professionally recorded speech. To validate our design choices, we present ablation studies of key components of our system and evaluate the impact of using mel spectrograms as the conditioning input to WaveNet instead of linguistic, duration, and $F_{0}$ features. We further show that using this compact acoustic intermediate representation allows for a significant reduction in the size of the WaveNet architecture.

Brandon Amos, J. Kolter

This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. In this paper, we explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, we show that the method is capable of learning to play mini-Sudoku (4x4) given just input and output games, with no a priori information about the rules of the game; this highlights the ability of our architecture to learn hard constraints better than other neural architectures.

Hang Zhao, O. Gallo, I. Frosio et al.

Jason Ku, Melissa Mozifian, Jungwook Lee et al.

We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion on high resolution feature maps to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produce state of the art results on the KITTI 3D object detection benchmark [1] while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles. Code is available at

Michael McAuliffe, Michaela Socolof, Sarah Mihuc et al.

Aaron Meurer, Christopher P. Smith, Mateusz Paprocki et al.

SymPy is an open source computer algebra system written in pure Python. It is built with a focus on extensibility and ease of use, through both interactive and programmatic applications. These characteristics have led SymPy to become the standard symbolic library for the scientific Python ecosystem. This paper presents the architecture of SymPy, a description of its features, and a discussion of select domain specific submodules. The supplementary materials provide additional examples and further outline details of the architecture and features of SymPy.

J. Zhao, Michaël Mathieu, Yann LeCun

We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.

K. Makarova, D. Haft, R. Barrangou et al.

M. I. Jordan, R. Jacobs

We present a tree-structured architecture for supervised learning. The statistical model underlying the architecture is a hierarchical mixture model in which both the mixture coefficients and the mixture components are generalized linear models (GLIM's). Learning is treated as a maximum likelihood problem; in particular, we present an Expectation-Maximization (EM) algorithm for adjusting the parameters of the architecture. We also develop an on-line learning algorithm in which the parameters are updated incrementally. Comparative simulation results are presented in the robot dynamics domain.

Camila Pereira Cagna, Cássio Antonio Tormena, Caroline Honorato Rocha et al.

ABSTRACT In sandy soils, low organic carbon content and weak structural stability often constrain pore functionality, water availability, and gas exchange, highlighting the need for management strategies that improve soil physical quality. The aim of this study was to quantify the impact of cover crops on SOC, soil physical properties, and soil processes in a sandy loam dystrophic Ferralsol (Latossolo Vermelho-Amarelo Distrófico). The experiment followed a randomized complete block design composed of five treatments and five repetitions. The experimental treatments were: (1) Control (fallow plots subjected to weed desiccation), (2) G (single grass: Urochloa ruziziensis), (3) GG (two grasses intercropped: Pennisetum americanum + U. ruziziensis), (4) GL (one grass and one legume intercropped: P. americanum + Mucuna pruriens), and (5) MIX (two grasses and one legume intercropped: P. americanum + U. ruziziensis + M. pruriens). Undisturbed samples were collected from the layers of 0.00-0.10, 0.10-0.20, 0.20-0.40, and 0.40-0.60 m to determine physical indicators such as bulk density (Bd), total porosity (TP), field capacity (FC), permanent wilting point (PWP), plant-available water (PAW), and the soil water retention curve, pore size distribution, water storage capacity (FC/TP), saturated hydraulic conductivity (Ksat), air permeability (Kair), and pore continuity index (K1). Results demonstrated that, compared with the Control (absence of cover crops), GL (grass + legume), MIX (mixed species), and GG (grasses) improved air conductivity by 8, 3.5, and 2.9 times, and pore continuity by 5.8, 2.9, and 2.2 times, respectively. The MIX system led to a 39 % increase in SOC relative to the Control. Additionally, intercropping two grass species with one legume (MIX) and combining one grass with one legume (GL) resulted in a 19 % increase in plant-available water compared with the Control treatment. Importantly, these improvements in pore architecture occurred without significant changes in soil bulk density (Bd) or total porosity (TP), underscoring that cover crops can reorganize the pore network independently of mass–volume relationships. Combination of grasses and legumes (GL, MIX) has substantial potential to improve plant-available water and the overall soil physical quality of sandy soils.

Crizaldy Tugade, Jopeth Ramis

Human airways maintain homeostasis through intricate cellular interactomes combining secretome-mediated signalling and mechanotransduction feedback loops. This review presents the first unified map of bidirectional mechanobiology–secretome interactions between airway epithelial cells (AECs), smooth muscle cells (ASMCs), and chondrocytes. We unify a novel three-component regulatory architecture: epithelium functioning as environmental activators, smooth muscle as mechanical actuators, and cartilage as calcium-dependent regulators. Critical mechanotransduction pathways, particularly YAP/TAZ signalling and TRPV4 channels, directly couple matrix stiffness to cytokine release, creating a closed-loop feedback system. During development, ASM-driven FGF-10 signalling and peristaltic contractions orchestrate cartilage formation and epithelial differentiation through mechanically guided morphogenesis. In disease states, these homeostatic circuits become pathologically dysregulated; asthma and COPD exhibit feed-forward stiffness traps where increased matrix rigidity triggers YAP/TAZ-mediated hypercontractility, perpetuating further remodelling. Aberrant mechanotransduction drives smooth muscle hyperplasia, cartilage degradation, and epithelial dysfunction through sustained inflammatory cascades. This system-level understanding of airway cellular networks provides mechanistic frameworks for targeted therapeutic interventions and tissue engineering strategies that incorporate essential mechanobiological signalling requirements.

Pragun Jaswal, L. Hemanth Krishna, B. Srinivasu

This paper proposes an low power approximate multiplier architecture for deep neural network (DNN) applications. A 4:2 compressor, introducing only a single combination error, is designed and integrated into an 8x8 unsigned multiplier. This integration significantly reduces the usage of exact compressors while preserving low error rates. The proposed multiplier is employed within a custom convolution layer and evaluated on neural network tasks, including image recognition and denoising. Hardware evaluation demonstrates that the proposed design achieves up to 30.24% energy savings compared to the best among existing multipliers. In image denoising, the custom approximate convolution layer achieves improved Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) compared to other approximate designs. Additionally, when applied to handwritten digit recognition, the model maintains high classification accuracy. These results demonstrate that the proposed architecture offers a favorable balance between energy efficiency and computational precision, making it suitable for low-power AI hardware implementations.

Zhipeng Liao, Kunming Shao, Jiangnan Yu et al.

With powerful and integrative large language models (LLMs), medical AI agents have demonstrated unique advantages in providing personalized medical consultations, continuous health monitoring, and precise treatment plans. Retrieval-Augmented Generation (RAG) integrates personal medical documents into LLMs by an external retrievable database to address the costly retraining or fine-tuning issues in deploying customized agents. While deploying medical agents in edge devices ensures privacy protection, RAG implementations impose substantial memory access and energy consumption during the retrieval stage. This paper presents a hierarchical retrieval architecture for edge RAG, leveraging a two-stage retrieval scheme that combines approximate retrieval for candidate set generation, followed by high-precision retrieval on pre-selected document embeddings. The proposed architecture significantly reduces energy consumption and external memory access while maintaining retrieval accuracy. Simulation results show that, under TSMC 28nm technology, the proposed hierarchical retrieval architecture has reduced the overall memory access by nearly 50% and the computation by 75% compared to pure INT8 retrieval, and the total energy consumption for 1 MB data retrieval is 177.76 μJ/query.