arxiv arXiv cs.LG · 6h ago

Diffusion Integrated Gradients: Controllable Path Generation for Flexible Feature Attribution

The authors propose Diffusion Integrated Gradients (DiffIG), a novel method that reformulates path generation as a conditional generative modeling problem to address limitations in existing attribution techniques. While integrated gradients are widely used, their reliance on fixed or hand-crafted paths often results in noisy or distorted attributions. To solve this, DiffIG trains a diffusion model to learn a distribution over paths derived from a Stick-Breaking Process. The method then employs guided sampling to allow for the embedding of user guidance during the inference-time sampling procedure. This approach enables flexible and controllable feature attribution by treating path selection as a generative task rather than a static choice. Experimental results demonstrate that DiffIG quantitatively matches or outperforms existing path-based methods in terms of attribution quality. Furthermore, the generated explanations are shown to be perceptually aligned with human expectations. The work introduces a new generative perspective for Explainable Artificial Intelligence that supports dynamic control over explanation paths.

arxiv arXiv cs.LG · 6h ago

First Finite-Time Analysis of Classical Adam for Nonsmooth Nonconvex Optimization

This study presents the first finite-time convergence analysis for the classical Adam optimizer, specifically addressing its behavior in nonsmooth nonconvex optimization settings. Previous research largely ignored Adam's bias-correction term or required extra algorithmic modifications like clipping, leaving the original method's guarantees unclear. The authors utilize the Online-to-Nonconvex Conversion framework to prove that a randomly scaled learning rate ensures a convergence rate of $1/T^{ rac{2}{13}}$. This theoretical result is significant because it applies to the modern heavy-tailed noise regime, which more closely reflects practical training conditions. Furthermore, the analysis establishes convergence under the parameter choice where $β_1=β_2$, aligning with recent empirical observations. These findings provide a rigorous explanation for Adam's effectiveness in real-world scenarios that were previously inadequately captured by smooth optimization theories.

arxiv arXiv cs.LG · 6h ago

Boundary-Aware Curriculum RL Expands LLM Reasoning Capacity Beyond Base Model Limits

The authors argue that mainstream Reinforcement Learning with Verifiable Rewards (RLVR) often fails to expand the reasoning capacity of large language models, merely reallocating probabilities among existing trajectories. To address this limitation, they introduce a boundary-aware Curriculum RL approach designed to move beyond the base model's empirical reasoning capacity boundary. The method first utilizes pass@k sampling to identify the current reasoning limits and then applies targeted teacher guidance to examples near or beyond that boundary. Reinforcement learning is subsequently used to consolidate these newly introduced reasoning patterns across Qwen, Llama, and DeepSeek base models. Experimental results demonstrate significant improvements in both pass@1 scores and pass@256 scores, which serve as a proxy for the reasoning capacity boundary. Specifically, average pass@256 improved by 9.8 percentage points over the base models and by 10.3 percentage points over Vanilla RLVR. These findings suggest that this curriculum-based strategy offers a scalable route for continuously improving LLM reasoning capabilities.

arxiv arXiv cs.LG · 7h ago

Attention Sinks and Collapse Are Universal Consequences of Content-Based Routing

The study demonstrates that attention sinks, representation collapse, and norm stratification are not unique to transformer architectures but are inherent consequences of content-based routing under a fixed similarity metric. It establishes an identity showing softmax attention functions as Boltzmann-weighted aggregation over Euclidean distances with constant key norms, rendering it blind to key magnitude due to the omission of a specific norm term. This framework predicts that any router utilizing a metric ill-matched to its representations will compensate by concentrating routing and collapsing the routed representations. The authors validate this hypothesis across diverse models including nine pretrained transformers, graph attention networks, selective state-space models, recurrent mixers, and learned residual layers. Experimental results confirm that all tested architectures exhibit this identical signature of collapse regardless of their specific domain or structure. Furthermore, within-model ablations isolate the routing mechanism as the primary cause rather than incidental training dynamics. The onset of this phenomenon is shown to be contingent on the strength of the positional brake accompanying the content score, which can shift the effect across its range. However, the underlying mechanism remains invariant and does not require norm stratification, as routers with norm-normalized keys exhibit the same concentration behavior.

arxiv arXiv cs.LG · 7h ago

No Reference-Free Generalization in Quantum Machine Learning

This study addresses the identifiability problem in quantum machine learning where training data lacks a preferred basis or reference frame. The authors formulate supervised learning without an external quantum reference frame, requiring classifiers to preserve unitary symmetries unbroken by the training data. They prove that if training states do not span the full Hilbert space, all pure states orthogonal to this span receive identical predictions. This limitation arises from missing reference information rather than state discrimination or computational constraints. The research establishes a robust version under weak symmetry breaking and shows that learning generic concepts requires exponentially many oriented training directions. Numerical illustrations visualize the resulting prediction collapse and its controlled relaxation. The results identify feature maps, measurement bases, and diverse training states as essential operational resources for generalization.

arxiv arXiv cs.LG · 7h ago

Null-Calibrated Conformal Selection via Target-Membership Scores

The article introduces Null-Calibrated Conformal Selection (NCCS), a method that utilizes target-membership probability scores to identify test candidates within a target region while controlling the false discovery rate. The authors argue that these membership scores provide a more natural ranking for selection tasks than conventional prediction-oriented nonconformity scores, particularly for complex targets. This distinction is critical for interval-valued, variance-driven, multimodal, or multi-condition targets where traditional scores may be misaligned with selection power. NCCS ranks test scores against confirmed non-target calibration examples to yield finite-sample valid null p-values under null exchangeability. These p-values can be combined with the Benjamini-Yekutieli procedure under arbitrary dependence or the Benjamini-Hochberg procedure under standard positive-dependence conditions. Experiments demonstrate that membership scores match conventional scores on mean-monotone targets but substantially improve performance on variance-driven targets. In rare-target regimes, NCCS trades power for finite-sample null validity, addressing issues where direct empirical-FDP thresholding can be anti-conservative.

arxiv arXiv cs.LG · 7h ago

Flow Annealing Posterior Sampling for Function-Space Regression and Inverse Problems

The authors introduce Flow Annealing Posterior Sampling (FAPS), a novel framework that unifies stochastic-process regression with PDE inverse problems in function space. Built upon pretrained function-space flow-matching priors, FAPS facilitates likelihood-guided posterior inference using sparse and noisy observations. The method supports variable query discretizations and avoids the need for explicit prior-density evaluation during sampling. It employs a Langevin correction mechanism that utilizes a low-rank covariance preconditioner to exploit dominant function-space correlations across different discretizations. Benchmarks on both Gaussian and non-Gaussian stochastic processes demonstrate that FAPS produces coherent posterior samples with accurate uncertainty quantification. The approach significantly outperforms existing functional regression baselines in these standard tasks. Furthermore, it achieves competitive or superior performance in noisy PDE inverse problems compared to diffusion-based samplers while reducing test-time sampling costs.

media r/LocalLLaMA · 8h ago

Backtrack Sampler and Verifier Drastically Improve Tiny Model Coding Performance

A new backtrack sampler combined with a verifier model significantly enhances the coding performance of tiny 0.5B parameter models, potentially making them competitive with larger 2-4B class models without weight changes. The approach theoretically addresses hallucination issues in large models by correcting errors during generation through re-sampling. However, this method incurs a 5-30% decode speed penalty due to the need for backward passes and requires training a verifier model of similar size to the original. This requirement doubles VRAM usage and increases compute demands by 1.5 to 3 times compared to standard inference. Despite these costs, the verifier generalizes across models of equal or lower weight classes if trained on diverse data distributions. Training the verifier is highly efficient, requiring only approximately 0.01% of the token size used for full pre-training.

arxiv arXiv cs.CL · 9h ago

Weave of Formal Thought: Uniting Rigorous Syntactic Validation with Learned Structural Representations

The authors introduce Weave of Formal Thought (WoFT), a paradigm combining rigorous syntactic validation with learned structural representations for code generation. The approach utilizes a formal engine and constrained decoder that is sound and complete regarding the full Tree-sitter specification. By augmenting generalized LR parsing with speculative lexing, the system maintains concurrent lexer-state hypotheses to admit valid program prefixes while rejecting invalid ones. Additionally, WoFT employs latent-variable fine-tuning to train models to interleave non-terminal grammar symbols directly into the generation process. This method uses the reweighted wake-sleep algorithm to optimize the importance-weighted evidence lower bound of the surface text. The model learns to selectively retain formal derivations as an adaptive structural scratchpad during inference. Experiments on Python show that fine-tuning StarCoder2-3B with this objective reduces per-token cross-entropy by 14.3% compared to a text-only baseline.

lab OpenAI News · 9h ago

OpenAI Research Shows AI Agents Transforming Work

A new research paper from OpenAI demonstrates how artificial intelligence agents are fundamentally changing the nature of work. The study highlights the capability of these agents to execute longer and more complex tasks than previously possible. This technological advancement is credited with expanding productivity across a wide variety of professional roles. The findings suggest a significant shift in how labor is organized and performed through automation. By handling intricate workflows, AI agents are enabling users to achieve greater efficiency. The paper serves as evidence of the growing impact of autonomous systems on modern employment.

arxiv arXiv cs.CL · 10h ago

Tatoxa: A Novel Text Detoxification System for Low-Resource Tatar

The paper introduces Tatoxa, a state-of-the-art system designed for automated text detoxification in the low-resource language of Tatar. This work addresses the lack of research attention given to abusive content mitigation in languages with limited digital resources. The authors present a new dataset specifically created for fine-tuning and evaluating detoxification models within these constrained settings. Comparative experiments demonstrate that Tatoxa outperforms both existing open-source and proprietary commercial large language models on key quality metrics. Furthermore, the study investigates cross-lingual transfer capabilities to assess the viability of using data from other languages. Results indicate that training on native Tatar data is significantly more effective than transferring knowledge from culturally close languages like Russian. Even when a large Russian corpus is available, cross-lingual approaches perform worse than models trained exclusively on native Tatar text.

arxiv arXiv cs.CL · 10h ago

Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoning

The study addresses the threat of training-time data poisoning during fine-tuning for abstractive text summarization models. Adversaries manipulate small task-specific datasets to induce persistent summarization failures while maintaining standard evaluation metrics. A unified post-hoc defense framework is proposed to detect and remediate poisoning across the machine learning supply chain. In white-box settings, detection relies on influence-function analysis identifying abnormally high training influence in poisoned pairs. Black-box defenses utilize behavioral auditing based on increased sensitivity to semantics-preserving perturbations. The authors introduce novel attacks targeting factual distortion and representational bias that evade conventional alarms. Experiments across nine architectures and six benchmarks show 85-92% detection precision for the proposed defenses. Gradient-ascent unlearning restores up to 96% of original behavior with less than 0.6% ROUGE degradation.

arxiv arXiv cs.CL · 10h ago

Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

A study identifies 'natural ungrokking,' a phenomenon where small language models lose learned grammatical rules midway through pretraining despite the evidence remaining in the data. Researchers observed that a model learning pronoun-gender agreement with Sue collapsed from 0.94 accuracy to near zero by step 3,500 without any corresponding spike in the loss curve. The survival of these rules is determined by support frequency within the training stream, while the data-to-parameter ratio only modulates the depth of the collapse. This emergence-then-collapse dynamic was replicated across multiple corpora, budgets, and seeds, and confirmed in public Pythia checkpoints where collapse depth correlated with model scale. The forgetting process acts as a displacement mechanism where a competing surface pattern out-competes the rule, causing the log-probability margin to cross zero within 100 steps of behavioral failure. Control over this fate is asymmetric; while injecting counter-evidence can destroy rules via a monotone dose-response, restoring support even at 450 times the sustaining level fails to recover them.

arxiv arXiv cs.CL · 10h ago

Keyword Lexicon Blindness Distorts Rhetorical Stance Measurement

A study analyzing 85 interviews with four public intellectuals reveals that keyword-based scoring can produce statistical artifacts regarding rhetorical stance. Initial analysis showed a robust negative-affect and emphatic-certainty co-occurrence pattern with high correlation coefficients ranging from r = 0.72 to 0.93. However, replacing this method with LLM-based zero-shot semantic classification on the full diarized corpus of 32,625 sentences significantly reduced these correlations. For instance, Dalio's correlation dropped from 0.851 to 0.206, while other speakers exhibited negative or null relationships between negativity and certainty. In contrast, the LLM analysis revealed a strong coupling between negative sentiment and hedging language, aligning with conventional expectations of pessimistic discourse. The discrepancy stems from three structural failures in keyword lexicons: syntactic blindness, polysemy blindness, and categorical absence. These flaws can invert semantic meaning, such as scoring 'never absolutely totally confident' as high certainty. The authors argue that keyword counts measure lexical co-occurrence tendencies rather than epistemic certainty, constituting a category error.

arxiv arXiv cs.CL · 11h ago

Structuring an Arabic-English Machine-Readable Dictionary Using Parsing Expression Grammars

This paper presents a method for structuring a machine-readable version of the Arabic-English Al-Mawrid dictionary, addressing the lack of standardization in printed formats. The approach converts unstructured streams of words and punctuation into explicit hierarchical structures that define entry components such as subentries, domain labels, and translation equivalences. Parsing serves as the central step within a cascaded design, implemented using the parsing expression grammars formalism. This technique allows for the automatic or semi-automatic organization of dictionary entries despite the absence of microstructure standardization in Arabic dictionaries. The study demonstrates that inducing microstructure enables plausible accuracy in structuring these complex lexical resources. By transforming raw text into defined formats, the work supports downstream natural language processing applications requiring machine-readable lexical data.

arxiv arXiv cs.CL · 11h ago

Automatic Generation of Highlights for Academic Paper Via Prompt-based Learning

This study investigates prompt-based learning for the automatic generation of academic paper highlights to address the lack of labeled training data in existing supervised methods. The researchers designed task-specific prompt templates combined with paper abstracts as inputs for several language models, including locally deployed GPT-2 and T5, as well as ChatGPT accessed via API. Experiments conducted on three datasets demonstrated that ChatGPT with prompt templates achieved performance comparable to previous supervised methods without requiring task-specific training samples. When a small number of examples were added to the prompts, the model significantly outperformed state-of-the-art methods on two of the datasets. The analysis revealed that while ChatGPT possesses strong language modeling capabilities, its performance is highly sensitive to the specific information provided within the prompt. Case studies indicated that the generated highlights are generally coherent, informative, and closely resemble those written by authors. This approach does not rely on domain-specific training corpora, supporting downstream text mining and bibliometric research for papers lacking existing highlights.

arxiv arXiv cs.CL · 12h ago

Measuring Research Difficulty in NLP: An Inverted U-Shaped Relationship with Academic Impact

This study proposes a comprehensive evaluation system for measuring the difficulty of academic research, focusing on Natural Language Processing as a case study. The authors extract internal and external features from papers, including collaboration, content, and references, to compute multiple difficulty indicators. These indicators are weighted using the entropy weight method and summed to generate a final research difficulty score. Academic impact is quantified by citation frequency, while expert assessments validate the reliability of the measurement approach. Empirical results indicate that page count, reference count, and high-level institutional participation significantly correlate with academic impact. Crucially, the analysis reveals an inverted U-shaped relationship between research difficulty and impact. This suggests that moderately difficult research tends to achieve the highest level of academic influence.

arxiv arXiv cs.CL · 12h ago

Data-Driven Evolution of Library and Information Science Research Methods (1990-2022)

This study analyzes the influence of data-centric research on Library and Information Science by examining methodological evolution from 1990 to 2022. Researchers automatically extracted four key categories of data-driven entities from academic papers: algorithms and models, data resources, software and tools, and metrics. The analysis evaluates trends across three dimensions, including temporal characteristics, topic-specific evolution, and cross-method features. Findings identify data resources as the primary driver of methodological changes within the discipline. The research reveals a cyclical pattern characterized by emergence followed by stability or practical application in LIS methods. This perspective highlights how big data advancements have reshaped the field's technical landscape over three decades.

arxiv arXiv cs.CL · 12h ago

iLLaDA: An 8B Masked Diffusion Language Model with Fully Bidirectional Attention

The authors introduce iLLaDA, an 8B parameter masked diffusion language model trained from scratch using fully bidirectional attention. This approach contrasts with the predominant autoregressive factorization and causal attention used in modern large language models. The model's pre-training scaled to 12 trillion tokens, followed by supervised fine-tuning on a 25 billion-token instruction corpus for 12 epochs. iLLaDA maintains the masked diffusion objective throughout both training phases and employs variable-length generation for efficiency. It also introduces confidence-based scoring to enhance performance on multiple-choice evaluation tasks. Benchmark results show significant improvements over its predecessor, LLaDA, including gains of 21.6 points on BBH and 14.9 points on ARC-Challenge for the base model. The instruction-tuned variant achieved increases of 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive nature, iLLaDA remains competitive with Qwen2.5 7B across several metrics.

arxiv arXiv cs.CL · 12h ago

Local Branch Routing: Efficient Trainable Test-Time Scaling for Language Models

The authors introduce Local Branch Routing (LBR), a token-level framework designed to improve language model reasoning through efficient test-time scaling. LBR expands a small local lookahead tree and forwards all sampled branches through the model, using a lightweight router to select the depth-1 subtree for commitment. This approach allows each token decision to utilize evidence from candidate local futures without incurring the computational costs of full solution-level search. The method employs a prune-shift-grow decoding process that preserves discrete branch identities and defines a tractable tree-trajectory likelihood. Consequently, LBR enables end-to-end reinforcement learning with verifiable rewards, jointly optimizing the base model and router under the same likelihood-ratio principle as discrete-token RLVR. Experimental results on synthetic hierarchical-planning tasks demonstrate that post-candidate hidden states provide useful routing evidence. Furthermore, benchmarks in mathematical reasoning show that LBR improves both Pass@1 and Pass@32 metrics compared to discrete chain-of-thought and other baselines.