This paper studies distributed Q-learning for Linear Quadratic Regulator (LQR) in a multi-agent network. The existing results often assume that agents can observe the global system state, which may be infeasible in large-scale systems due to privacy concerns or communication constraints. In this work, we consider a setting with unknown system models and no centralized coordinator. We devise a state tracking (ST) based Q-learning algorithm to design optimal controllers for agents. Specifically, we assume that agents maintain local estimates of the global state based on their local information and communications with neighbors. At each step, every agent updates its local global state estimation, based on which it solves an approximate Q-factor locally through policy iteration. Assuming decaying injected excitation noise during the policy evaluation, we prove that the local estimation converges to the true global state, and establish the convergence of the proposed distributed ST-based Q-learning algorithm. The experimental studies corroborate our theoretical results by showing that our proposed method achieves comparable performance with the centralized case.

Background

Hepatic arterial infusion (HAI) of oxaliplatin, leucovorin, and fluorouracil (FOLFOX) plus sorafenib has a more desirable effect versus sorafenib for hepatocellular carcinoma (HCC) patients with portal vein invasion. However, considering the high cost of hepatic arterial infusion of chemotherapy (HAIC), this study evaluated the cost-effectiveness of HAIC plus sorafenib (SoraHAIC) versus standard care for HCC patients from the Chinese health system perspective.

Methods

A Markov multi-state model was constructed to simulate the disease course and source consumption of SoraHAIC. Costs of primary therapeutic drugs were calculated based on the national bid price, and hepatic artery catheterization fee was collected from the Fujian Provincial Price Bureau. Clinical data, other costs, and utility values were extracted from references. Primary outcomes included life-years (LYs), quality-adjusted life years (QALYs) and incremental cost-effectiveness ratio (ICER). The robustness of model was verified by uncertainty sensitivity analyses.

Results

SoraHAIC gained 1.18 QALYs (1.68 LYs) at a cost of $65,254, while the effectiveness and cost of sorafenib were 0.52 QALYs (0.79 LYs) and $14,280, respectively. The ICER of SoraHAIC vs sorafenib was $77,132/QALY ($57,153/LY). Parameter that most influenced the ICER was utility of PFS state. The probabilistic sensitivity analysis (PSA) showed that SoraHAIC was not cost-effective in the WTP threshold of 3*Gross Domestic Product (GDP) per capita of China ($30,492/QALY). But about 38.8% of the simulations were favorable to SoraHAIC at the WTP threshold of 3*GDP per capita of Beijing ($72,000/QALY). When 3*GDP per capita of Fujian ($47,285/QALY) and Gansu Province ($14,595/QALY) were used as WTP threshold, the acceptability of SoraHAIC was 0.3% and 0%, respectively.

Conclusions

The study results indicated that SoraHAIC was not cost-effective in medium-, and low-income regions of China. In developed areas of China (Beijing), there was a 38.8% probability that the SoraHAIC regimen would be cost-effective.

The enhancement of re-endothelialisation is a critical therapeutic option for repairing injured blood vessels. Endothelial progenitor cells (EPCs) are the major source of cells that participate in endothelium repair and contribute to re-endothelialisation by reducing neointima formation after vascular injury. The over-expression of the inhibitor of differentiation or DNA binding 1 (Id1) significantly improved EPC proliferation. This study aimed to investigate the effects of Id1 on the phosphatidylinositol-3-kinase (PI3K)/Akt/nuclear factor kappa B (NFκB)/survivin signalling pathway and its significance in promoting EPC proliferation in vitro. Spleen-derived EPCs were cultured as previously described. Id1 was presented at low levels in EPCs, and was rapidly up-regulated by stimulation with vascular endothelial growth factor. We demonstrated that transient transfection of Id1 into EPCs activated the PI3K/Akt/NFκB/survivin signalling pathway and promoted EPC proliferation. The proliferation of EPCs was extensively inhibited by silencing of endogenous Id1, and knockdown of Id1 expression led to suppression of PI3K/Akt/NFκB/survivin signalling pathway in EPCs. In addition, blockade by the PI3K-specific inhibitor LY294002, Akt inhibitor, the NFκB inhibitor BAY 11-7082, the survivin inhibitor Curcumin, or the survivin inhibitor YM155 reduced the effects of Id1 transfection. These results suggest that the Id1/PI3K/Akt/NFκB/survivin signalling pathway plays a critical role in EPC proliferation. The Id1/PI3K/Akt/NFκB/survivin signalling pathway may represent a novel therapeutic target in the prevention of restenosis after vascular injury.

Cardiac stem cells (CSCs) can home to the infarcted area and regenerate myocardium. Stromal cell-derived factor-1α/C-X-C chemokine receptor type 4 (SDF-1α/CXCR4) axis is pivotal in inducing CSCs migration. However, the mechanisms remain unclear. This study set out to detect if SDF-1α promotes migration and engraftment of CSCs through the CXCR4/PI3K (phosphatidylinositol 3-kinase) pathway. In the in vitro experiment, c-kit+ cells were isolated from neonatal mouse heart fragment culture by magnetic cell sorting. Fluorescence-activated cell sorting results demonstrated that a few c-kit+ cells expressed CD45 (4.54%) and Sca-1 (2.58%), the hematopoietic stem cell marker. Conditioned culture could induce c-kit+ cells multipotent differentiation, which was confirmed by cardiac troponin I (cTn-I), α-smooth muscle actin (α-SMA), and von Willebrand factor (vWF) staining. In vitro chemotaxis assays were performed using Transwell cell chambers to detect CSCs migration. The results showed that the cardiomyocytes infected with rAAV1-SDF-1α-eGFP significantly increased SDF-1α concentration, 5-fold more in supernatant than that in the control group, and subsequently attracted more CSCs migration. This effect was diminished by administration of AMD3100 (10 µg/ml, CXCR4 antagonist) or LY294002 (20 µmol/L, PI3K inhibitor). In myocardial infarction mice, overexpression of SDF-1α in the infarcted area by rAAV1-SDF-1α-eGFP infection resulted in more CSCs retention to the infarcted myocardium, a higher percentage of proliferation, and reduced infarcted area which was attenuated by AMD3100 or ly294002 pretreatment. These results indicated that overexpression of SDF-1α enhanced CSCs migration in vitro and engraftment of transplanted CSCs and reduced infarcted size via CXCR4/PI3K pathway.

The proliferation, migration, and angiogenesis of endothelial progenitor cells (EPCs) play critical roles in postnatal neovascularization and re-endothelialization following vascular injury. Here we evaluated whether the over-expression of platelet-derived growth factor receptor-β (PDGFR-β) can enhance the PDGF-BB-stimulated biological functions of EPCs through the PDGFR-β/phosphoinositide 3-kinase (PI3K)/Akt signaling pathway. We first confirmed the expression of endogenous PDGFR-β and its plasma membrane localization in spleen-derived EPCs. We then demonstrated that the PDGFR-β over-expression in EPCs enhanced the PDGF-BB-induced proliferation, migration, and angiogenesis of EPCs. Using AG1295 (a PDGFR kinase inhibitor), LY294002 (a PI3K inhibitor), and sc-221226 (an Akt inhibitor), we further showed that the PI3K/Akt signaling pathway participates in the PDGF-BB-induced proliferation, migration, and angiogenesis of EPCs. In addition, the PI3K/Akt signaling pathway is required for PDGFR-β over-expression to enhance these PDGF-BB-induced phenotypes.

Rice paddy soil-associated microbiota participate in biogeochemical processes that underpin rice yield and soil sustainability, yet continental-scale biogeographic patterns of paddy soil microbiota remain elusive. The soil bacteria of four typical Chinese rice-growing regions were characterized and compared to those of nonpaddy soils. The paddy soil bacteria were significantly less diverse, with unique taxonomic and functional composition, and harbored distinct cooccurrence network topology. Both stochastic and deterministic processes shaped soil bacteria assembly, but paddy samples exhibited a stronger deterministic signature than nonpaddy samples. Compared to other environmental factors, climatic factors such as mean monthly precipitation and mean annual temperature described most of the variance in soil bacterial community structure. Cooccurrence network analysis suggests that the continental biogeographic variance in bacterial community structure was described by the competition between two mutually exclusive bacterial modules in the community. Keystone taxa identified in network models (Anaerolineales, Ignavibacteriae, and Deltaproteobacteria) were more sensitive to changes in environmental factors, leading us to conclude that environmental factors may influence paddy soil bacterial communities via these keystone taxa. Characterizing the uniqueness of bacterial community patterns in paddy soil (compared to nonpaddy soils) at continental scales is central to improving crop productivity and resilience and to sustaining agricultural soils. IMPORTANCE Rice fields provide food for over half of the world's human population. The ecology of paddy soil microbiomes is shaped by human activities, which can have a profound impact on rice yield, greenhouse gas emissions, and soil health. Investigations of the soil bacteria in four typical Chinese rice-growing regions showed that (i) soil bacterial communities maintain highly modularized species-to-species network structures; (ii) community patterns were shaped by the balance of integrated stochastic and deterministic processes, in which homogenizing selection and dispersal limitation dominate; and (iii) deterministic processes and climatic and edaphic factors influence community patterns mainly by their impact on highly connected nodes (i.e., keystone taxa) in networks. Characterizing the unique ecology of bacterial community patterns in paddy soil at a continental scale may lead to improved crop productivity and resilience, as well as sustaining agricultural soils.

The ESCAPED (Examining Sickle Cell Acute Pain in the Emergency vs Day Hospital) trial is an ongoing prospective study comparing outcomes of people with sickle cell disease (SCD) seeking care for acute pain management in either an emergency department or specialty infusion clinic. The objective of this paper is to describe the baseline characteristics and health care utilization of patients in the trial. This is a multicenter study across 4 US cities that enrolled all adults with SCD living within 60 miles (96.6 km) of a study site who were expected to have acute care utilization over the study period. Twenty-one percent of participants had no acute care visits in the first 12 months of follow-up. Using negative binomial regression, we describe subject characteristics that predict acute care utilization. Three hundred ninety-one subjects have completed 12 months of follow-up with a mean age of 34.5 years (standard deviation, 11.4), 60% are female. Fifty-four percent of subjects with hemoglobin SS disease and 46% with hemoglobin SC disease had 3 or more acute visits over the study period. The prevalence of chronic pain in this cohort was 68%. Predictors of higher rates of acute care utilization included being unemployed, having chronic pain, being on chronic transfusion therapy, having a history of stroke, and being on disability or on Medicaid. This is the first prospective cohort in the modern era, and it demonstrates much higher rates of acute care utilization than reported in the Cooperative Study of Sickle Cell Disease.

An amendment to this paper has been published and can be accessed via the original article.

Background

Paddy soil dissolved organic matter (DOM) represents a major hotspot for soil biogeochemistry, yet we know little about its chemodiversity let alone the microbial community that shapes it. Here, we leveraged ultrahigh-resolution mass spectrometry, amplicon, and metagenomic sequencing to characterize the molecular distribution of DOM and the taxonomic and functional microbial diversity in paddy soils across China. We hypothesized that variances in microbial community significantly associate with changes in soil DOM molecular composition.

Results

We report that both microbial and DOM profiles revealed geographic patterns that were associated with variation in mean monthly precipitation, mean annual temperature, and pH. DOM molecular diversity was significantly correlated with microbial taxonomic diversity. An increase in DOM molecules categorized as peptides, carbohydrates, and unsaturated aliphatics, and a decrease in those belonging to polyphenolics and polycyclic aromatics, significantly correlated with proportional changes in some of the microbial taxa, such as Syntrophobacterales, Thermoleophilia, Geobacter, Spirochaeta, Gaiella, and Defluviicoccus. DOM composition was also associated with the relative abundances of the microbial metabolic pathways, such as anaerobic carbon fixation, glycolysis, lignolysis, fermentation, and methanogenesis.

Conclusions

Our study demonstrates the continental-scale distribution of DOM is significantly correlated with the taxonomic profile and metabolic potential of the rice paddy microbiome. Abiotic factors that have a distinct effect on community structure can also influence the chemodiversity of DOM and vice versa. Deciphering these associations and the underlying mechanisms can precipitate understanding of the complex ecology of paddy soils, as well as help assess the effects of human activities on biogeochemistry and greenhouse gas emissions in paddy soils.