Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use AtomicView in the TxPool #1590

Merged
merged 55 commits into from
Jan 19, 2024
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
df6e1ae
Move storage traits implementation to the `fuel-core-storage` crate
xgreenx Dec 24, 2023
de82e25
Added comments to all newly added stuff. Made self-review and applied…
xgreenx Dec 26, 2023
7ccc722
Merge branch 'master' into feature/move-storage-implementation-to-own…
xgreenx Dec 26, 2023
33661bc
Updated CHANGELOG.md
xgreenx Dec 26, 2023
1b8295a
Merge remote-tracking branch 'origin/feature/move-storage-implementat…
xgreenx Dec 26, 2023
82df7e2
Moved insertion of the blocks into the `BlockImporter` instead of the…
xgreenx Dec 29, 2023
d4a4d0a
Update CHANGELOG.md
xgreenx Dec 29, 2023
5033d8c
Cleanup
xgreenx Dec 29, 2023
ab88c67
Moved insertion of the whole block with transactions
xgreenx Dec 29, 2023
a49d96d
Use `store_block` as a name instead of `block`
xgreenx Dec 30, 2023
c6fb1fe
Merge branch 'master' into feature/move-blocks-to-importer
xgreenx Jan 2, 2024
f5c38f2
Move `ChainId` to `fuel_core_importer::Config`
xgreenx Jan 5, 2024
4c3f18c
Apply suggestions from the PR
xgreenx Jan 5, 2024
0eaab98
Merge branch 'master' into feature/move-storage-implementation-to-own…
xgreenx Jan 5, 2024
c1ee5f8
Apply suggestions from the PR
xgreenx Jan 5, 2024
c594d45
Extract off chain logic from the executor
xgreenx Jan 4, 2024
bd60793
Added comments and linked todos
xgreenx Jan 5, 2024
8221443
Updated CHANGELOG.md
xgreenx Jan 5, 2024
810edee
Merge branch 'master' into feature/move-offchain-logic-from-executor
xgreenx Jan 5, 2024
4a580ed
Merge branch 'master' into feature/move-offchain-logic-from-executor
xgreenx Jan 6, 2024
0cea5bb
Merge branch 'master' into feature/move-storage-implementation-to-own…
xgreenx Jan 6, 2024
c9977a4
Merge branch 'master' into feature/move-storage-implementation-to-own…
xgreenx Jan 6, 2024
b0ed3e9
Fixed compilation
xgreenx Jan 6, 2024
541527e
Merge branch 'master' into feature/move-storage-implementation-to-own…
xgreenx Jan 6, 2024
c5a7b71
Merge branch 'master' into feature/move-offchain-logic-from-executor
xgreenx Jan 6, 2024
d2b5504
Use `BlockHeight` as a primary key for the `FuelsBlock` table
xgreenx Jan 6, 2024
1013f0e
Merge branch 'feature/move-storage-implementation-to-own-crate' into …
xgreenx Jan 6, 2024
359192e
Updated CHANGELOG.md
xgreenx Jan 6, 2024
34b5e6a
Merge remote-tracking branch 'origin/feature/block-height-as-primary-…
xgreenx Jan 6, 2024
e27996a
Fix compilation
xgreenx Jan 6, 2024
f3515d8
Merge branch 'master' into feature/move-offchain-logic-from-executor
xgreenx Jan 7, 2024
1be054c
Use `AtomicView` in the `TxPool`
xgreenx Jan 8, 2024
ea69532
Update CHANGELOG.md
xgreenx Jan 8, 2024
452418c
Merge remote-tracking branch 'origin/feature/move-storage-implementat…
xgreenx Jan 19, 2024
1fbc318
Merge branch 'master' into feature/move-storage-implementation-to-own…
xgreenx Jan 19, 2024
c5956a8
Use "blueprint" instead of "structure"
xgreenx Jan 19, 2024
cc9966c
Fix documents
xgreenx Jan 19, 2024
0bbeea1
Merge branch 'feature/move-storage-implementation-to-own-crate' into …
xgreenx Jan 19, 2024
2d3e471
Merge latest modifications from move storage PR
xgreenx Jan 19, 2024
f0df4f0
Merge branch 'master' into feature/block-height-as-primary-key
xgreenx Jan 19, 2024
688a24b
Merge branch 'master' into feature/move-offchain-logic-from-executor
xgreenx Jan 19, 2024
c7bce20
Fix conflicts
xgreenx Jan 19, 2024
0850b03
Merge branch 'feature/move-offchain-logic-from-executor' into feature…
xgreenx Jan 19, 2024
f607007
Merge branch 'feature/move-offchain-logic-from-executor' into feature…
xgreenx Jan 19, 2024
55424cb
Merge remote-tracking branch 'origin/feature/use-view-in-txpool' into…
xgreenx Jan 19, 2024
2b95d4e
Merge branch 'master' into feature/use-view-in-txpool
xgreenx Jan 19, 2024
993aa84
Merge branch 'master' into feature/block-height-as-primary-key
xgreenx Jan 19, 2024
36445b7
Merged master
xgreenx Jan 19, 2024
0e2abad
Merge branch 'master' into feature/block-height-as-primary-key
xgreenx Jan 19, 2024
6a503b9
Apply comments
xgreenx Jan 19, 2024
02a45a8
Merge branch 'master' into feature/use-view-in-txpool
xgreenx Jan 19, 2024
50f2ff0
Merge branch 'feature/block-height-as-primary-key' into feature/use-v…
xgreenx Jan 19, 2024
067792b
Apply comments
xgreenx Jan 19, 2024
6cead2b
Merge branch 'master' into feature/use-view-in-txpool
xgreenx Jan 19, 2024
8369cd5
Move arc wrapper into its own module
xgreenx Jan 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Description of the upcoming release here.
### Changed

- [#1591](https://github.com/FuelLabs/fuel-core/pull/1591): Simplify libp2p dependencies and not depend on all sub modules directly.
- [#1590](https://github.com/FuelLabs/fuel-core/pull/1590): Use `AtomicView` in the `TxPool` to read the state of the database during insertion of the transactions.
- [#1585](https://github.com/FuelLabs/fuel-core/pull/1585): Let `NetworkBehaviour` macro generate `FuelBehaviorEvent` in p2p
- [#1579](https://github.com/FuelLabs/fuel-core/pull/1579): The change extracts the off-chain-related logic from the executor and moves it to the GraphQL off-chain worker. It creates two new concepts - Off-chain and On-chain databases where the GraphQL worker has exclusive ownership of the database and may modify it without intersecting with the On-chain database.
- [#1577](https://github.com/FuelLabs/fuel-core/pull/1577): Moved insertion of sealed blocks into the `BlockImporter` instead of the executor.
Expand Down
12 changes: 6 additions & 6 deletions crates/fuel-core/src/graphql_api/api_service.rs
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
use crate::{
fuel_core_graphql_api::{
database::{
OffChainView,
OnChainView,
},
metrics_extension::MetricsExtension,
ports::{
BlockProducerPort,
ConsensusModulePort,
OffChainDatabase,
OnChainDatabase,
P2pPort,
TxPoolPort,
},
Expand Down Expand Up @@ -178,8 +176,10 @@ pub fn new_service<OnChain, OffChain>(
request_timeout: Duration,
) -> anyhow::Result<Service>
where
OnChain: AtomicView<OnChainView> + 'static,
OffChain: AtomicView<OffChainView> + 'static,
OnChain: AtomicView + 'static,
OffChain: AtomicView + 'static,
OnChain::View: OnChainDatabase,
OffChain::View: OffChainDatabase,
{
let network_addr = config.addr;
let combined_read_database = ReadDatabase::new(on_database, off_database);
Expand Down
64 changes: 58 additions & 6 deletions crates/fuel-core/src/graphql_api/database.rs
Original file line number Diff line number Diff line change
Expand Up @@ -57,25 +57,77 @@ pub type OnChainView = Arc<dyn OnChainDatabase>;
/// The off-chain view of the database used by the [`ReadView`] to fetch off-chain data.
pub type OffChainView = Arc<dyn OffChainDatabase>;

/// The GraphQL can't work with the generics in [`async_graphql::Context::data_unchecked`] and requires a known type.
/// It is an `Arc` wrapper around the generic for on-chain and off-chain databases.
struct ArcWrapper<Provider, ArcView> {
Copy link
Member

@Voxelot Voxelot Jan 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it would make sense to split this wrapper into a separate sub-module so it doesn't clutter the main database implementation?

inner: Provider,
_marker: core::marker::PhantomData<ArcView>,
}

impl<Provider, ArcView> ArcWrapper<Provider, ArcView> {
fn new(inner: Provider) -> Self {
Self {
inner,
_marker: core::marker::PhantomData,
}
}
}

impl<Provider, View> AtomicView for ArcWrapper<Provider, OnChainView>
where
Provider: AtomicView<View = View>,
View: OnChainDatabase + 'static,
{
type View = OnChainView;

fn view_at(&self, height: BlockHeight) -> StorageResult<Self::View> {
let view = self.inner.view_at(height)?;
Ok(Arc::new(view))
}

fn latest_view(&self) -> Self::View {
Arc::new(self.inner.latest_view())
}
}

impl<Provider, View> AtomicView for ArcWrapper<Provider, OffChainView>
where
Provider: AtomicView<View = View>,
View: OffChainDatabase + 'static,
{
type View = OffChainView;

fn view_at(&self, height: BlockHeight) -> StorageResult<Self::View> {
let view = self.inner.view_at(height)?;
Ok(Arc::new(view))
}

fn latest_view(&self) -> Self::View {
Arc::new(self.inner.latest_view())
}
}

/// The container of the on-chain and off-chain database view provides.
/// It is used only by `ViewExtension` to create a [`ReadView`].
pub struct ReadDatabase {
/// The on-chain database view provider.
on_chain: Box<dyn AtomicView<OnChainView>>,
on_chain: Box<dyn AtomicView<View = OnChainView>>,
/// The off-chain database view provider.
off_chain: Box<dyn AtomicView<OffChainView>>,
off_chain: Box<dyn AtomicView<View = OffChainView>>,
}

impl ReadDatabase {
/// Creates a new [`ReadDatabase`] with the given on-chain and off-chain database view providers.
pub fn new<OnChain, OffChain>(on_chain: OnChain, off_chain: OffChain) -> Self
where
OnChain: AtomicView<OnChainView> + 'static,
OffChain: AtomicView<OffChainView> + 'static,
OnChain: AtomicView + 'static,
OffChain: AtomicView + 'static,
OnChain::View: OnChainDatabase,
OffChain::View: OffChainDatabase,
{
Self {
on_chain: Box::new(on_chain),
off_chain: Box::new(off_chain),
on_chain: Box::new(ArcWrapper::new(on_chain)),
off_chain: Box::new(ArcWrapper::new(off_chain)),
}
}

Expand Down
20 changes: 19 additions & 1 deletion crates/fuel-core/src/service/adapters/graphql_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@ use crate::{
};
use async_trait::async_trait;
use fuel_core_services::stream::BoxStream;
use fuel_core_storage::Result as StorageResult;
use fuel_core_storage::{
transactional::AtomicView,
Result as StorageResult,
};
use fuel_core_txpool::{
service::TxStatusMessage,
types::TxId,
Expand Down Expand Up @@ -145,3 +148,18 @@ impl worker::BlockImporter for BlockImporterAdapter {
)
}
}

impl AtomicView for Database {
type View = Database;

fn view_at(&self, _: BlockHeight) -> StorageResult<Self::View> {
unimplemented!(
"Unimplemented until of the https://github.com/FuelLabs/fuel-core/issues/451"
)
}

fn latest_view(&self) -> Self::View {
// TODO: https://github.com/FuelLabs/fuel-core/issues/1581
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we should fix this TODO as well, now that the linked issue is no longer blocked. I'm fine with a follow-up PR as well.

self.clone()
}
}
24 changes: 3 additions & 21 deletions crates/fuel-core/src/service/adapters/graphql_api/off_chain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,9 @@ use crate::{
transactions::OwnedTransactionIndexCursor,
Database,
},
fuel_core_graphql_api::{
database::OffChainView,
ports::{
worker,
OffChainDatabase,
},
fuel_core_graphql_api::ports::{
worker,
OffChainDatabase,
},
};
use fuel_core_storage::{
Expand All @@ -18,7 +15,6 @@ use fuel_core_storage::{
IterDirection,
},
not_found,
transactional::AtomicView,
Error as StorageError,
Result as StorageResult,
};
Expand All @@ -36,7 +32,6 @@ use fuel_core_types::{
},
services::txpool::TransactionStatus,
};
use std::sync::Arc;

impl OffChainDatabase for Database {
fn owned_message_ids(
Expand Down Expand Up @@ -83,19 +78,6 @@ impl OffChainDatabase for Database {
}
}

impl AtomicView<OffChainView> for Database {
fn view_at(&self, _: BlockHeight) -> StorageResult<OffChainView> {
unimplemented!(
"Unimplemented until of the https://github.com/FuelLabs/fuel-core/issues/451"
)
}

fn latest_view(&self) -> OffChainView {
// TODO: https://github.com/FuelLabs/fuel-core/issues/1581
Arc::new(self.clone())
}
}

impl worker::OffChainDatabase for Database {
fn record_tx_id_owner(
&mut self,
Expand Down
30 changes: 6 additions & 24 deletions crates/fuel-core/src/service/adapters/graphql_api/on_chain.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,11 @@
use crate::{
database::Database,
fuel_core_graphql_api::{
database::OnChainView,
ports::{
DatabaseBlocks,
DatabaseChain,
DatabaseContracts,
DatabaseMessages,
OnChainDatabase,
},
fuel_core_graphql_api::ports::{
DatabaseBlocks,
DatabaseChain,
DatabaseContracts,
DatabaseMessages,
OnChainDatabase,
},
};
use fuel_core_storage::{
Expand All @@ -18,7 +15,6 @@ use fuel_core_storage::{
IterDirection,
},
not_found,
transactional::AtomicView,
Error as StorageError,
Result as StorageResult,
};
Expand All @@ -36,7 +32,6 @@ use fuel_core_types::{
},
services::graphql_api::ContractBalance,
};
use std::sync::Arc;

impl DatabaseBlocks for Database {
fn block_id(&self, height: &BlockHeight) -> StorageResult<BlockId> {
Expand Down Expand Up @@ -125,16 +120,3 @@ impl DatabaseChain for Database {
}

impl OnChainDatabase for Database {}

impl AtomicView<OnChainView> for Database {
fn view_at(&self, _: BlockHeight) -> StorageResult<OnChainView> {
unimplemented!(
"Unimplemented until of the https://github.com/FuelLabs/fuel-core/issues/451"
)
}

fn latest_view(&self) -> OnChainView {
// TODO: https://github.com/FuelLabs/fuel-core/issues/1581
Arc::new(self.clone())
}
}
5 changes: 0 additions & 5 deletions crates/fuel-core/src/service/adapters/txpool.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ use fuel_core_types::{
UtxoId,
},
fuel_types::{
BlockHeight,
ContractId,
Nonce,
},
Expand Down Expand Up @@ -139,8 +138,4 @@ impl fuel_core_txpool::ports::TxPoolDb for Database {
fn is_message_spent(&self, id: &Nonce) -> StorageResult<bool> {
self.storage::<SpentMessages>().contains_key(id)
}

fn current_block_height(&self) -> StorageResult<BlockHeight> {
self.latest_height()
}
}
2 changes: 2 additions & 0 deletions crates/fuel-core/src/service/sub_services.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ pub fn init_sub_services(
let last_block = database.get_current_block()?.ok_or(anyhow::anyhow!(
"The blockchain is not initialized with any block"
))?;
let last_height = *last_block.header().height();
#[cfg(feature = "relayer")]
let relayer_service = if let Some(config) = &config.relayer {
Some(fuel_core_relayer::new_service(
Expand Down Expand Up @@ -140,6 +141,7 @@ pub fn init_sub_services(
database.clone(),
importer_adapter.clone(),
p2p_adapter.clone(),
last_height,
);
let tx_pool_adapter = TxPoolAdapter::new(txpool.shared.clone());

Expand Down
1 change: 1 addition & 0 deletions crates/services/txpool/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ tracing = { workspace = true }
[dev-dependencies]
fuel-core-trace = { path = "./../../trace" }
fuel-core-txpool = { path = "", features = ["test-helpers"] }
fuel-core-types = { path = "../../types", features = ["test-helpers"] }
itertools = { workspace = true }
mockall = { workspace = true }
proptest = { workspace = true }
Expand Down
19 changes: 16 additions & 3 deletions crates/services/txpool/src/mock_db.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
use crate::ports::TxPoolDb;
use fuel_core_storage::Result as StorageResult;
use fuel_core_storage::{
transactional::AtomicView,
Result as StorageResult,
};
use fuel_core_types::{
entities::{
coins::coin::{
Expand Down Expand Up @@ -91,8 +94,18 @@ impl TxPoolDb for MockDb {
fn is_message_spent(&self, id: &Nonce) -> StorageResult<bool> {
Ok(self.data.lock().unwrap().spent_messages.contains(id))
}
}

pub struct MockDBProvider(pub MockDb);

impl AtomicView for MockDBProvider {
type View = MockDb;

fn view_at(&self, _: BlockHeight) -> StorageResult<Self::View> {
Ok(self.latest_view())
}

fn current_block_height(&self) -> StorageResult<BlockHeight> {
Ok(Default::default())
fn latest_view(&self) -> Self::View {
self.0.clone()
}
}
3 changes: 0 additions & 3 deletions crates/services/txpool/src/ports.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ use fuel_core_types::{
UtxoId,
},
fuel_types::{
BlockHeight,
ContractId,
Nonce,
},
Expand Down Expand Up @@ -55,6 +54,4 @@ pub trait TxPoolDb: Send + Sync {
fn message(&self, message_id: &Nonce) -> StorageResult<Option<Message>>;

fn is_message_spent(&self, message_id: &Nonce) -> StorageResult<bool>;

fn current_block_height(&self) -> StorageResult<BlockHeight>;
}
Loading
Loading