Bo QIU
d3a2118985
Sync file from neighbors by default ( #295 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* sync from neighbors by default
* change p2p protocol version
* minor fix for crash python test
* Fix pubsub message id issue
* fix python tests
2024-12-09 16:04:55 +08:00
Bo QIU
6b2420cac4
Refactor find file ( #294 )
...
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* refactor find file gossip
* refactor find chunks gossip
* refactor announce file gossip
* fix announce file issue
* refactor find chunks gossip
2024-12-06 14:54:01 +08:00
Bo QIU
afa471e9ae
Refactor file sync p2p protocol to NEW_FILE + ASK_FILE + ANSWER_FILE ( #293 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* refactor new file pubsub message
* extract common struct ShardedFile
* refactor file sync to ASK_FILE & ANSWER_FILE protocol
* simplify code
2024-12-06 10:04:12 +08:00
Bo QIU
c5ddcc1f17
Refactor announce shard config pubsub message ( #292 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* Refactor announce shard config pubsub message
* change topic name for announce shard config
* minor change
2024-12-05 15:59:29 +08:00
Bo QIU
b9e6431a4d
Add shard config in STATUS message and only dail to shard config matched peers ( #285 )
...
* Add shard config in status message
* verify shard config for status message
* Notify peer connected to sync layer after status message exchanged
* Do not dial to shard config mismatched peers
* Upgrade network protocol version
* disconnect peer instead of ban peer if shard config mismatch
* Add python test for TCP connection by shard config
2024-11-25 10:15:30 +08:00
Bo QIU
1de7afec14
Add more metrics for network unbounded channel ( #264 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* Add metrics for file finalization in chunk pool
* Add metrics for network unbounded channel
2024-11-12 17:25:49 +08:00
0g-peterzhb
fae2d5efb6
@peter/db split ( #262 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* do not pad tx during sync phase
* add pad data store
* support async padding after sync phase
* split misc
* add sleep for the next loop
2024-11-08 22:06:45 +08:00
Bo QIU
9b68a8b7d7
Implement file sync protocol V2 ( #249 )
...
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Add new P2P protocol NewFile
* Publish NewFile message when any file finalized
* handle NewFile message in router
* handle NewFile in sync servic to write in db
* use propagation source to handle NewFile message
* Disable sequential sync and store new file in v2 sync store
* Add shard config in FindFile
* Add AnnounceFile RPC message in network layer
* do not propagate FindFile to whole network
* Mark peer connected if FileAnnouncement RPC message received
* fix unit test failures
* Change P2P protocol version
* Ignore py tests of sequential auto sync
* Add py test for auto sync v2
* fmt code
* remove dummy code in py test
* fix random test failure
* Add comments
* Enable file sync protocol v2 in config file by default
2024-10-28 14:56:08 +08:00
Bo QIU
e701c8fdbd
Supports custom public ip to announce file ( #233 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* Supports custom public ip to announce file
* Fix comment
2024-10-14 14:57:42 +08:00
0g-peterzhb
20266e0a6c
@peter/async padding ( #219 )
...
* support async padding
2024-10-08 13:13:32 +08:00
Bo QIU
9cde84ae15
Refactor network peer db configs ( #209 )
2024-09-24 11:59:34 +08:00
Bo QIU
6685757a03
notify file announcement to sync layer only if shard config matches ( #174 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* notify file announcement to sync layer only if shard config matches
* use FromStr trait
2024-08-29 18:35:35 +08:00
Bo QIU
2fd9712d59
Enhance P2P network protocol to support batch messages for performance concern ( #173 )
...
* Add p2p protocol version in network identity
* Cache annouce file pubsub messages to publish in batch
* fix file location cache
* opt sync metrics
* opt file location cache default configs
* publish files announcements in batch
* enhance announce file pubsub msg metrics
* opt metrics
* fix ci
* fix clippy
* fix batcher
* minor fix
* opt batcher: publish all if expired
2024-08-29 09:55:24 +08:00
Bo QIU
f14a1b5975
add timestamp in p2p rpc request id for latency stat ( #167 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* add timestamp in p2p rpc request id for latency stat
* fix test compilation error
2024-08-22 14:34:10 +08:00
Bo QIU
82fef674ed
Add more metrics for file sync ( #164 )
...
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* add metrics in log sync package
* udpate auto sync metrics
* Add metrics for completed file sync
* add more metrics for serial file sync
* adjust default timeout value for auto sync
* fix metrics rpc for Timer type
* add metrics for channel
* refactor channel metrics
* add timeout metrics for segment sync
* refactor channel receiver
2024-08-22 10:42:15 +08:00
Bo QIU
c1f465e009
Do not verify announced ip address by default ( #163 )
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-08-20 14:39:58 +08:00
Bo QIU
f9120b1e4a
Add metrics for router and auto sync ( #161 )
...
* Add metrics configurations
* Add metrics in router package
* Add catch up info in sync service state
* Add metrics for auto sync
* update cargo lock
2024-08-19 09:54:52 +08:00
Bo QIU
22ed8f5f91
opt router debug logs ( #160 )
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-08-16 10:34:28 +08:00
peilun-conflux
12d0c6b675
Check network id in status and ban incompatible peers. ( #159 )
...
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Check network id in status and ban incompatible peers.
* Revert debug changes.
* Request chain id from the blockchain server.
2024-08-14 11:35:48 +08:00
Bo QIU
a5f95e2e7b
Verify announced IP with seen ip for pubsub msg ( #130 )
...
* Verify announced IP with seen ip for pubsub msg
* do not verify announced ip at local mode
2024-07-17 19:01:41 +08:00
peilun-conflux
ba8d065e73
Generate sync proof with the given tx seq. ( #123 )
...
* Generate sync proof with the given tx seq.
* Fill HistoryTree with new data.
* Fix clippy.
2024-07-12 15:55:57 +08:00
Bo QIU
821e4bd06e
Use public ip address to broadcast file announcement ( #115 )
...
* Use public ip address to broadcast file announcement
* auto detect public ip address for ENR address
* add more log
2024-07-09 17:39:14 +08:00
Bo QIU
68b5d27728
Filter out file announcement of private ip ( #113 )
...
* Filter out file announcement of private ip
* add log for listen address
2024-07-08 18:45:55 +08:00
Bo QIU
e44db25807
opt log level for file announcement relevant logs ( #109 )
2024-07-04 16:30:05 +08:00
Bo QIU
5bc5bbd696
Enhance log level ( #103 )
...
* reduce default find peers timeout for auto sync
* opt log level to avoid too many noisy info logs
2024-07-02 11:52:52 +08:00
peilun-conflux
4eb2a50b0e
Use inner lock in storage and use async lock. ( #92 )
...
* Use inner lock in storage.
* Remove mut.
* Remove async lock for storage.
* Fix tests and warnings.
* Use spawn_blocking for storage task.
* Fix clippy.
* Finalize the new tx at last.
* Revert "Finalize the new tx at last."
This reverts commit b56ad5582d
2024-06-29 17:08:02 +08:00
peilun-conflux
0f52325f67
Sync and finalize sharded data. ( #79 )
...
* Save test logs.
* Prepare to handle shard config in chunk pool.
* Allow files to be finalized with only sharded data.
* Handle shard config change.
* Only sync needed data in shard config.
* Fix clippy and tests.
2024-06-08 02:50:36 +08:00
peilun-conflux
c2c6e2d5fb
Store shard config for peers and choose sync peers accordingly. ( #77 )
...
* Implement Pruner.
* Put pruner in a crate.
* Fix clippy.
* Add rpc zgs_getShardConfig.
* Fix.
* Increase wait time.
* Add pruner_test and use max_num_chunks instead of size_limit.
* Store shard config for peers and choose sync peers accordingly.
* Add test and fix sync.
* Fix clippy and test.
* Fix some ut.
* Add AnnounceShardConfig gossip and fix tests.
* Add sharded tx finalize check in LogManager.
* Try,
* Rename.
* Longer timeout for mine_test.
* Save test logs.
2024-06-07 16:58:15 +08:00
MiniFrenchBread
8b410f2fc8
Chore: expose RPC modules, update dependency ( #35 )
...
* feat: expose rpc clients
* chore: expose rpc types
* chore: update dependency
2024-04-02 21:00:26 +08:00
Bo QIU
a4abe2b2a4
fix build dependency isuse ( #12 )
...
* fix build dependency isuse
* cargo fmt
* fix lint failure in CI
* update toolchain
2024-01-26 19:48:40 +08:00
MiniFrenchBread
7479f56474
feat: write cached segments to storage on log synced ( #7 )
2024-01-23 18:47:14 +08:00
Bo QIU
9b4b0436c3
Supports to sync partial chunks ( #4 )
...
* refactor p2p signed message
* add new pubsub messages in network layer to find chunks
* handle find chunks pubsub message in router
* Supports to sync partial chunks
* add admin rpc to sync chunks
* limit number of chunks to sync at a time
* refactor code to sync file and chunks
* add more switches to trigger file sync
* fix ut failure
* refactor code
2024-01-19 14:04:59 +08:00
Peilun Li
9058467be1
Initial commit
2024-01-04 18:05:32 +08:00
