Commit Graph

150 Commits

Author SHA1 Message Date
bruno-valante
aaba4b5da0
Fix issue found in auditions (#170) 2024-08-27 17:36:55 +08:00
bruno-valante
5757c98d2a
Config query context interval & Add config comments (#156) 2024-08-27 17:36:36 +08:00
Bo QIU
37344b9439
opt sync config and update config file (#169)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-08-26 15:29:52 +08:00
0g-peterzhb
780865b1b8
do not fail on prune error (#168)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* do not fail on prune error
2024-08-25 16:48:03 +08:00
peilun-conflux
1c72607fbc
Set sync start index based on data in db. (#166)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Set sync start index based on data in db.

* Fix test.

* nit.
2024-08-23 12:41:25 +08:00
Bo QIU
f14a1b5975
add timestamp in p2p rpc request id for latency stat (#167)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* add timestamp in p2p rpc request id for latency stat

* fix test compilation error
2024-08-22 14:34:10 +08:00
Bo QIU
82fef674ed
Add more metrics for file sync (#164)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* add metrics in log sync package

* udpate auto sync metrics

* Add metrics for completed file sync

* add more metrics for serial file sync

* adjust default timeout value for auto sync

* fix metrics rpc for Timer type

* add metrics for channel

* refactor channel metrics

* add timeout metrics for segment sync

* refactor channel receiver
2024-08-22 10:42:15 +08:00
Bo QIU
c1f465e009
Do not verify announced ip address by default (#163)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-08-20 14:39:58 +08:00
boqiu
c337ff90fb hotfix for admin rpc metrics
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-08-19 17:07:57 +08:00
Bo QIU
908ee156ca
Add admin rpc to exports metrics (#162) 2024-08-19 15:18:12 +08:00
Bo QIU
f9120b1e4a
Add metrics for router and auto sync (#161)
* Add metrics configurations

* Add metrics in router package

* Add catch up info in sync service state

* Add metrics for auto sync

* update cargo lock
2024-08-19 09:54:52 +08:00
Bo QIU
22ed8f5f91
opt router debug logs (#160)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-08-16 10:34:28 +08:00
peilun-conflux
12d0c6b675
Check network id in status and ban incompatible peers. (#159)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Check network id in status and ban incompatible peers.

* Revert debug changes.

* Request chain id from the blockchain server.
2024-08-14 11:35:48 +08:00
Bo QIU
03286ebd78
Supports HDD config file (#158)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Add debug log config

* Add config file for ssd and hdd

* Set most default value for ssd and hdd config file
2024-08-12 17:07:06 +08:00
0g-peterzhb
f03f97c609
more robust provider (#157)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* more robust provider
2024-08-12 11:35:21 +08:00
peilun-conflux
53449e1faa
Half log_page_size when it queries too many logs. (#152)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Half log_page_size when it queries too many logs.

* fmt.

* Increase log broadcast channel size.
2024-08-09 17:13:20 +08:00
Bo QIU
9189cabbb2
Supports batch randomly auto sync files (#154)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-08-08 18:36:08 +08:00
Bo QIU
e1df2f0868
Change hardcoded const variables to configs (#153) 2024-08-08 16:21:50 +08:00
rickiey
77d1b84974
fix:log config : invalid filter directive (#146)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-08-06 18:46:53 +08:00
peilun-conflux
d80e7e22ca
Prune no reward chunks. (#145)
* Prune no reward chunks.

* Add tests.

* Fix tests.

* Fix clippy.

* Revert test.

* Enable market in shard_sync_test.

* Add tx prune status.

* Fix tests.
2024-08-06 15:06:15 +08:00
Bo QIU
6ade66c086
Add admin rpc to return sync service state (#151) 2024-08-06 14:01:57 +08:00
Bo QIU
891e00fa80
Refactor auto sync to return sync status (#150)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* Refactor auto sync to return sync state

* Add auto sync manager to wrap multiple objects

* use auto sync manager

* fix python tests
2024-08-05 17:30:26 +08:00
peilun-conflux
dbd865fded
Change db_max_num_chunks to db_max_num_sectors. (#137)
* Change db_max_num_chunks to db_max_num_sectors.

* Update tests and config files.

* Revert contract change.
2024-07-29 22:31:19 +08:00
Bo QIU
ae9c52c0e6
Terminate file sync if connecting peers too long (#136) 2024-07-27 21:05:44 +08:00
bruno-valante
7d73ccd1e1
Update with the most recent storage contract (#142)
* Update with the most recent storage contract

* fix nits
2024-07-24 13:01:30 +08:00
MiniFrenchBread
533bacb234
fix: admin_getFileLocation; test: sync test (#141)
* feat: add all_shards in admin_getFileLocation

* fix: admin_getFileLocation

* test: improve sync test

* fix: lint
2024-07-23 15:47:44 +08:00
MiniFrenchBread
a782cf2d54
feat: add all_shards in admin_getFileLocation (#138) 2024-07-22 11:28:05 +08:00
MiniFrenchBread
085c34beb0
feat: file location admin rpc; refactor: all_shards_available (#134)
* feat: find file rpc

* refactor: all_shards_available

* fix: fmt

* chore: remove rpc trace
2024-07-19 11:55:18 +08:00
Bo QIU
a5f95e2e7b
Verify announced IP with seen ip for pubsub msg (#130)
* Verify announced IP with seen ip for pubsub msg

* do not verify announced ip at local mode
2024-07-17 19:01:41 +08:00
Bo QIU
920efe0b59
enhance admin rpc to return peers instead of dump (#128) 2024-07-16 14:57:36 +08:00
Bo QIU
0f9f1c7bf2
Add admin RPC to dump discovered peers (#127)
* add admin rpc to dump peers

* allow to specify file name

* return listen addreses in rpc

* add trace for admin rpc parameters
2024-07-15 18:19:30 +08:00
peilun-conflux
504a67fdd0
Handle chunk write of rpc in parallel. (#126)
* Handle chunk write of rpc in parallel.

* double mine period.

* Increase the log sync channel capacity.

* Increase capacity.
2024-07-15 15:50:51 +08:00
Bo QIU
b1c52c7ad6
disable auto sync until catched up (#125) 2024-07-12 17:45:20 +08:00
bruno-valante
cca14e246e
Support multiple mine submission (#124)
* Support multiple mine submission

* Update
2024-07-12 17:31:53 +08:00
peilun-conflux
ba8d065e73
Generate sync proof with the given tx seq. (#123)
* Generate sync proof with the given tx seq.

* Fill HistoryTree with new data.

* Fix clippy.
2024-07-12 15:55:57 +08:00
Bo QIU
2bc402f94b
Enhance sync logs (#122)
* enhance sync logs

* fix lint
2024-07-12 14:26:23 +08:00
Bo QIU
cc5f8c2da4
Terminate file sync if failed (#121)
* terminate file sync if failed

* fix fmt

* always add log for file termination result
2024-07-11 16:54:04 +08:00
peilun-conflux
e54b583b1c
Add zgs_getSectorProof. (#112) 2024-07-11 14:07:46 +08:00
peilun-conflux
3e23e81ae6
Repeat log catch-up until it's close to the latest height. (#114)
* Repeat log catch-up until it's close to the latest height.

* Add an oneshot channel to send catch up end complete.

* Fix comments.

* Fix tests.

* Fix ut.
2024-07-11 14:07:03 +08:00
Bo QIU
59d3a03487
Hotfix for cli default value (#118) 2024-07-11 11:54:13 +08:00
Bo QIU
727a9b8bb6
enhance config and cli (#116)
* Update default value in config file

* Add config file for testnet

* Change default value for testnet config file

* Supports environment config source

* Add cli args

* add default value for cli args

* fix fmt

* Fix cli args default value
2024-07-10 15:36:10 +08:00
Bo QIU
821e4bd06e
Use public ip address to broadcast file announcement (#115)
* Use public ip address to broadcast file announcement

* auto detect public ip address for ENR address

* add more log
2024-07-09 17:39:14 +08:00
Bo QIU
68b5d27728
Filter out file announcement of private ip (#113)
* Filter out file announcement of private ip

* add log for listen address
2024-07-08 18:45:55 +08:00
Bo QIU
84f455d1d4
Fix some issues in sync package (#111)
* Change default concurrent sync threads

* enhance log level in sync store

* Add rwlock in sync store to avoid concurrency issue

* enhance serial sync state machine log level

* wrap Instant for debug purpose

* add timestamp for connecting peers state

* Select random peer to sync chunks

* return sync peers states instead of connected peers only

* Use tokio async RwLock for sync store
2024-07-08 11:47:59 +08:00
Bo QIU
e44db25807
opt log level for file announcement relevant logs (#109) 2024-07-04 16:30:05 +08:00
Bo QIU
816353dfd4
Supports to concurrently sync files in sequence (#108)
* Supports to concurrently sync files in sequence

* add more comments

* refactor random auto sync
2024-07-04 14:04:17 +08:00
Bo QIU
5bc5bbd696
Enhance log level (#103)
* reduce default find peers timeout for auto sync

* opt log level to avoid too many noisy info logs
2024-07-02 11:52:52 +08:00
Bo QIU
055d1fa32d
Fix sync state machine issue for sharding case (#102)
* Fix sync state machine issue for sharding case

* Fix fmt lint
2024-07-01 16:35:27 +08:00
0g-peterzhb
2f3361fafc
fix pruning (#101)
* fix pruning
2024-06-30 08:31:06 +08:00
peilun-conflux
4eb2a50b0e
Use inner lock in storage and use async lock. (#92)
* Use inner lock in storage.

* Remove mut.

* Remove async lock for storage.

* Fix tests and warnings.

* Use spawn_blocking for storage task.

* Fix clippy.

* Finalize the new tx at last.

* Revert "Finalize the new tx at last."

This reverts commit b56ad5582d.

* Wait for old same-root txs to finalize.

* Use async storage in miner.

* Update rust version to 1.79.0.

* Use Vec to avoid stack overflow.

* Fix unused warning.

* Fix clippy.

* Fix test warning.

* Fix test.

* fmt.

* Use async storage in pruner.

* nit.
2024-06-29 17:08:02 +08:00
Bo QIU
4ee782e4bc
enhance default config value (#97) 2024-06-27 14:47:33 +08:00
Bo QIU
439314372b
Remove file location cache for connection timeout peer (#96) 2024-06-27 01:51:37 +08:00
Bo QIU
c6325f7643
Fix file sync pending due to peer connection timeout (#95)
* enhance logs for sync module

* report peer if connection timeout

* Use origin timestamp of sync state to determine if sync too long to terminate

* fix compilation error
2024-06-25 17:47:25 +08:00
Bo QIU
d8d8c28c64
Add discv5 config and enlarge the default rate limiter (#86) 2024-06-14 18:04:35 +08:00
Bo QIU
dea518a0aa
Supports discv5 related configurations (#84)
* Supports discv5 related configurations

* fix fmt
2024-06-13 16:14:20 +08:00
peilun-conflux
489d973347
Update run/config.toml and version. (#83)
* Update run/config.toml.

* Update version
2024-06-11 13:17:32 +08:00
peilun-conflux
0f52325f67
Sync and finalize sharded data. (#79)
* Save test logs.

* Prepare to handle shard config in chunk pool.

* Allow files to be finalized with only sharded data.

* Handle shard config change.

* Only sync needed data in shard config.

* Fix clippy and tests.
2024-06-08 02:50:36 +08:00
peilun-conflux
c2c6e2d5fb
Store shard config for peers and choose sync peers accordingly. (#77)
* Implement Pruner.

* Put pruner in a crate.

* Fix clippy.

* Add rpc zgs_getShardConfig.

* Fix.

* Increase wait time.

* Add pruner_test and use max_num_chunks instead of size_limit.

* Store shard config for peers and choose sync peers accordingly.

* Add test and fix sync.

* Fix clippy and test.

* Fix some ut.

* Add AnnounceShardConfig gossip and fix tests.

* Add sharded tx finalize check in LogManager.

* Try,

* Rename.

* Longer timeout for mine_test.

* Save test logs.
2024-06-07 16:58:15 +08:00
peilun-conflux
ef82f64393
Implement Pruner to delete unwanted data. (#70)
* Implement Pruner.

* Put pruner in a crate.

* Fix clippy.

* Add rpc zgs_getShardConfig.

* Fix.

* Increase wait time.

* Add pruner_test and use max_num_chunks instead of size_limit.

* Add back shard config and fix test.

* fix: serde format

* Remove unneeded PORA_CHUNK_SIZE.

* Fix tests.

---------

Co-authored-by: MiniFrenchBread <103425574+MiniFrenchBread@users.noreply.github.com>
2024-05-31 13:11:06 +08:00
Chenxing Li
c0f9d5cc31
Fix quality overflow (#75) 2024-05-30 21:49:47 +08:00
Bo QIU
0852b2355c
enhance rpc logs (#73) 2024-05-29 10:16:28 +08:00
Chenxing Li
2262bc3fb9
Support mining on sharded storage. (#64)
* Change PoraHash compute

* Change padSeed compute

* Refactor

* Support mining on sharded storage

* Detect single core performance and set correct params for test

* Fix clippy

* Fix an overflow bug
2024-04-30 14:34:44 +08:00
Chenxing Li
193e154361
Change miner id logic & request miner id automatically (#60)
* Change miner id logic & request miner id automatically

* Not enable all features in the test workflow.

* Auto configurable mining period

* Adjust test params for ci
2024-04-27 11:15:57 +08:00
Chenxing Li
c85fe38c59
Remove cli build from server build workflow (#54)
* Remove cli build from server build workflow

* Build cli for a single test
2024-04-23 14:35:24 +08:00
Chenxing Li
e5dc2d61cc
Configurable CPU usage on mine (#58)
* Configurable CPU usage on mine

* Cargo clippy
2024-04-23 12:52:39 +08:00
peilun-conflux
b9c6e43617
Enforce rear padding data check for segment proofs. (#55)
* Construct wrong proof.

* Enforce rear padding data check for segment proofs.
2024-04-17 18:46:04 +08:00
peilun-conflux
052e10cf8b
Add more log for debug. (#53) 2024-04-15 11:16:56 +08:00
0g-peterzhb
6b9ee41457
update default config (#52) 2024-04-15 10:50:13 +08:00
peilun-conflux
f5a71375be
Allow reverting the last tx with no flow data update. (#51)
* Allow reverting the last tx with no flow data update.

* Add details for panic.
2024-04-15 10:44:18 +08:00
Chenxing Li
8c955ed5aa
Fix an overflow bug in chunk sealing (#47) 2024-04-12 15:54:11 +08:00
Joel Liu
6c1b0b35ec
Wait SyncedBlock to be processed for revert block (#45)
* wait SyncedBlock to be processed

* remove retry counter

* use default parent block hash for missing case
2024-04-11 18:13:52 +08:00
peilun-conflux
bd4ebee2da
Change some log level. (#46)
* Change some log level.

* Add log for flow root commit.
2024-04-11 17:15:05 +08:00
Chenxing Li
163f843581
Configurable mine submission gas limit (#41)
* Configurable mine submission gas limit

* cargo fmt

* More log

* update

* cargo clippy
2024-04-11 11:52:03 +08:00
MiniFrenchBread
0047ab22d4
chore: update dependency (#39)
* chore: update dependency

* chore: update dependency
2024-04-08 13:54:01 +08:00
Joel Liu
b8a59e9222
query getLogs from batch blocks (#37)
* query getLogs from batch blocks

* resolve comments
2024-04-07 11:27:18 +08:00
Joel Liu
72e3e0a019
set uploaded_seg_num in FileInfo for rpc (#36)
* set uploaded_seg_num in FileInfo for rpc

* fix lint

* segments is 0 when not finalized

* refine code
2024-04-07 11:03:11 +08:00
MiniFrenchBread
8b410f2fc8
Chore: expose RPC modules, update dependency (#35)
* feat: expose rpc clients

* chore: expose rpc types

* chore: update dependency
2024-04-02 21:00:26 +08:00
peilun-conflux
588bf39d7d
Fix a possible log sync break after a node restarts. (#34)
* Fix a possible log sync break after a node restarts.

* Skip `sync_blocks` in blockchain node setup.
2024-03-29 16:15:01 +08:00
Joel Liu
f9ce286909
replace filter with eth_getLogs (#32)
* replace filter with eth_getLogs

* update log

* store block history into db

* fix lints error

* add free disk step

* set max worker to 3

* resolve comments and refact code

* resolve comments

* set max-workers to 4

* set max-workers to 3
2024-03-27 13:54:06 +08:00
Chenxing Li
16dfc56437
Test mining with the simple market (#31)
* Test mining with the simple market

* Support reward donation
2024-03-19 14:09:17 +08:00
Chenxing Li
306c43c9dc
Migrate to the new contract interfaces (#29)
* Fix compile warning & Search contract by name automatically

Migrate to the new contract interfaces

* Fix compile

* Fix lint
2024-03-14 13:56:17 +08:00
peilun-conflux
2ed6bca446
Process padding data in groups to avoid OOM. (#27) 2024-03-14 11:37:34 +08:00
0g-peterzhb
2e1d076460
@peter/update submodule (#25)
* update modules
2024-03-12 18:49:51 +08:00
peilun-conflux
3e2068d363
Fix a wrong assertion when we insert the last tx again. (#23) 2024-02-28 16:59:30 +08:00
peilun-conflux
310bf1c8dd
Save updated mpt nodes from proof to DB. (#20)
* Save updated mpt nodes from proof to DB.

* Free disk space for Github Actions.

* Use an alternative Rust cache action.

* Fix action usage.

* Do not free large packages.
2024-02-06 18:46:35 +08:00
MiniFrenchBread
01c2dd1135
feat: upload_segments (#21)
* feat: upload_segments

* feat: max request body
2024-02-06 17:51:31 +08:00
Bo QIU
c2b8b1cab3
Rpc enhancements (#15)
* add admin rpc to terminate file or chunks sync

* Disallow to change file sync goal when failed

* split admin rpc into separate endpoints

* Support to launch public and private rpc at one port

* use one rpc port for python test

* fix lint

* restart to sync file or chunks if sync already completed or failed
2024-01-30 16:50:35 +08:00
Bo QIU
a4abe2b2a4
fix build dependency isuse (#12)
* fix build dependency isuse

* cargo fmt

* fix lint failure in CI

* update toolchain
2024-01-26 19:48:40 +08:00
peilun-conflux
0c123500c9
Update the merkle tree with proof data. (#9)
* Add proof data for chunk proof.

* Support file proof.

* Update with segment proof and fix issues.

* Fix more issues.

* Fix the process of file proof.

* Merge branch 'main' into fix_proof

* Enable sync_test.

* Fix wrongly updated submodule.

* Fix bsc node version.
2024-01-26 10:29:09 +08:00
Bo QIU
2c2dba8730
fix ci failure (#10)
* fix ci failure

* fix ci failures

* fix py test issue

* disable --all-features for code coverage CI

* do not fail ci if failed to upload code coverage
2024-01-25 19:00:00 +08:00
MiniFrenchBread
7479f56474
feat: write cached segments to storage on log synced (#7) 2024-01-23 18:47:14 +08:00
boqiu
99ead6423b add python test for chunks sync by rpc 2024-01-22 18:19:28 +08:00
peilun-conflux
877f484dd0
Merge pull request #2 from zero-gravity-labs/filter_address
Filter address in log sync.
2024-01-19 19:45:27 +08:00
Bo QIU
9b4b0436c3
Supports to sync partial chunks (#4)
* refactor p2p signed message

* add new pubsub messages in network layer to find chunks

* handle find chunks pubsub message in router

* Supports to sync partial chunks

* add admin rpc to sync chunks

* limit number of chunks to sync at a time

* refactor code to sync file and chunks

* add more switches to trigger file sync

* fix ut failure

* refactor code
2024-01-19 14:04:59 +08:00
Bo QIU
971d344acd
fmt code (#3) 2024-01-19 10:56:59 +08:00
Peilun Li
a26cef470a Merge branch 'main' into filter_address 2024-01-17 10:57:59 +08:00
Peilun Li
986bfa412c Add to start_recover. 2024-01-17 10:57:07 +08:00
Peilun Li
e236c2f70a Filter address in log sync. 2024-01-16 17:54:23 +08:00
Peilun Li
6e15838060 Disable write window size check. 2024-01-10 17:29:11 +08:00
Peilun Li
9058467be1 Initial commit 2024-01-04 18:05:32 +08:00