Commit Graph

40 Commits

Author SHA1 Message Date
Peter Zhang
9a11e969e6 update config 2024-12-03 20:42:25 +08:00
Bo QIU
d322c38c11
Add script for dev test purpose (#288)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* Add script for local startup

* requires .env file

* fix bash syntax

* exit on exception
2024-12-03 19:25:07 +08:00
Bo QIU
b9e6431a4d
Add shard config in STATUS message and only dail to shard config matched peers (#285)
* Add shard config in status message

* verify shard config for status message

* Notify peer connected to sync layer after status message exchanged

* Do not dial to shard config mismatched peers

* Upgrade network protocol version

* disconnect peer instead of ban peer if shard config mismatch

* Add python test for TCP connection by shard config
2024-11-25 10:15:30 +08:00
0g-peterzhb
96f846073f
increase db_max_num_sectors to be about 1000GB (#272) 2024-11-15 13:10:07 +08:00
Bo QIU
9b68a8b7d7
Implement file sync protocol V2 (#249)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Add new P2P protocol NewFile

* Publish NewFile message when any file finalized

* handle NewFile message in router

* handle NewFile in sync servic to write in db

* use propagation source to handle NewFile message

* Disable sequential sync and store new file in v2 sync store

* Add shard config in FindFile

* Add AnnounceFile RPC message in network layer

* do not propagate FindFile to whole network

* Mark peer connected if FileAnnouncement RPC message received

* fix unit test failures

* Change P2P protocol version

* Ignore py tests of sequential auto sync

* Add py test for auto sync v2

* fmt code

* remove dummy code in py test

* fix random test failure

* Add comments

* Enable file sync protocol v2 in config file by default
2024-10-28 14:56:08 +08:00
Bo QIU
8f17a7ad72
Fix metrics config deser (#245)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-10-22 14:14:33 +08:00
0g-peterzhb
48868b60db
update ip (#223)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-10-11 17:07:43 +08:00
Bo QIU
ae6ecfec96
reduce default sync threads (#214) 2024-09-26 14:55:00 +08:00
Bo QIU
ad80b22a1b
Optimize rpc config (#213) 2024-09-25 16:48:40 +08:00
Bo QIU
9cde84ae15
Refactor network peer db configs (#209) 2024-09-24 11:59:34 +08:00
Bo QIU
a9f5169c15
Allow user to configure network bandwidth for file sync (#198)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-09-13 16:58:27 +08:00
Bo QIU
b6972b97af
Adjust default value for testnet configs (#180)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* Adjust default value for testnet configs

* Supports to disable sequential auto sync

* Add py tests for auto sync

* fix py file name

* rm dummy ;py code

* change default block confirm count
2024-09-05 10:09:29 +08:00
0g-peterzhb
508e787bd0
update boot nodes (#175)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-08-30 10:38:19 +08:00
Bo QIU
2fd9712d59
Enhance P2P network protocol to support batch messages for performance concern (#173)
* Add p2p protocol version in network identity

* Cache annouce file pubsub messages to publish in batch

* fix file location cache

* opt sync metrics

* opt file location cache default configs

* publish files announcements in batch

* enhance announce file pubsub msg metrics

* opt metrics

* fix ci

* fix clippy

* fix batcher

* minor fix

* opt batcher: publish all if expired
2024-08-29 09:55:24 +08:00
bruno-valante
5757c98d2a
Config query context interval & Add config comments (#156) 2024-08-27 17:36:36 +08:00
Bo QIU
37344b9439
opt sync config and update config file (#169)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-08-26 15:29:52 +08:00
Bo QIU
82fef674ed
Add more metrics for file sync (#164)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
* add metrics in log sync package

* udpate auto sync metrics

* Add metrics for completed file sync

* add more metrics for serial file sync

* adjust default timeout value for auto sync

* fix metrics rpc for Timer type

* add metrics for channel

* refactor channel metrics

* add timeout metrics for segment sync

* refactor channel receiver
2024-08-22 10:42:15 +08:00
Bo QIU
22ed8f5f91
opt router debug logs (#160)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
2024-08-16 10:34:28 +08:00
Bo QIU
03286ebd78
Supports HDD config file (#158)
Some checks failed
abi-consistent-check / build-and-compare (push) Has been cancelled
code-coverage / unittest-cov (push) Has been cancelled
rust / check (push) Has been cancelled
rust / test (push) Has been cancelled
rust / lints (push) Has been cancelled
functional-test / test (push) Has been cancelled
* Add debug log config

* Add config file for ssd and hdd

* Set most default value for ssd and hdd config file
2024-08-12 17:07:06 +08:00
Bo QIU
9189cabbb2
Supports batch randomly auto sync files (#154)
Some checks are pending
abi-consistent-check / build-and-compare (push) Waiting to run
code-coverage / unittest-cov (push) Waiting to run
rust / check (push) Waiting to run
rust / test (push) Waiting to run
rust / lints (push) Waiting to run
functional-test / test (push) Waiting to run
2024-08-08 18:36:08 +08:00
peilun-conflux
dbd865fded
Change db_max_num_chunks to db_max_num_sectors. (#137)
* Change db_max_num_chunks to db_max_num_sectors.

* Update tests and config files.

* Revert contract change.
2024-07-29 22:31:19 +08:00
MiniFrenchBread
085c34beb0
feat: file location admin rpc; refactor: all_shards_available (#134)
* feat: find file rpc

* refactor: all_shards_available

* fix: fmt

* chore: remove rpc trace
2024-07-19 11:55:18 +08:00
Bo QIU
727a9b8bb6
enhance config and cli (#116)
* Update default value in config file

* Add config file for testnet

* Change default value for testnet config file

* Supports environment config source

* Add cli args

* add default value for cli args

* fix fmt

* Fix cli args default value
2024-07-10 15:36:10 +08:00
Bo QIU
821e4bd06e
Use public ip address to broadcast file announcement (#115)
* Use public ip address to broadcast file announcement

* auto detect public ip address for ENR address

* add more log
2024-07-09 17:39:14 +08:00
Bo QIU
84f455d1d4
Fix some issues in sync package (#111)
* Change default concurrent sync threads

* enhance log level in sync store

* Add rwlock in sync store to avoid concurrency issue

* enhance serial sync state machine log level

* wrap Instant for debug purpose

* add timestamp for connecting peers state

* Select random peer to sync chunks

* return sync peers states instead of connected peers only

* Use tokio async RwLock for sync store
2024-07-08 11:47:59 +08:00
Bo QIU
816353dfd4
Supports to concurrently sync files in sequence (#108)
* Supports to concurrently sync files in sequence

* add more comments

* refactor random auto sync
2024-07-04 14:04:17 +08:00
Bo QIU
5bc5bbd696
Enhance log level (#103)
* reduce default find peers timeout for auto sync

* opt log level to avoid too many noisy info logs
2024-07-02 11:52:52 +08:00
0g-peterzhb
cc44c9db66
remove mine id (#98) 2024-06-27 16:19:40 +08:00
Bo QIU
4ee782e4bc
enhance default config value (#97) 2024-06-27 14:47:33 +08:00
Bo QIU
d8d8c28c64
Add discv5 config and enlarge the default rate limiter (#86) 2024-06-14 18:04:35 +08:00
Bo QIU
dea518a0aa
Supports discv5 related configurations (#84)
* Supports discv5 related configurations

* fix fmt
2024-06-13 16:14:20 +08:00
peilun-conflux
489d973347
Update run/config.toml and version. (#83)
* Update run/config.toml.

* Update version
2024-06-11 13:17:32 +08:00
Bo QIU
6ca05090c3
update config (#56)
* update config

* use evmos testnet binary all the time
2024-04-16 21:24:50 +08:00
0g-peterzhb
6b9ee41457
update default config (#52) 2024-04-15 10:50:13 +08:00
0g-peterzhb
7bcd9b1988
update default config (#44) 2024-04-09 17:23:17 +08:00
0g-peterzhb
0131de26c8
update default config (#42) 2024-04-09 15:48:35 +08:00
0g-peterzhb
b07e6e776a
update boot node to static ip (#38) 2024-04-06 20:38:53 +08:00
Bo QIU
c2b8b1cab3
Rpc enhancements (#15)
* add admin rpc to terminate file or chunks sync

* Disallow to change file sync goal when failed

* split admin rpc into separate endpoints

* Support to launch public and private rpc at one port

* use one rpc port for python test

* fix lint

* restart to sync file or chunks if sync already completed or failed
2024-01-30 16:50:35 +08:00
Bo QIU
9b4b0436c3
Supports to sync partial chunks (#4)
* refactor p2p signed message

* add new pubsub messages in network layer to find chunks

* handle find chunks pubsub message in router

* Supports to sync partial chunks

* add admin rpc to sync chunks

* limit number of chunks to sync at a time

* refactor code to sync file and chunks

* add more switches to trigger file sync

* fix ut failure

* refactor code
2024-01-19 14:04:59 +08:00
Peilun Li
9058467be1 Initial commit 2024-01-04 18:05:32 +08:00