KDB+效能玩票测试

动因

之前使用Man-Group的Arctic系统(老Arctic,基于MongoDB的那个)存储各种交易标的的数据,但发现比较慢。测试下传说中的KDB+看能否替换。目前并不期望能使用KDB+强项的内部计算(由于q语言学习成本过高),只是简单作为时序数据库使用,支持读取写入增删改查即可。

架构

使用一台Linux主机作为数据库服务器,与客户机在同一个局域网(千兆以太网)下。具体配置过程略。

简单测试

Read More

【Python知识库】Python中的PriorityQueue测试

Python官方提供一个PriorityQueue,可以按优先级取出Push入队的对象,但这个Queue不稳定(即相同优先级并不保证先进先出)。

测试代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
from collections import defaultdict
from queue import Empty, Queue, PriorityQueue
from threading import Thread, get_ident
from time import sleep
from typing import Any, Callable, List
from datetime import datetime



EVENT_SLEEP = "eSleep"


class Event:
"""
Event object consists of a type string which is used
by event engine for distributing event, and a data
object which contains the real data.
"""

def __init__(self, etype: str, data: Any = None, priority: int = 100) -> None:
""""""
self.type: str = etype
self.priority = priority
self.data: Any = data

def __lt__(self, other):
return self.priority > other.priority

def __le__(self, other):
return self.priority >= other.priority


# Defines handler function to be used in event engine.
HandlerType: callable = Callable[[Event], None]


class EventEngine:
"""
Event engine distributes event object based on its type
to those handlers registered.

It also generates timer event by every interval seconds,
which can be used for timing purpose.
"""

def __init__(self, interval: int = 1, using_priority = False) -> None:
"""
Timer event is generated every 1 second by default, if
interval not specified.
"""
self._interval: int = interval
self._queue: Queue = PriorityQueue() if using_priority else Queue()
self._active: bool = False
self._thread: Thread = Thread(target=self._run)
self._handlers: defaultdict = defaultdict(list)

def _run(self) -> None:
"""
Get event from queue and then process it.
"""
print(f"Event engine started. Thread ID: {get_ident()}")
while self._active:
try:
event: Event = self._queue.get(block=True, timeout=1)
self._process(event)
except Empty:
# logger.info(f"Event Queue empty. Thread ID: {get_ident()}")
pass

def _process(self, event: Event) -> None:
"""
First distribute event to those handlers registered listening
to this type.

Then distribute event to those general handlers which listens
to all types.
"""
if event.type in self._handlers:
for handler in self._handlers[event.type]:
handler(event)

def start(self) -> None:
"""
Start event engine to process events and generate timer events.
"""
self._active = True
self._thread.start()

def stop(self) -> None:
"""
Stop event engine.
"""
self._active = False
self._thread.join()

def put(self, event: Event) -> None:
"""
Put an event object into event queue.
"""
self._queue.put(event)

def register(self, etype: str, handler: HandlerType) -> None:
"""
Register a new handler function for a specific event type. Every
function can only be registered once for each event type.
"""
handler_list: list = self._handlers[etype]
if handler not in handler_list:
handler_list.append(handler)

def unregister(self, etype: str, handler: HandlerType) -> None:
"""
Unregister an existing handler function from event engine.
"""
handler_list: list = self._handlers[etype]

if handler in handler_list:
handler_list.remove(handler)

if not handler_list:
self._handlers.pop(type)

def process_sleep_event(event):
name, sleepseconds = event.data
print(f"In {get_ident()}, Task {name} sleep {sleepseconds} start @ {datetime.now().isoformat()}")
sleep(sleepseconds)
print(f"In {get_ident()}, Task {name} sleep {sleepseconds} end @ {datetime.now().isoformat()}")

if __name__ == "__main__":
print(f"main thread: {get_ident()} start @ {datetime.now().isoformat()}")
engine = EventEngine(using_priority=True)
print(f"using {type(engine._queue)}")
engine.register(EVENT_SLEEP, process_sleep_event)
engine.start()

engine.put(Event(EVENT_SLEEP, ("1st_1sec_event", 1)))
engine.put(Event(EVENT_SLEEP, ("2nd_1sec_event", 1), priority=50))
engine.put(Event(EVENT_SLEEP, ("3rd_5sec_event", 5)))
engine.put(Event(EVENT_SLEEP, ("4th_5sec_event", 5), priority=200))
sleep(13)
engine.put(Event(EVENT_SLEEP, ("5th_5sec_event", 3), priority=300))

engine.stop()
print(f"main thread: {get_ident()}, exit @ {datetime.now().isoformat()}")

输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
main thread: 117652 start  @ 2024-03-16T15:58:45.398626
using <class 'queue.PriorityQueue'>
Event engine started. Thread ID: 102312
In 102312, Task 4th_5sec_event sleep 5 start @ 2024-03-16T15:58:45.399626
In 102312, Task 4th_5sec_event sleep 5 end @ 2024-03-16T15:58:50.402557
In 102312, Task 3rd_5sec_event sleep 5 start @ 2024-03-16T15:58:50.402557
In 102312, Task 3rd_5sec_event sleep 5 end @ 2024-03-16T15:58:55.413506
In 102312, Task 1st_1sec_event sleep 1 start @ 2024-03-16T15:58:55.413506
In 102312, Task 1st_1sec_event sleep 1 end @ 2024-03-16T15:58:56.418677
In 102312, Task 2nd_1sec_event sleep 1 start @ 2024-03-16T15:58:56.418677
In 102312, Task 2nd_1sec_event sleep 1 end @ 2024-03-16T15:58:57.424852
In 102312, Task 5th_5sec_event sleep 3 start @ 2024-03-16T15:58:58.403006
In 102312, Task 5th_5sec_event sleep 3 end @ 2024-03-16T15:59:01.406562
main thread: 117652, exit @ 2024-03-16T15:59:01.406562

Read More

jupyterlab升级后的数个插件问题

第一,插件@jupyterlab/notebook-extension:tracker报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[W 2023-08-12 15:11:31.242 LabApp] Failed validating settings (@jupyterlab/notebook-extension:tracker): Additional properties are not allowed ('experimentalDisableDocumentWideUndoRedo', 'numberCellsToRenderDirectly', 'observedBottomMargin', 'observedTopMargin', 'remainingTimeBeforeRescheduling', 'renderCellOnIdle' were unexpected)

Failed validating 'additionalProperties' in schema:
{'additionalProperties': False,
'definitions': {'kernelStatusConfig': {'additionalProperties': False,
'properties': {'showOnStatusBar': {'default': False,
'description': 'If '
'`true`, '
'the '
'kernel '
'status '
'progression '
'will '
'be '
'displayed '
'in '
'the '
'status '
'bar '
'otherwise '
'it '
'will '
'be '
'in '
'the '
'toolbar.',
'title': 'Show '
'kernel '
'status '
'on '
'toolbar '
'or '
'status '
'bar.',
'type': 'boolean'},
'showProgress': {'default': True,
'title': 'Show '
'execution '
'progress.',
'type': 'boolean'}},
'type': 'object'}},
'description': 'Notebook settings.',
...

解决:从~\.jupyter\lab\user-settings\@jupyterlab\notebook-extension\tracker.jupyterlab-settings中删除以上选项。

第二,升级后部分Widgets无法正常显示,例如tqdm的进度条。

如果单单显示一个Error display widgets,那么首先要重新build插件。

Read More