skylarmt/postserve
1
0
Fork 0
Code Issues 6 Pull Requests 5 Releases Wiki Activity

Improve concurrency #14

Manually merged
skylarmt merged 1 commits from eyusupov/concurrency into master 2019-10-20 21:06:42 -06:00
Conversation 1 Commits 1 Files Changed 3 +50 -44
skylarmt commented 2019-10-20 19:36:24 -06:00
Owner
Copy Link

Author: @eyusupov

  • Update Python to use async/await
  • Use asyncpg
  • Use connection pool

I didn't have a powerful server to test it (on my machine the bottleneck was the query execution time), but it does seem to improve concurrency:

[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7592337.145509987), ST_Point(4891969.810251279, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 200.67ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7631472.903991994), ST_Point(4891969.810251279, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 304 GET /tiles/10/636/316.pbf (172.19.0.1) 120.94ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 304 GET /tiles/10/637/316.pbf (172.19.0.1) 87.38ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7592337.145509987), ST_Point(4852834.051769271, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 304 GET /tiles/10/635/317.pbf (172.19.0.1) 83.32ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7631472.903991994), ST_Point(4852834.051769271, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:40 server:97] calculated response
[I 191021 01:16:40 web:1971] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 882.31ms
[I 191021 01:16:40 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:51 server:97] calculated response
[I 191021 01:16:51 web:1971] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 10923.63ms
[I 191021 01:16:51 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215302, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:51 server:97] calculated response
[I 191021 01:16:51 web:1971] 200 GET /tiles/10/638/316.pbf (172.19.0.1) 235.96ms
[I 191021 01:16:51 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215302, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:51 server:97] calculated response

After:

[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7592337.145509987), ST_Point(4891969.810251281, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7631472.903991994), ST_Point(4891969.810251281, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7592337.145509987), ST_Point(4852834.05176927, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:24 server:86] received request
[I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7631472.903991994), ST_Point(4852834.05176927, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:27 server:89] calculated response
[I 191021 01:26:28 server:89] calculated response
[I 191021 01:26:28 server:89] calculated response
[I 191021 01:26:28 web:2246] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 4864.85ms
[I 191021 01:26:28 web:2246] 200 GET /tiles/10/637/316.pbf (172.19.0.1) 4869.97ms
[I 191021 01:26:28 web:2246] 200 GET /tiles/10/636/316.pbf (172.19.0.1) 4873.18ms
[I 191021 01:26:28 server:86] received request
[I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215301, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:28 server:86] received request
[I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215301, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:29 server:89] calculated response
[I 191021 01:26:29 web:2246] 200 GET /tiles/10/635/317.pbf (172.19.0.1) 5939.82ms
[I 191021 01:26:29 server:89] calculated response
[I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/316.pbf (172.19.0.1) 971.60ms
[I 191021 01:26:29 server:89] calculated response
[I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/317.pbf (172.19.0.1) 1254.28ms
[I 191021 01:26:37 server:89] calculated response
[I 191021 01:26:37 web:2246] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 14030.48ms
[I 191021 01:26:38 server:89] calculated response
[I 191021 01:26:38 web:2246] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 14568.89ms

Note hat individual SQL query times have increased - I'm not sure if it's because now several queries have to be processed at once by my underpowered PG server or something else.
Would be great if somebody else could check that. Postserve itself seems to be more concurrent now.

In the future, function created by generate-sqltomvt could be used instead of the built in one.

Author: @eyusupov - Update Python to use async/await - Use asyncpg - Use connection pool I didn't have a powerful server to test it (on my machine the bottleneck was the query execution time), but it does seem to improve concurrency: ``` [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7592337.145509987), ST_Point(4891969.810251279, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 200.67ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7631472.903991994), ST_Point(4891969.810251279, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 304 GET /tiles/10/636/316.pbf (172.19.0.1) 120.94ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 304 GET /tiles/10/637/316.pbf (172.19.0.1) 87.38ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7592337.145509987), ST_Point(4852834.051769271, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 304 GET /tiles/10/635/317.pbf (172.19.0.1) 83.32ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7631472.903991994), ST_Point(4852834.051769271, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:40 server:97] calculated response [I 191021 01:16:40 web:1971] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 882.31ms [I 191021 01:16:40 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:51 server:97] calculated response [I 191021 01:16:51 web:1971] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 10923.63ms [I 191021 01:16:51 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215302, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:51 server:97] calculated response [I 191021 01:16:51 web:1971] 200 GET /tiles/10/638/316.pbf (172.19.0.1) 235.96ms [I 191021 01:16:51 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215302, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:51 server:97] calculated response ``` After: ``` [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7592337.145509987), ST_Point(4891969.810251281, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7631472.903991994), ST_Point(4891969.810251281, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7592337.145509987), ST_Point(4852834.05176927, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:24 server:86] received request [I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7631472.903991994), ST_Point(4852834.05176927, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:27 server:89] calculated response [I 191021 01:26:28 server:89] calculated response [I 191021 01:26:28 server:89] calculated response [I 191021 01:26:28 web:2246] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 4864.85ms [I 191021 01:26:28 web:2246] 200 GET /tiles/10/637/316.pbf (172.19.0.1) 4869.97ms [I 191021 01:26:28 web:2246] 200 GET /tiles/10/636/316.pbf (172.19.0.1) 4873.18ms [I 191021 01:26:28 server:86] received request [I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215301, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:28 server:86] received request [I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215301, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:29 server:89] calculated response [I 191021 01:26:29 web:2246] 200 GET /tiles/10/635/317.pbf (172.19.0.1) 5939.82ms [I 191021 01:26:29 server:89] calculated response [I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/316.pbf (172.19.0.1) 971.60ms [I 191021 01:26:29 server:89] calculated response [I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/317.pbf (172.19.0.1) 1254.28ms [I 191021 01:26:37 server:89] calculated response [I 191021 01:26:37 web:2246] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 14030.48ms [I 191021 01:26:38 server:89] calculated response [I 191021 01:26:38 web:2246] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 14568.89ms ``` Note hat individual SQL query times have increased - I'm not sure if it's because now several queries have to be processed at once by my underpowered PG server or something else. Would be great if somebody else could check that. Postserve itself seems to be more concurrent now. In the future, function created by generate-sqltomvt could be used instead of the built in one.
skylarmt reviewed 2019-10-20 20:59:54 -06:00
skylarmt left a comment
Author
Owner
Copy Link

.

.
skylarmt closed this pull request 2019-10-20 21:06:42 -06:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: skylarmt/postserve#14
No description provided.
Delete Branch "eyusupov/concurrency"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?